Quantitative Evaluations

From CS1635 Spring 2016
Jump to: navigation, search

slides

Readings

Reading Critiques

Michael Oles 22:35:20 2/18/2016

I didn't really get too much out of the beginning of the chapter which explained the different types of variables because I was already familiar with these concepts from previous science classes but it might be helpful for review. The randomization and the confounding variables were new to me and it shows the importance of making sure your study does not have outside factors influencing it. I thought the Coke/Pepsi example was really interesting. And the other examples like the shaving cream were helpful as well. Basically this has allowed me to realize how careful you have to be to conduct an accurate experiment because so many factors can skew your results. I think using the chart given at the end would be helpful for this.

Luke Kljucaric 19:48:40 2/22/2016

I have had many science classes throughout my academic career so the sections on how to perform experiments was nothing new for me. The parts of the threats to internal validity were new to me. A majority was a good refresher; however, specifically the mortality section. It's not something you would technically think of affecting an experiment but it could have potentially affect the experiment's results. Like the reading said though, this is typically not something we would want to worry about. Additionally, I like the examples that were given in the reading. I thought they were a vital point of the reading that help convey important topics.

Nate Patton 8:56:25 2/23/2016

The reading for this week was very interesting to me. What really peeked my interest was during the Selection section. I never thought of how a non-random could possibly effect the outcome of an experiment. This may play a part in our group assignment.

Daniel Hui 1:13:04 2/24/2016

Maturation is an interesting topic in the article. Pretty much the concern is that when a participant/participants participating in a experiment spread at over a long period of time, some participants may be subject to 'maturation' or the idea that a participant may change over the course of the experiment possibly altering the outcomes of the experiment by ultimately changing control variables/characteristics of ones self. I wish the article discussed what happens to these people who mature. Are they considered outliers of the experiment? Are they automatically dropped from the experiment to save the outcome of the experiment? It would probably best if participants who matured parts of themselves are tossed from the experiment if other variables in the experiment depended on the changed properties of the participant.

Robert Webb 11:44:06 2/24/2016

I appreciate the reading of how to test users and what to look out for to help our tests be valid. For example, mortality that's differential can be really bad - if all of your users are giving great feedback that they love your features, but their number is declining after every release, then it's quite possible that it's your features that are causing users to leave. Also, there can be lots of confounding factors in testing users. You may find that you're not getting the feedback that you want because the users are using a different app instead of your app for the features that you have poorly implemented. Overall, a good reading.

Bogdan Kotzev 12:18:36 2/24/2016

I learned about a lot of things that you want to considered when doing experiments. I was already aware of independent and dependent variables but did not know about control variables. I thought having a lot of variables that were controlled was a good thing, but it actually leads to reduced external validity. Fortunately the article suggests dealing with that through by using random variables or using a technique called randomization within constraints. The article also pointed out things that can affect the conclusion of an experiment known as internal validity. Things like confounding variables, changes in history that spiked the interest about a certain topic in recent times, maturation of test subjects, test subjects dropping out of experiments, and how the test subjects are subjected. There are even more things to consider when doing experiments. The article almost made me not want to conclude experiments, because of the too many things that you have to worry about.

Charlotte Chen 15:41:38 2/24/2016

The articles provides an interesting perspective on the elements and challenges faced during experiments. The first few terminologies (independent, dependent, control variables) are all familiar to me since it appeared a lot in my psychology and biology classes before. The threats to internal validity section is very interesting. Maturation: the increasing maturity and experience of the participants can greatly affect the validity of the result in long-term experiments. Selection is also a noteworthy challenges; I always see city of Pittsburgh posting ads searching for research participants, which definitely could fall under the self-selection problem since people who choose to volunteer their time to participate in research might share common traits (such as being unemployed) that might negatively affect the experiment result. Overall the article provides a very extensive explanation on how to conduct a good experiment by helping us to understand the experimental variables we should choose, and be aware of the different factors that might distort our outcome.

Jonathan Blinn 20:06:18 2/24/2016

While the first part of the reading was a little mundane, I found the section about testing to be rather interesting. In this scenario, it seems extremely difficult to see how well an advertising campaign worked. By checking the consumer base's knowledge of the product before the ads air can skew the results. However, by not checking this before airing the ad, you have nothing to compare it to. Asking the right question is extremely important as well as the wording and the timing of it.

Matthew Reinhold 20:32:42 2/24/2016

The reading provided a lot of information regarding how to do formal studies and what they can mean, but only a few things were useful for the type of experiments we will be doing in our groups. The information regarding confounding variables will be relevant for determining the features that are actually causing a negative impact on the application as well as the section about threats to internal validity.

Tiffany Martrano 20:46:13 2/24/2016

The article that we read for today talked about how to perform test and set up an environment to ensure a good test. I thought this article was interesting because I've learned a lot of these concepts in my psychology classes, but never knew they could be applied to computer science. A lot of these concepts, like different variable types and how to go through a test are familiar to me. I'm interested in seeing how we can apply these concepts to building a mobile application and what benefit it can serve.

John Phillips 20:51:03 2/24/2016

The reading talked about how to design an experiment. It started with the basics: the independent variable, the dependent variable, randomization, control groups, etc and later moved into different ways that experiments can fail. Some of these most people have heard of before, such as confounding variables while some may be unfamiliar such as maturation and statistical regression. I think the most important part of this reading was the threats to internal validity as sometimes a badly done experiment can be worse than no experiment at all. When using experiments for interface feedback, the experimenters need to be careful to reduce these threats.

Max Benson 21:15:41 2/24/2016

It's good to know all the different ways experimental results can be skewed by the context of the experiment, as well as the way variables are controlled (or not controlled). Particularly, I was interested to see that it is not always beneficial to control the environment as much as possible, since that will only produce results applicable to that environment. Randomizing variables within certain constraints seems like a good approach to overcoming this "over-sterilization" problem.

Jason Naughton 0:17:52 2/25/2016

Having taken research methods and psychology courses, this reading was more of a refresher than an education. It's interesting how much of the formal research methods is applicable in human testing in software. For example, the reading mentions how testing itself can be a threat to internal validity. So if you practice A/B testing and the experimental group broadcasts the changed features, the A group may develop a bias, and the control would be ruined. One interesting practical difference between researching software and researching the natural sciences is that you can programmability: select random samples from your userbase (with constraints, even, based on their information), enroll them in the experiment, and monitor their interactions with ease. However, although it's much easier to test subjects remotely, that invites threats to internal validity because the experimental conditions cannot be controlled and there are many confounding variables in the real world.

Joshua Fisher 0:43:47 2/25/2016

I really enjoyed the reading today. I remember learning some of this material in my Intro to Psychology class I took during my freshman year here. Reading about the different types of variables was interesting, and it make me think about how this relates to the experiments we have to conduct to test our group project user interfaces. We want to make sure that we keep as many variables consistent when testing the user interface so that we can compare the results from the different people.

chris finestone 0:50:51 2/25/2016

Statistics taught me how important studies were, but I never quite realized what it really took to actually carry one out. It's about finding the things that can be said about a particular group of different people all doing the same thing compared to a group of equally different people doing a placebo. If the two groups are different enough in the right way, the data may be used to suggest a correlation between a theory and the common trait or action from the study participants. Or, a study may find no significant difference between two groups. When that happens, we need to go back to the drawing board and think about what it is we're trying to say and if it is what really happens in the world, regularly.

Casey Nispel 0:55:46 2/25/2016

I found the most interesting part of this reading all the different ways that the internal validity of an experiment can be threatened. The experimental method aims to clear results with every variable accounted for and the control variable directly affecting the independent variable, but in the real world this is not always possible and we have to be aware of the various ways our experiments can be influenced. Most of the threats mentioned are not things I would immediately consider when doing an experiment, but the examples provided showed how each could influence the results and create confounding variables. I thought the example about the left handed children disappearing was interesting because it forces you to think about outside influences that at first don’t seem to be relevant to the experiment but on closer review hold a great deal of significance.

Adhyaksa Pribadi 2:06:18 2/25/2016

The readings go through the different type of variables that are used during an experiment. In an experiment, there is an attempt to establish a causal relationship between at least one circumstance and one behavior. Well-designed experiments have no confounding variables, which are those that change along with the levels of the independent variable.

Xinhai Xu 3:02:43 2/25/2016

This reading introduces method of science experiments. It talked about terms like independent variable, dependent variable, control variables and random variables in detail. Random variables improve the external validity of an experiment and allow it to be generalized to other people, settings, and times. Experimenters should attempt to eliminate or minimize confounding variables.

Amukher14 3:03:31 2/25/2016

I thought this was a concise overview what factors to consider during psychological experimentation. A lot of this reading had overlap with content from an intro to psychology class I took earlier. I think the article did a good job of informing the reader of the importance of internal and external validity in experiments and what variables to account for to achieve a quality experiment with quality results. This type of information is useful when trying to get quality feedback on the application interface development process because it helps filter out good and bad data from users in scientific way.

Mengqi Wu 6:57:39 2/25/2016

It's much hard than I thought to design an experiment. it's impossible to control all the variables and there are so many things need to be taken into the consideration. Also, the example of confounding variables from Coke is very interesting. And I never thought that the letter of the glasses will affect people on their reference decisions. So, in real life, you never know what little thing is actually affecting your experiment result.

John Riesenberger 7:27:47 2/25/2016

Todays reading showed us, quite literally, how to do experiments. We were shown how to handle and account for a wide breadth of different factors, including several types of variables, such as independent and dependent variables, controls, random variables, and confounding variables, which was a new concept to me. After the brief introduction to the variety of variable types, we were shown common pitfalls, or "threats to validity". I can only very loosely see how we might apply these techniques to interface design.

Yijia Cui 8:20:23 2/25/2016

This article introduces the elements of experiments. The first variable is independent variable, as its name referring, which is independent of participant’s behavior. Those variables are chosen by the designer of the experiment, and the participants have no way to change it. Correspondingly, there is a variable called “dependent variable”, which is dependent of participant’s behaviors. In the experiment, there is always a hypothesis, which is consisted of the expected future of relationship. Control variables are the variables being controlled by the designer to make them not vary from a single value. Random variables are based on the random selection/assignment, which can validate the findings from the experiments. In some cases, designers would like to conduct experiments neither too being controlled nor randomized. Thus, randomization within constrains can support the needs to control part of the assignments and randomize the rest of parts. Furthermore, because the experiments can never be perfect, and also it’s impossible to be perfect and simple in the real-world, the confounding variables are brought up and indicate the changes as the independent variables are manipulated, and also introduces the low internal validity. There are some threats to internal validity, including history, maturation, selection, morality, testing, statistical regression, and interactions with selections. Overall, this article gives a great introduction to terminologies used in the experiments, and elaborates them to introduce the experimental method.

Clark Nicolas 8:36:55 2/25/2016

This reading was kind of dry because it discussed topics that we've all heard throughout schooling, but I still think it was useful as a refresher, and I can see how it's applicable to mobile app/interface development.

Ish Davis 8:58:37 2/25/2016

The reading today was a synopsis of the scientific method and how it could be used to formulate viable experiments. The concepts that were presented were things that we've learned over and over while being in school, but it was very concise and detailed the components very effectively. The scientific method is an invaluable resource for anyone doing an experiment because it provides a specific framework for testing the things that you're looking for. Independent variables, dependent variables, randomization, and bias are all things that we know very well that can be easily added to any experiment that we do. In this class we will use this method when testing our mockups and designs in order to find what we need to improve and where.

Sarah Dubnik 9:57:14 2/25/2016

I was worried that this reading would just be about basic components of an experiment, with which I am already very familiar, but the discussion of all of the unexpected ways that an experiment could be compromised was very interesting. Perhaps it's just more applicable to psychology experiments, where complex humans are the subjects, but I was slightly worried about the prospect of so many ways that external and internal validity could be threatened. The examples given by the author had an incredible number of points to consider that could affect results, almost to the point that it seems impossible to conduct a valid experiment in psychology. It makes me much more wary of all of the studies I read about in the news. I am curious as to how this information can be applied to our projects, as it seems much more intense than what I was expecting from our own testing.