Experimental Design

From CS2610 Fall 2016
Jump to: navigation, search

slides

Readings

Reading Critiques

Tazin Afrin 0:29:17 9/26/2016

Critique of “How To Do Experiments”: In the 2nd chapter of his book ‘Doing Psychology Experiments’, David Martin addresses the difficulties during testing a hypothesis. An experimental method of hypothesis allows to make causal statement. He discussed different types of variables and threats to validity. The independent variable is manipulated as the circumstances and the dependent variables is to be determined and measured as the behavior change. An independent variable may have different levels on which the dependent variable is dependent. An independent variable does not change. If some circumstances need to be accounted is some way, those may become control variables, and a value for them may be set. But at the same time it should be done with care because we may not want a lot of control variables because in that case the experiment may not become generalizable and issue the threat of external validity. Random variables improve the external validity when those are set to vary in a random fashion. A confounding variable may distort the result the experiment because it systematically changes with the change of independent variable. So we need to closely observe a confounding variable and try to eliminate it. An internal validity is a threat that refers to if anything other than the independent variable caused the change in dependent variable. Internal validity includes history, maturation, selection, mortality, testing etc. A threat to internal validity may introduce a confounding variable. This book chapter clearly identifies the key elements that has to be taken care of while doing an experiment and it gives a structure to systematically perform an experiment. ------------------------------------------------------------------------------------- Critique of “How to Decide Which Variables To Manipulate and Measure”: In the 7th chapter of his book ‘Doing Psychology Experiments’, David Martin discussed about independent and dependent variables. Because when planning any experiment, whether it is the simplest one or the very complex one, one has to decide the two most common variables – independent and dependent variables. This is very important, because an independent variable effects the behavior. Choosing an independent variable in psychology experiment is difficult because psychology researchers have more difficulty to come to an agreement. But a lot of chaos can be avoided when the experiment has an operational definition. Hence choosing an independent should be realistic and should have a well-defined range that shows effect. Often the best thing to find a range is to do a pilot project. A pilot experiment is a small scale version experiment of the original experiment and it helps to find many pitfalls of the experiment in practice. Also a pilot experiment helps to eliminate some extravagant designed that are not really helpful and one can come up with a simpler and effective experiment. The researcher must decide what he wants to measure from the infinite number of things to measure. So it is most important to follow a systematic way to find dependent variables. A dependent variable should have an operational definition to remove any conflict of the result. Also a researcher has to take care if the same experiment is repeated it will give the same result. This is the reliability of the dependent variable. Also there can be an indirect link of the dependent variable to some other variables. To improve the chance of using a valid dependent, one may using multiple dependent variables or composite dependent variables. Indirect dependent variables include physiological measures and behavioral measures.

Haoran Zhang 14:27:21 9/26/2016

How To Do Experiments: After design phase or even implementation phase, we need to evaluate how good is our design. In this article, author tell us how to do experiments, that is a way to evaluate our design. To do experiments, we need to define variables, what are independent variables in your experiment? What are dependent variables? What are control variables? And what are random variables? To find out these variables, we need to figure out what we want to test, so that we can manipulate some of them to do experiments. Sometime, we don’t want to make a circumstance into either a random or a control variable, then, we can randomize constraints. Also confounding variables also helps for a good experiment. The author not only tell us how to do experiments, but also tells us what are threats to internal validity. They are history, maturation, selection, mortality, testing, statistical regression, and interaction with selection. If you don’t care about these things, they may ruin your experiments, and make your work useless. Because, it may let the experiment provide a wrong result. May be this is what you want, but not accurate or don’t have any representative. Or it gives you a bad result, but actually, your job works. In addition, we need to make a good summary of the experimental method, so that we can have records of different variable settings. Then we can have a good summary of our work. How To Decide Which Variables To Manipulate And Measure: To do experiments, we need to manipulate independent variables, and observe dependent variables. But it raises another problem is that, how to decide which variables to manipulate and measure. This article gives us an answer. To choose an independent variable, we need to defining our independent variable. Then based on characteristic of different independent variables, we need to choose the range of our independent variable. Just remember to be realistic, select a range that shows effect, and do a pilot experiment. Then we need to choose a dependent variable. The same, there are several ways to choose them, operational definitions again. We also need to care about the reliability and validity. Directly observable dependent variables. We have directly, then we have indirect dependent variables. To measure them, we can do physiological measures, behavioral measures. They can help us measure our experiment and provide a reliable and validity result of our experiments.

Zhenjiang Fan 2:18:00 9/28/2016

How to Do Experiments::::::::::::::::::::::::::::::::::::::::::::::::::The work's main purpose is to demonstrate ways on how to conduct an experiment. One of the most important steps it to define variables in terms of circumstances around. An independent variable is set to manipulate the experiment's environment to multiple levels so that we can test our experiment based on them. The outcome that generated based on an independent variable is defined as a dependent variable because it is dependent on that independent variable. If the outcome is impacted by the independent variable, then we can say we have improved our hypothesis is right, in that sense, our experiment is successful. Control variables are set to maintain the environment of an experiment consistent. Usually, all control variables are supposed to maintain consistent for a period of time, so that we could make sure all the changes on the experiment outcomes are determined by independent variables. There are many variables in an experiment and most of them are control variables because we want to test a specific variable's impact on the experiment. But in some case, the cases that we want our experiments to be applicable to a general situation or real-world scenario, we do not need to maintain all control variable stable. The generalizability of an experiment is sometimes highly valued by many experimenters. In the case of generalizability, we sometimes need to simulate our real-world scenarios to conduct our experiments by generating some unexpected circumstances. Random variables are used to ensure this generalizability or external validity. The author so far has stated two almost opposite experiment simulation scenarios, one set by control variables and one by random variables. But there are also some cases in between, cases that we want some elements in the experiment in control and others not. The author calls it randomization within constraints. Then it comes to the confounding variables or internal validity, which are variables change systematically due to repeated changes caused by independent variables. To understand how to eliminate internal validity, we need to know possible threats to internal validity. These threads are history maturation, selection, mortality, testing, statistical regression and interactions with the selection.::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::How to Decide which Variables to Manipulate and Measure:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::In the last assignment reading, the author discusses some variables important to conduct an experiment, in this reading the author focuses on which variables are we going to choose to manipulate and measure the experiment. Choosing independent variables for an experiment is the most important decision you have to make, given the fact that the independent variable is the one that the experimenter manipulates and the purpose of the experiment is to find the effect of this independent variable on behavior. Since the independent variable is so important, first you need carefully to choose it, define it(operational definitions) and then set the range of the independent variable. In choosing a dependent variable, first, we also need to go through the operational definition and see if it is reliable and valid. Measuring a dependent variable's validity is even more difficult than measuring its reliability. Then the author goes on talks about different dependent variables, for example, directly observable dependent variables which include single dependent variables, multiple dependent variables, and composite dependent variables. Indirect dependent variables are those impossible to observe directly, so we need to some ways to measure it, for example, physiological measures and behavioral measures.

Steven Faurie 13:21:38 9/28/2016

Steve Faurie How to Do Experiments: This paper describes the experimental process. It describes the way experimenters should try to choose variables to measure, fix and randomize. The first variable type described is independent variables. These are the variables that experimenters manipulate during an experiment. Dependent variables are the variables experiments measure. The example given is a participant’s reaction to changes in the independent variable. The next variable type described are control variables. These are the variables an experimenter will attempt to keep constant between changes to their independent variable. For example, if you’re testing concentration relative to lighting you would want to keep the sound level in the room the same. Random variables are intended to fight some bias that your sample selection might introduce. For example, if you’re trying to say things about the population of the united states you would want to select participants randomly from all over the country instead of just Sophomores at Pitt. Confounding variables were described as some variable other than the independent one that may be contributing to the results. For instance, you ask someone to do something three times in different conditions they might be better the third time because of practice and not the condition that was changed. The next section of the paper described threats to internal validity, things outside of the experiment that may be changing the results. History was the first threat. The example given was conducting experiments in a class room using two different school years as your separate test groups. Maybe the university got more selective between years. Maturation, the devilment of your experimental subjects was another example. Selection bias was also described. As mentioned above this can be combatted through random sampling. Mortality, people dropping out could also affect results. Perhaps there was some commonality between all the people who could no longer participate in the experiment. Testing and statistical regression were also both described. Testing could change a subject’s perception of something by making them more aware of the thing the test was about. Statistical regression simply says that extreme results are likely to be more average if you were to repeat the experiment a second time. The chapter and the more human centered testing described in it was a good thing to review for this class. Many of us are used to simulating something and gathering direct results. Or making some observations about how a process works, but I doubt many students in the computer science department regularly do testing on subjects. How to Decide Which Variables to Manipulate and Measure: This chapter discusses several issues about choosing variables and measuring them. Choosing independent variables can be a little more difficult than one would expect. If you are trying to classify things and use different classes of those things as your independent variables it could be difficult to come up with a valid classification scheme. The same is true for measuring dependent variable. The chapter gives the example of observing a child’s behavior. This isn’t something you can measure with a ruler, but instead requires judgments to be made by another human. They also discuss directly observable vs indirectly observable dependent variables. Directly observable variables are better if they’re an option. But they might not be if you’re trying to observe something like feelings or thoughts of participants. The chapter also describes using physiological measurements to interpret some indirect dependent variables. However, this technique assumes we know certain physiological states correspond to the dependent variable in question. Behavioral measures were discussed as well. It is subject to the same issues as physiological measurements. Are the assumptions we’re making about what a behavior indicates accurate?

Xiaozhong Zhang 19:25:25 9/28/2016

How to do experiments The chapter was about how to distinguish different variables in a psychological experiment. First, basic variables types were introduced like independent variable, dependent variable, control variable and so on. Then the author argued that in real word scenario, it is impossible to avoid the happening of cofounding variables. Since cofounding variable can also impact the experiment result, it is very important to be aware of them in order to avoid erroneous experiment observation or explanation. Therefore, the author in turn introduced many types of cofounding variables. First, different history periods can have different characteristics, which can be ignored. Second, maturation can also be a factor. The longer a user is involved in the experiment, the maturer he/she will be, which can also change the experiment result with changing the dependent variables. Third, selection refers to the picking of user groups. Selection is obviously a cofounding variable, if it is not considered as a dependent or control variable. Fourthly, general mobility can commonly be ignore, however, mortality of specific group can affect the experiment result greatly, thus should be considered a confounding variable as well. Finally, the author gave an example to illustrate different types of variables using his teaching example and again emphasized the the experiment executor should be aware of the cofounding variable in order to control the experiment result integrity and quality. How to Decide Which Variables To Manipulate and Measure The chapter introduced the ways to determine different dependent variables in the experiment design. First is choosing an independent variable. In order to do that, we need to first have operational definitions which needs to be defined through operations. Then, we need to choose the range of the independent variable. In the process, we need to be realistic and the range needs to cover the area where the variable has effect. The author further argued that pilot experiment helps defining the dependent variables. The second step is to choose a dependent variable. Again these variables need to be operationally definable. Besides, they need to be both reliable and valid. A dependent variable is reliable if we fix the independent variables, the value of the dependent variable needs to be the same no matter how many times we do the experiment. A dependent variables is valid if we are actually measuring it in when we do the measurement. In the third part, the author talked about different types of dependent variables, both directly observable and indirectly observable. It also talked about dependent variables that rely on one independent variable, multiple independent variable as well as the combined by several dependent variables i.e. composite dependent variables. Finally, it introduced several methods to measure indirect dependent variables, like using physiological measures as well as behavioral measures.

nannan wen 23:07:18 9/28/2016

“How to do experiments” by David W review: In this book, the author talked about how to build the basis of experiments in psychology. First of all, the author described different types of variables that can appered in the experiment, which includes independent variables, dependent variables, control variables and random variables. Psychology is a new topic in this area, a lot of variable needs to be take into consideration when design such a experiment, variables like the history, gender, and personality etc. In many situations, all other variables should be clear and easy to find out, but if the circumstances has changed, then the whole effort of the experiment would be wasted. “How to decide which variable to manipulate and Measure” review: In this part, the author mainly talked about how to choose an independent variable. If I were to chose a variable in my experiment, I should be able to specify and operational definition of the variable so that others can duplicated my experiment without much difficulties. The author gives an example regarding to this requirement. The experimental scientists requires specific definitions for these variables.

Keren Ye 0:12:36 9/29/2016

How to do experiment The authors discussed the experimental method in the reading. The benefit is that it allows us to make causal statement, that a circumstance caused a change in behavior. Given the example of the a person to press a button in response to a light when the light has a particular intensity, the authors explained some concepts to us: 1) Light intensity is called an independent variable because it is independent of the participant’s behavior; 2) The behavior we choose to measure is the dependent variable because it is dependent on what the participant does. Thus the dependent variable is the time from the onset of the light until a button is pressed. 3) If other circumstances in an experiment that need to be account for in some way, we could control them using control variables. 4) Finally, random variables are circumstances we do not want to control. 5) Randomization within constraints falls between the control variable and random variable. 6) Confounding variable is the circumstance that changes systematically as the independent variable is manipulated. After discussing the concept of all the circumstances, the authors state that confounding variables can cause low internal validity and make it difficult to say that only the independent variable caused a change in the dependent variable. They then talk about threats to internal validity including history, maturation, selection, mortality, testing, and statistical regression, furthermore, variables such as maturation and history may have interactions with selection. The authors discuss these factors in details. Finally, summary of the experimental method is made. The authors provide a systematic way of analyzing all of these circumstances and the relations among them. The most important thing is that the experimental method allows causal statements to be made. How to Decide Which Variables To Manipulate and Measure This paper discusses how to decide which variables to manipulate and measure. It firstly talks about how to choose an independent variable. Some suggestions are given: 1) Defining the independent variable; 2) Choosing the range of the independent variable; 3) Be Realistic; 4) Select a range that shows effect; 5) Do a pilot experiment. Then the authors explain how to choose a dependent variable. Suggestions they gave includes: 1) Defining the dependent variable; 2) Analyze the reliability and validity. To further explain the ideas, the authors discuss directly observable dependent variables and indirectly observable dependent variables in details. For directly observable dependent variables, they introduce about single, multiple, and composite dependent variables. When mention about the indirect dependent variables, they gave some ways to measure them, including physiological measures, behavioral measures. In sum, to choose an independent variable for the experiment, we must firstly specify an operational definition of the variable. It is also important to choose the level of the independent variable so that the range is large enough. A trial run, or pilot experiment will some time help the decision.

Alireza Samadian Zakaria 0:18:43 9/29/2016

The Second chapter of the book is about how to do an experiment. According to the reading, there are several types of variables in an experiment. Independent variable is the one that we want to change and see the effects of that. Dependent variable, is participant’s behavior in response to manipulations of independent variables. Sometimes we make a statement about the expected relationship between these two variables which is called hypothesis. Another type of variables is control variables; control variables are fixed to a single value so that we make sure that they don’t affect the result of experiment. Another type of variables is random variables which are tried to be random in different cases by randomize selection. Random variables improve generality and external validity of an experiment. Sometimes, we need to randomize some variable in a limited range or limited condition, these variables are called randomized within constraints. If some variable is not in mentioned categories, we would not be sure about its effect and it can cause that our results would be countered. The author also provide us with some threats to internal validity of an experiment. For example, one of these threats is biased selection which can affect our randomization. Another threat is history which is about uncontrolled events during the experiment. Maturation, which is mainly concerned in young participants. Mortality, which is about nonrandom loss of individuals (not always caused by death). Statistical regression is another threat which is term used for the movement of scores toward the average for groups which has gotten extreme scores. Another threat is interaction with selection which is about different conditions of some groups mainly because of age. The example for the last thread was interesting which was about less rate of left handed population among older people which should not be used to conclude that left hand people die sooner. ------- The seventh chapter of the book is about how to set variables learned in the second chapter in a psychological experiment. The author first talks about how to choose independent variable. It is important to have operational definition of the variable since many of the variables are hard to measure for instance what is a violent movie? After defining the variables, we should decide about range of these variables. Range should be large enough to affect the dependent variables and it should also be realistic. The author suggests to run a pilot experiment which is a trial run so that we can see any possible defect in our experiment and variables. Furthermore, the author talks about how to choose dependent variables. Again we need some operational definition and this time we also should think about reliability and validity. Reliability means getting the same result when we repeat the test. There are some ways to determine about reliability, test-retest is one of the ways and it means to repeat the test on the same group later. Sometimes repeating a test on the same group can have different result due to the previous experience. Thus, there are two other methods which are alternative-form and split-half which can be used. In addition to reliability, validity is another thing that we should concern. It refers to whether we are measuring what we want to measure. There are different forms of validity; the weakest form is face validity which is so subjective and it is not of much scientific use. There are three other forms of validity which can be used and are introduced. Generally, measuring validity is more difficult than measuring reliability. There are two types of dependent variable based on being observable. The first category is directly observable which are easier to work on; however, we should still be worried about validity of measurements. Indirect dependent variables are those we cannot measure directly and there are two types of measurement that we can use for them: physiological measures which are popular but hard to interpret and behavioral measures.

Tazin Afrin 2:00:01 9/29/2016

Critique of “How To Do Experiments”:  In the 2nd chapter of his book ‘Doing Psychology Experiments’, David Martin addresses the difficulties during testing a hypothesis. An experimental method of hypothesis allows to make causal statement. He discussed different types of variables and threats to validity.  The independent variable is manipulated as the circumstances and the dependent variables is to be determined and measured as the behavior change. An independent variable may have different levels on which the dependent variable is dependent. An independent variable does not change. If some circumstances need to be accounted is some way, those may become control variables, and a value for them may be set. But at the same time it should be done with care because we may not want a lot of control variables because in that case the experiment may not become generalizable and issue the threat of external validity. Random variables improve the external validity when those are set to vary in a random fashion. A confounding variable may distort the result the experiment because it systematically changes with the change of independent variable. So we need to closely observe a confounding variable and try to eliminate it. An internal validity is a threat that refers to if anything other than the independent variable caused the change in dependent variable. Internal validity includes history, maturation, selection, mortality, testing etc. A threat to internal validity may introduce a confounding variable.   This book chapter clearly identifies the key elements that has to be taken care of while doing an experiment and it gives a structure to systematically perform an experiment.   -------------------------------------------------------------------------------------  Critique of “How to Decide Which Variables To Manipulate and Measure”:   In the 7th chapter of his book ‘Doing Psychology Experiments’, David Martin discussed about independent and dependent variables. Because when planning any experiment, whether it is the simplest one or the very complex one, one has to decide the two most common variables – independent and dependent variables. This is very important, because an independent variable effects the behavior.  Choosing an independent variable in psychology experiment is difficult because psychology researchers have more difficulty to come to an agreement. But a lot of chaos can be avoided when the experiment has an operational definition. Hence choosing an independent should be realistic and should have a well-defined range that shows effect. Often the best thing to find a range is to do a pilot project. A pilot experiment is a small scale version experiment of the original experiment and it helps to find many pitfalls of the experiment in practice. Also a pilot experiment helps to eliminate some extravagant designed that are not really helpful and one can come up with a simpler and effective experiment.   The researcher must decide what he wants to measure from the infinite number of things to measure. So it is most important to follow a systematic way to find dependent variables. A dependent variable should have an operational definition to remove any conflict of the result. Also a researcher has to take care if the same experiment is repeated it will give the same result. This is the reliability of the dependent variable. Also there can be an indirect link of the dependent variable to some other variables. To improve the chance of using a valid dependent, one may using multiple dependent variables or composite dependent variables. Indirect dependent variables include physiological measures and behavioral measures.  

Anuradha Kulkarni 7:48:14 9/29/2016

Doing Psychological Experiments -- Chapter 2: The main focus of this chapter is introducing the basic terms and notices that are seen in experiments. First part of the paper describes types of variables can appear in the experiment, which includes: independent variable, dependent variable, control variable, random variable, random variable with constraint, and confounding variables. These variables have been explained clearly with help of examples. In many situations, all other variables should be clear and easy to find out. But with little change in circumstance, the whole effort of experiment will be wasted without noticing confounding variables. The second part discusses about the threats that could effect the experiment process. For Example, selection of the samples and mortality of the participants. The chapter is explained well with good set of examples. --------------------------------------------------------------------------------------------------------------------------------------------------- Doing Psychological Experiments -- Chapter 7: This chapter is the continuation of chapter 2 discussing about the factors needed to keep in mind while choosing the independent and dependent variables. This chapter even discuss about how to manipulate the dependent variable and how to strategize an experiment with valid dependent variable. There are two things that need to be focused on while choosing the dependent variable i.e. they should be reliable and valid. There are few methods that have been discussed: face validity, content validity, predictive validity and concurrent validity. I liked the basic idea of this chapter and in my opinion it serves good in the line of research.

Zuha Agha 8:52:21 9/29/2016

How to do Experiments This chapter discusses the methodology of designing experiments and the factors influencing experimental design. Some of the important factors discussed include cofounding variables variable selection and control, cofounding variables and validity threats prevalent in the experiment environment. It describes how the choice of variables and correlation or causality between variables can dictate the outcomes of an experiment and often times could reflect the biases of the experimenter. Randomization and constraints can also affect the behavior significantly. Moreover, external changes over time threaten the validity of experiments including maturation that implies an increase in participant’s experience, unsuitable or non-generalizable selection of participants, mortality due to the change in number of participants, pitfalls in testing of experiment and so on. Overall I think the chapter provided a very useful analysis on designing experiments. It was very relatable because as graduate researchers we are actively engaged in the process of experiment design and should be well-aware of the important elements to consider for successful experiment design. ======================================================= How to decide which variables to Manipulate and Measure This chapter emphasize on how to ensure that the experiment is reliable and what can be done to increase transparency of the validity of experiments. The objective is to make the experiments easily reproducible for other to test and verify. It is important to have an operational definition for all variables in the experiment so that other experimenters can easily understand and use them. Moreover, several methods of reliability are discussed including test-retest, alternative-form and split-half as well as the pros and cons of each one of these methods. It also describes tests for validation including predictive validity, content validity, concurrent validity and so on. In addition, it describes ways of measuring variables going into the details of measuring different classes of dependent variables including directly or indirectly dependent variables as well as single and composite dependent variables. Lastly, measuring behavioral and psychological responses in experiments is also discussed. What I like about the chapter is how it provides a comprehensive workflow for identifying the different types of variables and testing/validation, clearly highlighting the difficulties, advantages and disadvantages of each one of them.