Evaluation 2

From CS2610 Fall 2014
Jump to: navigation, search

slides

Contents

Readings

Reading Critiques

nro5 (Nathan Ong) 22:25:21 10/28/2014

Review of the three chapters of Doing Psychology Experiments by David Martin The author in these three chapters explains how to experiment and evaluate human subjects on a given task or set of tasks. He also provides examples in explaining different concepts relating to experimental design and evaluation choice. Finally, he presents a list of metrics that may or may not be relevant to certain experimental designs that are standard in evaluating performance. The concepts that the author provides in these chapters extend beyond the field of psychology; these methodologies are also applicable in many experiments in computer science, especially in human-computer interaction. When dealing with user interfaces, the problem tends not to be on the interface itself (aside from design/programming errors), but tends to be the user interacting with them. Understanding how research is performed when observing or analyzing human psychology (or human interaction) is important in determining the effectiveness of any user interface system. It does not make sense to build a whole experimentation infrastructure for human-computer interaction, when the psychology experimentation infrastructure is more than sufficient for evaluating user interface systems. The readings provided this time were extremely helpful in solidifying and consolidating the base of evaluation techniques available to HCI researchers. Not only do these chapters provide the complete list of metrics and experimentation designs, but also they provide a solid foundation for psychology and other human-based research. It allows readers to examine what is the accepted standard for research, and determine the reasons why certain metrics can be used in certain situations as well as the validity of comparison with other experiments with similar designs. These ideas are not readily taught to researchers when they should be. Instead, researchers are asked to "learn on the job," which can lead to many fatal errors committed by novice researchers.

yubo feng 17:52:57 10/29/2014

In the second chapter, author mentions several kinds of variables that should be focus on int he experiment, which are: #independent variables: kind of variables that is independent of the participant’s behaviors #dependent variable: result of the experience is dependent of the participant’s behaviors; when we try to use the dependent, then we get a hypothesis which forecast the reaction that participants will make. #control variables: account for other circumstances; however, these control variables probably influence the result of our experience, then we need to carefully limit them or measure them. #confounding variables: any circumstance that changes systematically as the independent variable is manipulated is a confounding variables. Then author talks about threats to internal validity, which are: #history: to some even that takes place between the testing of the levels of the independent variables. #maturation: caused by participants’s growing older or perhaps more experienced. #selection: when participants are assigned non randomly, particularly when they are self-selected. #testing: when a pretest or multiple test designed is used. #statistical regression: this term refers to the fact that when participants are chosen on the basis of having scored very high or very low on a particular test, their scores tend to move toward the mean on a second test. Finally, in order to make these variables visible and easy to manage, the author come up with a kind of method: by using table, marking every possible variables into this, we write two column that we need, circumstances and behaviors with in depend variable aside in left side and dependent variables aside in the right hand, then we coordinate each kind of variables into these circumstances and behaviors. In the seventh chapter, author talks several ways to decide which variables to manipulate and measure: define independent variable: make clear operational definitions make sure this is the dependent variable you want to measure and control. Choosing the range of independent variable: this range should be easy to see and measure which shows effect. If it is not hard to choose the range, then could use pilot experiment to figure it our. For other variables, principles keeps the same, after we come up with all the variables, it is the time to get start.

Bhavin Modi 20:14:18 10/29/2014

Reading Critique on Doing Psychology Experiments Chapters 2, 7 and 12 The book on Doing Psychology Experiments by David Martin explains the methods to be used for successful experimentation such that the setup and the inferences from them are valid and cannot be refuted. These are guidelines that one must keep in mind and follow are closely as possible. The approach defined has been outlined after years of research in the field and learning from various flaws. These points are at a conceptual level for naïve researchers in the field to learn and understand the uphill task of experimentation. There is a common misunderstanding that thinking of an idea and implementation is enough, but as we see the efforts required for proving your hypothesis is no joke and requires a lot of skill. The situations, conditions and other aspects both environmental, social, human have to be accounted for as they are as important as the research itself, to lend credibility to the hypothesis. The book provides a lot of knowledge and is much like a textbook on doing experiments, and maybe we can add to it from our own experiences in the future. How to do Experiments – This chapter lays the foundation for the understanding the coming chapters. This discussion closely relates to the paper read for the previous lecture Methodology Matters: Doing Research in the behavioural and social sciences. Where previously we learned about all the methods and what flaws and strengths they have. Here we discuss in generality the factors to keep in mind while performing experiments. The analogy is the black box, where the input are the independent variables and output is the dependent variable. We learn the terminology for each these, independent, dependent, control. Confounding, random and random with constraints variables. After understanding this, we take into account maintaining the validity of experiments, internal and external. Increasing one leads to decreasing the other. The threats to validity are history, maturation, selection, mortality, testing, statistical regression, and interactions with selection. Randomization does not incur additional cost as people think but leads to random selection and mostly assignments. The cost remains the same, giving us more generality and the drawback is as mentioned increase in external validity with a corresponding decrease in internal validity. How to decide which Variables to Manipulate – Now we move onto laying out the foundations for carrying out the experiments. Initially we need to decide our hypothesis, which involves confirming your independent and dependent variables. The first challenge comes in the form of operational definition of the variables. Independent are a little easier to define than dependent in this respect. To agree is upon definitions is a problem though, for some experiments relaxed definition will also work. We need to maintain the reliability and validity though for acceptable inferences from results, using test-retest, alternative form and split-half. For validity we can us face validity (fails until experienced), content, predictive, and concurrent validity. The presence of multiple dependent variable is also a challenge but the use of physiological indications for the indirect variables was an interesting method though not exactly unbreakable. With advent in technology and new user interaction standards will create a challenge as traditional approaches will not work and we will have to define new parameters for experimentation. Physiological methods will have to become more refined, the availability of base rates is another aspect and finding correlations in the presence of confounding variables is a challenge. As mentioned this is not an exact science and will keep on evolving with our intuition and knowledge coming into play. How to Interpret Experimental Results – This chapter is more on statistics and result interpretation and manipulation to an informative form. We can learn many techniques to understand the correlation between the independent and dependent variables. The nomenclature of frequency distribution graphs with the use of mean, median, mode, variance and standard deviation are discussed and explained with the same purpose so as to draw logical inferences from the data and provide significant results. The factor of maintaining inferential stability is the challenging task, with the probability of difference being less than 0.05 or 0.01 for declaring statistical significance. We have come across Meta-analysis before in papers like the multi-touch interfaces by Bill Buxton and a paper on input devices by Stuart Card and Balakrishnan. The problems with this have been shown and I agree with them. Personal choices are an important factor and creating an unbiased and sound meta-analysis is a difficult task. It is much needed though as a quick study will save you a lot of time as mentioned. Overall the reading has been enlightening and there are even more things to learn from what is mentioned here, a thorough understanding will lead to good experimentation and significant results.

SenhuaChang 20:32:51 10/29/2014

How to do interpret Experimental Results This Chapter gives lots typical methods to analyze the data and how to use the result. The first method called plotting frequency distributions which are treated as the first step in finding Difference between conditions. The common kinds of distributions include normal, bimodal, skewed and truncated. Next, the author talks about the statistics for describing distributions. Fist kind is the descriptive statistic which is a number that allows the experimenter to describe some characteristics of the data rather than having to report every data point. Fist type is central tendency describing the typical behavior. Three common ways to express the central tendency are mode, median and mean. The second method is dispersion which shows how spreads out score are. Further, we need to plot relationships between variables and graph is always a good way. <111111111111111> How to Decide Which Variables to Manipulate and Measure This article shows us how to choose variable in a psychology experiment. It said that defining a problem is very important at beginning of an experiment, since variables play critical role in an experiment. When choosing an independent variable for the experiment, we must first specify an operational definition of the variable so that other experimenters will be able to go through the same operations when they conduct similar experiments. It is also important to choose the levels of our independent variables so that the range is large enough to show the experimental effect but small enough to be realistic. And also, a trial experiment will help us in decision. Dependent variable has two properties, which are reliable and valid. The dependent variable is valid if it agrees with a commonly accepted standard. <111111111111111> How to Do Experiments This article show s five variables that we should and have to consider in experiments. The first one, independent variable which is the circumstance of major interest tor experimentation. Second one, is dependent variable which is the behavior we choose to measure. Then is the control variables. Which are some of the other circumstances for controlling. And, random variables are other circumstances to vary in a random environment. Any circumstance that changes systematically as the independent variable is manipulated is a confounding variable. As threats to internal validity, there are history which is the occurrence of an uncontrolled event, maturation which is the change in age or experience, selection which is the biased assignment, mortality which is the nonrandom loss, testing which is the change due to the testing process, statistical regression which is the movement of scores toward the mean for groups selected on the basis of extreme scores, and interactions with selection which is the differential effects of a treat on nonequivalent groups. This chapter gives the necessary factors during the experiments. We must consider those the five type of variables when experimenting. Also, we have to care about several threats to internal validity. Experimenters should attempt to eliminate or minimize confounding variables, which change systematically with the independent variable and distort the relationship between and independent and dependent variables

phuongpham 20:32:48 10/29/2014

How to do experiments: this chapter presents different kinds of variable in a psychology experiment, i.e. independent, dependent, control, random, and confounding. Moreover, the author also pointed out possible threats for experiment settings. I find the chapter interesting because I have already had some ideas about setting up an experiment and had the same 'feeling' as well as how to deal with these variable types. However, my ideas are tuitive and unorganized. Reading this chapter gives me an organized knowledge about different variable types. I can write them down, analyze them to make a better experiment setting. The possible threats are very important. Some threats, e.g. regression and testing, are unexpected, to me. ***How to decide which variables to manipulate and measure: this chapter gives a deeper information about setting a psychology experiment. It is interesting that research questions in this field are often subjective. Researchers have to give an explicitly and widely accepted operational definitions of both independent and dependent variables. I have had encountered some of these cases in the NLP area where the performance of a system, e.g. conversation generator, will have to be subjectively judged by human. This gives an opportunity for researcher to derive an automatic evaluation technique, at least I have seen in NLP area. With such auto evaluation techniques, we can do research projects in a larger scale and widely accepted by many other researchers. The difficulty of such subjective definitions is that researchers have to argue about the experiment's outcome. It is the researchers' job to convince readers that the results are valid and contributive. That's why the author has focused on evaluation, e.g. validity, reliability, in the later half of this chapter. It is shown that we have not had an accurate tool to read human's mind. Therefore, all experiments have to be considered very careful in order to convince readers. Even in the examples of indirect dependent variables, physiology measures have been used by many researchers yet some results will become invalid as new research shows that the physiology measurements indicate different aspects. The research areas show challenges as well as interesting questions for researchers. ***How to interpret experimental results: this chapter guides us how to present and interpret results from psychology experiments. Although many techniques here may not applicable for otehr research areas, the author has done a good introduction to basic plotting techniques. I find it is helpful to know the assumptions and limitations of various statistics mentioned in the chapter. In particular, I think the meta-analysis may be very helpful for researchers when starting a new research project and need to get foundation knowledge about the field. This is the first time I have heard about such technique. Another comment about the chapter is there are many techniques for many kinds of output data. However, I think another factor should be considered is the current research question. Because the objective of a study is answering some research questions. Even though the research questions would decide the form of output data, experiment result but I believe this factor will play a critical role when chosing the statistical tool for analyzing the results.

Mengsi Lou 22:16:26 10/29/2014

Chapter 2: How to do experiments ------------This chapter discusses the experimental methods. It can be seen in the design of variables. ------------The first is the independent variable that is independent of participants’ behavior. And it manipulates the experimental circumstance. The second is the dependent variable that is dependent on what the participant does. The dependent variable measures the behaviors. The third is the control variable that is set at the particular level and not allowed to vary. The concept of control is vital that it light the point in the experiment. And here are two traps about the control variables. One is that it is not possible to control all the variables and the other is that we do not wish to control all the variables in experiments. The fourth is the random variable that is the circumstances can vary randomly. For example, the samples had been randomly selected that is random samples. It claim the external validity. The fifth is the confounding variables. ------------Here are some threats to internal validity that we should avoid. The first is the history that is some events take place between the testing of the levels of the independent variables. The second is the maturation that is the participants may grow old and have more experience than before. The third is the selection that is the participants are selected nonrandomly. And the mortality, testing and statistical regression are also threats for experiment. ------------The last is two figures shows the concepts and relations about different variables and how they consist the experiment. ///////////////////////////////////// Chapter 7: How to decide which variables to manipulate and measure ------------This chapter tells about how to choose the variables. ------------As for the independent variables the first thing we should do is to specify the operational definitions. Then later other experiments will follow the definitions. The second step is to choose the range of independent variables. There is no certain ways for how to choose but here are some guidelines that is be realistic, select a range that shows effect and do a pilot experiment. The last one doing a pilot experiment is to do a small range of experiment that can iron out any problems before proceed. ------------As for the dependent variables the first thing is also to give operational definitions. And the second we should be able to show the dependent variables are reliable and valid. For the reliability can be test through test-retest, alternative-form and split-half. The validity is the variables agrees with the commonly accepted standard. There are several ways to test validity that is face validity, content validity, predictive validity and the concurrent validity. And there are two ways of measuring one is directly observable dependent variables and the other is indirect dependent variables. ////////////////////////////////// Chapter 13: How to interpret experimental results ------------This chapter discusses the display of experimental results. ------------The first step is to plot a frequency distribution illustrating the data points of dependent variables. Three distributions are used one is the normal distribution, and the second one is the bimodal model. The third one is the mode, median, and mean. And the two statistics are used are the range and the standard deviation and the variance. And the graphs helps illustrate the relationship between the independent and dependent variables, such as bar graph, function line graph. To determine whether there is the main effect is the important point for interpreting the results of a factorial experiment. Last, the meta-analysis is a statistical technique for combining the results of experiments.

Eric Gratta 22:35:59 10/29/2014

How to Do Experiments David W. Martin Since this is not a research paper, this response will consist of summarized definitions and notes from the text. The examples and diagrams throughout the readings were really useful. The last reading on how to interpret results did not contain any novel information, at least for me. Independent variable: the circumstance of major interest in an experiment, something independent of your participants’ control. Each different manipulation of the variable is a “level,” and at least two levels are needed to be conducting an experiment. Dependent variable: something that is measured during the experiment because it depends on the behavior that occurs. Usually we want to see the relationship between the independent variables and the dependent variables; a prediction about that relationship would be the hypothesis, but the hypothesis could be vague and just ask what the relationship is. Control variables: independent variables that stay the same across all trials/participants so as to reduce complexity in terms of what might affect the measurements of the dependent variables. External validity: how generalizable an experimental finding is. If too many variables in an experiment are controlled, special circumstances have been created that are too specific to be useful. Random variables: those variables that were not controlled and were allowed to vary randomly when the experiment took place. Some of these, particularly details of participant background, we have no control over, but others may be levels of our independent variables that we randomly assign to incoming participants, possibly with constraints. Randomness can improve generalizability of results but decrease precision. Confounding variable: something that varies in relation to an independent variable, such that no claim can be made about the independent variable. This is one of many threats to internal validity. Threats to internal validity: 1) changes that have occurred due to “history” or a relevant gap in time, 2) maturation of participants in age or experience or both, 3) non-random selection of participants, 4) differential mortality or participant drop-out from experiments from certain levels of the independent variable, 5) testing (or surveying) that sensitizes and informs participants, 6) statistical regression, 7) interactions with selection and some other variable that weren’t anticipated. Some independent and dependent variables will require operational definitions so that they have a meaning precise enough so that others may measure those variables in the same way. However, it is important to refer to definitions already made by other researchers when they exist. The range of the independent variable is just the span from the lowest to the highest level. To make findings generalizable, a meaningful range should be chosen; avoid the “sledgehammer” effect where the range chosen is so wide and difference in conditions so extreme that there must be an effect. Pilot experiments are small-scale experiments that you can run to check for and fix any problems in the experimental setup, like choosing the range of your independent variable. When measuring your dependent variable, if you have come up with a new test or method of measuring something you gave an operational definition for, you need to (at least informally) prove the reliability of your measurements. For tests, there are a variety of methods for determining the test-retest reliability. Using multiple dependent variables improves the chance that you have measured something valid or meaningful for your experiment. However, to make interpreting these many results easier, the variables should be combined into some composite dependent variable. Also, you may be trying to observe behavior that cannot be measured directly, and so you will have to make a convincing argument that the indirect dependent variables you are measuring (and the ways you are measuring them) have a strong correlation with the behavior you want to measure.

Longhao Li 22:53:14 10/29/2014

Critique for Doing Psychology Experiments Chap. 2 This chapter of the book talked about different variables that need to care about when doing experiment. It also talked about the validity of the experiment. In this chapter, the author talked about what is independent variable, what is dependent variable, what is random variable and what is confounding variable. Basically, independent variable is the variable that will not be changed due to the change of other variables. Dependent variable is the variable that will be changed due to the change of other variables. Smart control of independent variables may lead to the success of the experiment. Also confounding variable need to be taken into consideration since it may make the result be not valid. Understanding the knowledge of the variables is important for doing experiment. It will give you the opportunity to clear define the experiment so that you can conduct it correctly and carry out reasonable result. I think doing experiment without control will lead to no result since we cannot see what the experiment want to see in the result so that understanding the materials in this chapter is important. Critique for Doing Psychology Experiments Chap. 7 In this chapter, the author talked about how to select and control variables. It also includes some suggestions for the preparation of experiment. At the beginning of this chapter, the author talked about how to choose an independent variable, which include how to define your independent variable and how to select the range of the independent variable. Then the author talked about how to choose dependent variable. It includes operational definitions and different kind of dependent variable, single dependent variable, multiple dependent variables, composite dependent variable, and indirect dependent variables. Also the author talked about the reliability and validity of experiment. All the materials are very helpful for researchers to do experiments since it include guidance for the key parts of doing experiment. In this chapter, the author introduces pilot experiment. I think it is a great part that experiment conductors need to think about it. It seems that doing some rehearsal of experiment is somehow waste time. But actually it is very important, since you can find something wrong about your design of experiment early. Find error during the experiment will not be acceptable since you need to redo the experiment again. It is wasting researchers’ times and also participants’ time. Critique for Doing Psychology Experiments Chap. 12 This chapter talked about how to deal with experimental result by using statistics’ knowledge. Graphical explanation makes the article much easier to understand. To my understanding that this chapter is important, since it introduced the statistical method that is needed for experiment. First is how to deal frequency distribution, which included normal distribution, bimodal distribution, truncated distribution, and skewed distribution. Then the author introduced statistics for describing distributions, which include the idea of mode, median and mean etc. The author also introduced different type of graphs, bar graph, histogram and line graph. At the end of the article, the author talks about inferential statistics, which include how to prove significance of difference. In these statistical methods, I think the most important part is the inferential statistics, since it determined the result of experiment. Once you carry out some experiment results that cannot be prove significance of difference, the experiment needs to be conducted again. This is happening for a lot of PhD students. Some experiments need to be conduct for hundreds of times to prove the significance of difference. Thus, I want say that patient is a basic requirement for a PhD student.

Yanbing Xue 0:20:26 10/30/2014

The author mainly talks about three aspects of psychology experiments: experiments, manipulation and result interpretation. When doing experiments, one of the most important thing is how do set variables. According to different kinds of variables, we adopt different methods. For independent variables, as experiments, we manipulate it -- that is, choose two or more levels to present -- and nothing the particiant does can change the levels we have chosen. For example, if our independent variable is light intensity, we might select a high-intensity light and a low-intensity light as ourvtwo levels and observe behavior under both circumstances. Without at least two levels, we are not doing an experiment, but we are free to choose many more levels or to have more than one independent variable. For dependent variables, in the reaction-time experiment, for example, we want to find out whether a relationship exists between light intensity and time to respond. Thus, our dependent variable is the time from the onset of the light until a button is pressed. It is sometimes useful to make a statement about the expected nature of the relationship; such a statement is called a hypothesis. In the example,we might hypothesize that the more intense the light, the quicker the response will be. The outcome of the experiment will determine whether the hypothesis is supported and becomes part of the scientific body of knowledge or whether it is refuted. For control variables, we can control such circumstances by seeing that they do not vary from a single level. For example, in our reaction-time experiment, we might require that the lighting conditions in the room be constant, all participants be right-handed, the temperature be constant, and so on. Ideally, all circumstances other than the lighting conditions in the room be constant throughout an experiment. We would then know that any change in the dependent variable must be due to changes that we had brought about in the independent variable. However, even though many variables in your experiments will be control variables, you should realize that, especially in psychology, not all variables assigned as control variables. First, it is impossible to control all the variables. Not only is it impossible to force cooperative attitudes, attentional states, metabolic rates, and many other situational factors on our human participants. For random variables, there is a population of items and some random process is used that makes the selection of any one item from that population as likely as the selection of any other item. Random selection is used to ensure external validity -- that is, to ensure that the sample of the items randomly selected from the population is generalizable to that population. So, if you wanted to generalize your results from an experiment to all people in the United States, ideally you would use a means of selection that was equivalent to putting the name of everybody in the country into a very large hat and drawing out a sample of names. You could then say that your sample had been randomly selected and you could claim good external validity of your findings.

Wei Guo 0:23:53 10/30/2014

Reading critique for How to Do Experiment This paper first introduces types of variables: independent variables, dependent variables, control variables, random variables, randomization within constraints, and confound variables. Paper also addresses the threats to internal validity: history, maturation, selection, mortality, testing, statistical regression, and interaction with selection.  Reading critique for How to Decide Which Variables to Manipulate and Measure This paper introduces the choosing of independent and dependent variables. Independent variable is the one that the experimenter manipulates. To choose independent variable, we should define the independent variables. That is specifying the operations anyone must go though to set up the independent variable in the same way they did. After defining, we should choose a range of the variable--the difference between the highest and lowest level of the variables. And then select a range that shows effect. Finally determine the best range for an experiment. For the dependent variable, defining it is the first step, too. Then we need to judge the reliability and validity. For direct dependent variable and indirect dependent variable, the processing will be different. Reading critique for How to Interpret Experimental Results With the experiment results, there are some ways to plotting them: Plotting frequency distributions. We can analyze this kind of plotting by considering central tendency and dispersion. Plotting relationships between variables. By drawing graphs and doing analysis, we can describe functions between variables. This paper is basically the same as black-box metaphor professor talked in class. With the independent input, we will get the dependent output. We need to control those we do not interested in, and try to eliminate the threats. For our project evaluation part, we can follow the above instructions to choose independent and dependent variables. And when having the result, we can analyze it by either plotting them, or plotting relationships of them.  Although this is a phycological paper, the method for evaluation is fitting to CS area. With these basic knowledges, I hope we can improve our evaluations in final project.

Wenchen Wang 0:26:23 10/30/2014

<Chapter 2 How to do experiments> <Summary> This chapter introduces the experiment method, in aspect of variables and threats to internal validity. <Paper Review> There are five kinds of variables, such as dependent variable, independent variable, control variable, random variable and confounding variables. Random variables are the variables that allow some circumstances to vary randomly. Those kind of variable involve random sample and random assignment. Another aspect is threats to internal validity. There are three kinds of threats, maturation, selection and mortality. Maturation is a threat to internal validity caused by participants. Selection is a threat when a participant is assigned nonrandomly. When a participant drops out of an experiment, it is also a kind of threats to experiment. <Chapter 7 How to decide which variables to manipulate and measure> <Summary> This chapter introduces how to choose independent variable and dependent variable. <Paper Review> There are five kinds of variables, such as dependent variable, independent variable, control variable, random variable and confounding variables. No matter what kinds of variable, we need to define them in different circumstances first. Then we need to select a range for each variable value. It is very important because for some high security project, such as nuclear power plant. We need to define the safe range for each variable, such as temperature, position and flow rate. <Chapter 12 How to interpret experimental results> <Summary> <Paper Review> There are several ways to interpret experimental results. First method is plotting frequency distributions, such as normal distribution and bimodal distribution. Second way is to describe distributions by statistics. Third way is to plot relationships between variables by drawing graph and function description. Another way is to describe the strength of a relationship by scatterplots and correlation coefficients. Meanwhile, we also need to interpret results from factorial experiments in terms of main effects and interactions. In addition, we can also use computers and software to help us interpret results, such as Matlab and processing. To be specific, I think Matlab is a very powerful software to analysis experiment data.

Brandon Jennings 0:51:34 10/30/2014

How to Do Experiments This paper is about how to properly conduct experiments. Goes through the different components of experiments and defines the terms and explains how they relate and what they mean. It discusses the different kinds of variables that make up an experiment and how they relate to each other (i.e. independent and dependent variables). It also talks about constraints and how they help to narrow and pinpoint experiments. It goes on to talk about how to maintain validity of the experiment while testing and using statistical regression to analyze the results. How to Decide Which Variables to Manipulate and Measure This paper is about choosing the dependent and independent variables. It says you want to operationally define your variables such that the experiment may be repeated the same way by others. When picking an independent variable you want a range wide enough to show effects. Sometimes preliminary testing might be necessary to ensure a good set of independent variables. When choosing a dependent variable you have to take into account reliability and validity. Reliability is important when using things like instruments. It is as critical to measure what it is you intend to measure. There also different types of dependent variables such as directly observable and indirectly observable and understanding their relationship with the independent variable is important. How to Interpret Results This paper is about interpreting the results from an experiment. It describes the many ways to analyze data from experiments. Plotting frequency distributions can help to visualize trends and recognize patterns over some metric. Statistics are a common tool used to analysis. Statistics like mean, medians, and modes help to describe the normality of particular variables, as well as the range. Different graphs can be used for different types of experiments, or different types of analyses. Scatter plots help show the strength of relationship between variables, a metric measured by correlation coefficients. Statistical tests like parametric and nonparametric but be chosen carefully. Use parametric when it is assumed a normal distribution of frequency for the population of interest. Nonparametric should be used otherwise.

changsheng liu 0:54:12 10/30/2014

Doing Psychology Experiments --chap 2, 7, 12 --- In chapter 2, the author presents method to do experiments by explaining different types of variables. It also introduces the threats to internal validity. There're six types of variables: The first one is Independent variable, which is the variable that independent of the participant's behavior; The second one is dependent variable, which is the participant's behavior we choose to measure in response to manipulations of independent variable; The third one is control variable, which is the variable controlled by the researcher and is set to be constant; The forth one is random variable, which is the variable that cannot be controlled. The fifth one is randomized with partly control; The final one is confounding variable, which is any circumstance that changes systematically as the independent variable is manipulated. The paper also talks about the threat to internal validity. For example, the factor History, one can usually collect data at all levels of the independent variable over a relatively short time span. Any changes in the dependent variable is likely to have been due to history, that is to some event that takes places between the testing of the levels of the independent variables. Also, maturation is an interesting factor. The participants involved in the experiment might grow older or perhaps more experienced. Also, selection of the participant should be random. One subtle threat is statistical regression. This term refers to the fact that when participants are chosen on the basis of having scored very high or very low on a particular test, their scores tend to move toward the mean on a second test. The 7th chapter describes how to choose variables to manipulate and measure. We can first choose the independent variable. Due to the difference in precision between what the general public will accept in defining a term and what experimental psychologists will accept. Experimental psychologists require operational definitions of the independent and dependent variables. Once you have defined your independent variable, you still have to choose the range of the variable. We should be realistic when choose the range. Meanwhile, the range should large enough to show the effect of independent variable. Choosing dependent variable is similar to the previous one. We should define the variable at first. In some special case, even when a dependent variable seems quite straightforward, there can be problems with operationally defining it. The variable must be reliable and valid. “Reliable” means we should get exactly the same result when we repeat the experiment. Dependent variable is much more complicated than independent variable in terms of multiple dependent or composite dependent, indirect dependent. Finally , the chapter points out, physiological measures may provide an indication of internal states, but they are often difficult to interpret. Behavioral measures such dual-task methodology also offer the possibility of determining a participants’ internal state. Chapter 12 explains how to interpret experimental results. Frequency distribution is always the first step to show the result of the experiment. It illustrate the number of point occurring within categories of the dependent variable. Sometimes these distributions are similar to a symmetrical bell-shaped distribution called a normal distribution. Two statistics are commonly used is range and standard deviation. The range is the difference between the highest and lowest scores. The standard deviation is the dispersion of distributions. A bar graph can be used to illustrate data points that represent qualitatively different categories. Histogram or functional line can be used to illustrate continuous variables. To do factorial experiment, you must determine whether there is a main effect. In some case, we need to prove the difference between data samples is due to chance rather than due to a real difference in populations. This is called inferential statistics. Finally, the author introduces meta analysis, which is a statistical techniques for combing the results of many experiments. As the book points out, we can use computer to help us interpret the data, which is much efficient.

zhong zhuang 2:45:35 10/30/2014

This chapter of book is about experiments in psychology. Experiments are very important in every research area, but the theory of how to conduct experiment is sometimes ignored by researcher. In this book chapter, the author introduces some basic theory in how to conduct experiments in psychology, but these theory is actually applicable in all research area. The structure of an experiment is simple, a circumstance is manipulated it causes a change in some behavior. The term for the circumstances are called independent variable which is relatively controllable by the experimenter, the behavior is called dependent variable which responses to the circumstance change. Besides these variables, there are other variables, control variables is something that is also controllable but remains unchanged during an experiment. Random variable is something that may be allowed to change in a random fashion. variable randomized within constraints, these variables are like random variables but the experimenter can impose some limitations on them. The last variable is confounding variable, this variable normally is unwanted in an experiment because they change systematically with the independent variable and distort the relationship between the independent and dependent variables. Because of these confounding variables, internal validity exists. it will make it difficult to say that only the independent variable caused a change in the dependent variable. There are some threats to internal validity. First is history, during a long experiment, some uncontrolled event may happen. Second is maturation, during an experiment, the participants may change in experience and skill. Third one is mortality, this is caused by participants quitting and reentering the experiment. The last one is statistical regression, the movement of scores toward the mean for groups selected on the basis of extreme scores.

Xiaoyu Ge 3:44:17 10/30/2014

Chapter 2 This chapter introduced that the influential circumstances with human behavior can be classified as variables using experiment method. Variables include independent variables, dependent variables, control variables, random variables, variable randomized with constrains and confounding variables. And history, maturation, selection and mortality were also introduced as threat. Independent variable considering the manipulated circumstance, control variables indicating circumstance that might be set but not allowed to vary, and other circumstances were random variables. The catalog and analysis of circumstances that influence human behavior construct a good direction for human computer interface design. Since interface design is based on human behavior, the classification of variables becomes very useful. By classifying all these circumstances, HCI interface design will be able to consider only general solutions to cover the same type of influential circumstances. Even though the general solution might not cover some circumstances, it will still greatly reduce the design and implementation complications and difficulties. In that case, the author`s metrologies regarding phycology should be carefully studied for the purpose of HCI design. Since I am learning the HCI design, I will make used of the circumstance variables classification to design better HCI projects. Chapter7 This chapter focused on defining a procedure for experiments. As for choosing independent variable, firstly the author specified an operational definition to standardize similar experiment. Secondly, choose a level of range using pilot experiment. Thirdly, validation tests were introduced. There are several ways for test: face validity, content validity, predictive validity and concurrent validity. And all these measures provided by physiological experiment lead to human behavioral measurement. Consequently, the experiment methodology and procedure along with the method on how to preform effective validation on human behavior should be greatly adopted by HCI design and testing. Since HCI is not only considering software and hardware technical difficulties, user experience is the most important aspect of it. Human behavior validation methodology can be use to create test cases for HCI product. And by adopting the physiological methods, the HCI testing procedure can be adjusted and systemizes which will leads to better and more satisfied HCI product. Chapter 12 This chapter introduced data analysis methods for experimental results. It basically introduced basic frequency domain signal process and stochastic process methods such as using Gaussian normal distribution, extract skew, mean, mode, standard deviation and variance correlation coefficient as features for the evaluation. And plot the frequency domain data into bar graph, line graph and curvilinear graph to analyze the trend of experimental data. These data analysis techniques introduced in this chapter have already been deeply utilized in signal process research areas, and all these frequencies domain process were the most basic techniques. Since it is frequency domain analysis there are a lot of advanced algorithms developed nowadays in signal process area to deal with not only frequency domain problems but also in other domains such as s domain. Even though frequency domain data analysis method is functional in some cases, there is always a possibility that analysis in other domain may lead to a better and accurate result. I think it will be useful to use these data analysis techniques for HCI experiment results. However, as my work experience in several different domain analysis process, this probably not the best analyze method. And it needed to be further investigated.

zhong zhuang 4:34:04 10/30/2014

This chapter teaches how to decide which variables to manipulate and which are to measure. The first step will always be define the variable, if we want to measure successfulness, then we need to define what success means. This technique is called operational definition, it specify the operations anyone must go through to set up the variable in the same way they did before. For the independent variable, we need to determine the range, it needs to be large enough to reflect change but it can not be at extreme case, otherwise the experiment will be meaningless. For example, if we select temperature as the independent variable, it can’t be in the range of 20-21 degree because it is not enough to reflect change, but I will be meaningless if we choose the range as -30 to 100 degree. To determine the range, the experimenter can conduct a pilot experiment. For the dependent variable, the definition needs to be valid, in other words, it needs to be commonly accepted by other researchers. There are several ways of doing the validity process: face validity, content validity, predictive validity and concurrent validity. Besides, the researcher needs to know which category does the dependent variable falls into, directly observable variable is easy to measure, indirect variables are for some not publicly observable behavior.

zhong zhuang 5:09:59 10/30/2014

This chapter tells about how to interpret experiment results. It is very introductory and focuses on very basic statistic theory. It briefly introduced normal distribution, and some other distributions like bimodal, skewed and truncated. It explains how to calculate mean and standard deviation and some other basic metrics of a statistic graph. Then it jumps to how to draw various kinds of graphs, of course the most basic one is just the abscissa, then bar graph, histogram, and scatter plot. Besides statistic, it also explains how to interpret the results of a factorial experiment which normally can’t be represented by numbers. The main step is to determine whether there is a main effect. Then the experimenter needs to decide if there is interactions between factors. The experimenter also needs to pay attention to whether data samples are different due to chance or due to real difference, this can be done by inferential statistics. Finally, the authors talks about meta-analysis, this is a statistical technique for combining the results of many experiments. Also computers can be used to help statistical analysis, the most famous tool is matlab.

Jose Michael Joseph 6:27:26 10/30/2014

Chapter 2: How to do experiments This chapter deals with the various considerations that we must take while doing an experiment. This involves the various primary components of an experiment such as an independent variable and a dependent variable. There are other secondary components such as control variables and random variables. The point of the experiment is to increase the external validity of the experiment. Researchers should at the same time try to avoid confounding variables which are variables that distort the data that is obtained from the experiments. In this chapter we learn that an independent variable is something who’s effect we wish to see. In the chapter the example given is of children watching violent TV shows. Thus in this context violent TV shows are the independent variables. Dependent variables are those that change with the independent variable. They serve to show the influence that the independent variable has on the dependent variable. In the example in the book, the dependent variable was the child’s aggressiveness. The chapter also discusses control variables which are variables used to ensure that an experimental set up has some constant values. These constant values are needed to ensure that the various people participating in the experiment have some common baseline. If everyone was participating in a scenario that is unique to itself then this data would not co-relate well and thus would be essentially useless. Random variables on the other hand are present to introduce an element of randomness in the experiment. This helps us to bring more realism into the experiments as most real world situations have a high degree of randomness present in them. Random variables thus improve the external validity of the experiment. The external validity of the experiment is the degree to which the results of the experiments can be generalized. Results obtained from a certain sample of population may not be sufficient to generalize over the whole population and thus external validity is a very important aspect of experiments. The drawback of this chapter is that it has not talked in detail the effect of random variables and confounding variables on the experiment as a whole. It does not advise us on how to choose these variables and what should they extent of contribution be in the whole experiment. Chapter 7: How to decide which variables to manipulate This chapter talks about the various ways in which we can choose the different variables for our experiment. This is a crucial step as variables play a very important part in our experiments and often define the results to some extent. We go about the various ways we can choose different values for the variables defined in chapter 2. The first thing that the author asks researchers to choose is an operational definition to choose a variable. An operational definition is a procedure by which the variable was chosen. For the example of children watching violent TV shows an operation definition would be something like adults watching the show and then classifying whether it is violent or not. Such operational definitions are crucial as it gives a baseline for future experiments to use the same methods to generate their variables. This is extremely important to ensure that the results of the various experiments are comparable. If the experiment we are conducting has never been conducted before then we must conduct a pilot experiment. A pilot experiment is one where we conduct the same experiment with relaxed constraints and sometimes only in parts. This is done so that we may understand the inherent difficulties of this experiment and thus use this knowledge to prevent such problems from appearing in this real experiment. Thus it is a trial run of sorts to familiarize ourselves with the conditions of the actual experiment. In choosing a dependent variable we must also ensure that it is reliable and valid. It is said to be reliable if it produces the same result on multiple runs of the experiment under the same conditions. This is a very important factor because if with the same input we see the experiment producing different outputs then we can never co-relate our findings to any real application as we see that our output is unstable. Thus reliability forms a very important aspect of deciding our dependent variable. Another aspect that is equally important is validity. A variable is said to be valid if it agrees with a commonly accepted standard. There are various ways to measure the validity of a variable and some of them are face validity, content validity, predictive validity and concurrent validity. In some experiments we may have to measure multiple dependent variables. In such a case it is hard to determine which is the variable that we should put focus on or how much should we distribute our focus between the various variables. These multiple variables can also be combined to form a single composite variable which in essence would be easier to understand as it would put all the different conditions into a single measurable unit. Indirect dependent variables are used when the quality we are looking for is not directly measurable. In real life most qualities will be indirect dependent variables. The drawback of this chapter is that while the author has discussed the possibility of having multiple independent variables, the author has left out the possibility that there could be multiple independent variables that influence a single dependent variable. Such a relation would be much harder to map and needs more emphasis. Chapter 12: How to interpret experimental results This chapter mainly deals with the various ways we can interpret the results that we have obtained after conducting our experiment. This is an important aspect as generally we are quick to obtain results but are then confused on how to interpret them. This chapter provides the different ways we can visualize and interpret the results into real life applications. One of the primary ways to interpret the results would be by using a frequency distribution. We could segregate the data points depending on the categories in which they occur and display it using a frequency distribution. The different types of frequency distribution are normal distribution, bimodal, with two most frequent categories, skewed, with more values in one side of the distribution and truncated where there are no values on one side of the distribution. The commonly used variables to describe results from these graphs are mode, median and mean which each having their advantages and disadvantages. Range, standard deviations and variance are used to describe the dispersion of data in the result. We may also use graphs to visualize data. Bar graphs are useful when we have data that explicitly resides in different categories. Whereas histogram or functional line graphs can be used to describe continuous variables. The functions can be linear or curvilinear, positive or negative, monotonic or nonmonotonic, positively accelerated or negatively accelerated or asymptotic. The results of the experiment can also be described using a scatterplot. While interpreting the results of a factorial experiment we must decide which the main effect is and which are the interactions. Interactions occur when the effect of one variable is different depending on the levels of the other variables. We also have crossover interactions where the lines of interaction for these variables cross over. For a result to be accepted as statistically significant, we must show that it cannot occur by chance in more than 1 in 20 times. If it does then the results we produced could be due to random luck rather than due to some specific condition. Finally the chapter discuses Meta Analysis which is a technique that combines the results of all other experiments to give a unified picture of the results. This is important as we understand the general trend followed by the experiments as opposed to seeing the results as the byproduct of a single experiment.

Xiyao Yin 8:41:38 10/30/2014

Doing Psychology Experiments has now been available for 30 years and it still seems to be fulfilling its original function: to teach students having little or no background in experimentation how to do simple experiments in psychology. In chapter 2 ‘How to Do Experiments ’, it discusses about several different conditions and aspects in doing experiments. The circumstance of major interest in an experiment is called an independent variable. Dependent variable is things dependent on what the participant does. Control variable is also another circumstance in an experiment. Things we should do with the remaining circumstances in our experiment is to permit some of the circumstances to vary randomly and people also need to control part of the event assignments. Confounding variables can cause low internal validity and make it difficult to say that only the independent variable caused a change in the dependent variable. Threats to internal validity include history, maturation, selection, mortality, testing, statistical regression and interactions with selection. In chapter 7 ‘How to Decide Which Variables to Manipulate and Measure ’, it considers two decisions that have to be made when planning any psychology experiment, from the simplest to the most complex. Choosing an independent variable is about the most important decision people have to make. Experimental psychologists require operational definitions of the independent and dependent variables and an operational definition is a bit like a recipe, except the procedures and ingredients create a variable rather than a cake. After defining independent variable, people need to choose the range of the variable. Formally determining reliability is particularly important when the dependent variable is the score from a test instrument such as a test of achievement, aptitude, or personality traits. Test-retest reliability, alternative-form method and split-half technique are good ways of establishing reliability. Validity refers to whether we are measuring what we want to measure. Face validity, content validity, predictive validity and concurrent validity are different procedure of establishing validity. Although directly observable dependent variables are relatively easy to measure, deciding which single dependent variable to use is still sometimes difficult. Indirect dependent variables are used when the behavior we are interested in is not publicly observable. Physiological measures are good to provide an indication of internal states. In chapter 12 ‘How to Interpret Experimental Results ’,it gives us an understanding of the logic underlying data analysis. Interpret the data is an important work after completing an experiment. A normal first step is to plot a frequency distribution, sometimes it is also called a normal distribution. Skewed by having more scores in one tail of the distribution or truncated by having one tail of the distribution missing are two frequent categories in interpreting the data. Mode, median and mean are significant mathematical concepts in analyzing the data. Using these three ideas, people can understand the change and frequency in different data. Graphs illustrate the relationship between the independent and dependent variables. Bar graph, histogram and functional line graph are useful material material and technology to show functions of data, by describing whether they are linear or curvilinear, positive or negative, monotonic or nonmonotonic, positively accelerated or negatively accelerated, or asymptotic. These seem to be the most important ideas in interpreting results.

Nick Katsipoulakis 8:52:09 10/30/2014

How To Do Experiments :: In this chapter of the book, the author is introducing readers on the proper way of doing experiments. A number of important definitions are provided. Initially, different types of variables are presented that should appear in an experiment. Independent variables are used to set the basic dimension in which a user participant's behavior is measured. Dependent variables are used to model user's actions during an experiment and Control Variables are independent variables that are controlled by the person conducting the experiment. The latter variables are important when there exist a number of independent variables in the experimental conditions. Furthermore, Random Variables represent random samples or random behaviors of the test subjects. In order to get representative samples of a population special attention should be paid on the way a sample is formed, so that it is a representative subset of the population. Finally, confounding variables and they are defined as circumstances that change systematically as the independent variable changes. In the next part of the test, a number of aspects for internal validity are presented. In addition, examples of experiments are provided in order to understand how can the validity be compromised if special attention is not given to the test subjects. //---------------------------END OF FIRST CRITIQUE --------------------------------------// How To Decide Which Variables To Manipulate and Measure :: During the experimental evaluation process it is important to be able to isolate and modify parameters of the testing environment in order to get a better understanding of causality. This chapter provides helpful information on the proper methods for choosing independent variables in an experiment. Initially, specific definition of the problem and the independent variables should be used, to avoid fuzzy terms and poorly defined criteria. Also, independent variables should be realistic, have values to make the effect visible and before starting formal experimentation, pilot tests should be conducted. Turning to dependent variables, operational definitions need to be established with precision and validity of test results should be reinforced by examining the correctness of experimental outcomes. Finally, cases in which a plethora of variables is involved are presented, and methods for handling are discussed. //--------------------------END OF SECOND CRITIQUE------------------------------------// How To Interpret Experimental Results :: The author presents the statistical tools needed for interpreting experiment results. A thorough analysis of mathematical ways to visualize and characterize results are provided. Next, the author demonstrates a method for interpreting results according to main effects and interactions. These methods are important to understand interactions between variables and main causes of different phenomena. On top of that, inference methods for experimenting are analyzed and the idea of Meta Analysis is presented. Overall, I enjoyed reading this chapter because it combined mathematical tools and what indications a scientist can get from them. //--------------------------END OF THIRD CRITIQUE------------------------------------//

yeq1 8:53:19 10/30/2014

Yechen Qiao Review for 10/30/2014 Doing Psychology Experiments Chapter 2 In this chapter, the author gave an overview of experimental variables, namely independent, control, dependent, and confounders. The chapter also discussed several common confounders in psychology experiments. For the students who may not be familiar with scientific terminologies, I think this chapter may be especially useful as they start to write a proposal. It is very easy for some students to think that there’s nothing in the methodology and experimental design, and we should always create the easy research questions and hypothesis. However, due to the fact that human research is potentially difficult to setup and time consuming, choosing the correct questions and hypotheses (independent and dependent variables), as well as setting up the experiment correctly (reliable and valid) is important and a potentially difficult task. For example, I just had this argument given to me yesterday: it is much easier for us to argue for user acceptance. I asked: how do you plan to do that? The response I got was: just let them do the tasks and give them a questionnaire. Suppose I got back plenty of answers that says they like the system and like the system a lot, what does this actually tells me? Since in this project, we are doing a pilot study, the sampling technique we use is not probabilistic. We may already have some relationships with many of these subjects. The experimental design leads the respondents to positive answers by letting them do the experiment with an obvious existing solution and an obvious baseline. Furthermore, studies have found that existing relationships and compensations often leads the respondents in answering questions positively. The answer for this question may actually be hard to extract without having some types of deception, and questions related to physical ease of use may be easier to answer and more interesting than this. It is important to be careful in selecting independent and dependent variables. Doing Psychology Experiments Chapter 7 In this chapter, the author discussed operationalization, reliability, and validity. After selecting a hypothesis for research, the investigator needs to pay attention on how the study can be performed to test the concepts. Operationalization depends on having an appropriate concept. In computer science, the concepts we study is generally easier to define than in social science as either the concept has been widely adopted by social scientists, or the concepts we are studying has a rigid engineering or scientific definition. The difficult part is how to create a sequence of operations that can produce reliable and valid results. This is important as when intersubjectivity is not reached, the papers are often either rejected or ignored by other researchers. A pilot study may be performed if some of the variables studied are new. This is typically the case in computer science as we introduce a new method or technology (a new algorithm, a new system, a new device, etc.). Pilot studies, when properly designed, allows qualitative measurement that quickly narrows down the number of variables of interest. If the variables are interesting, the researcher may decide to conduct a formal quantitative study later. However, when pilot studies are designed incorrectly, this may leads setting up an expansive experiment later which is likely to produce a disappointing result. See the example above for details. Reliability and validity tests discussed in this chapter can be useful in detecting these improper setups and leads to faster and higher quality research. Doing Psychology Experiments Chapter 12 In this chapter, the authors described some of the useful evaluation techniques. In particular, the authors mentioned some common distributions, some popular aggregates and plots, and some statistical tools that allows inferences of correlations. I’m not sure what else to say about this chapter, since the materials are typically found in statistics tests in undergraduate courses.

Qiao Zhang 8:58:32 10/30/2014

In Doing Psychology Experiments, the author introduces the way to do experiments that leads to making causal statements. In chapter 2, the author explains what is independent variables (the input of the experiments), dependent variables (the output of the experiments), control variables (the constants), random variables (to ensure the sample is statistical representable) and confounding variables (the ones the experiment conductors need to avoid). Internal validity is a property of scientific studies which reflects the extent to which a causal conclusion based on a study is warranted. Such warrant is constituted by the extent to which a study minimizes systematic error (or 'bias'). There are several threats to internal validity, such as maturation, selection, mortality, testing and statistical regression etc. The last one is particularly interesting to me, because I know that regression can mislead people by showing strong correlation between the confounders and the dependent variables. For example, I once read a paper published on New England Journal that claims chocolate consumption can predict the number of Nobel prizewinners. Yes, the correlation does exist, but I still feel doubtful towards the explaination although it is published on a authority journal. Such cases should be paid enough attention by the researchers before making any conclusions. In chapter 7, the author introduces how to decide which variables to manipulate and measure. On how to choose independent and dependent variables, researchers must first specify an operational definition of the variable so that other experimenters will be able to go through the same operations when they conduct similar experiments. This ensures a very important character of experimental science: a result can be repeated by using the same procedure. On choosing dependent variables, one must also operationally define those variables. The researcher must be able to show that the dependent variable is reliable and valid. Reliability means the same result is obtained every time a measurement is taken; validity means that the dependent variables agrees with a commonly accepted standard. There are several ways of establishing different kinds of validities. In chapter 12, the author talks about how to interpret experimental results. This is the most interesting part of an experiment. To analyze the data and provide a reasonable conclusion is my favorite part of all research procedures. On this topic, even though I haven't read this book before, I find myself using the exact techniques in this chapter (using the R programming language). When I get a set of data, normally I would plot it to get a high level idea of how the data looks like. If I find some interesting patterns, I will dig into it by plotting pairwise relationships. Strong correlations in data will be shown in this process. Interpreting the results is the rigorous part. One must determine whether there is a main effect and whether there is interactions (a situation in which the simultaneous influence of two variables on a third is not additive) between variables. Several other important statistical techniques such as significant test are also explained in this chapter.

Vivek Punjabi 9:40:15 10/30/2014

Chap 2: How to do experiments This chapter explains all the variables that can be used in an experiment. Independent variables are the circumstances that causes change in behaviours when they are manipulated. The dependent variables are the measure of the behavior. Control variables are used to set constraints on variable measures. Random variables maybe used to generalize the results. They can also improve the external validity of the experiment. Variables randomized within constrains can be used to vary randomly but within limits. We should also try to minimize or eliminate confounding variables which change in the background due to the independent variables and distort the results. This chapter gives a thorough analysis of various kinds of variables that can be considered while performing an experiment. This will help to evaluate the results an accurate way while considering all the variations and manipulations possible. Chap 7: How to decide which variables to manipulate and measure How to choose independent and dependent variables in order to get best results for the experiments. You choose an independent variable the most important thing is to define it in such a way experimenter performs the same operations. It is also important to choose the range of independent variables as realistic and effective. Pilot experiment or a trial run can also help. The dependent variable must also be operationally defined. Are the same time it should be reliable as well as valid. Reliability can be determind in several ways such as test-retest, alternative-form split-half. To determine validity, we can consider it's face validity, content validity, predictive validity and concurrent validity. We can also easily measure directly observable dependent variables. But it can be difficult to decide single dependent variable and so we can use multiple dependent variables or composite dependent variables. There are also independent dependent variables to measure the internal behavior. We can measure them using physiological measures and behavioral measures. So this paper gives a detailed analysis of the variables that we can manipulate while performing an experiment. It also provides examples for each different kind of variable and their manipulations. Chap 12: How to interpret experimental results This chapter describes various ways of interpreting the data after the evaluation experiment is performed. The data is assumed to be listed on the response sheets. One way is to plot a frequency distribution graph which can illustrate the number of the data points occurring within categories of the dependent variable which. The shapes of the graphs can be used to interpret the results such as normal distribution which is bell-shaped, bimodal which contains two categories, skewed which has a long tail of distribution, or truncated with a missing tail. Three commonly used for a distribution are mode which means most frequently occurring category, median which gives in the middle score and mean which gives the center of gravity for the distribution. To describe the dispersion of the distribution, commonly statistics are range which gives the difference between the highest and lowest scores, standard deviation and variance. These graphs gives the relationship the independent and dependent variables. We can use a bar graph, a histogram, a scatterplot or a functional line graph to illustrate discrete and continuous variables. With these graphs we can the results, which has the most effect along with tge interactions and crossover interactions. Inferential statistics can be used the find difference between data samples. We can also combine results of many experiments using a statistical technique called meta-analysis. We can also use computers to carry out statistical analysis, which can give better results the assumptions and the data are entered correctly. The chapter gives a thorough analysis of the methods to interpret the results and also it's consequences. It is useful to interpret the results in the correct manner in order to determine desirable results. However, there was lack of some non-statistical methods that can also be used to determine results of an experiment. Also, there should some ways to interpret results when there is much less data available or where large amount of data is not possible.

Yingjie Tang 9:50:23 10/30/2014

“Doing Psychology Experiment” is a book that is very interesting and it helps us build the basis of experiments in psychology. Since human beings are an important component of user interface systems, so we can reference some basic principles from the experiment of psychology to the evaluation of user interface systems. The 2nd chapter of the book tells us how to do experiments in psychology. Psychology experiment is pretty new compared to some natural sciences, thus the abstraction of the basic elements is not that obvious. Although we can generalize the variables like independent variables, dependent variables, control variables and random variables. The experiments are not that easy in psychology because there exists so many variables compared with natural science. Thus we have to classify these variables into these classes and handle them correctly. Someone may argue that we can make a specific experimental environment, but that will minimize the generalization ability of the experimental result. The tricky point in social science is that the environment varies a lot once we fix the experiment environment. In other words, there is a tradeoff between the expressive and the flexibility. When I read this chapter, I found out that the many of the concepts have already introduced by professor Jingtao Wang in the last lecture. Like the external validity and the internal validity. We should be careful to treat our experiment since there are a lot of factors could threat our external validity and internal validity. External validity refers to that the result generalized in our experiment can apply to other subject groups. Thus the limited number of subjects in the experiment could have a bad effect on the external validity. That’s because us humans vary from each other a great in some specific terms. Thus the coverage of the subjects to the whole domain should be as large as it can. The internal validity is a property of scientific studies which reflects the extent to which a causal conclusion based on a study is warranted. There are several things can threat the internal validity. Like the history, gender, and personality ect. In this point, I learned that in our class project, we should be very careful in the user study design. Our topic is to learn about the collaborative approach of the front and back of a smartphone to the unreachable problem. In order to make a valid experiment, we should do our best to recruit those who are the same skillful to use the back and front of a smartphone or to make some reimbursement in this case.————————————————————————- The 7th chapter “How to decide which variable to manipulate and Measure” is useful for us to referenced in human user interface systems. It tells us how to choose an independent variable or how to choose a an dependent variable. This is very useful because in psychology and human interface systems, the variables are in a large quantity where we are easily to lost ourselves. For example, when I decide to choose an independent variable for my experiment, I must specify and operational definition of the variable so that other people will easily go through the same operations when they want to verify my experiments. I used to design some user interfaces in web services which is in the framework of Service Oriented Architecture. I remember there are a lot of papers describing how to take use of the SOA. However, very limited number of them make a specification operational definition of the variable like what kind of operation system are they experimental in or what kind of web server did they experiment on. And the dependent variable must also be operationally defined. People must be able to show that the dependent variable is reliable and valid where the same results can be obtained on the same measurement. This is quite different from natural science where we don’t have to consider about the generalization of the results because nearly every experiment we conduct on the same object are in the same experiment. The experimental objects will not vary from the component to component. However, in psychology or human user interface systems, it is quite different. Since human is a component of the system, human’s ability and personality will change from time to time from space to space. Thus, when conducting test scores as a dependent variable, reliability of the test can be determined in “test-retest”, “alternative-form”, and “split-half” ways.————————————————————————————————— “How to Interpret Experimental Results” is very important to us and help us to convey the results to the readers. we must interpret the data listed on the response sheets. It can be frequency distribution to describe the number of data points occurring within categories of dependent variables, Or sometimes the distributions are similar to symmetrical normal distribution. Moreover, we can take the advantage of graphs which illustrate the relationship between the independent and dependent variables. Graph is very clear to convey the relationship between the dependent variables and independent variables. But we should be very careful because choosing what kind of graph and the exact implementation of a specific graph is very tricky. I remember when I was writing the graduation design for my undergraduate studies, I made a mistake to draw the graph which illustrate the correlation of the popularity of a web service and it’s design. I had a bias to choose what kind of variable to draw in the graph. In order to interpret the results of a factorial experiment, we should determine whether there is a main effect. And we must also determine whether the effect of one variable is different depending on the levels of other variables. These kind of differences are particularly with crossover interactions and can make interpretation of main effects difficult.

Christopher Thomas 12:15:27 10/30/2014

In How to Do Experiments, the author discusses experiments in the domain of psychology and human subjects. The author decomposes the discussion into experimentation, manipulation of parameters, and result interpretation. For experimentation, the author discusses that one of the critical elements of this is how to set the variables. The piece reminded me of the discussion that we had in class on Tuesday, where we discussed the importance of knowing the difference between different variable types, such as independent variables, dependent variables, constants, etc. So, this relates directly to Tuesday's lecture. Independent variables are those variables which get set by the experimenter during the experimental design. The participant's responses to various things will not alter or change the independent variables. One interesting observation that the author made which I didn't realize before is that without at least some variation (such as multiple levels) or multiple independent variables, we are not conducting a formal experiment and thus are measuring the amount of variation not subject to any dependencies. In contrast, dependent variables are those variables that we are measuring which depend on our independent variable settings. For instance, if the experimenter is attempting to measure accuracy, the independent variable would be the approach the subject uses, which we would alter between subjects to test. The dependent variable would be the accuracy we observe from the subject's choices in the experiment. It is important also to note, as we did in class on Tuesday that there can be variables that are not immediately apparent. For instance, if we first have subjects use one approach, then test the second approach, it may be that the subjects learn a little and gain some experience on the first approach which helps them do better on the second approach. In other words, the second approach gets a boost because it comes second. This kind of latent variable is known as a confounder and controlling them is extremely important because they can cause unexpected results. Another important type of variable is the control variable. The control variable is a variable which we try to set constant so that it doesn't affect our dependent variable because it is not something we are measuring. A good example of this is trying to control for gender. In other words, we will give our experiment only to men and this would prevent any gender differences from affecting our dependent variable and hence our results. Other control variables include lighting, time taken, time of day, even location. If one experiment is conducted on the top of a high mountain with less Oxygen, the experiments are likely to do worse than if it was conducted in a normal environment. Thus, it is critical to keep as many controls as possible to minimize unnecessary randomness from the experiment. Another type of of variable discussed is the concept of random variables, which connected to the lecture also. The second paper also discusses different types of variables and how to choose which is which and to decide different variable levels. In this way, there was a TON of overlap with this paper and the first paper. Importantly, though, this paper discusses how to select a range for the independent variable that will reasonably demonstrate the effect but is not unreasonable. Finally, this paper discusses also how the experiment can judge the reliability of the experiment and the concept of validity. Reliability is how well you think the measuring devices were working, which has an impact on how good the evaluation is. If the measuring devices aren't that accurate, it isn't OK to say that our approach has an effect that is even less likely than that attributable to measuring device or approach errors. The final paper discusses how we can interpret experimental results. This paper is focusing on evaluation of results and showing us various ways to analyse the data we collected after our experiment. One way that is discussed is a frequency distribution. This is interesting because we can model the distribution of frequencies of scores, etc. usually using a Gaussian or something similar. As such, we can make predictions and then measure the central tendencies of the distribution and the standard deviation. Also, the paper discusses how we can plot relationships between variables. It discusses 2d graphs to show how one variable effects another. Finally, it also mentions concepts of internal and external validity which were mentioned in class. Again, internal validity concerns the validity of the experimental design itself (whether or not the experiment is effective in measuring what we claim it is measuring or whether we are getting more random noise or something else). Again, we must always remember that correlation does not imply causation, which is something easy to overlook. All in all, I hope that I can use some of the approaches I read in this paper to enhance our project proposal with some of these new ideas.