Experimental Design

From CS2610 Fall 2017
Jump to: navigation, search

[ slides ]

Readings

Reading Critiques

Krithika Ganesh 18:50:48 10/2/2017

How to do experiments: This chapter explains in detail what kind of variables are required to define an experiment. In an experiment, when a circumstance is manipulated it causes a change in behavior. This circumstance is called independent variable and the behavior that is measured is called the dependent variable. Control variables can be set for an experiment which is not allowed to vary vs random variables that can vary. Random variables improve the external validity (how well a causal relationship can be generalized) whereas confounding variable changes the independent variable systematically as the circumstance changes there by reducing the internal validity (whether the manipulated change in the independent variable caused the change in the dependent variable or something else caused the change). The chapter also explains different threats to internal validity: historical events, maturation (more experienced higher the threat), selection (non-randomness), mortality(particular kind of participants drop off) , testing (sensitizing participants), statistical regression. What interested me is that how a decision to control increases the precision but decreases generality of the results and how a decision to randomize decreases the precision but increases the generality, hence one should carefully trade-off between generality and precision when it comes to designing the experiment. In the lecture pace experiment, it was assumed that if the background noise level was low, the students were most attentive. Is the right assumption made that the background noise level alone would be a major dependent variable? As measuring attentiveness is a psychological experiment one should consider using multiple dependent variables and indirect dependent variables. ------------------------------------------------------------------------------------------------ How to Decide Which Variables to Manipulate and Measure: This chapter explains how one needs to choose independent variables and dependent variables by defining them operationally so that it would be helpful to experimenters performing similar experiments. While choosing the independent variable one needs to choose the range such that it is large enough to show some effect and small enough to be realistic. While choosing the dependent variable one needs to see to that they are ‘reliable’ (if a test score is used as a dependent variable) by performing test-retest (first test items are same as second), alternative form (first test items are similar to second), split-half(single test split into 2) and see to that they are ‘valid’ by performing face validity, content validity (forming questions based on the content), predictive validity (predicts some specific criterion) and concurrent validity (predicts some criterion by taking the 2 measures as the same time). When one needs to observe human behavior, one must choose multiple, composite and indirect dependent variables as opposed to single dependent variable as one cannot conclude draw conclusions from a single dependent variable and one may doubt the validity of the experiment. What is interesting is that though one uses indirect dependent variables in the case of polygraph tests and pupillometries, they have not been successful as one could not be sure of the reliability and validity of the result. Even though using fMRI one can get picture showing the blood flow to various areas if the brain corresponding to the tasks performed by the person, can this concept be used to improve accuracy of lie-detectors or systems which predict the emotional state of the person? As the author states, indirect behavioral measures are only as good as the assumptions that underline it bringing down the confidence in our inferences! Could we in the future come up with ways to measure indirect behaviors with reduced assumptions?

Sanchayan Sarkar 12:05:00 10/4/2017

CRITIQUE 1 (How to Do Experiments)************** In this chapter, the author introduces a wide range of vocabulary on variables required to setup experiments, their uses and their level of influence toward the results. The merit of the paper is distinct categorization of variables. The first of this kind is Independent variable. The author mentions clearly this is the variable that is the direct cause of the effect that the experiment is trying to achieve. This is similar to the “x” variable of a time-series function. The next type of variable is the dependent variable which is the one that is caused by the change in the independent variable. This is similar to the “y” variable of a time-series function. The author gives clear illustrated examples to demonstrate this. The third category of variables are the control variables. These variables account for the circumstances that the experiment must face. Ideally, these variables should remain constant throughout the experiment. This is to get an entire cause-effect relationship from the independent to the dependent variable. However, the author cautions that fixing this too much would damage the generality of the experiment and make it extremely specific which is not good. The author discusses the use of random variables and random assignments to increase generality and external validity to the problem. However, the ideal solution lies somewhere in the middle of the random-control spectrum. This is where the author introduces the “Constraints”. Finally, the author asserts Confounding Variables”, which is where the circumstances change during the experiment and threatens the internal validity. Scenarios like history, maturation (experience), selection, dying out within the lifespan of experiment are circumstances that can affect the internal validity of the experiment. One of the advantages of this paper is the use of illustrated examples specifically using a diagram (Figure 2.1), where the different categories of variables are shown along with the relation from independent to dependent variable. The diagram gives a deep insight on how the whole range of vocabularies come at play. This paper is critical for those who are trying to form user studies in detecting behavioral patterns over a set of constraints and situations. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ CRITIQUE 2 (How to Decide Which Variables to Manipulate and Measure)**************** In this chapter, the author assert the considerations that a psychologist must take in order to choose the variables for the experiment. This chapter can be seen as an extension to the previous chapter. A merit of this chapter is that the author discusses several aspects of choosing an independent variable. First aspect is defining the operations where the scientist need to think the steps that the variable must go through to cause the effect. Second is the range of it. The range must be optimal such that the effect is noticed but not too large that it’s unable to specify the dependency to that variable. The third aspect is to construct a pilot experiment. In cases of subjective experiments like psychology and Human Computer Interaction, a scientist must look for a pilot experiment in order to learn the optimum range of independent variables. The author asserts that pilot experiments can also prune away certain absurd values for the independent variable. When it comes to dependent variables, the author introduces Reliability and Validity. These are cornerstones for selection of dependent variables, specifically for subjective experiments. Three ways to achieve this are: test-retest reliability, alternative form and split-half. The later two methods are based on positive correlations between the two groups of test to ensure reliability. This is relevant to my work in Face Recognition, where two different user studies were done on the AR Database to ensure a high reliability for maintaining structural consistency while changing the lighting factor (independent). Another interesting feature I liked is idea of multiple Dependent Variables to ensure validity. This is true in case of recognition problems where Sensitivity and Specificity are evaluation metrics to determine the overall accuracy of the algorithm. Similarly Rank-1 accuracy and recall both are “composite” dependent variables estimating a similar evaluation. Hence, the categorization of dependent variables is extremely useful. Finally, One of the hottest topics in modern age HCI is the use of indirect dependent variables like physiological variables like EEG signals, MRI scans, etc. and behavioral variables. Rise in multi-modal technology makes this paper more relevant. This paper is filled with clear illustrative examples that interested readers across disciplines would be interested in. However, had the paper, given more pictorial representations of the terminologies and its’ associated scenarios, it would have served the purpose better. Nevertheless, it is a critical chapter for understanding the crux of experimentation.

Tahereh Arabghalizi 16:25:22 10/4/2017

How to Do Experiments - Chapter 2: the author proposed experimental method in this chapter and different types of variables are defined. Experimental method helps us simulate changes in different situations and circumstances. Independent variables have two different levels: dependent variable (might be varied) and control variable (cannot change). Random variables are the ones that are selected randomly. Using theses variables we can increase the external validity of an experiment. In experiments we are not looking for changes and variations, but some situations can change under experimenter control. This is called variable randomized within constraints. As an experimenter we should eliminate confounding variables. Using confounding variables we know that independent variables make change in dependent variable which results in decrease in internal validity. We need high internal validity but there are some different factors that can affect internal validity. These factors include: history, maturation, selection, mortality and testing. ---------------------------------------------------------------------------------------------- How to Decide Which Variables to Manipulate and Measure - Chapter7: this chapter elaborates more details about the complications related to the variables. Independent variable selection is very important because the main goal of a particular experiment is to analyse the effects that a specific independent variable have on the environment. It begins with defining the independent variable and selecting the range of this variable. On the other hand, reliability and validity are the main parts in depending variable. Depending variables have various types including single, multiple and composite depending variables. Moreover, variables which change along the internal behaviour are called indirect dependent variables. Different measures such as psychological ones are used to gain knowledge about the internal event and to produce forms of inference. The other type is the behavioral measures that can be used as indirect variables as well. Although this chapter refers to experiments in psychology, the premises seem valid and can be employed in other areas of research.

Kadie Clancy 18:41:53 10/4/2017

How to Do Experiments: This chapter discusses in detail how to choose and set variables in an experiment. The differences between independent and dependent variables is also discussed, along with what situations call for control variables versus random variables. The author also speaks to the importance of eliminating or minimizing confounding variables, which change systematically with the independent variables and can distort the results of an experiment. Finally, the chapter addresses threats to internal validity which refers to whether the manipulated change in independent variables actually caused the change in dependent variables or whether the change was caused by a confounding variable. Threats to validity include: history, maturation, testing, and statistical regression. This paper can be an important resource for individuals who want to carry out valid experiments, as it provides examples of how to select variables. I think that the the reminder from the author that setting too many control variables decreases the generality of the experimental results is important advice to keep in mind when designing experiments. Results that aren’t general enough to be applied to a more general population won’t be usable in real life situations. Finally, I think that the advice to be aware that confounding variables have an impact on experimental results aids individuals who may be new to experiments in being careful that their claims are actually backed up by their experimental results. How to Decide Which Variables to Manipulate and Measure: This chapter discusses the two most important decisions that need to be made when planning an experiment: how to choose the independent and dependent variables. To choose independent variables, one must specify an operational definition for repeatability reasons. Choosing a suitable range of independent variables is also stressed by the author. Dependent variables must be operationally designed as well, with the added requirements that they must be reliable and valid. Reliability refers to and obtaining the same results each time a measurement is taken, while validity refers to measuring what we actually want to measure. Dependent variables can be of the form directly or indirectly observable. This chapter provides more insight on constructing valid psychological experiments and is, therefore, an important tool for those interested in testing how well a design suits user interaction or determining models of human cognition for HCI use. As HCI research deals heavily with cognitive research, I think it is important to keep in mind that we are often dealing with indirect dependent variables and that it is important to ensure that the results are valid, i.e. that we are measuring what we actually intend to measure.

MuneebAlvi 20:35:41 10/4/2017

Critique of both readings since they directly follow one another. Summary: The readings show how psychologists approach experiments from which a causal relationship is trying to be determined. These experiments involve variables and selection. Other aspects of conducting an experiment involve determining what parts should be variables and the kind of variables they should be. This reading showed me the intricacies of trying to perform experiments involving humans. There are a surprising number of variables which must be considered such as what condition the researcher is trying to change and what outcome they expect. There are also lots of other aspects that most people don't consider such as all of the other conditions occurring at the same time such as the weather, temperature, how much sleep the researcher or participant got the night before, etc... This can relate to human computer interfaces because trying to test intricate or complicated features of an interface with users can involve many variables. For example, a researcher might want to know if a pop up window with a help message will shorten the time it takes for a participant to perform a complicated task. However, there are lots of things to consider such as the age of the participant, how familiar they are with the software/hardware, the actual hardware used, the operating system, if the user has any problems using their arm or hand, the age or experience of the user with respect to computers, etc... However, if research could be done which shows which areas of a user interface are helpful vs not helpful, the interface could be improved after every study. In the modern world many software programs can send feedback to the designers about a users usage patterns. Users also talk on online forums or comments about their thoughts. In this way, the software is essentially always conducting experiments on users in the real world instead of a predefined setting. This allows a lot of user feedback and an iterative design for the interface.

Mingzhi Yu 21:54:01 10/4/2017

How to do Experiments: This chapter talked about how to design experiments and what the important variables are. This is an experimental design guide for researchers. It gives the readers some concepts how to manipulate experiments. In general, I will think this is a guide for all the nature science and maybe some areas of engineering. It mentioned that in order to have some generalization results that are more representative, we don't want to control all the variables but leave some space to let random factors play roles. I deeply agree with the point of view as I worked on some experimental data. It is the random acts let us to explore other potential latent variable and gives researchers the space to think and make a hypothesis. Thi article also provides some overview of common statistic analysis methods. Those are the methods we are very useful and very familiar to me. In general, this is a very good introduction level guide for researchers. I am guessing the reason that we are required to read this article is that the area of HCI is closely connected to psychology and human behavior. It is necessary to have a good sense of how to design a good psychology experiment. How to decide which variable to manipulate: This is the 7 chapter in the same book. This one is much deeper than the chapter and explained how to choose dependent and independent variables. It systematically describes the meaning, importance, tips of different experiment variables. It I他mentioned an interesting measurement physiological measures, which I encountered before. Also, the dual task measurement is very interesting. They are both under psychological assumption but it seems work pretty well so far. I felt there are many factors in the test that will change the test results but those factors are not necessary within the range of examining.

Jonathan Albert 21:54:53 10/4/2017

CH 2: This chapter discusses the types of variables present during experiments. It focuses on "confounding variables" and other factors that degrade the validity of tests. The author's discussion of confounding variables was humorous and insightful. It is important to recognize and seek to eliminate biases present explicitly or arbitrarily in the organization of the experiment. By explicit biases I mean, for example, only surveying persons in a small geographic area when a general statistic is desired, whereas arbitrary biases are such that occur with the letters "Q" and "M" when sampling colas. Nevertheless, the author's definition of the term could use some rewording. "Changing systematically with the independent variable" is a clause with a not-so-apparent meaning, despite being well explained later. I also think the author's choice of "testing" for threats to internal validity through testing was redundant at best. It tends to conflate the meaning and inhibit recall, in my opinion. The principle itself is sound, but perhaps associating it with "artificial sensitization" or some other distinguished but less verbose term would help. ---- CH 7: Building off of the prior reading, this chapter details how to choose an independent variable for experiments. It goes on to explain how tests involving that variable should look like in addition to measures of their validity. The author touched on a key subject when he mentioned a variable's "operational definition." Without a common understanding of the terms of discourse, knowledge transfer will be hindered, if not inhibited completely. The consequences may range from equivocation or misleading phrases (e.g., attributing "great" performance increases when metrics increase by factors of ten where competitors excel by factors of hundreds) to general unintelligibility (e.g., mistaking "morose" for "Morse Code" when someone describes their mood). While these strictures are ignored in common parlance, they are vital for technical communications. The author's recognition of the limitations of physiological inputs was also refreshing. It is a fortuitous leap to suggest "we found love" or "we can read minds" based on fMRI signals. Such tabloid taglines should appear less frequently with a conservative assessment of a variable's explanatory power. These guidelines are therefore quite germane to today.

Ahmed Magooda 22:29:12 10/4/2017

How to do experiments. In this chapter the author dives deep into the different aspects of an experiment. The author quantize the spectrum of experiment into different categories of variables, the author then starts defining each category and at the same time he provides some examples to illustrate the different types. The author also introduces some threats to internal validity. The types of variables are that can be part of an experiment are: - Independent variable, is the variable that we want to change and see its effect, it should have at least two different values. - Dependent variables, are variables that are affected due to the manipulations of independent variables. Usually what governs the relation between these two variables is the expected or proposed hypothesis. - Control variables, control variables are fixed to a single value so that we make sure that they don’t affect the result of experiment. - Confounding variables, which are variables that may change with the independent variable and induce some distortion to the relation between the independent and dependent variables, we usually try to minimize or eliminate these variables. - Random variables, variables that we hope change randomly. Random variables helps to improve experiment generalization coefficient. - Randomized within constraints, are random variables that we keep within a set of limits. Some of the threats to internal validity are: - History, which is the occurrence of uncontrolled events during the experiment. - Maturation, which is the change of age or experience that can happen to individuals during the experiment. - Mortality, which is the non-random loss of individuals from groups. --------------------------------------------------------- How to decide which variables to manipulate and measure. In this chapter the author discuses the best practice of how we can incorporate the variables (independent/dependent) in experiments. The author emphasises that we need to choose variables wisely, and discusses how it can be an indirect task to define independent or dependent variables, so It is important to have operational definition of variables. When choosing an independent variable the author argues that the choice has to be realistic and the variable range should be well-defined to highlight the effect of change. one way to define the range is by setting up a pilot experiment, which is a small scale version experiment of the original one. The author then talks about how to choose dependent variables. In choosing dependent variables there are some aspects to consider like (reliability and validity). Reliability means getting the same result upon repeating the experiment. Some ways to check for reliability are (test-retest,alternative-form and split-half). It then mentions some ways of measuring validity like (face validity, content validity, predictive validity and concurrent validity).

Charles Smtih 0:23:40 10/5/2017

On: Chapter 2 The author in this chapter describes what to think about before conducting an experiment. He goes into detail on what to look for, what to avoid, and common failures. It is always good to keep these kinds of ideas on your mind when conducting experiments. It becomes very easy to not stop, and take a look at the whole picture to make sure you’re not missing anything. These ideas can be transferred into almost any topic. On: Chapter 7 In this chapter the author talks in more details about the variables in an experiment. More specifically, how to pick independant variables, how to measure dependant variables, and how everything works together. This is also a very important, transferable, and timeless chapter. When conducting an experiment, it’s easy to get stuck by the question, or your hypothesis, and fail to perform an actual experiment that is correct.

Ronian Zhang 1:27:53 10/5/2017

Doing psychology experiments: In the chapter of the book, the author shows how to make causal statements. He introduces the critical conception and what we should pay attention in doing the experiment, he uses a lot of examples and make the book highly understandable: the experiment conductor should set independent var. to at least 2 level and use them to measure dependent var. There are control var. which should be set to a level and not allowed to vary, but sometimes the random var. appear in a random distribution which could improve the validity. The conductor may control randomized variables with in constrains. To make the experiment valid, we should try to remove the confounding var which may led to internal validity and lead to a unreliable result. The internal validity could also be threatened by unexpected events happened during the experiment, the getting more experienced participants, the unfair selection of participants, the nonrandom selection, the change of the awareness during testing, the extremely behavior, the various effect of the nonequivalent group. I think the skills is very practical and useful. Whenever something wrong, we should look back to the experiment itself. And the same logic applies to program testing, even though a lot of the criterion can’t apply, still some are the same when choosing the test cases.————————————————————————————— How to decide which variable to manipulate and measure: In the chapter of the book, the author shows the way of choosing independent and dependent variable. Since when choosing the independent var. may be both time consuming and causing disagreements easily, there should be rule or standard to follow to avoid them. The best way is to make a clear operational definition first. And should carefully chose the range, if it’s too tight, the experiment may hard to show differences, but if it’s too large, it’s unrealistic and impossible to actual conduct. The same thing goes for dependent var., for it should also be give a clear operational definition at the starting stage. In addition, they should be reliable (re-conductable in my own understanding ) and valid (follow a widely accepted standard). To improve the possibilities of getting a valid dependent, we could use multiple or composite variable. The chapter is very useful and also have a clear skeleton. It introduces the problem, teach how to make it work along with emphasizing the difficulties.

Xiaoting Li 1:43:07 10/5/2017

1.How to Get an Experimental Idea: In this chapter, the author introduces us with several important concepts including different types variables in an experiment, threats to internal validity, and experimental methods. The good take-away message in this chapter is that the author lists out several threats to internal validity. Before reading this chapter, I knew little about how all of these threats and sometimes even mistakenly applied some of them into experimental measurements. Besides the good take-away message, another impressive aspect of the paper is that the author does not only explain the related concepts but also shows us simple examples to help us better understand how to carry out an experiment in a better way. 2. How to Decide Which Variable to Manipulate and Measure: In this chapter, the author shows us the importance of selecting variables for manipulation and measurement in an experiment. The author also explains how we can select independent variables and dependent variables in efficient ways. Selecting effective independent variables and deciding the range of independent variables is always a difficult task. If we cannot identify independent variables in an effective way, we can hardly get a correct model. Therefore, it is interesting and useful that the author introduces us with the idea of carrying out pilot experiments. With the help of the pilot experiments, we can get some general ideas about different variables and we can identify the range of independent variables in a more cost-effective way.

Yuhuan Jiang 3:34:46 10/5/2017

Paper Critiques for 10/05/2017 == How to Do Experiments == This is a chapter from the book Doing Psychology Experiments written by David W. Martin. The chapter describes various important aspects of the methodology of conducting experiments. The chapter begins by introducing a few types of variables, namely, independent variables, dependent variables, control variables, and random variables. Then, the chapter moves on to the common pitfalls that can make experiments invalid. For example, the effect of history events can be easily ignored, and wrong attributions of improvement may be made. The maturation of participants (especially young children) can threat the validity of experiments. When participants are not assigned randomly, the validity is also threatened by the selection. Statistical regression is also a threat, as participants who are chosen on the basis of having scored high/low some test will tend to move toward the mean on a second test. The experiment method introduced in this chapter enables causal statements and correct attribution of the changes in participants’ behavior. == How to Decide Which Variables to Manipulate and Measure == This is another chapter from the book Doing Psychology Experiments. The main question this chapter addresses is how to choose variables in psychological experiments. The first thing the author mentioned is to define the independent variables. The key to choosing it is to specify the operations needed to set up the independent variables. Use operational definition like receipt. Then, the range of the variable must be decided. There are a few principles in this aspect. First, the range should be realistic. Second, the rage should be large enough to show effect. Thirdly, a pilot experiment may be helpful in determining the range. The next thing is to define dependent variables. The operational definition is again needed. In addition to that, we must show that the dependent variable is reliable and valid. Being reliable means that the same results can be obtained every time a new measurement is taken. Being valid means that the measurement agrees with a commonly accepted standard, including face validity, content validity, predicative validity, and concurrent validity. Many concepts (such as the different types of variables) are not only applicable to psychological experiments, they apply universally to all scientific experiments.

Akhil Yendluri 4:01:16 10/5/2017

DOING PSYCHOLOGY EXPERIMENTS CHAPTER 2: HOW TO DO EXPERIMENTS This paper talks about how to conduct experiments and he various variables involved while doing an experiment. The author talks and explains about independent variables which are of major interest in the experiment. He then talks about dependent variables, control variables, random variables and confounding variables. The author explains them with simple examples making it easier to understand. The author than talks about the effects of being too generic or too fixed. The author explains how maturation, morality, testing, statistical regression and selection can be a major threat to Internal Validity. The author finally summarizes that the decision to increase control can improve precision of results but would affect generality. While increasing generality would increase randomness but affects precision. DOING PSYCHOLOGY EXPERIMENTS CHAPTER 7: HOW TO DECIDE WHICH VARIABLES TO MANIPULATE Here the author builds on the content of chapter 2 by explaining about how to choose an independent variable for doing an experiment. He asks us to clearly specify the various parameters involved like range, pilot experiment while choosing the independent variable so that it can be successfully re-evaluated by others. He then continues by explain how to choose a dependent variable. He mentions the use of reliability and validity to measure dependent variables. The author explains various techniques like test-retest reliability, alternative-form method and split-half technique. The author than tells that the dependent variables should be valid by commonly accepted standards using face validity, content validity, predictive validity and concurrent validity. Then the author talks about physiological measures and behavioral measures that might help in determining the internal state of the participant.

Ruochen Liu 8:54:38 10/5/2017

1. How to Do Experiments? This is the second chapter of book “Doing Psychology Experiments”. In this chapter, the author mainly talks about the important factors of psychology experiments and how this kind of experiments can be achieved. So, why psychology experiments are important to human-machine interaction? In my opinion, the psychology experiments allow us to find the truth about relation between human and interfaces, which means human beings can design and build better interfaces basing on these truth. Using an example of an experiment measuring the time it takes a person to push a button in response to the light when the light has a particular intensity, this chapter introduces variables like independent variables, dependent variables, control variables, random variables and so on. When talking about confounding variables, the part of “Coke-Pepsi slugfest, 1976” interests me a lot. I am surprised to see that the word on the cup can make more influence on the experiment than the soft drinks themselves. That shows the confounding variables can cause low internal validity and make it difficult to say that only the independent variable causes a change in the dependent variable. Using the experimental method that mentioned in this chapter can help to make causal statement, which is that when a circumstance is manipulated it causes a change in behavior. 2.How to Decide Which Variables to Manipulate and Measure: This is the seventh chapter of book “Doing Psychology Experiments”. The author illustrates the two decisions to make in the psychology experiments: choosing independent variables and dependent variables. In choosing an independent variable for the experiment, an operational definition of the variable must be specified first. Choosing the levels of independent variable is also important. And sometimes, a trial run or pilot experiment can help choose an independent variable. For the dependent variables, they are valid if they agree with the commonly accepted standard. Also, there several ways to test the validity, such as face validity, content validity, predictive validity and concurrent validity. In some specific areas, multiple dependent variables are required or a composite dependent variable as a combination of dependent variables is needed.

Xingtian Dong 10:25:17 10/5/2017

1. Reading critique for ‘How to do experiments’ I think this chapter is useful for us. Though computer science is not based on experiments, it is still worthy for us find what is ‘dependent variable’, what is ‘independent variable… in our codes. It will make both designing and testing much easier. Actually I major in electrical engineering and focus on control. These words are not unfamiliar for me. But I never thought about using it computer science. This chapter inspire me to use knowledge from other areas. It is important to detect connections between different areas. 2. Reading critique for ‘How to Decide Which Variables To Manipulate and Measure’ This chapter is based on the last one and has a further explanation about how to use the variables. It tells me how to find the variables for baseline, how to choose the values for them and how set the constraints. I still have to say, this is quite similar to what I learned in control. It also teaches how to define a real word problem into a math problem. And how to set the variables and constraints and so on. I think I already have a lot of knowledge, but I still need to practice and learn how implement if in computer science area.