Professor Pamela Oliver Department of Sociology

FIELD EXPERIMENT EXERCISE

Reading:

Singleton, Chapters 7 and 8; recommended Stern Chapter 3. You must understand what a manipulated independent variable is, what randomization is and how it works (and how it differs from random sampling), and what it means to gain experimental control from holding constant. You continue to need to know what it means to operationalize a variable. You may wish to review Singleton Chapter 18 on writing a research report.

For this exercise you design and carry out a small field experiment using a randomized posttest-only between-subjects (control group) design. You devise a manipulable independent variable and a measurable dependent variable. You may use any hypothesis and procedures that you are comfortable with. Some people prefer to do strange or unusual things and observe the reaction, while others prefer to do quieter things. Both can easily be accomodated, as long as you make sure your partners share your attitude. Class members may wish to suggest general theoretical hypotheses that different teams can operationalize in different ways, thus permitting the class as a whole to generate tests of the external validity of results.

About Teams.

You are strongly encouraged but not required to do this exercise with another class member. Teams have two options: (1) write a joint report, or (2) each person write the whole report individually. Option 1 is appropriate when team members are truly working and learning together. It is unfair and unethical for one student to do most of the studying and writing while another "free rides" under the guise of option 1. Those doing the work have the right to refuse to "give" partners papers they did not help write. If you choose option 2, you work together until the data are collected and, if you wish, put into a statistical table, but you must not collaborate in writing your separate reports. If you find yourself in an ambiguous position about these options because of unforseen problems, speak to me and I will help you to determine the fairest thing to do.

Planning Your Experiment

Experiments are planned in detail in advance. You should not do your "official" experiment until your planning is complete. In planning your experiment, you need to figure out each of the following:

1. A "setup," the basic thing that will be happening in your experiment (e.g. a survey, a petition, a request for help).
2. A manipulable independent variable, with a careful operationalization of the differences among the categories or treatments. There should be only one variable with two or three categories.
3. An operationalized dependent variable which can be objectively and consistently measured for all subjects, regardless of the treatment they receive.
4. A plan for randomly assignment subjects to treatments that is properly random and that is practical in the situation you have devised.
5. A plan for holding constant all aspects of the situation and your behavior except the independent variable or things which can be randomized.
6. A sampling plan. (You do not need a random sample, although you may use one if it is possible with your setup.)

It is often very useful to run small scale pretests of parts of your idea, to see if it will work. If you are doing something new or unusual, I suggest you try out your setup, independent variable, or dependent variable before wasting too much time on something that might not work.

Your experiment should have only one independent variable and one dependent variable. You may do additional independent or dependent variables only if you are sure you know what you are doing. Unfortunately, people who do not understand experiments often throw in extra variables by mistake, so I must have the rule that extra variables must be handled correctly to get credit. If you think you can handle this and there is something that interests you, speak to me and I will explain how to do a factorial design correctly.

Doing the Experiment

Follow your plan and obtain a minimum of 10 cases (i.e. units of analysis: subjects or trials or groups) for each category of the independent variable. If the independent variable has 2 categories, this will be 20 cases. Fifteen or even 20 cases per category of the independent variable (i.e. 30 or 40 total cases) is better. This is a very easy goal to obtain in the vast majority of field experiments. It is common for people to have so much fun they do more. Do not go below 10 cases per category unless you have written permission from me. Usually when it seems unreasonable to do so many cases it is because your idea is impractical and you need advice.

Record the data according to your plan, and make a note of any unexpected things that happen. Make sure you turn in your data with your report.

WRITTEN REPORT

PLEASE FOLLOW THIS FORMAT EXACTLY. This is based on Chapter 17 of the Singleton book, but includes some specifics for this class.

About Truthfulness.

Science depends on researchers telling the truth about what really happened in their research, not what they wish had happened. At the same time, students worry that they will be graded down if they tell the truth. So, for each question, I insist that you tell the truth about what really happened in the research, but then follow it with an opportunity to explain what you now think you should have done. If there was a mistake and your self-criticism gives a correct statement about what you should have done, you will receive full credit as if you had done things right in the first place.

1. Title page. Title of report, author(s), date. Put partner's name in parentheses at the bottom of the page if you worked with someone but wrote reports separately.
2. Abstract. Write one paragraph which summarizes your hypothesis, experimental research methods, hypotheses, and findings. You may include this on the title page if you wish.
3. Body of paper.
1. Introduction. Write a paragraph stating your bivariate hypothesis and why it is worth researching. Sometimes teams agree on the variables but disagree on what they predict will happen. This is OK for the purposes of this assignment; just say who makes what prediction. (Note: Citations to readings are not needed, but go here if something you read went into your thinking on this project.)
2. Methods of research. (Note: To aid grading, number each section of this discussion as it is numbered here.)
1. Setting.
1. Details on the "set up" of your research, the context within which the independent and dependent variables operate.
2. Why you chose it.
3. How it worked out in practice.
4. Evaluation: why you think it was good, or what you now believe should have been done differently.
2. Independent variable. (Make sure this is manipulable)
3. Complete details on your operationalization as you planned it
4. Why you chose this operationalization.
5. How the operationalization actually worked out.
6. Evaluation: why you think these procedures were good, or what you now believe should have been done differently.
3. Dependent variable
1. Complete details on your operationalization as you planned it.
2. Why you chose this operationalization.
3. Why subjectivity is not a problem with this operationalization, or what you did to reduce the possibility of bias due to subjectivity.
4. How the operationalization actually worked out.
5. Evaluation: why you think these procedures were good, or what you now believe should have been done differently.
4. Experimental Control.
1. Details on the things about your procedures or behavior you consciously brought under experimental control through holding constant.
2. How this actually worked out.
3. Evaluation: why you think these procedures were good, or what you now believe should have been done differently.
5. Sampling.
1. Specify your unit of analysis (individual, group, time period, trial). Ask me if there is any doubt.
2. Describe your sampling procedures, including location and time, any restrictions placed on eligible subjects, or other procedures for deciding whom to study.
3. Evaluation: why you think these procedures were good, or what you now believe should have been done differently. Please note, this evaluation is in terms of what was actually possible in this assignment, and is not about the standards you believe professionals should adhere to.
6. Randomization. In real professional experiments, randomization is usually taken for granted and not discussed. Because this is a course assignment, I want you to spell these procedures out.
1. Describe how you randomly assigned units of analysis to categories of the independent variable. Say what your random device was (slips of paper, coin flips, shuffling, etc.) but, more importantly, explain how you worked the randomization into the flow of the whole experiment.
2. Describe any additional steps you took to prevent selection biases from entering after the randomization, or explain why they were not necessary.
7. Results. (Attach the original messy data collection sheet to the back of your paper as an appendix.)
1. Prepare a bivariate statistical table to show the relationship between your independent variable and dependent variable. Do either a contingency table or a difference of means table. If the dependent variable is a dichotomy (has only two categories), you can show a difference of proportions either as a contingency table or as a means table; they are equivalent. (Refer to "bivariate association" in the statistics part of this course. Or ask for help, if necessary.)
2. Write a paragraph discussing your statistical results saying what they show and whether your hypothesis is confirmed or disconfirmed.
3. Discuss anything else worth mentioning that you learned in your research, including unexpected events or surprising findings.
8. Conclusions and interpretations. This is where you talk about the larger issues your research raises, about the extent to which you think the findings apply to other situations, and about future research by yourself or others that would seem to follow from your findings. For this class, it is also a chance to talk informally about what you liked or didn't like about the assignment or the way you did your research.
4. Questions testing your understanding of the logic of experiments. This would not normally be part of a research report, but is used because this is a class. Everything in this section refers to extraneous variables that might threaten the internal validity of your results, that is, to things that might make people fall into different categories of your measured dependent variable.
1. What threats to internal validity were eliminated through randomization? (Discuss the several different kinds of extraneous variables that were controlled, using a variety of examples of things that might actually have been relevant to your experiment.)
2. What threats to internal validity were eliminated through holding things constant? (Discuss the several different kinds of extraneous variables that were controlled, using examples of things that might actually have been relevant to your experiment.)
3. What threats to internal validity remain uncontrolled in your experiment? (NOTE: I do not mean "random error," the possibility that randomization failed to equate the groups. I mean factors that were not randomized and not held constant, that might have affected the bivariate relation between the independent and dependent variables.)
5. Group process report. Pick the category that applies to you and answer the relevant questions.
1. No partner.
1. How did you feel about working alone? Would you do it again, or would you prefer a group?
2. How much effort did you have to put into this project?
3. How well prepared did you feel in terms of course materials and understanding what to do.
4. Tell me if there is anything I should know about your or your life that you want me to know, especially if it might affect your grade or my ability to be fair in grading your work.
2. Had partner, wrote separate papers.
1. Compare you and your partner in the effort you put into the project.
2. Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
3. Who did your statistical analysis?
4. Did you start trying to work together before deciding to write separate papers? How far did you get?
5. Were there some things you found necessary to discuss in preparation for writing your papers? What?
6. How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
7. Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.
3. Wrote joint paper.
1. Do you stand by the paper as written, or is there something you feel should have been said differently? Any corrections you offer at this point will be factored into your grade. This answer may be as long or short as you feel is appropriate.
2. Compare you and your partner in the effort you put into the project.
3. Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment.
4. Who did your statistical analysis?
5. How did you go about getting the writing done?
6. How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future?
7. Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.
   

Top

Questions or Comments? Email Oliver -at- ssc -dot- wisc -dot- edu. Last updated December 25, 2004 © University of Wisconsin.