Professor Pamela Oliver Department of Sociology

Sociology 357/8 Prof. Pamela Oliver (revised 6/02)

DUE DATE:

For this exercise, you plan and carry out a structured field observation. You may observe people in public places, television shows or commercials, or advertisement or letters to the editor or other short articles in newspapers or magazines. You may behave in some mildly "odd" fashion and observe reactions. You will decide what to observe, select a hypothesis, operationalize the independent and dependent variables, and do structured observation to test your hypothesis. The central point of this assignment is to develop the operationalization of a relatively-complex variable and conduct a reliability test of your operationalization.

Reading: For assignment, Singleton, Chapter 11; also review Chapters 4 and 5, paying special attention this time to the discussions of reliability (pp. 114-122). For writing a research report, Singleton Chapter 18.

You may use any appropriate topic for this research, but if you cannot think of anything you like better, I have two suggestions: (1) Test whether women pause and scan or stop and groom themselves just before entering a room more often than men do. (2) Test whether advertisements show women in more sexually-oriented poses or in less business-oriented settings than men. It is fine for different teams to test the same or related hypothesis. If you have a different idea, discuss it in class. You always get better information if several teams work on the same general idea than if each team tries to be unique. This is because replication is essential before any finding can be treated as trustworthy.

You will need to plan your observation procedures so that two of you can independently observe the same people or objects. To have a valid reliability check, you cannot collaborate or check with each other while you are doing the observations, but you have to observe the same things or the reliability check is meaningless. If you are observing people in public places or things broadcast on television, you need to be watching the same things at the same time. If you are observing printed matter, you can take turns observing. To observe people in a public, you need a setting where people move through a few at a time. If there are too many people to observe at once, there is no way to know whether you are observing the same people at the same time, too few and you'll never get the project done. A reasonable range is as fast as three a minute or as slow as one every three minutes, on average

About Teams.

You are strongly encouraged but not required to do this exercise with another class member. Teams have two options: (1) write a joint report, or (2) each person write the whole report individually. Option 1 is appropriate when team members are truly working and learning together. It is unfair and unethical for one student to do most of the studying and writing while another "free rides" under the guise of option 1. Those doing the work must refuse to "give" partners papers they did not help write. If you choose option 2, you work together until the data are collected and, if you wish, put into a statistical table, but you do not collaborate in writing your separate reports. Option 2 is a good choice if you have busy schedules which make it difficult to meet with others, but want the advantage of having a partner in thinking up the original idea and collecting data. If you find yourself in an ambiguous position about these options because of unforeseen problems, speak to me and I will help you to determine the fairest thing to do.

Preliminary Unstructured Observation and Pretesting

1. Spend some time in preliminary observation. If you are observing people or television, this is before your "real" data collection. If you are observing printed matter, do your preliminary observation on different examples than your "real" data collection. Take brief notes on the things you see. Try to get beyond your high-level interpretations to noticing what exactly it is that you can see (or hear) that you could study. Look at behavioral details like patterns of movement through space, hand gestures, posture, positions of legs or arms, ways of eating or drinking, eye or head movements, amount or volume of talking. Notice and write down at least five possibilities of things you might want to operationalize with short notes about what details you would need to observe. Also consider whether this seting seems feasible. It is ok if you change settings or objects until you find something that seems interesting and feasible. TURN IN A TWO-PARAGRAPH REPORT ON THIS STEP A HOMEWORK ASSIGNMENT AS SHOWN IN THE SYLLABUS. It is perfectly OK if this is in your normal illegible handwriting with spelling and grammar errors. I just want to know that you did this step. It is NOT worth recopying or typing it.

2. Talk over your observations and ideas with your partner. Select one to develop for this project. You will need to plan and write down in advance a sampling procedure and operationalizations of your dependent and independent variables. At least one of these must involve complex enough judgments that it is not "obvious" how to do it. You must submit this plan the way it was written before your data collection as an appendix to the assignment. It is ok if this is in messy handwriting with sentence fragments or spelling/grammar errors.

3. Operationalization of observable variables. You need to operationalize each variable. An operationalization tells both what to look at and how to categorize what you see into a set of exhaustive and mutually exclusive categories so it will be a variable. This often involves refining what you are looking for so that it is something observable. You choose a level of measurement, decide on the categories, and then carefully spell out what observable cues you will use for categorizing people's behavior. Basically you will either count how many times (or how long) a person does something, or you will categorize their behavior. If you categorize, your operationalization is focused on defining the differences among the categories. If you count, your operationalization is focused on defining exactly when the behavior you are counting or timing begins and ends. This must be explained in observational detail. You need to write this down in sufficient detail that someone other than yourself could follow these instructions and record the variable the same way you would. At least one of your variables must be complex enough that it takes at least five sentences to explain the details of how you operationalized it. The other one can be obvious and just briefly explained, but you must be observational in your explanation.

4. Sampling procedure. Determine the necessary rules for determining who or what is a "subject" in your study. Will you observe "everyone" or only adults? Only people alone? Will people have to enter a certain space or spend a certain amount of time in the setting to count as subjects? Will you exclude certain people (e.g. those wearing employee uniforms)? What will you do if too many people come in at once for you to observe? Or if commercials or printed material, determine which ones you will be studying. Consider the problem of subjectivity in your sampling criteria and the need to operationalize the variables which define your sample. For example, if you are studying only "adults," what are the criteria for "adult"? Or only ads for personal care products, or only sports articles, what are the criteria for inclusion? Spell that out. If you follow a procedure in which you select subjects by agreement in advance, you must explain this in your report.

5. Create a recording sheet for your structured observations. PLEASE NOTE: This format is almost always the best one; students often invent their own formats which are more error-prone and unreliable than this one. USE THIS FORMAT UNLESS I TELL YOU THAT ANOTHER IDEA IS BETTER FOR YOUR PARTICULAR PROJECT. Let each line be a different subject (unit of analysis). Select consistent shorthand symbols for each category of the independent and dependent variables. Suppose you agreed to use M and F for male and female, and to use L for licked ice cream cone, B for bit it, and O for anything else. Then your recording sheet might look like this:

Regardless of how you measure your dependent variable, you will need "identifiers" for the individuals observed. These identifiers are so you and your partner(s) can go back over your individual data later to check your reliability. Generally, use hair, skin, and shirt/blouse colors or unusual behaviors as identifiers for people, and whatever seems appropriate for other kinds of subjects.

Before the observation, agree upon the symbols and set up the columns, being sure to have a few extra made up in case you get more subjects than you expect. You will use the comment whenever it is difficult to decide how to categorize someone on the independent or dependent variable, or to explain "other" codes.

TURN IN THE ORIGINAL MESSY DATA COLLECTION SHEET YOU ACTUALLY USED IN THE FIELD AS AN APPENDIX.

6. Pretest your operationalization a little by trying it out on a few cases, where you are allowed to talk to each other. If you make changes at this stage, add appropriate notes to your written operationalization.

Carry Out Your Research

Using the form you have developed, you and your partner observe the same subjects for a minimum of 30 minutes or until you obtain a minimum of 30 observations. If you realize part way through that there is a problem with your operationalization or sampling, use the comments column to note the details for difficult cases and keep going. If you add a decision rule part way through, record it on your sheet and keep observing. Quit observing early only if the problems are so bad that you just cannot use your scheme at all, in which case you need to fix the problem and start over. If you had to start over, discuss this in your report. You will not be penalized for having to start over.

Calculate Your Reliability

Even when two people observe the same thing and follow the same rules, there will usually be some disagreement in the observation. This is more likely when you have done a "hard" variable to observe, rather than a trivially easy one. Inter-coder reliability is an excellent way to assess the accuracy with which it is possible to record data. This is somewhat tedious to do, but has the potential of being a valuable learning experience.

1) Compare your sheets and match up subjects, using the identifiers and any variable that is not ambiguous). Mark as "sampling error" subjects who do not match up. For each subject that does match up, check whether the independent and dependent variables are the same; mark everyone for whom you disagree about a variable. (Note: if you are counting or timing something, give yourselves a reasonable margin of error such as being within 1 or 2 seconds still counts as "the same," so you do not have to include too many errors. See me if you need help with this.

2) Calculate the following numbers for your most complex variable (usually the dependent variable)

A = number you agree on: same subject, same variable code.

C = coding difference: number of times you observed the same subject but coded the variable differently.

S1 = sample difference 1: number of times partner 1 observed a subject partner 2 did not see.

S2 = sample difference 2: number of times partner 2 observed a subject partner 1 did not see.

N = total number of distinct people seen by either partner 1 or partner 2 (or both).

NOTE: If both your independent and dependent variables are complex, you may receive extra credit for doing two reliability calculations.

3) Reliability computations:

SE = (S1 + S2)/N = sample selection error (proportion of total cases that one person saw but not the other).

CE = C/(A+C) = coding error (proportion you both saw that you disagree about in the variable).

SHOW YOUR COMPUTATIONS IN AN APPENDIX AND MAKE SURE IT IS POSSIBLE TO TELL FROM YOUR DATA SHEETS AND YOUR COMPUTATIONS WHAT YOU DID. ADD NOTES EXPLAINING ANYTHING THAT MAY BE UNCLEAR.

NOTE: If you have a three-person team, there are three possible pairs for which you can do this analysis. Either do the above for all three possible pairs (if it does not make you feel too oppressed) or do it for two of the three pairs, the two partners that seem most alike and the two partners that seem most different. SEE ME if you do not understand this or want to negotiate.

4) Prepare your data for hypothesis testing. If you have any coding or sampling error, you need to decide what data you will analyze. You have four choices: a) each partner analyzes the data s/he collected (appropriate only if you are writing separate papers), b) use the data from the partner which you believe was most accurate, c) create a composite data set using the good data from each partner, d) do the analysis twice, for each data set. You may choose whichever seems most reasonable to you, but you must explain what you did.

It is also OK at this stage (after the reliability test) to re-categorize cases before further analysis. If you had cases classified as "other" because they were unexpected, you can decide now how to classify them. You can also group categories that you now think are similar and eliminate categories that were not used.

PLEASE FOLLOW THIS FORMAT EXACTLY. This is based on Chapter 17 of the Singleton book, but includes some specifics for this class.

About Truthfulness. Science depends on researchers telling the truth about what really happened in their research, not what they wish had happened. At the same time, students worry that they will be graded down if they tell the truth. So, for each question in your report, I insist that you tell the truth about what really happened in the research, but then follow it with an opportunity to explain what you now think you should have done. If there was a mistake and your self-criticism gives a correct statement about what you should have done, you will receive full credit as if you had done things right in the first place.

I. Title page. Title of report, author(s), date. Give your project a real title as if you were taking it seriously, not just "assignment 1" or some such. If you worked with someone but wrote reports separately, put "Partner: Lee Li" in parentheses under your name.

II. Abstract. Write one paragraph which summarizes your hypothesis, research methods,, and findings. You may include this on the title page if you wish.

III. Body of paper.

A. Introduction. Write a paragraph stating your topic and why it is worth researching. Summarize observations relevant to the topic that came up in your unstructured research. Explicitly state your bivariate hypothesis and why you believe it is true. (If you are just guessing about the hypothesis, or team members disagree, just write about the reasoning in your guess, or what the disagreement is.) (Note: we will NOT normally do literature reviews in our course assignments, but this is where it would go, and if something you read went into your thinking on this project, this is the appropriate place to discuss it.)

B. Methods of research. (Note: We will write this section in a more closely structured format than the usual research article. This is so I can more easily grade your paper. Number each section of this discussion as it is numbered here, e.g. 2b for operationalization of dependent variable; this is essential for grading, as part of the grade is putting the correct information under the correct heading!)

1. Sampling. a) Say what the units of analysis are (e.g. people, advertisements). b) Describe the setting of your research, the time of day you conducted it, and any details relevant to understanding your data. Obviously this will vary a lot depending on the type of subjects. c) Describe your sampling procedures, including any restrictions placed on eligible subjects, or other procedures for deciding whom to study within the setting. d) Discuss the problem of possible subjectivity in the sampling criteria; how did you operationalize the sample selection variables? e) Evaluation: why you think these procedures were good, or what you now believe should have been done differently.

2. Dependent variable. a) Why you chose your particular operationalization. b) Complete details on your operationalization as you planned it. This should be consistent with the notes in the appendix. (Note: This is never correctly a one-sentence answer. I'm looking for details of wording that tell me you know what is important about operationalization, as well as for indications that you did things properly. Many people do it right but cannot explain it right.) c) How the operationalization actually worked out. d) Evaluation: why you think these procedures were good, or what you now believe should have been done differently.

3. Independent variable. a) Why you chose your particular operationalization. b) Complete details on your operationalization as you planned it. (Note: This answer is typically short, but it must be worded to show that you know what an operationalization is.) c) How the operationalization actually worked out. d) Evaluation: why you think these procedures were good, or what you now believe should have been done differently.

4. Ethics. Discuss your evaluation of the ethics of doing this research. Do you feel you invaded anyone's privacy? How did you feel about doing covert observation?

C. Results. (Attach the original messy data collection sheet to the back of your paper as an appendix. You will be graded down if this is missing.)

1. Sampling Reliability Analysis. a) Present the results of your reliability calculations. (Include the computations in an appendix so that I can check your work.) b) Discuss these results. Did you have more than trivial disagreements (more than 1 case) over sampling? Can you figure out what caused the problem(s)? Discuss in some detail. c) Evaluation: why you conclude your procedure were good, or what you now believe you should have done differently.

2. Measurement Reliability Analysis. a) Present the results of your reliability calculations. (Include the computations in an appendix so that I can check your work.) b) Discuss these results. Did you have more than trivial disagreements (more than 1 case) over coding? Can you figure out what caused the problem(s)? Discuss in some detail. c) Evaluation: why you conclude your procedure were good, or what you now believe you should have done differently.

3. Explain how you chose/constructed the data you analyzed for your hypothesis test.

4. Test of hypothesis. a) Prepare a bivariate statistical table to show the relationship between your independent variable and dependent variable. Do either a contingency table or a difference of means table. (Refer to "bivariate association" in the statistics part of this course. Or ask for help, if necessary.) b) Write a paragraph discussing your statistical results saying what they show and whether your hypothesis is confirmed or disconfirmed.

5. Discuss anything else worth mentioning that you learned in your research, including unexpected events or surprising findings.

D. Conclusions and interpretations. This is where you talk about the larger issues your research raises, whether you feel that your findings are likely to be more generally true, and what research, if any, you would like to see pursued by yourself or others as a consequence of your research. For this class, it is also a chance to talk informally about what you liked or didn't like about the assignment or the way you did your research.

GENERAL NOTE: The point of this assignment is to learn from a reliability analysis. You do not dismiss error as small, you try to figure out how procedures could have been improved to reduce it. In writing your report, imagine this is a pretest report for a larger study: you are describing what still needs to be fixed and what is ok and ready to roll. Please note that it is entirely possible to make an A even if you had very serious problems with your reliability, if you discuss it well and explain how it could be improved. Conversely, it is possible to make a C or worse with perfect reliability if you cannot explain well how you got it.

IV. Appendices

A. The written operationalization you planned in advance before data collection. (The original messy version.)

B. Your original structured observation data collection sheet, the one you actually used in the field to get the data. DO NOT recopy or retype this sheet. I want to see the real data.

C. The work showing the computations for your reliability analysis and test of hypothesis. This can be an unrecopied "rough draft" (I certainly would NOT type it!!), but I DO need to be able to follow your computations and check them against your data sheet to be sure you did not make an error.

V. Group process report. Pick the category that applies to you and answer the relevant questions. PARTNERS MUST HAND THESE IN SEPARATELY SO THEY CANNOT POSSIBLY HAVE ACCESS TO EACH OTHER'S ANSWERS. I WILL NORMALLY NOT RETURN THESE STATEMENTS, BUT WILL KEEP THEM FOR MY RECORDS.

A. No partner. 1) How did you feel about working alone? Would you do it again, or would you prefer a group? 2) How much effort did you have to put into this project? 3) How well prepared did you feel in terms of course materials and understanding what to do. 4) Tell me if there is anything I should know about you or your life that you want me to know, especially if it might affect your grade or my ability to be fair in grading your work.

B. Had partner, wrote separate papers. 1) Compare you and your partner in the effort you put into the project. 2) Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment. 3) Who did your statistical analysis? 4) Did you start trying to work together before deciding to write separate papers? How far did you get? 5) Were there some things you found necessary to discuss in preparation for writing your papers? What? 6) How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future? 7) Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.

C. Wrote joint paper. 1) Do you stand by the paper as written, or is there something you feel should have been said differently? Any corrections you offer at this point will be factored into your grade. This answer may be as long or short as you feel is appropriate. 2) Compare you and your partner in the effort you put into the project. 3) Compare you and your partner in the extent to which you studied course materials and knew what to do for the assignment. 4) Who did your statistical analysis? 5) How did you go about getting the writing done? 6) How did the group process work out? Was it a positive or negative experience? Would you do things differently in the future? 7) Tell me anything else I should know that might affect your grade or your partner's, or that I should know to be fair in grading your work, or that you would like me to know even if it is not relevant to your grade.

Top

Questions or Comments? Email Oliver -at- ssc -dot- wisc -dot- edu. Last updated December 25, 2004 © University of Wisconsin.