Professor Pamela Oliver Department of Sociology

ARTICLE ANALYSIS ASSIGNMENT

DUE: SUMMARY PERIOD.

READ THIS HANDOUT CAREFULLY! You must do this analysis by answering the specific questions listed. Keep your answers as brief as possible using an "outline" style rather than an elaborate writing style whenever possible.

Stern's Evaluating Social Science Research is highly recommended for teaching you how to read and evaluate a research article. It will help you learn what you need to know to do this assignment.

Criteria for Article Selection

The articles reviewed for this assignment must report the results of someone's research in an area of social research. The research should have been carried out by the author(s). The article must be directed at a scholarly audience.

Your review must be on an article reporting structured research, that is, one with variables, statistical analyses, relationships among variables, etc.

The article may be about any topic you choose, as long as it is basically social research. Check with me if you have any doubts about your topic. Research in sociology, political science, psychology, or social work are fine. (But remember you need research articles; not all articles in any field are research articles.)

The following types of articles may NOT be used:

1. Purely theoretical papers which discuss concepts and propositions, but report no empirical research;
2. Statistical or methodological papers where data may be analyzed but the bulk of the work is on the refinement of some new statistical or modelling technique;
3. Review articles, which summarize the research of many different past researchers, but report no original research by the author;
4. Popularizations or abridged reports, commonly found in popular newsstand magazines such as Psychology Today or books of readings designed for use by undergraduates;
5. Extremely short reports with less than four pages devoted to methods and findings, unless I determine that adequate information is given.

Most research reports begin with sections on theory and reviews of others' research, so skim the whole article or read the abstract, if there is one, to determine whether the author reports actual research he or she has done.

All articles must receive my OK. No two students may review the same article. It is OK to use articles you have to read for another class, if they meet all of the above criteria, but you may not use the articles in the Golden reader.

Where and How to Find an Article

You must use scholarly articles for this assignment; these are found in professional journals, not general circulation magazines. The University of Wisconsin subscribes to a large number of such journals. Recent issues are kept in the periodicals room of Memorial Library. Past issues are bound in hardcover by volume and kept on the first and second floors of the south stacks of Memorial Library. Bound volumes of some journals are in the reserve room of Helen C. White library. To find the call number of a specific journal, look up the journal's title in the card catalogue, or in a copy of the specific list.

If you want to find articles about a particular topic, use the Social Science Index, Sociological Abstracts, or some other appropriate reference book to get citations of articles relevant to your interest. There are reference books in other social sciences or disciplines such as social work, political science, and psychology, and in special topics such as urban affairs, poverty, human resources, women, family, etc. Many of these are listed in The Student Sociologists' Handbook. Another place to get citations of articles in a topic area is in the bibliographies of other books or articles in the topic area.

If your interests are wide, general, eclectic, or uncertain, you may prefer to locate a supply of journals, either new ones in the periodicals room or older ones in the stacks or the reserve room and flip through them until you spot an article that looks interesting to you. The major general sociology journals are American Journal of Sociology, American Sociological Review, and Social Forces. Some other journals in sociology are: Journal of Marriage and the Family, Crime and Delinquency, Social Psychology, Sociology and Social Research, Social Problems, Journal of Political and Military Sociology. There are dozens more specialized journals.

Final approval will be given only on the basis of the photocopy of the whole article; I will write approval on the copy itself. When you have found the article(s) you want, photocopy it, and write right on the photocopy the journal name, volume number, issue number, month, year of publication, and pages. The author's name and the article's title should be on the first page; if they are not, copy these down too. (You should get into the habit of writing the full citation on everything you photocopy. This saves having to return to the library for the information when you later decide to use the material in a term paper or, worse, not being able to find it.)

If you wish to save money, you may show me a copy of the abstract before you copy the whole article. I can tell you whether the article has been chosen by another student, and can tell you with 90% confidence whether it will be OK for the review. You can also check out bound periodicals from Helen C. White to show a whole article to me.

Outline for Your Article Review

PLEASE NUMBER THE SECTIONS OF YOUR REVIEW TO CORRESPOND TO THE NUMBER OF MY QUESTIONS. It is not in your interest for me to have to guess what you're writing about. Answer the questions as briefly as possible. This is not a literary essay. An "outline" style, tables, and other devices to keep your answers brief while complete are all acceptable.

1. Introduction

NOTE: Make sure that the full citation is written on your photocopy or you will not get credit for the review. Attach the photocopy to your review. I simply cannot grade your review without the photocopy.

1. What is the problem or question(s) this research concerns? You should be able to identify the central focus. If there are additional secondary problems, identify these too. (1-4 sentences)
2. What is the source of the data? (That is, questionnaire, intensive interview, documents, existing statistical information, observations, laboratory manipulations, field manipulations, etc.) In some studies there are two or more sources of data. Give a brief overview of how the data were acquired. (2-5 sentences. DO NOT "dump" your whole report here!)
3. Briefly, what do the key findings turn out to be? (1-5 sentences)
2. Construct Validity of Measures of Variables
1. Using words or, perhaps a sketch or flow diagram, list all the operationalized variables in this research and very briefly indicate how they relate to the concepts or variables of theoretical or substantive interest. You should discuss all the operational variables, but it will be often easiest to write your answer by starting with the concepts, and explaining how each is measured. Do NOT "dump" all the measurement details here. This is just a summary that lists all the measured variables and what their logical relation is to the purposes of the research.

The reason it is hard for me to explain this question clearly is that how to do it depends very much on what your article is like. Probably the best thing is for me to give you an example of what I mean. For the horn-honking article, the answer would be: The independent variable is status of frustrator. This is operationalized as the type of car and the driver's clothes. The dependent variable is aggression, which is operationalized as horn-honking. The frustrating situation is operationalized with being blocked at a green light. Sex of subject was a control variable.

Different articles have different logical structures, and the best way to do your article is to describe what is happening in it. Some have no distinction between the concepts and the operationalizations, everything is just operationalization. Others have bivariate relations but are noncausal, and do not have independent and dependent variables. Still others have complicated and convoluted steps getting from the original concepts to the measured variables.

2. Select the two (2) most problematic, complicated or difficult variables in the article and do the following detailed analysis for each.
   
1. State the concept and give a brief summary of what (if anything) the author says about general principles of measuring this. (NOT the measurement details themselves.)
   
2. State exactly how the relevant data was measured on the units of analysis. That is, what were the original measured variable(s), and what were their categories? (Make sure you answer this in a way that shows me you know the difference between variables and categories.) (For the horn-honking article, the authors recorded how long before each of the first two honks, but left after 15 seconds had elapsed. In the Goldberg or Ransford articles, the specific questions asked on the survey would go here, along with the response catetgories.)
   
3. Explain how the original measured variables and categories were combined or modified to create the specific operational variable that was used in the statistical analysis. This is where you describe index or scale construction. Discussions of how cases were grouped or regrouped belongs here, too. Again,make clear the difference between variables and categories. (Sometimes the original measured variable requires no modification; if this is true, just say so.) (For the horn-honking article, you end up with three operational variables: (1) honked or not; (2) number of honks (0, 1, 2); and seconds before first honk. Also in this section you would discuss issues about regrouping everybody who honked after 12 seconds at the loger light. For Ransford, this would include how the original 5-category questions were summed and then dichotomized. For Goldgerg, this would be about how the original nine questions and six topics were regrouped into indices.)
   
4. Summarize any discussion by the author of why this is a good measure or of what its problems are, including statistics like factor analyses or reliability coefficients, but NOT material on the bivariate relations with other variables.
   
5. Give your own evaluation of how good you think this measurement is, explaining your reasons.

NOTE: The format of the above questions works best when the variable that gets into the statistics is a composite of several original measured variables. In some articles, what is more interesting is to start with a concept that has several related measures (each of which is fairly simple) which are then analyzed to see which is "best," in which case you might want to discuss them as a group and treat the matter of choosing among them in d). I suggest you ask me if there is any doubt in your mind about which two variables would be good choices. I should note that in some articles, all of the variables are pretty straightforward. In this case, just pick any two of them. You will not be graded down because your article is less complicated. However, I do expect people with very uncomplicated variables to analyze them perfectly, while I might decide that a mistake in analyzing some complicated variable is not that bad.

3. Internal Validity
1. Identify two of the most important hypotheses (explicit or implicit) or questions of the research. For each hypothesis or question, list those findings which are most centrally relevant to it. If there are only a few relevant findings, list all, but if there are many, list only the few which you think are most important. NOTE: A finding is the actual number(s) from the statistics, not just the author's word summary. Often a particular hypothesis is supported by several different findings which show that the bivariate relation holds true after other variables have been statistically controlled, or when the research design is altered, or when the variables are measured in different ways. If so, you would list several different findings as relating to the same hypothesis or question, but if there are many different relevant numbers for the same hypothesis, you would pick out only the few most important ones.

When articles list more than two hypotheses or goals, it can be difficult to decide which is most important. Think about the central purpose or argument of the article. Four common approaches lead to long lists of implicit or explicit hypotheses or questions.

1. They are all variations on the same general idea. In this case, pick the two variations that seem most central in the discussion.
2. The author actually believes in only one or two of the hypotheses, and the others are set up as alternates to be proved wrong. In this case, pick the ones the author seems to believe in.
3. The argument has a series of logical steps and there are hypotheses about each step. In this case, all the steps do matter, but pick out the ones that seem to you or the author to be most central in this article.
4. The article does not really have a central point and there is just a laundry list of hypotheses, questions or topics. In this case, pick out the ones that your or the author think are most interesting.
2. If there are additional findings that you or the author found interesting or surprising, list them here. (Again, a finding is not just the verbal summary, but the number that backs it up.) If you already wrote a lot for #1, you may say "no additional findings" here.

NOTE: If your article has only a few statistics, you may end up writing about all of them in #1 and #2, but if your article has a lot of statistics, do NOT write about everything. Instead, try to figure out what is really important. I do want you to learn to read the numbers, and you may ask me for help translating them.

3. Evaluate the internal validity of the findings. It is OK to make summary statements which are true for all findings, where appropriate, but be very sure to discuss the findings separately where necessary.
   
1. Is the conclusion supported by an appropriate bivariate statistical result? That is, you're just looking at the statistic copied above to be sure it is actually relevant to the hypothesis it is supposed to be relevant to. Sometimes in a bad article, the relevant finding is actually not reported! (Remember that a bivariate association of zero supports a conclusion of no effect.)
   
2. Is there adequate justification given or implied for the presumed direction of causality, e.g. for why A causes B instead of B causing A)? If yes, say why in one sentence. If no, say in one sentence what you think the problem is.
   
3. List the potential extraneous variables which have been controlled for in multivariate statistical tests. (This is simply a matter of being able to read your tables.)
   
4. What kinds of extraneous variables are simply irrelevant for this finding and could not possibly be a problem? (Examples: on stage effects for research on historical documents, maturation or other time-tied variables for research that is conducted in one short period.) Just list general classes of variables.
   
5. Which potentially significant extraneous variables have been controlled in the design of the research? List general classes of variables, mentioning specifically only those which would otherwise be a special problem.
   
6. Are there other possible problems or extraneous variables which the author discusses and gives reasons why they should not be problems? Summarize the discussion.
   
7. Are there other possible problems or extraneous variables which the author believes have not been adequately eliminated? Summarize the discussion.
   
8. Are there any remaining possible problems or extraneous variables which you can see that have not already been discussed above? Are there variables that should be controlled that were not? Could a different designed have eliminated problems? Are there things you can see as problems that you wouldn't know how to fix? If yes to any of these, discuss your concerns.
   
9. Overall, how much internal validity do you attach to the findings? Why? (Be sure to say whether your answer varies from finding to finding.)
4. External Validity
1. Give the following information about the sampling procedures in outline form, saying "not given," if it is not:
1. The units of analysis (e.g. people, organizations, sentences);
2. Definition of the population of theoretical or substantive interest;
3. Geographic areas, organizational units, or other primary sampling units and how these were chosen (probability or nonprobability?);
4. Sampling frame or operationalization of the actual population studied;
5. Restrictions imposed on the actual population;
6. Method of selecting units of analysis from the actual population (probability or nonprobability?);
7. Response rate;
8. Sample size;
9. Analyzed sample size, after cases dropped for missing data or other reasons, and why cases were dropped.
10. Information given relevant to judging the typicality or representativeness of this sample.

If you feel that this outline does not adequately demonstrate your understanding of the sampling, or that there is something important about the sampling that does not fit in this outline, write an additional paragraph that provides any necessary extensions or clarifications. (Do not, however, omit the outline.)

Often articles which use one of the well-known large national probability samples do not give much information about the sample because they assume that professionals will recognize the sample title and already know the basic information. Check with me if you suspect this is the situation with your article.

2. Evaluate the sampling procedures.
1. Do the geographic or other restrictions imposed on the actual population (c, d, e, above) seem justified in light of the purposes of the research and practical constraints?
2. Were units of analysis selected randomly from the actual population? If not, were the selection procedures especially prone to bias, or only uncontrolled?
3. Are you aware of anything in the research procedures that added any implicit restrictions to the sample (e.g. interviewing only during the day)?
4. Do the frequency distributions or other results suggest that the sample is reasonably representative, or do they point to problems or biases?
5. Overall, how good do you feel the sampling was?
1. Strictly speaking, to what population can the results of this research be generalized?
2. To what population would you feel reasonably confident the results probably apply? Why?
3. At what point would you be very hesitant to apply these results?
5. Overall Evaluation
1. Give your overall evaluation of the methods used in this article: what things were done well? what were done poorly? How much trust do you put in the findings?
2. Look at this article's "packaging," that is, the theoretical introduction and the discussion or interpretation at the end. Do you feel that the actual methods and results support the theoretical and interpretive claims of the author? Why?
3. What possible ethical issues might have arisen in the process of doing this research? Do you think the researcher's ethical decisions were all justified, or are some questionable? Why?
4. To sum up, what do you feel you've learned worth knowing from this article? (If your answer is "nothing", explain why.) (Please note: this question is about the article and refers to the quality of information it contains.)
5. Tell me anything you would like me to know about your experiences doing this analysis, or any suggestions you have for future revisions of this assignment.
   
   

END OF REVIEW

Some Remarks on Grading Standards

1. The key to this assignment is to apply the methodological concepts you have learned in this course to the evaluation of a research article. You demonstrate your ability by specifically linking the procedures discussed in the article to the concepts. Think of it as a take-home test, not as an opinion essay. You have the burden of proof to demonstrate that you know what you are doing. In particular:
1. Never answer just "yes" or "no"; always explain your answer.
2. Never state some general methodological term or principle without linking it up specifically to something in the article (or to something missing in the article).
3. Never give a vague or evasive answer in which you avoid sticking your neck out (hoping you won't be marked "wrong"); if you don't commit yourself to a specific answer, I will assume you do not know what it is. But try to say what is needed as briefly as possible. Long-winded rambling answers are evidence that you do not know precisely what is important.
2. Questions of "fact" will be graded by comparing what the article says with what you said it said, along with your ability correctly to use the relevant methodological terms. Questions requiring evaluation will be graded according to these criteria:
1. you take some position
2. you defend your position by talking about your article in ways which raise issues that we discussed in class.
3. If the article fails to give some information the review asks for, you get credit by saying that the article fails to give the information. Note that this failure should then become part of your evaluation of the relevant section. (I will try to avoid approving articles that are missing too much of the relevant information.)
4. If the article is unclear or ambiguous, or if you are ambivalent in your evaluation of something, it is fine to give an answer that expresses these problems.
5. Don't blindly assume the author is using the correct methodological terms for what s/he did. For example, Ransford describes his sample as "disproportional stratified" (p. 298 of Golden reader). But if you carefully read the paragraph on p. 298 and the extended description of the sample on pp. 309-310, you will discover that the sample was not stratified at all: three clusters (Watts, South Central, Crenshaw) were chosen purposively; blocks were chosen randomly within clusters; and households were chosen purposively within blocks, after a random start on block corner and an overall quota of 8 households per block. The use of the term "random methods," rather than "random sample," is the sort of thing you'll see when the procedures are less than ideal.
   

Top

Questions or Comments? Email Oliver -at- ssc -dot- wisc -dot- edu. Last updated December 25, 2004 © University of Wisconsin.