12 - The illusion of learning from observational research  pp. 251-273

The illusion of learning from observational research

By Alan S. Gerber, Donald P. Green and Edward H. Kaplan

Image View Previous Chapter Next Chapter


Empirical studies of cause and effect in social science may be divided into two broad categories, experimental and observational. In the former, individuals or groups are randomly assigned to treatment and control conditions. Most experimental research takes place in a laboratory environment and involves student participants, but several noteworthy studies have been conducted in real-world settings, such as schools (Howell and Peterson 2002), police precincts (Sherman and Rogan 1995), public housing projects (Katz, Kling, and Liebman 2001), and voting wards (Gerber and Green 2000). The experimental category also encompasses research that examines the consequences of randomization performed by administrative agencies, such as the military draft (Angrist 1990), gambling lotteries (Imbens, Rubin, and Sacerdote 2001), random assignment of judges to cases (Berube 2002), and random audits of tax returns (Slemrod, Blumenthal, and Christian 2001). The aim of experimental research is to examine the effects of random variation in one or more independent variables.

Observational research, too, examines the effects of variation in a set of independent variables, but this variation is not generated through randomization procedures. In observational studies, the data generation process by which the independent variables arise is unknown to the researcher. To estimate the parameters that govern cause and effect, the analyst of observational data must make several strong assumptions about the statistical relationship between observed and unobserved causes of the dependent variable (Achen 1986; King, Keohane, and Verba 1994). To the extent that these assumptions are unwarranted, parameter estimates will be biased.

Achen, Christopher H. 1986. The Statistical Analysis Of Quasi-Experiments. Berkeley: University of California Press
Angrist, Joshua A. 1990. “Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records.” American Economic Review 80(3): 313–36
Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin. 1996. “Identification of Casual Effects Using Instrumental Variables.” Journal of the American Statistical Association 91(June): 444–55
Berube, Danton. 2002. “Random Variations in Federal Sentencing as an Instrumental Variable.” Unpublished ms, Yale University
Box, George E. P. and G. C. Tiao. 1973. Baysian Inference in Statistical Analysis. Reading, MA: Addison-Wesley
Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin
Cook, Thomas D. and Monique R. Payne. 2002. “Objecting to the Objections to Using Random Assignment in Educational Research,” in Frederick Mosteller and Robert Boruch (eds.), Evidence Matters: Randomized Trials in Education Research. Washington, DC: Brookings Institution Press
Eldersveld, Samuel J. 1956. “Experimental Propaganda Techniques and Voting Behavior.” American Political Science Review 50 (March): 154–65
Freund, John E. 1971. Mathematical Statistics, 2nd ed. New York: Prentice-Hall
Gerber, Alan S. and Donald P. Green. 2000. “The Effects of Canvassing, Direct Mail, and Telephone Contact on Voter Turnout: A Field Experiment.” American Political Science Review 94: 653–63
Gosnell, Harold F. 1927. Getting-Out-The-Vote: An Experiment in the Stimulation of Voting. Chicago: University of Chicago Press
Green, Donald P. and Alan S. Gerber. 2003. “Reclaiming the Experimental Tradition in Political Science,” in The State of the Discipline IIIHelen Milner and Ira Katznelson (eds.), Washington, DC: American Political Science Association
Heckman, James J. and Jeffrey A. Smith. 1995. “Assessing the Case for Social Experiments.” Journal of Economic Perspectives 9 (Spring): 85–110
Howell, William G. and Paul E. Peterson. 2002. The Education Gap: Vouchers and Urban Schools. Washington, DC: Brookings Institution Press
Imbens, Guido W., Donald B. Rubin, and Bruce I. Sacerdote. 2001. “Estimating the Effect of Unearned Income on Labor Earnings, Savings, and Consumption: Evidence from a Survey of Lottery Winners.” American Economic Review 91(4): 778–94
Katz, Lawrence F., Jeffrey R. Kling, and Jeffrey B. Liebman. 2001. “Moving to Opportunity in Boston: Early Results of a Randomized Mobility Experiment.” Quarterly Journal of Economics 116: 607–54
King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton: Princeton University Press
Kramer, Gerald H. 1970. “The Effects of Precinct-Level Canvassing on Voting Behavior.” Public Opinion Quarterly 34 (Winter): 560–72
Levitt, Stephen D. 1997. “Using Electoral Cycles in police hiring to Estimate the Effect of Police on Crime.” American Economic Review 87(3): 270–90
McConahay, John B. 1982. “Self-Interest versus Racial Attitudes as Correlates of Anti-Busing Attitudes in Louisville: Is it the Buses or the Blacks?” Journal of Politics 44(3): 2–720
Morton, Rebecca. 2002. “EITM: Experimental Implications of Theoretical Models.” The Political Methodologist 10(2): 14–16
Rosenstone, Steven J. and John Mark Hansen. 1993. Mobilization, Participation, and Democracy in America. New York: Macmillan Publishing Company
Sherman, Lawrence W. and Dennis P. Rogan. 1995. “Deterrent Effects of Police Raids on Crack Houses: A Randomized, Controlled Experiment.” Justice Quarterly 12(4): 755–81
Slemrod, J., M. Blumenthal, and C. Christian. 2001. “Taxpayer Response to an Increased Probability of Audit: Evidence from a Controlled Experiment in Minnesota.” Journal of Public Economics 79(3): 455–83
Weiss, Carol H. 2002. “What to Do until the Random Assigner Comes,” in Frederick Mosteller and Robert Boruch (eds.), Evidence Matters: Randomized Trials in Education Research. Washington, DC: Brookings Institution Press