Randomized Experiments

Randomized experiments have become a key method for identifying the effects of treatments or programs on outcomes in criminology and criminal justice. They have also become an important method for identifying how people’s perceptions and attitudes change in different scenarios that are created in the laboratory. We begin the chapter by describing the structure of a randomized experiment and then illustrate why randomized experiments provide a very strong ability to make causal inferences without concern for confounding. We then turn to selected design types and associated analyses. We pay particular attention to block randomized studies and illustrate how they help the reader to maximize equivalence and statistical power in randomized experiments. Finally, we discuss the approach of using covariates in experimental studies as a method of increasing statistical power.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic €32.70 /Month

Buy Now

Price includes VAT (France)

eBook EUR 67.40 Price includes VAT (France)

Softcover Book EUR 84.39 Price includes VAT (France)

Hardcover Book EUR 116.04 Price includes VAT (France)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Notes

Statistically, the groups are equivalent in the sense that the expected value of the mean for any baseline characteristics is the same across conditions. However, the observed groups will differ, but these differences will conform to known probability distributions, enabling us to differential between outcome differences that were plausibly due to these random differences or likely due to the experimental manipulation (e.g., treatment).

See Boruch (1997). Econometrics includes other methods, such as instrumental variable analysis, that allow for an unbiased estimation of a treatment effect. These are beyond the scope of this text. However, these methods often rely on naturally occurring random processes, thus mimicking what is discussed here.

See Weisburd and Gill (2014). Stata programs were developed to run a randomization sequence (blocked or naïve) on the JCE dataset and then run a t-test comparing the treatment and control group means at baseline on the three outcomes of interest. Stata’s simulation function was then used to run each program 10,000 times and create a dataset containing the group means, t-values, p-values, an indicator showing whether or not the two groups were significantly different at baseline for each iteration, and the absolute average mean group difference across all iterations.

Of course, this is about what we would have expected given a .10 significance threshold and a fair randomization procedure. But the important point is that the block randomization approach allows us to do better.

In factorial experimental designs, it is ideal to have fully balanced designs. This both simplifies the analysis, as explained below, and maximizes statistical power given a fixed sample size. For block randomized designs, the sample sizes across the levels of the blocking factor are typically unequal. However, it is ideal to ensure balance on the experimental or treatment factor within each level of the blocking factor. That is, block randomized designs are ideally at least partially balanced.

For a factorial experiment where two (or more) factors are manipulated, a Type III model will usually be preferred. In this situation, the unbalanced nature of the design is merely an experimental artifact and should be small in magnitude. Thus, any difference in the sample sizes across cells is random, and giving each cell equal weight in the analysis makes the most sense. That is, conceptually, we are interested in the effects that would be estimated if the design were balanced. However, if the main effects for a Type II versus Type III ANOVA differ, it is wise to explore why that is the case and carefully assess which makes most conceptual sense for your research question.

References

Author information

Authors and Affiliations

  1. Institute of Criminology, Faculty of Law, Hebrew University of Jerusalem, Hebrew University of Jerusalem Inst. of Criminology, Fac. of Law, Inst. of Criminology, Fac. of Law, Fairfax, VA, USA, Jerusalem, Israel David Weisburd
  2. Department of Criminology, Law and Society, George Mason University, Manassas, VA, USA David B. Wilson
  3. Department of Criminal Justice, Temple University, Philadelphia, PA, USA Alese Wooditch
  4. Department of Sociology, Iowa State University Department of Sociology, Ames, IA, USA Chester Britt
  1. David Weisburd