RCTS FOR POLICY INTERVENTIONS
RCTs for policy interventions? A review of reviews and meta-regression
Background
The aim of this project was to investigate the implications of choice of study design when estimating the effect of policy interventions. We searched for systematic reviews that had included both randomised and non-randomised studies to estimate the effect of a policy intervention. Methods used by reviewers to try to identify and/or address bias/confounding that are uncontrolled for in non-randomised designs were investigated. We then focused on a smaller set of systematic reviews, where we had access to the data from the primary studies. Multi-variate regression was used to investigate heterogeneity in the findings of randomised and non-randomised studies.Findings
From the resampling studies we have no evidence that the absence of randomisation directly influences the effect size of policy interventions in a systematic way. At the level of individual studies, nonrandomised trials may lead to different effect sizes, but this is unpredictable. Many of the examples reviewed and the new analyses in the current study reveal that randomisation is indeed associated with changes in effect sizes of policy interventions in field trials. Despite extensive analysis, we have identified no consistent explanations for these differences. Researchers mounting new evaluations need to avoid, wherever possible, allocation bias. New policy evaluations should adopt randomised designs wherever possible.
Conducted by: S Oliver1, AM Bagnall2, J Thomas1, J Shepherd3, A Sowden2, I White4, J Dinnes3, R Rees1, J Colquitt3, K Oliver1, Z Garrett11. Social Science Research Unit, Institute of Education, University of London; 2. Centre for Reviews and Dissemination; 3. Southampton Health Technology Assessments Centre, Wessex Institute for Health Research & Development, University of Southampton; 4. Medical Research Council Biostatistics Unit, Institute of Public Health, Cambridge
Publications
Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z. Randomised controlled trials for policy interventions: a review of reviews and meta-regression. Health Technol Assess. 2010;14(16)
Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z. RCTs for policy interventions?: a review of reviews and meta-regression. Birmingham: University of Birmingham. 2008