Evaluating non-randomised intervention studies


In the absence of randomised controlled trials (RCTs), health care practitioners and policy-makers rely on non-randomised studies to provide evidence of the effectiveness of healthcare interventions. However, there is controversy over the validity of non-randomised evidence, related to the existence and magnitude of selection bias. The objective of this project was to carry out three reviews to consider methods and related evidence for evaluating bias in non-randomised intervention studies.


Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when the treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results.

Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Six tools were considered potentially suitable for use in systematic reviews, but each requires revision to cover all relevant quality domains.

Health care policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made.

The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies which are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are not feasible or are unethical.

Conducted by: JJ Deeks1, J Dinnes2, R D'Amico1, AJ Sowden3, C Sakarovitch1, F Song4, M Petticrew5, DG Altman1

1. Centre for Statistics in Medicine, Institute of Health Sciences, Oxford; 2. Southampton Health Technology Assessments Centre, University of Southampton; 3. NHS Centre for Reviews and Dissemination; 4. Department of Public Health and Epidemiology, University of
Birmingham; 5. MRC Social and Public Health Sciences Unit, University of Glasgow

Further details

Project page on HTA Programme website


Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, Petticrew M, Altman DG. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):1-186


Dinnes J, Sowden AJ, Deeks J, Altman D. A review of quality assessment tools for non-randomised intervention studies. Annual Meeting of the International Society of Technology Assessment in Health Care. Philadelphia. 2001


Commissioned by the HTA Programme