Evaluation of the Yorkshire and Humber Excellence Hub ‘Find Your Way’ programme

The Excellence Hub for Yorkshire and Humber is a partnership between the Universities of Hull, Leeds, Sheffield, and York. This regional partnership works in collaboration with local authorities across the region to develop opportunities for secondary pupils identified as Gifted and Talented. Find Your Way (FYW) is a four-year programme of enhanced activities within the Hub, which aims to address the needs of a cohort of 2007/8 Year 9 pupils who have high potential and who are from a disadvantaged background. The evaluation will be completed in 2012 when the cohort leave school.

The aim of the evaluation is to determine the impact of FYW on a number of outcomes, including GCSE, A/S and A level results, and to provide a comprehensive and detailed examination of all aspects of the programme. Our primary outcome measure will be post-18 participation in Higher Education.

The evaluation will be undertaken by researchers from the IEE, led by Peter Rudd.  The research team, which is independent of the Excellence Hub, will be advised and supported at all stages of the research by a Steering Group comprising experts in the substantive field (Gifted and Talented programmes), including a representative from the Excellence Hub Collaboration. 

Method

FYW will be evaluated using a mixed methods approach, combining quantified outcomes in an Impact evaluation with qualitative outcomes in a Process evaluation:

  • Impact evaluation
    • Primary Analysis: Case-Control Design with Multiple Comparison Groups
    • Secondary Analysis: Case-Control with Longitudinal Follow-up (Potentially ITS with Matched Control)
  • Process evaluation
    • Focus Groups with Students
    • In-Depth interviews with Teachers and other Stake-Holders
    • Evaluation of Questionnaires routinely provided by ‘Find Your Way’

For the Impact evaluation, the primary analysis will be a case-control design with multiple comparison groups. Quasi-experiments occur when studies are unable to use random assignment to allocate participants to groups; instead researchers seek to make the groups as comparable as possible through matching based on known characteristics (Bryman, 2008, Shadish et al., 2002). Including design elements from randomised experimentation, such as control groups or longitudinal observation, can help reduce the likelihood of bias, confounding and other threats to validity ― although it cannot eliminate them (D'Agostino and Kwan, 1995, Shadish et al., 2002). If the enhanced activities programme is effective we should observe a statistically significant difference in outcomes when compared to the non-intervention control groups.

The first ‘control’ group will be an equivalent, matched group of individual pupils from schools who have pupils taking part in FYW. The control pupils will be selected by the teachers using the same methods (or as similar as possible) as the methods that is, those students from the year above in each participating school who would have been eligible for FYW if the programme had been available a year earlier. The second ‘control’ group will also be an equivalent, matched group of individual pupils, but selected from the year below rather than the year above. That is, using the same methods, pupils would be selected by the teachers who would have been eligible for FYW as if the programme had been going to continue for another year.

The intervention and control cohorts of pupils from each school will be followed up in Year 11, 12, and 13 and at post-18 destination.

The secondary analysis will investigate the impact on whole schools using interrupted time series (ITS) to plot school-level data from each of the schools participating in FYW, looking historically at the post-18 destinations of all pupils, helping identify longer term trends that may affect decision-making. Its greatest strength is that it allows the analysis to control for the “threat of history”, such as broader social trends and cycles which can undermine inferences from the analysis (Shadish and Cook, 2009, Shadish et al., 2002) . Comparing the intervention group time series against ‘no-intervention’ control time series reinforces the analysis by strengthening inferences about whether an observed change is due to the intervention itself, or due to other factors. A comparison group can be formed by plotting a ‘control’ time series using whole school-level data from matched schools who are not participating in FYW.

For the Process evaluation, we will analyse qualitative data to assess the influence of the programme on students’ confidence, aspirations and awareness of the opportunities available to them. We will investigate the views of teachers, schools and other stake-holders involved in the programme.

Sample size

Approximately 280 pupils: 130-140 FYW and 130-140 control students in fourteen schools in North Yorkshire, York, NE Lincolnshire, Hull, Rotherham, Bradford, Kirklees, and Leeds.

Results so far

Report to the Excellence Hub Advisory Board, 30 November 2009: Hendry VL, Ainsworth HA, & CJ Torgerson (2009), 2009 Report on the First Set of ‘Find Your Way’ Focus Groups.

Advisory Board papers, including IEE updates

References

BRYMAN, A. (2008) Social Research Methods. 3rd ed. Oxford, Oxford University Press.

D'AGOSTINO, R. B. & KWAN, H. (1995) Measuring Effectiveness: What to Expect without a Randomized Control Group. Medical Care, 33(4): AS95-AS105.

SHADISH, W. R. & COOK, T. D. (2009) The renaissance of field experimentation in evaluating interventions (advance online version). Annual Review of Psychology, 60: 1.1-1.23.

SHADISH, W. R., COOK, T. D. & CAMPBELL, D. T. (2002) Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, Houghton Mifflin.

Last Updated: November 10, 2011 | Webmaster, iee

Back to the Top