6.4 ASSESSMENT OF QUALITATIVE RESEARCH 

The application of quality criteria to qualitative research is widely debated, although many accept the need for clear and transparent approaches for judging the quality or credibility of research. For example, it has been noted that the distinguishing mark of all ‘good’ research is the awareness and acknowledgement of error and, that what flows from this is the necessity of establishing procedures which will minimise the effect such errors may have on what counts as knowledge.26 It is less clear whether consensus can be reached over an agreed set of principles for judging quality. Qualitative researchers from different disciplines and from different theoretical backgrounds may have different criteria for assessing the quality of a study.27 Some argue that quality cannot be determined by following prescribed formulas28 or that it is fruitless to try to set standards for qualitative research as such.29 Others, accepting the need for structured procedures, argue for more rigorous use and reporting of analytical approaches which improve reliability and validity.30 Others have suggested there are general questions that can be asked to judge validity and reliability in qualitative research, but that these are not readily codified.31 It has also been argued that quality assessment should take account of theory in the design of the research, analysis, and interpretation of the data.32

6.4.1 How should quality be assessed?

Despite lack of consensus about quality assessment a number of different tools and techniques are now available. Over one hundred sets of proposals on quality in qualitative research have been identified,33 a subset of which have been reviewed,34 including five that were developed specifically for use in systematic reviews.35, 36, 37, 38, 39 The majority of tools available are generic, and to date there have been few attempts to develop method specific approaches. This is despite arguments that different qualitative methods need to be appraised in different ways.40

Some issues in using structured approaches were illustrated in a recent study.41 Two structured methods – the Critical Appraisal Skills Programme (CASP) tool,42 and the Quality Framework34 – were systematically compared with an approach based on unprompted judgement (where experienced qualitative researchers relied on their own expertise to make judgements of quality). Each approach was used to assess twelve qualitative studies investigating support for breastfeeding. Agreement between researchers and between methods was slight, and importantly researchers disagreed on the quality of the studies, whether papers were actually reporting qualitative research and whether the study was relevant to the review question. Because answering questions about quality is largely a subjective process involving judgement, it may lead to differences both between researchers and methods.

In addition, the Quality Framework was criticised for its length and complexity, which is likely to impact on its use in future systematic reviews. The authors identified a need for continued debate and empirical research into the use of quality assessment. Similarly, the authors of a recent review who attempted to apply two different quality frameworks, concluded that further methodological work is needed to produce clear guidance about how quality appraisal should be undertaken.32

An innovative approach, developed to appraise qualitative studies for inclusion in a set of reviews focusing on peoples’ experiences and perspectives, uses generic methodological quality criteria tailored to the specific review question.43 It is designed to help researchers assess to what extent studies may have distorted, misrepresented or simply missed people’s experiences and perspectives. The authors have published a series of reports that outline how the approach has been applied in practice.44, 45

A structured review of reports published between 1988 and 2004, appraising and synthesising qualitative studies in health and health care is available.46 The authors found that over 60% of the 42 reviews included either explicitly stated that quality appraisal was not carried out or failed to report any appraisal of studies. Interestingly where quality appraisal was used, in all but one case the instrument or criteria were modified, suggesting that available methods are difficult to apply in practice. Others have opted to construct their own criteria for assessing rigor as part of the review process.47

Box 6.1 lists some of the different appraisal tools that have been developed explicitly for use in systematic reviews and/or have been used for that purpose (this is not a comprehensive list). Researchers interested in carrying out quality assessment, might consider using one or more of these tools.

Box 6.1: Appraisal tools

Popay, Rogers & Williams (1998)39

 

Primary question relates to the appropriateness of the methods used. This is followed by a detailed assessment of methodological soundness.

Critical Appraisal Skills Programme (1998)42

 

10 questions relating to rigour, credibility and relevance.

Quality Framework (2003)34

 

18 questions relating to 9 key areas: findings; design; sample; data collection; analysis; reporting; reflexivity and neutrality; ethics and auditability.

Prompts for appraising qualitative research (2004)33

Generic set of prompts relating to aspects of reporting and aspects of study design and execution.

Long & Godfrey (2004)38

 

A tool to explore descriptive and evaluative elements of a study. 34 questions across 4 key areas: phenomenon studied and context; ethics; data collection, analysis and potential researcher bias; policy and practice implications.

Walsh & Downe (2006)48

 

Set of prompts relating to 8 key areas: scope and purpose; design; sampling strategy; analysis; interpretation; reflexivity; ethical dimensions; relevance and transferability.

6.4.2 How should quality assessment be used?

Quality assessment has been used to establish a quality threshold below which studies will be excluded, or to distinguish between studies in terms of overall contribution.32, 36, 49 There is no consensus as to which approach is preferable. Quality assessment can also be used to gain an understanding of the relative strengths and weaknesses of the body of evidence and taken into account during the process of synthesis.

Some have reported that better quality studies appear to make stronger contributions to the synthesis19, 49 or that weaker studies contribute nothing substantially different from the stronger studies.32 Sensitivity analysis has been used to explore the relationship between the quality of qualitative studies and contribution to review findings.50 The analysis was based on 62 primary studies from five reviews, and suggested that studies judged to be of low quality contributed little to the overall review findings. This appears to be the first attempt to apply sensitivity analysis to the question of quality in qualitative research and further assessment is required. However, the findings are consistent with the more descriptive accounts offered about study quality and overall contribution to synthesis.19, 32, 49

6.4.3 When should quality assessment be carried out?

The use of quality assessment is further complicated by debate around when it should be carried out. The need for appraisal of studies before the synthesis has been queried.51 The authors of one qualitative synthesis reported that the necessity of prior quality appraisal was a moot point.36 They did go on to comment however that the appraisal process was a useful prelude to the synthesis because it helped to screen out inappropriate and poor quality studies. Clearly, if quality assessment is to be used to establish a quality threshold, then assessment will need to take place before the synthesis.