Accessibility statement

Methodology guidance: where are the methods?

Thursday 18 July 2019, 11.00AM

Speaker(s): Professor Tony Ades, University of Bristol

Abstract: There is an increasing stream of published 'methodology guidance' covering the conduct and reporting of almost every kind of observational study, randomised trial, systematic review and meta-analysis. In some cases, journals require that this guidance is followed as a condition for publication. But what are the methods that generate this guidance?

Using Network Meta-analysis (NMA) as an example, we look at guidance issued by GRADE (Grading of Recommendations Assessment, Development, and Evaluation) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). We find that, in common with the majority of 'methodology guidance', the GRADE and PRIMA extensions for Network Meta-analysis are based on no more than opinion, and issued without any explicit empirical or theoretical under-pinning. We identify aspects of GRADE-NMA and PRSIMA-NMA that are incorrect or even incoherent, and illustrate alternatives. 

With scientific method set aside, the door is opened to informal approaches described as: 'meetings', 'emails', 'discussion', 'iterative refinement', 'brainstorming', 'Delphi survey'. But these, too, are 'methods' whose properties can be studied. The social psychology literature has shown them to be unreliable and vulnerable to a range of group dynamic biases, including 'groupthink', 'false consensus' and 'echo-chamber' effects.

The current approach to methodology guidance is leading to ossification and hegemonism, and feeds a 'post-truth' agenda in which the investigator’s priority is conformity with guidelines rather than a search for understanding. We distinguish between prescriptive and descriptive methodology guidance, and suggest that a debate is needed on what form methodology guidance should take.

Location: Alcuin A Block A019/20

Who to contact

For more information on these seminars, contact:
Alfredo Palacios
Shainur Premji

If you are not a member of University of York staff and are interested in attending a seminar, please contact 
so that we can ensure we have sufficient space

Economic evaluation seminar dates

  • Tuesday 28 November 2023
  • Thursday 14 December 2023