Methodology guidance: where are the methods?

Abstract: There is an increasing stream of published 'methodology guidance' covering the conduct and reporting of almost every kind of observational study, randomised trial, systematic review and meta-analysis. In some cases, journals require that this guidance is followed as a condition for publication. But what are the methods that generate this guidance?

Using Network Meta-analysis (NMA) as an example, we look at guidance issued by GRADE (Grading of Recommendations Assessment, Development, and Evaluation) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). We find that, in common with the majority of 'methodology guidance', the GRADE and PRIMA extensions for Network Meta-analysis are based on no more than opinion, and issued without any explicit empirical or theoretical under-pinning. We identify aspects of GRADE-NMA and PRSIMA-NMA that are incorrect or even incoherent, and illustrate alternatives. 

With scientific method set aside, the door is opened to informal approaches described as: 'meetings', 'emails', 'discussion', 'iterative refinement', 'brainstorming', 'Delphi survey'. But these, too, are 'methods' whose properties can be studied. The social psychology literature has shown them to be unreliable and vulnerable to a range of group dynamic biases, including 'groupthink', 'false consensus' and 'echo-chamber' effects.

The current approach to methodology guidance is leading to ossification and hegemonism, and feeds a 'post-truth' agenda in which the investigator’s priority is conformity with guidelines rather than a search for understanding. We distinguish between prescriptive and descriptive methodology guidance, and suggest that a debate is needed on what form methodology guidance should take.