REGISTER OF ONGOING SYSTEMATIC REVIEWS
International register of ongoing systematic reviews
Delphi exercise: summary of second round responses
Introduction
In response to growing concern about reporting biases, and calls for the prospective registration of systematic reviews, the Centre for Reviews and Dissemination (CRD) has taken the initiative in establishing an international register of ongoing systematic reviews. The register will use the existing IT platform and infrastructure that supports production and provision of the CRD databases and use an approach similar to that adopted for clinical trials registers, to record prospectively key features of systematic reviews.
The aim of the Register is to provide a freely accessible web-based register of systematic reviews in health and social care. Registration will be free of charge and take place at the protocol stage (or its equivalent). The Register will provide a record of systematic reviews underway and thereby help avoid unplanned or unnecessary duplication, for example, by funding agencies or students planning to do a systematic review as a research project. By recording key methodological information early in the review process, maintaining an audit trail of any changes, and providing a permanent publicly accessible record, the register will facilitate assessment of potential bias within a review after it has been completed and published. It will also provide advance notice about reviews in progress to users of the findings of systematic reviews, such as guidelines developers, who may wish to contact the researchers for information on their completion date and current findings.
In order to arrive at an internationally agreed minimum data set for registration, a Delphi consultation exercise has been undertaken. The opinions of international experts in systematic reviewing, methodology, commissioning, and guideline development in health and social care and journal editors around the world were sought. A detailed report is being prepared for publication. This report gives a summary of the responses to the second round questionnaire. These will be used by the Register Advisory Group to inform the development and implementation of the Register.1. Information about responses and respondents
There were 194 respondents in the first round and 209 in the second round of the Delphi exercise. 113 of those who took part in the second round said they had also taken part in the first round; 72 said they had not; and 24 could not remember.
There were no significant differences between role designations; areas of health interest; review methods of interest; authorship of, or involvement in systematic reviews; or proportion of work related to research methodology between the first and second round respondents.
In the first round, 66% of respondents said English was their first language. This was 65.3% in the second round. Respondents to both the first and second rounds were based in the same 34 countries, with an additional six countries represented in the first round only, and a different five countries represented in the second round only.
2. Fields for the minimum dataset
The aim of registration is to provide enough information about the review to help people planning reviews in a similar area to decide whether or not to proceed and also to capture those attributes of a review that are likely to be important in the avoidance of bias. Therefore, this Delphi process was undertaken to establish the minimum data set that will allow this. It did not seek to capture the attributes of the wider information that should be included in a protocol for a systematic review, or to determine all the variables that people might wish to record in registers of systematic reviews that would be used for other purposes.
In the first round of the Delphi process, opinions were sought on some practical aspects of implementing the Register and whether individual elements of a review protocol were considered ‘Essential’, ‘Desirable’ or ‘Not needed’ for the purposes of the Register. In the second round, participants were asked to indicate if they agreed or disagreed with the designation of ‘Required’ or ‘Optional’ to each field. Alternatively, they could indicate that the field is not needed or that they had no opinion about an item. A summary of the comments made in the second round is given here.
2.1 Minimum dataset: identification
| 1. REVIEW TITLE: (Required field – 98% of 2nd round respondents agreed) This is the working title of the review. Ideally it should succinctly state the intervention or exposure being reviewed and the associated health or social problem. A separate field will be available in the record for the final title of the review (for example, as used when it is published). |
Summary of 14 comments:
Comments generally indicated agreement that this field should be required, but that flexibility be allowed in what the title includes (a) in order to avoid duplication of information in other fields, (b) to facilitate inclusion of the planned wide range of review types and topics, and (c) in recognition that review titles might change over time and that a review might be published more than once.
It was suggested that a structure be suggested, such as GRADE or key elements for inclusion in the title.
t was noted that different groups have different intended users for reviews and titles should be specific enough (e.g. on population, comparators, outcomes) to minimise inappropriate duplication.
| 2. NAMED CONTACT: (Required field – 97% agreed) The Named Contact acts as the guarantor for the accuracy of the information presented in the Register record. Ideally this should be the lead reviewer or a representative of the review team, such as the corresponding researcher. This person is also responsible for submitting details of any amendments while the review is ongoing and publication details after the review is completed. The Named Contact is the person to whom users of the Register would send questions or comments. |
Summary of 9 comments:
There was concern that review teams could be inundated with enquiries.
It was suggested that in order to encourage the registration of pharmaceutical industry reviews they would want to retain commercial sensitivity and therefore contact details would need to be kept confidential.
t was suggested that this field be editable in order to allow for absences and changes.
| 3. ORGANISATIONAL AFFILIATION OF THE REVIEW: (Required field – 84% agreed) Full title of the organisational affiliation for this review, and web address if available. (This may be completed as ‘None’) |
Summary of 21 comments:
It was suggested that there be an option to keep details confidential.
Guidance about the affiliation of the review versus the reviewers was requested (for example, whether it would include the organisations of each member of a Cochrane team).
It was felt important to have the option of ‘none’ as stated.
| 4. NAMED CONTACT ADDRESS: (Optional field – 77% agreed) Full postal address for the Named Contact. |
Summary of 11 comments:
Comments ranged from those who felt postal address was totally unnecessary to those who felt it was essential. It was suggested that
an organisation’s address might be more reliable than that for a individual person.
| 5. NAMED CONTACT PHONE NUMBER: (Optional field – 78% agreed) Telephone number for Named Contact. |
Summary of 9 comments:
Respondents who commented had no major problems with this as an ‘Optional’ field, but were likely to decline to provide a phone number as the potential nuisance value is too high. It was also suggested that an organisation’s telephone number might be more reliable than that for a individual person.
Others felt this should be mandatory if no email address was provided.
t was noted that everyone's communication preferences should be accommodated (for example that while people with impaired hearing may prefer email, others may prefer to communicate by phone).
| 6. NAMED CONTACT EMAIL: (Required field – 93% agreed) Electronic mail address of the Named Contact. |
Summary of 8 comments:
Comments ranged from a review author who would not be happy if everyone could email them, to those who would find email contact invaluable.
It was suggested that as not all reviewers will have a reliable email so a telephone contact should also be given.
he alternative of providing a generic email of the group or organisation undertaking the review was suggested. The provision of a secondary email address was felt prudent; along with the regular verification of all email addresses. In addition, it would be essential to protect against ‘phishing’ attacks on the register seeking to collect email addresses for spam messaging.
| 7. REVIEW TEAM: (Optional field – 67% agreed) Names of all members of the review team. |
Summary of 23 comments:
It was agreed that this information gives the opportunity to recognise all those involved in the review. However, some respondents noted that the review team is likely to change over the course of a review; making it important that this field can be edited.
The information could be used as an indicator of the different skills involved in a review team which might be apparent from the names or included as details of ‘contribution’.
his was thought to be an important field for assessing potential conflict of interest and to minimize ghost and honorary authorship.
| 8. REVIEW TEAM MEMBERS' ORGANISATIONAL AFFILIATIONS: (Optional field – 76% agreed) Affiliations of each member of the review team. |
Summary of 13 comments:
It was felt that knowledge of the organisations involved would help (a) identify the different skill sets involved in the review, and (b) identify potential conflicts of interest.
here were conflicting opinions on the need for the organisation details versus the names of the members of the review team.
| 9. COLLABORATORS: (Optional field – 76% agreed) Details of any individuals or organisations who are collaborating on the review but who are not review team members. |
Comments ranged from this being important information about contributorship and the engagement of stakeholders, to it not being needed.
1.2 Minimum dataset: general information
| 10. ANTICIPATED OR ACTUAL START DATE: (Required field – 88% agreed) The date of commencement of the systematic review. For the purposes of the Register this is defined as when screening of the identified studies against the eligibility criteria begins. |
Summary of 16 comments:
There were a number of comments about the definition of the ‘start’ of the review, particularly in relation to the date of the searches, which was felt by some to be a more appropriate ‘start’ date.
It was suggested that the option of indicating whether the date submitted was 'anticipated' or 'actual' would be advantageous.
As this information could be an indicator of whether a review will be delivered or not, the ability to amend or explain discrepancies between actual and proposed start dates was requested.
t was proposed that the registration submission date could be the required field, with the anticipated start date as an optional field only.
| 11. ANTICIPATED COMPLETION DATE: (Required field – 79% agreed) The date by which the review is expected to be completed. |
Summary of 18 comments:
There was general concern in the comments that this would be difficult to estimate. An option would be to state the date in broad terms (e.g. last quarter 2011) and be able to edit the date and record reasons for delays. An automated email reminder as the ‘time’ approached would help ensure up-to-date information was provided. If no revised date was forthcoming, this could be used as an indicator that a review was not going to be completed and therefore another group could get started.
Some respondents were uncertain of the value of this date, and suggested that the start date alone was sufficient.
t was suggested that the completion date and the date on which the review becomes available are potentially different dates, the latter being under the control of the funders or commissioners of the review or the publishers, and that this would need to be made clear.
| 12. FUNDING SOURCES/SPONSORS: (Required field – 93% agreed) Details of the individuals, organizations, groups or other legal entities who take responsibility for initiating, managing, sponsoring and/or financing the review. |
Summary of 11 comments:
The need for explicit transparency was voiced by a number of respondents; tempered with the need for flexibility in responses to allow the inclusion of ‘None’ and details of support other than financial (e.g. "study leave", or acknowledgement that the research component of an academic contract might focus on review work).
| 13. CONFLICTS OF INTEREST: (Required field – 90% agreed) Any conditions that could lead to actual or perceived undue influence on judgements concerning the primary interest of the review. |
Summary of 21 comments:
A number of respondents felt that conflicts of interest were important, but more relevant in the final publication. It was felt that it might be difficult to gather conflicts of interest information at the time of registration in some instances.
uggestions included the need to clarify who and what conflicts of interest were required, for example by using a standardised format like the ICMJE form; and the possibility of flexibility to reflect the evolving nature of conflicts of interest. It was suggested that having this as a required field would be a barrier to registration for some reviews.
| 14. OTHER REGISTRATION DETAILS: (Optional field – 70% agreed) Anywhere else the systematic review is registered (such as with The Cochrane Collaboration, Campbell Collaboration, Joanna Briggs Institute). |
Summary of 15 comments:
The comments indicated that this information would be valuable, for reasons of transparency, identification of protocol details elsewhere, and avoidance of duplication of reviews and registrations. The ability to record that the review is not registered elsewhere is required.
here possible it was requested that inclusion of other organisation’s protocol registration information be automated, to minimise the workload for the person registering the review.
| 15. ORGANISATION REFERENCE NUMBER: (Optional field – 72% agreed) Unique identification assigned to the review by the funding, sponsoring or hosting organization, (e.g. a grant number or an accession number). |
Summary of 9 comments:
Some respondents indicated that the reason for asking for other reference numbers should be made clear; while others recognised the value for reasons of transparency, avoidance of duplication, and consistency.
| 16. LANGUAGE: (Optional field – 54% agreed) The language(s) in which the review will be written and will be made available. |
Summary of 14 comments:
Opinions varied from not seeing the need for this until the review was completed, to considering this as an essential item helping users to avoid waiting for a report that could only be read if translated.
A drop down list of languages, with English as the default was suggested.
The ability to select multiple languages and specify which parts of the review will be in different languages was suggested. For example, there might be an abstract or executive summary in English and full report in Spanish.
| 17. COUNTRY: (Optional field – 71% agreed) The country in which the review is being carried out. For multi-national collaborations include all the countries involved. |
Summary of 8 comments:
The comments mostly reflected a feeling that this information was not necessary (or, possibly, only necessary on completion of the review), and that it could be inferred anyway from the details of the named contact. However, its value for retrieval purposes was also noted.
| 18. KEY WORDS: (Optional field – 59% agreed) Words or phrases that best describe the review. Keywords help users find reviews in the Register, if they are as specific and precise as possible. They should avoid acronyms and abbreviations. Information specialists at the Centre for Reviews and Dissemination (CRD) will assign MeSH terms. When the review is published and indexed, NLM assigned terms will be added where these are available. |
Summary of 20 comments:
Inclusion of key words was thought to be useful to give enhanced searchability, though it may be a burden for registrants and accuracy of key word selection at the start of a review was thought to be questionable. It was suggested that as part of registration process key words could be assigned by the register administrators for checking by the registrant.
ome respondents indicated that expertly assigned MeSH headings and free text searching would be sufficient, negating the need for key words, unless assignment of MeSH built in a significant delay to the registration process. Furthermore, the use of MeSH would give consistency in those records for whom MeSH could be applied. Inconsistency of searching across the register may be the result of MeSH not being available for some review types.
| 19. ANY OTHER INFORMATION: (Optional field – 89% agreed) Any further information the review team consider relevant to the registration of the review. |
1.3 Minimum dataset: key information from the protocol
| 20. REVIEW QUESTION(S): (Required field – 97% agreed) The question(s) to be addressed by the review. |
Summary of 7 comments:
Comments highlighted the issue of the iterative nature of review question development and the need to accommodate this in the registration process and provide specific guidelines for what was required.
| 21. SEARCHES: (Required field – 81% agreed) Details of the sources to be searched, and any restrictions (e.g. language or publication period). The full search strategy is not required, but may be supplied as a link or attachment. |
Summary of 9 comments:
Comments included: not seeing the need for this in the register, but leaving it until the review is completed; limiting search information to month and year of anticipated final search; relying on the inclusion of the search strategy in a published protocol; caution about the maintenance of links long term; and concern that search details may change over time.
| 22. URL TO SEARCH STRATEGY: (Optional field – 75% agreed) A link to the search strategy or an example of a search strategy for a specific database if available (including key words for use in the searches). Alternatively, an electronic file could be supplied which will be linked to from the Register record. |
Summary of 10 comments:
The importance of the search strategy was emphasised in the comments, particularly for social sciences. A number of respondents indicated support for the inclusion of a link to the protocol or the search strategy in order to minimise effort and maximise usefulness. The alternative of providing contact details for someone who could provide (or had a commitment to provide) the search strategy was suggested.
t was also suggested that search strategies could be saved within the register as they were done, providing long term availability.
| 23. TYPES OF STUDY TO BE INCLUDED: (Required field – 87% agreed) Details of the study designs to be included in the review. If different study designs are needed to address different questions within the review, this should be made clear. Where qualitative evidence will be incorporated in or alongside a review this should be stated. If the authors have no restrictions on the types of study design eligible for inclusion, this should also be stated. |
Summary of 14 comments:
The need to be able to amend details in this field was stressed; for example inclusion of observational studies if no randomised trials were found.
Some thought this information would already have been provided within the review question or the search strategy; or could be incorporated into those fields to reduce number of questions.
It was suggested that the use of tick box options could assist with consistency of this information. Although, conversely, other respondents suggested this would be an impossibly heroic task. Opinions on the amount of detail required ranged from none to very specific study designs relevant to all disciplines.
| 24. CONDITION OR DOMAIN BEING STUDIED: (Required field – 93% agreed) A short description of the disease, condition or healthcare domain being studied. This could include health and wellbeing outcomes. |
Summary of 11 comments:
Some thought this information would already have been given in the review question and/or could be incorporated into Participants/Population field to reduce number of questions.
The need for specific advice on requirements for this field was highlighted. It was suggested that classification codes could be offered. Inclusion of why the review question is important in that condition was proposed.
| 25. PARTICIPANTS/ POPULATION: (Required field – 93% agreed) Summary criteria for the participants or populations being studied by the review. The preferred format includes lists of inclusion and exclusion criteria. |
Summary of 5 comments:
As with items 23 and 24, some suggested that this should be covered in the research question or search strategy.
The need to be able to edit this field was highlighted.
| 26. INTERVENTION(S), EXPOSURE(S): (Required field – 96% agreed) Details of the nature of the intervention(s) or the exposure(s) to be reviewed. Clear definitions, particularly for complex interventions, including which elements are necessary should be stated. If appropriate, an operational definition describing the content and delivery of the intervention should be given. |
Summary of 9 comments:
Some suggested this should be addressed in the research question or the search strategy.
Some respondents highlighted the need to consider the relevance of this field for reviews of diagnostic questions and prevalence or incidence studies, and the need for guidance for those circumstances. It was also thought there could be complications for reviews that include meta-synthesis, where it might be difficult to prescribe such detail in advance.
Tne proposal was to use a semi structured format with tick box selection of ‘area’ (e.g. a. Pharmacological; b. Surgical; c. Devices; d. Vaccines; e. Educational; etc...), followed by an open text field.
| 27. COMPARATOR(S)/ CONTROL: (Required field – 94% agreed) Where relevant, details of the alternative(s) against which the subject/topic of the review will be compared (e.g. another intervention or a non-exposed control group). |
Summary of 8 comments:
Some felt this should be addressed in the research question or the search strategy.
Some respondents highlighted the need to consider the relevance of this field for reviews of diagnostic questions, prevalence or incidence studies and micro-econometric studies, and the need for guidance for those circumstances.
One proposal was to use a semi structured format with tick box selection of ‘area’ (e.g. a. Pharmacological; b. Surgical; c. Devices; d. Vaccines; e. Educational; etc...), followed by an open text field.
Opinions ranged from only including key information to requiring justification for the choice.
28. CONTEXT: |
Summary of 25 comments:
Respondents who felt this field should be ‘Required’ cited the crucial nature of this information in the health policy and systems research area; for diagnostic and prognostic questions; in the social sciences field, where interventions for the same problem such as school truancy can take place in schools, courts, or community-based centers; and the relevance of setting to those working in different clinical areas such as primary versus secondary or tertiary referral centres. It was suggested that without context, understanding and generalisability would be limited.
Of those who commented in support of their opinion that this field should be ‘Optional’, they noted reasons such as minimising burden on registrants; the information being of interest but not essential; that there should be a re-assessment after time; and that for reviews where this information is critical and/or could lead to bias, it should be Required.
Some respondents felt they needed greater clarity about the meaning and operation of this field in order to make a judgement. Others felt the information should be addressed in the research question, the search strategy or the full protocol.
| 29. PRIMARY OUTCOME(S): (Required field – 93% agreed) The most important outcomes, including information on timing and effect measures, as appropriate. |
Summary of 12 comments:
Some felt this should be addressed in the research question, the search strategy or the protocol. Others suggested that as the distinction between primary and secondary outcomes in systematic reviews is controversial or overstated, and that different outcomes can be important to different users, the Primary and Secondary outcome fields be merged into one; possibly with a structured format (e.g. as per the Cochrane Summary of Findings table) .
| 30. SECONDARY OUTCOMES: (Required field – 76% agreed) Any additional outcomes that will be addressed, including information on timing and effect measures, as appropriate. |
Summary of 13 comments:
Some felt this should be addressed in the research question, the search strategy or the protocol.
As for Primary outcome field, other respondents suggested the Primary and Secondary outcome fields be merged into one; possibly with a structured format.
Other suggestions were: to limit the number of secondary outcomes to three; to allow additions as the review progresses; and to include in the searches section how they will be searched for.
It was recognised that secondary outcomes should be predefined but there was scepticism about how often this is done and a feeling that making it a ‘Required’ field could be a barrier to registration.
| 31. DATA EXTRACTION, (SELECTION AND CODING): (1st round: Essential – 51%; Desirable – 30%; Not needed – 19%) (2nd round: Optional field – 53%; Required field – 40%; Not needed – 6%; No opinion – 1.0%) The procedure, including the number of researchers involved, blinding and how discrepancies will be resolved. Information on whether authors of primary studies will be contacted to provide missing or additional data, and how these data will be collected. For reviews of individual participant data, this would include the data to be collected and how this will be done. |
Summary of 21 comments:
Comments from respondents who felt this field should be ‘Required’ highlighted that data extraction is a key element for methodological assessment, for example how issues of publication bias and errors are dealt with.
It was suggested that the field be split to mirror the requirements for search, i.e. into principles and practice. This would make it possible to make some information ‘Required’ and the other ‘Optional’; e.g. Number of researchers and blinding could be required; but the methods for dealing with discrepancies, for collecting further information from authors and for collecting data could be optional.
Respondents who opted for this to be an ‘Optional’ field commented that they did so in order to minimise burden on registrants, and in recognition that these issues may not have been decided at the point of registration.
It was suggested that for organisations with set procedures these could be recorded once for that organisation and reproduced for each of their reviews rather than entering the same data each time. Alternatively, standard wording could be used (e.g. as happens within the protocols for some Cochrane Review Groups). Another option suggested was to provide a checklist for registrants to use.
Some respondents felt this was not necessary or too much detail for a register; or that the information should already be available in the search field, or via a link to the protocol.
| 32. RISK OF BIAS (QUALITY) ASSESSMENT: (Required field – 74% agreed) Whether and how risk of bias will be assessed, how the quality of individual studies will be assessed, and whether and how this will influence the planned synthesis. |
Summary of 18 comments:
Some of those who commented felt this was too much information to ask for at registration, or should be provided via a link to the protocol.
It was suggested that ‘quality assessment‘, as used by The Cochrane Collaboration, was a better term than risk of bias. However some respondents pointed out potential problems for some types of meta-synthesis, where the overall quality of a study is less important than its potential contribution to the review; and for systematic reviews of some topics such as adverse effects and intended or hoped for effects.
Some respondents acknowledged that quality assessment is subjective, and that the use of a formal scale, particularly one with a numeric scoring system, is not recommended by some guidance. The possibility of making this an optional field was proposed, with the need to assess more carefully those reviews that do not record this information.
| 33. STRATEGY FOR DATA SYNTHESIS: (Required field – 71% agreed) The planned general approach to be used, for example whether the data to be used will be aggregate or at the level of individual participants, and whether a narrative (descriptive) or quantitative synthesis is planned. Where appropriate, details of the planned analytical approaches (e.g. fixed or random effects model; Bayesian or frequentist approach; one or two stage IPD approach; details of model selection; approach to categorising studies within a narrative synthesis) should be given. Whether and how statistical heterogeneity will be explored and how any observed heterogeneity will impact on or modify the planned approach to analysis should be stated. |
Summary of 17 comments:
Respondents suggested that this may be too vague to be useful in practice; was too much information to ask for at registration; should be limited to the basic approach to be taken (e.g. narrative synthesis, a structured qualitative synthesis (perhaps using a known method) or a quantitative synthesis, recognizing that the available data may dictate what is actually done in the end); or should be provided via a link to the protocol.
Some respondents acknowledged that this field may be difficult for some people to respond to. It was suggested that some key analysis options, i.e. those that if not declared prospectively could be directly linked to an increased risk of bias in the review, could be given as tick box options; and planned comparisons should be clearly stated.
The possibility of making this an optional field was proposed, with the need to assess more carefully those reviews that do not record this information.
| 34. ANALYSIS OF SUBGROUPS OR SUBSETS (Required field – 70% agreed) Any planned exploration of subgroups or subsets within the review. For example, whether a priori separate presentation, exploration or analyses are planned of different types of individuals (e.g. by age, sex, disease status, ethnicity, socioeconomic status, presence or absence or co-morbidities); different types of intervention (e.g. drug dose, presence or absence of particular components of intervention); different settings (e.g. country, acute or primary care sector, professional or family care); different types of study (e.g. randomised or non randomised). Details of approach to be taken, for example, whether subgroup analyses, meta-regression or modelling of covariates is planned. Where appropriate, information on how data will be categorised (e.g. age <40, 40-50, 50-60, >60 years). |
Summary of 15 comments:
Some respondents felt this was too much detail for a register; or that the information could be provided in the Condition or domain or Participants/population field, or via a link to the protocol.
Some suggested that the requirement should be limited to key options and not include details of the approach to be taken. The option of allowing organisations to have a set procedure recorded for all their records was proposed.
It was noted that for reviews of qualitative studies, the subgroups may emerge as a result of the review and cannot be known at registration.
One person felt there was a general ‘lack of exactly moderator/mediator analyses in reviews’ and it would be good to prompt people's thoughts on this issue.
| 35. DISSEMINATION PLANS: (Optional field – 79% agreed) Brief details of plans for communicating essential messages from the review to the appropriate audiences. |
Summary of 15 comments:
Respondents felt that while important, dissemination was not essential to the aims of the register and questioned the value of the information likely to be recorded (e.g. "We plan to publish.")
Conversely, more specific information about where and when a review would be published was proposed as a method of 'pushing' researchers to publish.
Given the evidence of publication bias at systematic review level, it was suggested that it would be important to know if reviews were not going to be made available in the public domain, but, for example, submitted as a government report.
| 36. DETAILS OF ANY EXISTING REVIEW OF THE SAME TOPIC BY THE SAME AUTHORS: (Optional field – 65% agreed) Details of earlier versions if an update of an existing systematic review is being registered (full bibliographic reference if possible). |
Summary of 15 comments:
Although likely to be helpful for those registering to think about, this was felt by some to be too much for the register.
Others felt it was important for transparency, to avoid duplication of effort, for keeping track of reviews splitting or merging and for consideration when planning an update. If a review was an update, this field was considered particularly important.
Rather than ‘details’ it was suggested that "summary" information be asked for.
While it was thought that the CRD databases should provide this information, others recognised that registrants should have searched for existing systematic reviews and guidelines and that it may be useful to have an additional field for 'any existing review of the same topic by other authors'.
3. Timing of registration
To facilitate assessment of the risk of bias within a completed or published review due to changes to the methods that were made while it was being conducted, an indication of the stage of the review at the time of submission will be required. In addition, the date of submission of the registration form will be recorded automatically and displayed.
This will be a ‘Required field’. Ideally, reviews should be registered before studies are screened against the eligibility criteria (an early point at which bias could be introduced) but we recognise that this might not always be possible and that reviewers are often aware of some of the potentially eligible studies and whether they are likely to include or exclude these, before they start the formal screening.
Summary of 22 comments:
There was concern that degree of completion for registration would vary, and if timing of registration was different to other registers, database users might get confused. It was suggested that early registration be made easy. Conversely, in order to ‘protect a topic’ reviewers may be tempted to register once they have completed screening and know the scope and the expected due date.
While supporting registration prior to formal screening of the literature, some respondents said that it is highly likely that reviewers are aware of some of the potentially eligible studies. There were concerns over deciding the options for ‘stage of review’; the reliability of the information supplied; and how information would be checked.
It was suggested that it would assist registrants if a tick box or drop down list of review stages could be provided with a free text box to include any qualification of the stage selected.
It was proposed that different levels of ‘requirement’ could be activated based on stage, because different items can be completed when a review has been finalized compared to when it is starting. For example, if the ‘timing’ field could be linked to the start/end date fields and screening has already started (which is the working definition of 'start date' for the register), then this field could be required.
There was concern that the Register could be used to ‘discipline’ reviewers and that the perception that this might happen was likely to increase the risk of bias in responses to this item, undermining the purpose of the Register and potentially its success.
4. Amendments to protocols, publications on completion, and updates to reviews
4.1 All submitted amendments to Register records will be recorded and dated to provide an audit trail.
Summary of 112 comments:
There were 83 short notes of agreement in the submitted comments.
This proposal was judged to be worthwhile for transparency in the process and as a tracking device for the review team. It was thought important to ensure that the existence of the audit trail be made clear but not intrusive in the current version and that flexibility was paramount.
It was suggested that the focus could be on substantial amendments e.g. any changes to the eligibility criteria or the research questions. This would make it less ‘messy’ and onerous for registrants and, therefore, more likely to be done.
Respondents supported the recognition that amending the protocol does not mean that you are biasing the review. There was a request for inclusion of the reason for an amendment.
As some updates are done by different authors, it was thought this may be a problem as the first registrant could be different (different ID and password) from the registrant of the update.
There was concern that publishing amendments could be taken by reviewers as a means of ‘policing’ reviews methods rather than achieving the aims of the Register. It was also thought the Register would require considerable support to achieve this.
4.2 The default view will show the most up-to-date version of the record, with changes incorporated. Users will be able to view the audit trail of changes made if they wish.
Summary of 101 comments:
There were 75 short messages of agreement in the submitted comments.
It was thought important to ensure that it was clear where changes had been made.
There was a request for the ability to include the reason for an amendment.
There was concern that publishing amendments could be taken by reviewers as a means of ‘policing’ reviews methods rather than achieving the aims of the Register.
4.3 Links to subsequent publications will be included in the Register record.
Summary of 107 comments:
There were 75 short messages of agreement in the submitted comments.
There was concern that these would be hard to maintain, but thought to be necessary if not essential to understanding the ‘bigger picture’. Conversely, it was felt that the Register should not become a new database of completed reviews.
There was concern that in practice this would not be possible as links and their respective websites change with time and can be very difficult to keep up with. Instead it was proposed that documenting the organizational location of the products would be more helpful.
The question of who would provide the links and how this would be achieved was felt to be a barrier to this working in practice.
4.4 In the absence of a publication, details of availability of the unpublished results, or reasons for the termination of a review, will be documented.
Summary of 104 comments:
There were 77 short messages of agreement in the submitted comments.
It was thought likely that if reviewers had not published the review, it was likely they would have lost interest and would not provide this information.
There was concern over whose responsibility this would be; that it would be too much for the reviewers and possibly too costly for the Register administration.
A few opinions differed in the extreme, with some fearing this information would deter enthusiastic participation and others wishing it to be a mandatory requirement.
4.5 An ‘Optional field’ will be available if reviewers wish to include their results in the Register record.
Summary of 98 comments:
There were 62 short comments of agreement in the submitted comments.
A structured format to include results was considered important; particularly to see the final results for the outcomes specified in the protocol.
Concern was expressed that publishing results on the register could jeopardise subsequent peer review publication. Other respondents thought it was a good idea, given the length of time publication can take, but that it may be seen as an alternative to publication and delay or prevent more formal publication by some review teams (e.g. where funding has run out).
The space available was queried, with the suggestion that a ‘summary’ word limit be given (e.g. as for an abstract). The option of linking to a site where the results are available was proposed.
Concern was expressed about the usefulness of this field as it will be impossible to check the validity of the data, and results could be presented out of context. It was also thought to be unnecessary and duplicative work.
A worry was expressed that users of the Register would read the record and not the actual review publication and that some of the fields could act as an appraisal of the review before it has even finished. The question was asked: although these fields are important in themselves, do they contribute to the desired result of reducing bias?
5. General comments
Summary of 33 general comments:
A number of general comments were made after reflection on the number of fields that had been ‘agreed’ to. While all the fields were thought worthy, respondents advised that the registration requirements be kept as brief and flexible as possible in order to encourage use and make the Register useful. The need to balance a rigorous and comprehensive approach against practical success was highlighted. The use of ‘optional’ rather than ‘required’ where ever possible was advocated. The use of tick lists was encouraged. It was suggested that if a link to a full protocol was available, registrants should not be required to complete many of the fields.
A few suggested a minimalist approach so the information provided helped users find reviews of interest and gave contact details for the team conducting the review, possibly with a link to protocols (one suggested this should be a ‘Required’ field, negating the need for the minimum dataset).
The question of who should register a systematic review, (e.g. the sponsor or the researcher), was asked.
There was concern that the register could be used to ‘topic-occupy’ or ‘title bag’ reviews as has been observed with Cochrane protocols. This would be a particular problem if the registered review was never completed.
Clarity over the permanency of records and the use of the term "ongoing" in the project title was requested, along with more detailed explanations of the information that should be entered in each field and a bibliography.
The plans for registration of rapid reviews were queried e.g. the short semi-systematic reviews that are performed rapidly to evaluate genomic applications.
One respondent expressed concern that if the Register was adopted widely it would contain essentially the same information as a Cochrane protocol (more in fact) and it would be difficult to get reviewers to enter the same information a second time in a Cochrane protocol. However, as Cochrane protocols counts as a 'publication' this may be incentive enough to make data entry in Cochrane the priority.
The need to include as much information as for a published protocol was questioned; particularly as this meant that reviewers would be working on a review for several months before they were at the stage required for registration.
One respondent expressed the opinion that although reviewers may be happy to register their reviews, many organizations and government agencies commissioning reviews wish them to be treated confidentially until they are ready to announce that the review is being done, and do not allow the protocols to be made publicly available. The respondent went on to say they also felt that registration would be unlikely to be sanctioned for the many systematic reviews commissioned by industry. They therefore proposed that the protocols and amendments be held in confidence, and only be posted once the review became public, if it did. This would then facilitate assessment of risk of bias as intended.
A respondent requested that therapeutic uncertainties be highlighted by authors and sent to the UK database of uncertainties of effects of treatments (UKDUETs).
A number of enthusiastic messages of congratulations and general support for the initiative were given.
Respondents who commented had enjoyed the opportunity to contribute and looked forward to detailed feedback about response rates, respondents and the denominators.
Acknowledgement
We would like to thank everyone who participated in this consultation, your time and thoughts are much appreciated. All the views, opinions and suggestions submitted will be carefully considered by CRD and the Register Advisory Group and used to inform the development and implementation of the Register.
If you have any further thoughts or comments on this summary report please contact us at
crd-register@york.ac.uk
Alison Booth and Lesley Stewart, Centre for Reviews and Dissemination, University of York, York, UK
Mike Clarke, UK Cochrane Centre, Oxford, UK
Davina Ghersi, International Clinical Trials Registry Platform, WHO, Geneva, Switzerland
David Moher, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
Mark Petticrew, Public and Environmental Health Research Unit, London School of Hygiene and Tropical Medicine, London, UK