REGISTER OF ONGOING SYSTEMATIC REVIEWS

International register of ongoing systematic reviews

Delphi exercise: Summary of first round responses

The aim of the Register

The aim is to provide a freely accessible web based register of systematic reviews in health and social care. Registration will be free of charge and take place at the protocol stage (or its equivalent). The Register will provide a record of systematic reviews underway and thereby help avoid unplanned or unnecessary duplication. By providing key methodological information early in the review process, it will facilitate an assessment of potential bias within a review after it has been completed and published.

Establishing a minimum dataset

The aim of registration is to provide enough information about the review to help people planning reviews in a similar area to decide whether or not to proceed and to capture those attributes of a review that are likely to be important in the avoidance of bias. Therefore, this Delphi process is being undertaken to establish the minimum data set that will allow this. It does not seek to capture the attributes of the wider information that should be included in a protocol for a systematic review or to determine all the variables that people might wish to record in registers of systematic reviews that would be used for other purposes.

In the first round of the Delphi process, opinions were sought on some practical aspects of implementing the Register and whether individual constituents of a review protocol were considered ‘Essential’, ‘Desirable’ or ‘Not needed’ for the purposes of the Register. A summary of those opinions is given here. They have been used to inform the second round of the Delphi, and will be considered in the development and implementation of the Register.

1. Scope for inclusion
2. Minimum dataset
3. Allocation of unique identification numbers
4. Timing of registration
5. Amendments to protocols, publications on completion, and updates to reviews
6. Other systematic review protocol registries

1. Scope for inclusion

Responses from Round one:
The majority of respondents supported the principle of a wide scope for the Register. There were a few requests for Reviews of Reviews and Reviews of Methods to be included, and a small number of people also asked for Scoping Reviews to be included.

Some respondents sought clarification on whether a variety of review topics and types were to be eligible.

Scope
The long term aim for the Register is broad inclusion of systematic reviews in health and social care, where any review that is systematic and has a health related outcome would be eligible.

However, to achieve this aim, a stepped approach will be taken. The Register will begin with a focus on ongoing systematic reviews of the effects of health and social care interventions, and inclusion of, and support for, other reviews will be phased in over time.

2. Minimum dataset

Responses from Round one:
Several respondents expressed concerns that if the Register asks for too many details this will become a barrier to registration. It was suggested that the preferred minimum data set should specifically address issues relevant to the primary purpose of the register to reduce inappropriate duplication of effort (hence the need for information about the eligibility criteria) and selective outcome reporting (hence the need to define key outcomes).

Minimum dataset
The fields are designed to be flexible enough to facilitate the inclusion of the many different types of systematic review that would fit within the broad scope planned.

In almost all cases, we used the majority vote for ‘Essential’ or ‘Desirable’ in the first round to categorise fields as ‘Required’ or ‘Optional’, respectively, in the second round. None of the fields in the first round had a majority in favour of ‘Not needed’. Taking into account the feedback on the need to have as few barriers to registration as possible, we downgraded two fields in making our suggestions (see below) in the second round of the Delphi.

Fields that are ‘Required’ in the final dataset will have to be completed before a record can be submitted for possible inclusion in the Register. These would constitute the ‘Minimum data set’ for a register of systematic reviews.

2.1 Minimum dataset: identification

Review title (Required field)
Named contact (Required field)
Organisational affiliation of the review (Required field)
Named contact email (Required field)
Named contact address (Optional field)
Named contact phone number (Optional field)
Review team (Optional field)
Review team members' organisational affiliations (Optional field)
Collaborators (Optional field)

2.2 Minimum dataset: general information

Anticipated or actual start date (Required field)
Anticipated completion date (Required field)
Funding sources/sponsors (Required field)
Conflicts of interest (Required field)
Other registration details (Optional field)
Organisation reference number (Optional field)
Language (Optional field)
Country (Optional field)
Key words (Optional field)
Any other information (Optional field)

2.3 Minimum dataset: key information from the protocol

Review question(s) (Required field)
Searches (Required field)
URL to search strategy (Optional field)
Types of study to be included (Required field)
Condition or domain being studied (Required field)
Participants/ population (Required field)
Intervention(s), exposure(s) (Required field)
Comparator(s)/ control (Required field)
Context (Optional field)*
Primary outcome(s) (Required field)
Secondary outcomes (Required field)
Data extraction, (selection and coding) (Optional field) *
Risk of bias (quality) assessment (Required field)
Strategy for data synthesis (Required field)
Analysis of subgroups or subsets (Required field)
Dissemination plans (Optional field)
Details of any existing review of the same topic by the same authors (Optional field)
 
* The majority of respondents in the first round selected these fields as ‘Essential’

2.4 Minimum dataset: availability of the full protocol

There was strong agreement in the first round with the following statements:

In relation to the availability of published protocols for registered reviews, concern was expressed that reviews might be delayed while the authors wait for acceptance of a protocol for peer review publication and that the increasingly short turn round times for many commissioned reviews means that they do not have a published protocol. Respondents suggested that the important issue is to have sufficient information to minimise the likelihood of systematic reviews that are prone to bias and that this should be collected in ways that authors would not view as barriers.

It was suggested that ‘publication’ of protocols should be considered in a wider sense than just peer reviewed journals.

The notion that publication of full protocols in scientific journals would become less relevant with a repository for protocols was expressed. However, although the planned Register will contain key information about a review to assist with the assessment of the risk of bias in its final report, users of the Register should not expect the Register record to include all of the details that would be expected in a protocol. They would be directed to the published protocol or the contact person for the Register record for further information.

The current lack of opportunities to publish the protocol for a systematic review was identified as a problem; along with funding to pay the author fee for publication in an open access journal. However, it was also suggested that by offering to host unpublished protocols, CRD would discourage people from seeking publication elsewhere.

A number of respondents made the point that the challenges and opportunities for publishing protocols vary across different areas of health and social care and in different countries.

Concern was expressed about making the plans for a review available in a published protocol too soon before the review itself would be ready, balancing the desire for transparency with the risk of being ‘scooped’ once the methods were made public.

The option of entering a minimal amount of information in the Register and adding a link to a full protocol was seen as less burdensome for researchers, while still being informative for users of the Register.

3. Allocation of unique identification numbers

The principle of a unique identification number was almost unanimously supported.

In response to requests to allow appropriate cross-linkage and avoid duplication where possible, an additional field has been proposed in the minimum dataset for the recording of other registration details.

Suggestions that updates to previously registered reviews should retain the same unique identifying number and be given a suffix (such as the year) might be an alternative to the creation of a new number and links between the records for updates of reviews. As one respondent pointed out, having multiple records for the same review might be confusing. What constitutes an update will have to be defined.

The question of what to do with the record for a review which is discontinued was raised. The intention is to note this fact within the record.

The potential problem of duplicate registration of the same review was raised, with the suggestion that, in keeping with experienced gained in the registration of trials, an early assessment be made to see how big an issue this might be for reviews. It is hoped that this will be a minimal problem, if the lead reviewer takes responsibility for registration.

4. Timing of registration

There were a few concerns expressed in the first round about being ‘too strict’ in setting a specified time in its life cycle at which a review should be registered. These concerns ranged from practical issues while the Register is being established (such as the inclusion of reviews that have already started); to the iterative nature of systematic reviews, making it difficult to define a ‘start’ time and introducing the possibility that registering a review too soon might lead to multiple amendments to the Register record as the protocol and the plans for the review are finalised. However, there was strong support for the principle of the timing proposed and the following revised wording aims to clarify the intention:

In order to facilitate assessment of the risk of bias within a completed or published review due to changes to the methods that were made while it was being conducted, it will be necessary to give an indication of the stage at which the details about the plans for the review were recorded in the Register. Therefore, in addition to the date on which a registration was submitted being available in the record, an indication of the stage of the review at the time of submission will be required.

This will be a ‘Required field’. Ideally, reviews should be registered before studies are screened against the eligibility criteria (an early point at which bias could be introduced). However, we recognise that this might not always be possible and that reviewers are often aware of some of the potentially eligible studies and whether they are likely to include or exclude these, before they start the formal screening.

We hope that the length of time that it will take to complete the Register record for a review will not prove to be a barrier to registration, especially if the necessary information can be cut and paste from the written plan for the review. The average amount of time it will take is unknown until the minimum dataset is agreed and the registration interface is available. However, some pilot registrations will be undertaken to estimate this. Furthermore, the Centre for Reviews and Dissemination (CRD) will work with organisations that already maintain repositories of protocols (such as The Cochrane Collaboration) to explore ways to facilitate the inclusion of information from these in entries for the Register. There will not be a quality assessment and feedback step for applications, which means that reviewers would not need to stop working on the review while awaiting registration.

5. Amendments to protocols, publications on completion, and updates to reviews

Respondents felt that the explanation of the default view for records in the Register was unclear, and most of those who commented on this suggested that this view should be the most current version of the record, with the ability to look at any preceding amendments if desired. The explanation has been amended to clarify this.

It was suggested that the reasons for any amendments to information in the Register record should be given in sufficient detail so as to be able to judge potential for bias.

It was suggested that consideration be given to providing the option of recording changes, such as post hoc statistical analyses after the review had started.

It was proposed that regular updates on progress with the review would be useful to users of the Register, up to and including the addition of a link to its publication. However it was also suggested that the burden this would place on the Register and on reviewers was likely to lead to limited compliance.

Consideration of other approaches to tracking publication which do not rely on input by the authors was suggested.

A number of respondents were in favour of the inclusion of a summary of the results in the Register record after completion and publication of the review. An optional field to facilitate this might be particularly useful for reviewers who do not publish their findings elsewhere.

It was felt essential that clear definitions and guidance be provided for issues such as ‘lumping’ together existing reviews when these are being updated, updating part of a review; defining the point at which changes to the methods for a review when it is being updated constitute a new review; the identification of abandoned reviews, etc.

6. Other systematic review protocol registries

We were previously aware that the Cochrane and Campbell Collaborations and the Joanna Briggs Institute maintain public records of protocols produced within their organisations and of the All Wales Register of Systematic Reviews. Respondents were asked to provide details of any other existing or planned registers for ongoing systematic reviews. Plans to establish registries of systematic reviews in Italy and China were reported.

Work in the US on a repository of study data (extracted during reviews) was also highlighted.

Some people mentioned HuGE Net, which used to have system of recording intended or planned reviews, but found that this led to m any topics being "reserved" with the registered reviews not being completed. As noted above, the recommendation that the Register should include a way to indicate if a review is abandoned or not completed, along with the date and the reason for this, will be considered in the development of the Register.

We were also informed of the following protocol collections: the AHRQ Effective Health Care program posts shortened versions of protocols of their reviews on their website; the EPPI-Centre; Guidelines International Network; the German Institute for Quality and Efficiency in Health Care (which posts German language protocols of all its reviews on the institute's website); and the Canadian Agency for Drugs & Technologies in Health (CADTH) is Secretariat to "The Exchange" - a network of Canadian HTA producers. CADTH maintains a database of planned and ongoing HTA-related work (in Canada).

CRD’s HTA database includes information about health technology assessments in progress, including systematic reviews. These bibliographic records will continue to be held separately from the Register; but the intention is to include a link to the registration record in the relevant HTA database record.

If you have any comments please email crd-register@york.ac.uk