The conduct of randomized, controlled trials of nonpharmacologic treatments presents specific challenges that are not adequately addressed in trial reports.
To develop an extension of the CONSORT (Consolidated Standards of Reporting Trials) Statement for trials of nonpharmacologic treatments.
A consensus meeting was organized to develop an extension of the CONSORT Statement that addresses randomized trials of nonpharmacologic treatments. To prepare for the meeting, a survey was conducted to identify the specific issues for discussion.
Consensus meeting in Paris, France.
A total of 33 experts attended the meeting. The experts were methodologists (n = 17); surgeons (n = 6); editors (n = 5); and clinicians involved in rehabilitation (n = 1), psychotherapy (n = 2), education (n = 1), and implantable devices (n = 1).
Experts indicated which of the 22 items on the CONSORT checklist should be modified or which additional items should be added specifically for nonpharmacologic treatments. During a 3-day consensus meeting, all items were discussed and additional methodological issues related to nonpharmacologic research were identified.
The consensus was that 11 items on the CONSORT checklist needed some modifications for nonpharmacologic trials: item 1 (title and abstract), item 3 (participants), item 4 (interventions), item 7 (sample size), item 8 (randomization), item 11 (blinding), item 12 (statistical methods), item 13 (participant flow), item 15 (baseline data), item 20 (discussion: interpretation), and item 21 (generalizability). In addition, the meeting participants added 1 item related to implementation of the intervention.
Evidence was not always available to support the inclusion of each checklist item.
The methods and processes used to develop this extension could be used for other reporting guidelines. The use of this extension to the CONSORT Statement should improve the quality of reporting randomized, controlled trials assessing nonpharmacologic treatments.
Randomized, controlled trials (RCTs) are considered the gold standard for evaluation of drugs, devices, and procedures. To help improve the quality of reporting of these trials, the CONSORT (Consolidated Standards of Reporting Trials) Statement, a 22-item checklist and flow diagram, was developed. Use of this evidence-based guideline is associated with improved quality of reporting of RCTs (1–2). The original CONSORT Statement proposed guidelines for reporting 2-group parallel RCTs (3–4). The CONSORT Statement has subsequently been extended to cover specific variants of this design, such as cluster randomized trials (5) and noninferiority and equivalence trials (6); certain interventions, such as herbal therapies (7); and data, such as reporting of harms (8).
Nonpharmacologic treatments cover a wide range of interventions, including surgery, technical procedures (for example, angioplasty), implanted devices (for example, pacemakers), nonimplantable devices, rehabilitation, physiotherapy, behavioral therapy, psychotherapy, and complementary and alternative medicine. Although the CONSORT Statement can be applied to reports of these trials, certain issues, such as the complexity of the intervention, expertise of the care provider, and difficulties with blinding (9), present specific challenges that the revised CONSORT Statement and the accompanying explanation and elaboration document do not address in depth (3–4, 9–13).
A steering committee was ultimately responsible for the development of this reporting guide. They secured funding, reviewed the literature, identified participants to invite to a consensus meeting, conducted a survey to identify specific issues related to the assessment of nonpharmacologic treatments, and organized the CONSORT consensus meeting (agenda, identification of the topics to be discussed, organization of presentations, and methods of consensus).
Funding is essential to enable the development of any reporting guideline. For this guidance, we obtained funding to develop a Web-based survey and have the consensus meeting. The funding was used solely to cover the expenses of participants coming to the meeting. Members of the steering committee were funded by their respective home institutions.
Funding of this extension was obtained primarily from nonindustry sources: the Department of Clinical Research of the Assistance Publique des Hôpitaux de Paris (Paris, France); the Department of Epidemiology, Biostatistics, and Clinical Research of the Bichat Hospital (Paris, France); INSERM (Institut National de la Santé et de la Recherche Médicale) (Paris, France); and the Eli Lilly Institute (Suresnes, France), which is a French association providing unrestricted research and education grants.
Identification of Participants
Our aim was to bring together clinical epidemiologists and statisticians who published methodological papers on issues involved in the assessment of nonpharmacologic treatments and clinicians in various fields who participated in the design, conduct, and analyses of RCTs of nonpharmacologic treatments. In addition, we invited editors of journals that publish trials of nonpharmacologic treatment and other editors with expertise and experience in developing reporting guidelines.
On the basis of previous experiences in developing reporting guidelines, we limited the number of participants to 40. This was done because funding was limited and to allow maximum interaction during the meeting. When an originally nominated invitee suggested additional invitees, we invited them if their domain of expertise was not already sufficiently represented and the total number of participants was still fewer than 40.
Thirty-seven experts were invited to participate in the consensus meeting. The 33 attendees comprised methodologists (n = 17); surgeons (n = 6); medical journal editors (n = 5); and clinicians involved in rehabilitation (n = 1), psychotherapy (n = 2), education (n = 1), and implantable devices (n = 1). These are nominal frequencies for each category, as several participants had more than 1 domain of expertise.
Identification of Specific Issues
Before the meeting, the steering group surveyed meeting invitees by using a Web-based questionnaire to identify specific issues that should be discussed during the meeting. Invitees were asked to suggest which of the 22 items on the CONSORT Statement might need to be modified for the proposed nonpharmacologic treatments extension. The Web-based interface is available at http://www.nonpharmacological.com/ConsortExtension/consort05.php.
On the basis of the results of a systematic review of the topic (9), respondents were asked whether they believed we should add the following 7 items.
Methods section: 1) eligibility criteria for care providers (for example, surgeons, physiotherapists, or psychologists) included in the trial and 2) the centers' volume for the procedure or similar procedures (as a proxy for experience)
Results section: 1) the number of care providers performing the treatment in each group, 2) the number of participants treated by each care provider, 3) participants' expectancies or preference for the treatments at baseline, 4) baseline data on care providers, and 5) care providers' compliance with the planned procedure.
Invitees were also asked to nominate additional items they felt were particularly important when evaluating nonpharmacologic treatments.
When more than one third of the respondents rated an item as needing modification or said that an additional item should be added, that item was selected by the steering committee as a high priority for discussion during the meeting. Items rated by fewer than one third of the respondents were given a lower priority.
A 3-day consensus meeting was held in February 2006 in Paris, France. The meeting began with several presentations on specific topics related to the reporting of nonpharmacologic treatments (complexity of the intervention, influence of centers and care providers, clustering effect, blinding, assessment of harms, external validity). These presentations were selected by the steering committee to facilitate additional discussion during the meeting.
Participants then introduced and discussed each item proposed for modification or as an addition to the checklist until they reached a consensus. At least 1 member of the steering group moderated these discussions. The participants then considered the remaining CONSORT checklist items, which were given lower priorities, to see whether additional modifications were needed. All of these discussions were recorded, and minutes of the meeting were prepared.
The meeting concluded with a discussion of optimal dissemination and publication strategies in light of other CONSORT extensions.
After the meeting, the steering committee circulated a draft of the CONSORT extensions for nonpharmacologic treatments to all meeting participants for feedback. The steering committee collated the participants' comments and suggested revisions and developed a paper describing the new extension to the CONSORT Statement. The participants subsequently revised the document to ensure that it accurately represented the decisions made during the meeting.
Twenty-six of the 30 meeting invitees (apart from the 3 members of the steering committee) completed the survey. Eight CONSORT checklist items were selected for further discussion during the meeting (Table 1). These items dealt mainly with reporting the methods of a nonpharmacologic treatment trial. No checklist item in the results section was selected for modification by more than one third of the participants. In contrast, there was substantial concurrence (77% to 100%) among the survey respondents regarding the 7 specific issues suggested for further discussion, except for “details on patients' expectation or preference for the treatments at baseline” (Table 1).
Table 1. Results of the Survey on Revisions and Additions to the CONSORT Checklist
Respondents provided several comments to explain their choice. These comments were grouped by the steering committee into 5 sections: 1) the level of complexity of nonpharmacologic treatments that implies the need to report all the components of the intervention, co-interventions, method of standardizing the treatment, and compliance of care providers with the planned protocol; 2) the influence of care provider and center volume on estimates of treatment effect, with the difficulties of having an adequate proxy to describe care providers—the possible proxy could address care providers' experience, qualification, years of practice, specific training, skill, and learning curve (that is, the relationship between the experience of a surgeon and 1 or more performance indicators); 3) the statistical analyses, with the need to take into account the clustering of outcomes of patients treated by the same care provider or center in the sample size calculation and the statistical analyses; 4) the difficulties of blinding; and 5) the generalizability with regard to the comparator and care providers and centers.
The meeting participants recommended modifications to 11 checklist items (Table 2). In addition, the group agreed to add 1 new checklist item, related to implementation of the intervention. Some of these recommendations begin with “when applicable” to indicate that not all the information authors are encouraged to report is applicable for every nonpharmacologic treatment trial. For example, the reporting of centers' volume is probably more appropriate for surgical and technical procedures than for rehabilitation. Item 11 (blinding) was modified to reflect the change in wording of this item in the proposed revision to the 2001 CONSORT Statement, as discussed during the last CONSORT meeting in Montebello, Québec, Canada, in January 2007 (Moher D. Personal communication.).
Table 2. Checklist of Items for Reporting Trials of Nonpharmacologic Treatments
Meeting participants modified the CONSORT flow diagram to include data on the number of care providers and centers in each group, as well as the number of patients treated by each care provider (Figure)
Modified CONSORT flow diagram for individual randomized, controlled trials of nonpharmacologic treatment.
An extra box for each intervention group relating to care providers has been added. For cluster randomized, controlled trials, authors should refer to the appropriate extension. IQR = interquartile range; max = maximum; min = minimum.
The checklist and flow diagram are available on the CONSORT Web site (http://www.consort-statement.org). They can be freely downloaded and copied for noncommercial purposes without any modifications, as long as a full citation is provided. For full details on conditions of use, consult the CONSORT Web site.
During the meeting, participants encouraged the development of an explanation and elaboration document similar to those developed for other reporting guidelines (4, 20, 22). We felt that such an effort would facilitate the understanding and dissemination of the nonpharmacologic treatment checklist and flow diagram. We developed the document by using a standard template (19). Each item is introduced, after which the rationale and evidence for the need to report the item (when available) are presented, and an example of reporting in tabular format is provided. To clarify the extra information sought for in nonpharmacologic trials, this CONSORT extension presents the additional text separately from the text of the original CONSORT item rather than as merged text, as has been done previously.
The meeting participants also proposed developing specific explanation and elaboration documents for different categories of nonpharmacologic treatments, such as surgery, rehabilitation, and behavioral interventions. We will develop these documents by using an approach similar to the one described here.
Finally, the meeting participants discussed in detail the issue of how best to use the CONSORT Statement and its extensions to report trials that cut across specific reporting guidelines (for example, trials on the use of both pharmaceutical and surgical interventions, or trials using a cluster or equivalence design). The CONSORT Group is currently developing a Web-based interface to allow authors to select combinations of specific checklists and flow diagrams to help them report their trials. That interface will be available from the CONSORT Web site (http://www.consort-statement.org).
Although journal adoption of the CONSORT Statement is associated with improved quality of reporting (1), the reporting of trials of nonpharmacologic treatments remains suboptimal, especially with regard to such issues as the description of the interventions and the volume of care providers and centers (14, 14, 23–27). To help improve this situation, the CONSORT Group has developed this extension of the CONSORT Statement.
This article describes the current methods used by the CONSORT Group to develop their reporting guidelines. Development of evidence-based reporting guidelines is a new and evolving area. Some initial ideas have been proposed to enhance the development process, and therefore the validity of the guidelines it produces (19).
First, development should involve a broad range of participants representing different perspectives, areas of expertise, and experiences. The participants in this meeting included clinical trialists in various fields, methodologists, statisticians, and journal editors. Second, substantive preparations and data gathering should precede the meeting. For this extension, we completed an extensive review of the literature on bias when assessing nonpharmacologic treatments (9). We also completed a survey of the surgical literature by assessing the quality of reports of 158 surgical intervention trials and developed a quality tool (14, 28). All of these steps helped to ensure that the foundation for developing this extension was as evidence-based as possible.
This reporting guideline builds on the 2001 version of the 22-item CONSORT checklist and flow diagram. A Web-based preliminary survey was very useful in identifying which items needed further discussion and which new items should be included. Finally, the meeting organized to develop this extension took the form of very structured discussions led by different chairs.
The publication of this extension of the CONSORT Statement is an essential step to further its dissemination. To provide a more thorough understanding of the checklist, we developed an explanation and elaboration document (29) to be used in conjunction with this extension. That explanation and elaboration document should be considered in conjunction with the 2001 CONSORT explanation and elaboration paper (4).
In conclusion, this article provides a detailed description of the methods and processes used to develop the CONSORT extension for nonpharmacologic treatments. Describing these methods allows readers to judge the validity of the process and the subsequent reporting guideline.
We believe that the use of this extension will improve the reporting of trials on nonpharmacologic treatment. As with all CONSORT guidelines, these reporting recommendations are evolving and require periodic reevaluation. The CONSORT Group will continue to survey the literature, and we invite authors of any interesting articles to notify us about them.
Many journals have endorsed the CONSORT Statement, modified their instructions to authors, and asked authors to adhere to the checklist and flow diagram when submitting a report of their RCT. We hope that these journals will also endorse this CONSORT extension for reporting nonpharmacologic treatments and ask authors to adhere to it. In addition, we invite journals that have not yet endorsed the CONSORT Statement and its extensions, including CONSORT for nonpharmacologic trials, to do so by modifying their instructions to authors accordingly. They can alert the CONSORT Group of such actions through the CONSORT Web site (http://www.consort-statement.org).