Evidence-based practice is the cornerstone of healthcare decision-making today. Providers are consistently seeking evidence to support current practices or drive practice changes to improve care and achieve better patient outcomes. Barriers to evidence-based practice, such as lack of time or access, still exist. For this reason, providers often actively seek systematic reviews or clinical practice guidelines as a source of synthesized evidence to guide practice. Where these do not currently exist, many healthcare professionals conduct their own systematic reviews to answer the clinical questions that affect their current practices. The number of published systematic reviews is constantly rising. A search of PubMed through 30 March 2016 showed 51,763 citations with "systematic review" in the title.
What happens when the questions we seek answers for in the current evidence base come up empty? So-called empty reviews are the results of a comprehensive search in which no studies are located that meet the a priori inclusion criteria for a systematic review.1-4 These empty reviews are more likely to be subject to publication bias and not published in scholarly journals. However, organizations such as the Joanna Briggs Institute and Cochrane understand the benefits of publishing empty reviews. A search of the Cochrane Library though 2010 revealed that 8.7% of the 4320 reviews published at that time were empty reviews.5
An empty review may arise when review questions are asked about an area of practice with a limited research base.1 Empty reviews may also be the result of review questions that are highly specific in terms of the population, intervention or outcome criteria compared with questions asked in primary research studies.2-4 In addition, the use of highly specific methodological inclusion criteria, as can be seen when systematic reviews focus solely on randomized controlled trials as the highest level of evidence but the review question being asked cannot easily be answered by a randomized design, may produce no eligible studies for inclusion.5 Some of these issues may be foreseen at the time of protocol development when reviewers conduct initial pilot searches to identify existing literature to support and inform the inclusion criteria and establish the need for conducting the review. If the area of research is too new to provide adequate evidence for synthesis, the need for conducting a systematic review should be reconsidered.
Although they are unable to provide recommendations for practice related to the review question, empty reviews still have important implications; however, there are inconsistencies in their reporting. Lang et al.6 first introduced the issue of inconsistent reporting of the results in empty reviews in 2007. The review report needs to be clear to the consumer that there are no eligible studies that meet the inclusion criteria detailed in the a priori systematic review protocol; therefore, no conclusions related to the review question can be drawn. Lang et al.6 identified numerous phrases used in published reviews to indicate an empty review and recommended standardizing the reporting of empty reviews by simply stating "no eligible studies found" in the results section. Despite this recommendation, a search of the JBI Database of Systematic Reviews and Implementation Reports revealed no reviews using this exact terminology; a search of the Cochrane Database of Systematic Reviews identified 160 reviews that used this terminology when one or more of the review questions came up empty.
Another issue in the reporting of empty reviews occurs in the statement of conclusions and recommendations for practice when there are no eligible studies found that meet the inclusion criteria. Review authors may be tempted to state conclusions based on excluded or non-eligible studies, based on other evidence or not based on any evidence.3,4 The conclusions drawn from a systematic review need to be based on evidence that answers the review question and that has been assessed for methodological quality through a rigorous review process. The conclusions reported in some empty reviews raise a concern for bias as conclusions that are unsupported by evidence to answer the review question may misguide healthcare professionals in their decision-making.
If your review does come up empty, there is still important information to report from your efforts. Empty reviews highlight major gaps in the current evidence base. Review authors are often experts in the field who have first hand knowledge of research that needs to be conducted to guide practice. An empty review should provide a direction for research to fill this knowledge gap, with recommendations to guide the types of research designs needed. Recommendations should also be provided regarding eligibility criteria for sample selection as well as specifics regarding the ideal intervention, comparator and outcome measures for future research. These same implications could also apply to systematic reviews with included studies in which there is insufficient evidence from which conclusions can be drawn to guide practice or in cases in which the evidence is conflicting.
The implications of empty reviews continue to be discussed but guidelines for reporting on empty reviews have yet to be developed. These conversations will continue among review groups. Review authors must remember to present conclusions based on the available evidence so as not to mislead healthcare professionals in their decision-making.
References