– Written by Mike Clarke, United Kingdom

Over the past few decades, systematic reviews have become widely accepted as one of the key elements in evidence-based health and social care¹. A growing evidence base has also appeared to help people conducting and using reviews to identify good and bad methods for doing them. Systematic reviews seek to bring together the relevant evidence to answer a specific question, assess the eligibility of and appraise the quality of the identified studies and compare, contrast and, if appropriate, com-bine their findings to provide a summary of the evidence base. This can then be used by patients, practitioners, policy makers and the public to make well-informed decisions and choices. In regard to questions relating to the effects of interventions, actions and strategies, systematic reviews provide the means to identify which are beneficial (and for whom and by how much), which interventions are ineffective or harmful and which remain unproven. They are seen as a key source of knowledge for clinical medicine in high-income countries and are increasingly promoted as such in more challenging areas, such as the humanitarian sector and low-resource settings^2-4. This article discusses the general principles of systematic reviews and how these can be applied, which is increasingly important in sports medicine, where a growing number of systematic reviews are appearing^5-7.

Without systematic reviews, decision-makers are faced with an often overwhelming number of individual studies and, in recent years, similar concerns have even been raised in relation to the increasing number of systematic reviews appearing in the literature. Recent estimates suggest that health and social care reviews are being published at the rate of at least 8000 per year⁸ and more than 10,000 ongoing reviews are included in the prospective register, PROSPERO (www.crd.york.ac.uk/PROSPERO).

Systematic reviews can be used to tackle any topic that can be subject to research in a single study, since their ultimate aim is to bring together all studies of the same topic. For instance, among many others, there are systematic reviews of the effects of prevention⁹, treatment¹⁰ and rehabilitation¹¹ of sports injuries which rely on randomised trials for comparing different interventions; reviews of test accuracy research to identify the value of tuning forks to diagnose fractures¹²; and reviews of observational research to examine factors predictive of progression to surgery after non-operative management of anterior cruciate ligament ruptures¹³ or the prevalence and risk factors for lower limb tendon pathology and tendinopathy in children and adolescents¹⁴.

BACKGROUND

Individual studies are often too small to provide reliable answers to the uncertainties faced by decision-makers and people making choices about their own or someone else’s care. They might also be subject to selective reporting, which can restrict the availability of any information about the study through publication bias (which suppresses all the findings of the study)¹⁵ or outcome reporting bias (which restricts the availability of the findings to a sub-set determined by those findings)¹⁶. Regardless of how large a single study is, chance may lead to an overestimate or underestimate of the true effect but, in small studies, this is more likely to produce effects that are large enough to be mistakenly deemed to be clinically meaningful¹⁷ and bias from selective reporting provides decision-makers with a distorted view of the evidence base¹⁶. Even worse, a combination of chance and bias can lead to findings that appear convincing but are fundamentally false^18,19.

Systematic reviews are not immune to the effects of chance and bias but they can reduce both in comparison to individual studies. The averaging of the results of the studies in a review, using meta-analyses, will minimise chance. The use of comprehensive searching, clear application of eligibility criteria, consistency in how studies are appraised and analysed and a neutral interpretation of the subsequent data all help to minimise bias.

Systematic reviews are needed before new research is done, to provide the scientific, ethical and environmental justification for that research²⁰; and after the study has finished, to place its findings in proper context²¹. As noted above, systematic reviews are increasingly common, which is in some part due to the work of the Cochrane Collaboration over the last few decades. The Collaboration was established in 1993 as a global effort to prepare, maintain and promote the accessibility of systematic reviews of the effects of interventions^22,23 and the full text of more than 6000 full Cochrane people and protocols for a further 2000 that are at earlier stages of preparation are now available online, in the Cochrane Library (www.cochranelibrary.com). For much of the past decade, approximately 400 to 500 Cochrane Reviews have appeared in full for the first time each year and a similar number of existing Cochrane Reviews have been updated annually. However, the number of non-Cochrane systematic reviews now far outstrips this: more than 7000 non-Cochrane reviews were published in 2014, a substantial increase over the past decade⁸.

The following sections outline key steps in the systematic review process to help with both their conduct and use.

QUESTION FORMULATION AND ELIGIBILITY CRITERIA

Systematic reviews should begin with the formulation of a clear question that can be used to underpin the eligibility criteria and the searching for and selection of eligible studies. The question should also reveal the aim of the researchers doing the review. For example, they might be seeking to:

• Derive an estimate of the relative effects of two treatments,
• Assemble or catalogue all the relevant research on a particular topic in order to describe what has already been done,
• Learn from past studies when designing a new one or
• Identify the key gaps or uncertainties in the research base that need to be filled by new studies²⁴.

The question might adopt a structure such as PICO (or PECO): Participants, Intervention (Exposure), Comparator and Outcomes but, in general, it should provide the information to identify the population to be studied, what happened to them and what their outcomes were. It should be concise, with extra detail provided in the eligibility criteria as necessary.

The eligibility criteria for a systematic review set out the rules for the research that will be included, if such studies have been done and can be identified. They usually describe:

• The types of study design that would be eligible.
• The patients (or other participants) who were studied.
• The interventions, exposures, actions or strategies that they were subject to.
• The outcomes that were measured for them.

STUDY IDENTIFICATION

When the question has been established and the eligibility criteria are decided, a systematic process for searching for relevant studies begins. If all studies are identified and included, irrespective of their results, this would eliminate any biases due to selective reporting. Finding and using the results of all relevant studies would also minimise chance effects by maximising the amount of analysed data and increasing the precision of any quantitative results in the review. However, reviewers might struggle to achieve this ultimate aim of including all relevant studies and, instead, they need to get as close as they can to this by using comprehensive search strategies, avoiding language restrictions²⁵ and trying to avoid gathering a sample of studies that is inherently biased because of selective reporting¹⁶. Ideally, their searching might include:

• Several bibliographic databases such as PubMed, EMBASE and CINAHL.
• Repositories of studies in specific areas such as PEDRO for physiotherapy²⁶ or of specific types such as CENTRAL for controlled trials²⁷.
• Prospective registries of trials such as ClinicalTrials.gov and the World Health Organisation’s portal for trial registries²⁸.
• The hand searching of journals or conference proceedings that have not been included in indexed databases²⁹.
• The checking of references in included studies³⁰.

APPRAISING STUDIES FOR INCLUSION IN A SYSTEMATIC REVIEW

The phrase ‘garbage in, garbage out’ is apt for describing the rationale for the next step in conducting a systematic review: assessing the quality of the potentially eligible studies to determine whether they will help or hinder the reviewers in their effort to provide a reliable and robust answer to their research question. A variety of numeric scales have been developed for assessing the quality of specific types of study, particularly randomised trials, but caution is needed in using these since different quality assessment tools can give widely different findings³¹. An alternative approach is for reviewers to decide on the key areas of study quality for their review and then to appraise each study in each of these areas, which might include a consideration of both quality and risk of bias. How each study performs in each area can then be described in the review. Reviewers also need to consider how they will use their assessments of study quality in their review³². This might link back to their underlying aim, since systematic reviews seeking a reliable estimate on the effects of an intervention might wish to exclude poor quality studies or those at high risk of bias from their meta-analyses. On the other hand, reviewers seeking to collate all research on a specific topic, learn from successes and failures in past studies or identify key gaps and areas of uncertainty might wish to retain such studies to explore these issues in depth.

COLLECTION OF DATA

When the eligible studies have been identified, reviewers need to gather information to summarise these studies in their review, including any data that they will use to compare, contrast and/or combine the studies' findings. This extraction and clear presentation of information on each study should make it easier for users of the review and may require the reviewer to gather material that is not readily available in the report of a study. This might be details that are available in a study protocol or entry in a trial registry²⁸ or might extend to their collection of individual participant data from the included studies³³.

STATISTICAL ANALYSIS

Having extracted or collected the data for each study, reviewers have a variety of options and methods for combining these results in a meta-analysis^34-36. Typically, each study is analysed separately to generate its summary statistic and these are then combined in the meta-analysis. The results of the meta-analysis might be shown as a forest plot³⁷, which reveals the relative contribution of each study and allows an exploration of differences across the results of the individual studies including tests for statistical heterogeneity³⁸, helping the reviewer and the review user to determine whether the average result that comes from the meta-analysis is a good guide for making decisions in their setting.

Systematic reviews might include sensitivity analyses to determine how sensitive the results of the systematic review are to the methods used for the review. They might be used to investigate the consistency of the findings from using different statistical techniques for the meta-analysis or the impact of including studies that were published in particular languages, assessed to be at high risk of bias or of uncertain eligibility. However, a key point to note is that these sensitivity analyses can only be performed if the reviewers have done the necessary work and then remove it. For example, the impact of including studies published in languages other than English can only be assessed if such studies were searched for and found.

REPORTING SYSTEMATIC REVIEWS

Reporting guidelines now exist for many different types of research study and these have been collated by the EQUATOR network. Use of these guidelines should help systematic reviewers by improving the quality of the reporting of the studies that they will include and help users of systematic reviews by improving the quality of the reports of the reviews. PRISMA guidelines now exist for the protocols³⁹ for systematic reviews and for the full reviews⁴⁰, as well as versions for specific types of review, such as those based on individual participant data⁴¹.

UPDATING SYSTEMATIC REVIEWS

Systematic reviews are, by their nature, retrospective. They look back in time and can be thought of as historical research, looking back on the evidence that existed at the time of the review. However, given that the intention for most reviews is that they will provide the knowledge needed by decision-makers today and by policy makers for the future, they need to be kept up-to-date to consider emerging evidence. Ideally, all relevant research is included in a review at the time that it is being used to inform a decision, but this is impractical without continual updating of the review to incorporate new evidence⁴². Instead, reviews need to be updated periodically through new searching, appraisal and analyses. Updating the review can also help to maintain its relevance to the contemporary setting, reflecting, for example, changes in how care is organised, costs and values and preferences about outcomes.

APPRAISING AND USING SYSTEMATIC REVIEWS

The factors that are important when conducting a systematic review can also form the basis for appraising the quality of systematic reviews. Users need to consider the risk of bias or imprecision in the review from a failure to include all relevant research, the possibility of selective reporting of the findings of the review and how the authors have interpreted their findings and drawn their conclusions. The AMSTAR (www.amstar.ca) tool has been validated for the assessment of reviews and provides a framework for doing this^7,43,44. Users of reviews also need to decide how to use the findings from the accumulated research that was done at other times and in other places. They should ask themselves whether the populations and interventions in the studies in the reviews are so different from their own setting that they are not applicable. And, whether the outcomes that were measured and reported provide sufficient information on the potential benefits and harms that matter to them, which might be helped through increased adoption of core outcome sets^45,46.

CONCLUSIONS

Decisions and choices in sports medicine, as in all health and social care should be based on the best available evidence. The evidence should be reliable and robust. It should have minimised the effects of bias and chance. Systematic reviews provide busy users, who might otherwise be overwhelmed by a vast number of individual studies, with a summary of the evidence for a particular research question. However, these reviews need to be of high quality and need to have been able to draw on research that is itself good enough to be informative. Therefore, whether or not someone is able to use a systematic review as one part of the knowledge they need to make a decision depends on the prioritisation of the conduct of the underlying research and review and the quality of both.

Mike Clarke D.Phil.

Director, Northern Ireland Methodology Hub

Centre for Public Health

Institute of Clinical Sciences

Queen’s University Belfast

Belfast, Northern Ireland

Contact: m.clarke@qub.ac.uk

References

1. Clarke M. History of evidence synthesis to assess treatment effects: personal reflections on something that is very much alive. J R Soc Med 2016; 109:154-163.
2. Chinnock P, Siegfried N, Clarke M. Is evidence-based medicine relevant to the developing world? Systematic reviews have yet to achieve their potential as a resource for practitioners in developing countries. PLoS Med 2005; 2:e107.
3. Antes G, Clarke M. Knowledge as a key resource for health challenges. Lancet 2012; 379:195-196.
4. Gerdin M, Clarke M, Allen C, Kayabu B, Summerskill W, Devane D et al. Optimal evidence in difficult settings: improving health interventions and decision making in disasters. PLoS Med 2014; 11:e1001632.
5. Handoll H. Work of the Cochrane Bone, Joint and Muscle Trauma Review Group: making sense of complexity. J Evid Based Med 2013; 6:208-212.
6. DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB et al. Systematic reviews in sports medicine. Am J Sports Med 2016; 44:533-538.
7. Weir A, Rabia S, Ardern C. Trusting systematic reviews and meta-analyses: all that glitters is not gold! Br J Sports Med 2016; 50:1100-1101.
8. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016; 13:e1002028.
9. Lauersen JB, Bertelsen DM, Andersen LB. The effectiveness of exercise interventions to prevent sports injuries: a systematic review and meta-analysis of randomised controlled trials. Br J Sports Med 2014; 48:871-877.
10. Fitzpatrick J, Bulsara M, Zheng MH. The effectiveness of platelet-rich plasma in the treatment of tendinopathy: a meta-analysis of randomized controlled clinical trials. Am J Sports Med 2016 [Epub ahead of print].
11. Mason DL, Dickens VA, Vail A. Rehabilitation for hamstring injuries. Cochrane Database Syst Rev 2012; 12:CD004575.
12. Mugunthan K, Doust J, Kurz B, Glasziou P. Is there sufficient evidence for tuning fork tests in diagnosing fractures? A systematic review. BMJ Open 2014; 4:e005238.
13. Eggerding V, Meuffels DE, Bierma-Zeinstra SM, Verhaar JA, Reijman M. Factors related to the need for surgical reconstruction after anterior cruciate ligament rupture: a systematic review of the literature. J Orthop Sports Phys Ther 2015; 45:37-44.
14. Simpson M, Rio E, Cook J. At what age do children and adolescents develop lower limb tendon pathology or tendinopathy? A systematic review and meta-analysis. Sports Med 2016; 46:545-557.
15. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009; 1:MR000006.
16. Dwan K, Altman DG, Clarke M, Gamble C, Higgins JP, Sterne JA et al. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med 2014; 11:e1001666.
17. Clarke M, Halsey J. DICE 2: a further investigation of the effects of chance in life, death and subgroup analyses. Int J Clin Pract 2001; 55:240-242.
18. Counsell CE, Clarke MJ, Slattery J, Sandercock PAG. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994; 309:1677-1681.
19. Clarke M, Halsey J. Dicing with chance, life and death in systematic reviews and meta-analyses: D.I.C.E. 3, a simulation study. J R Soc Med 2014; 107:116-119.
20. Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One 2014; 9:e102670.
21. Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc 2013; 24:145-148.
22. Chalmers I. The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann N Y Acad Sci 1993; 703:156-163.
23. Starr M, Chalmers I, Clarke M, Oxman AD. The origins, evolution, and future of The Cochrane Database of Systematic Reviews. Int J Technol Assess Health Care 2009; 25 suppl 1:182-195.
24. Alderson P, Roberts I. Should journals publish systematic reviews that find no evidence to guide practice? Examples from injury research. BMJ 2000; 320:376-377.
25. Pilkington K, Boshnakova A, Clarke M, Richardson J. “No language restrictions” in database searches: what does this really mean? J Alt Complement Med 2005; 11:205-207.
26. Stevens ML, Moseley A, Elkins MR, Lin CC, Maher CG. What searches do users run on PEDro? An analysis of 893,971 search commands over a 6-month period. Methods Inf Med 2016; 55:333-339.
27. Dickersin K, Manheimer E, Wieland S, Robinson KA, Lefebvre C, McDonald S. Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials. Eval Health Prof 2002; 25:38-64.
28. Ghersi D, Pang T. From Mexico to Mali: four years in the history of clinical trial registration. J Evid Based Med 2009; 2:1-7.
29. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev 2007; 2:MR000001.
30. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev 2011; 8:MR000026.
31. Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282:1054-1060.
32. Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L’Abbe KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol 1992; 45:255-265.
33. Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M et al. Individual participant data (IPD) meta-analyses of randomised controlled trials: guidance on their use. PLoS Med 2015; 12:e1001855.
34. Cooper HM, Rosenthal R. Statistical versus traditional procedures for summarizing research findings. Psychol Bull 1980; 87:442-449.
35. Deeks JJ. Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes. Stat Med 2002; 21:1575-1600.
36. Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol 2008; 61:857-865.
37. Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001; 322:1479-1480.
38. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003; 327:557-560.
39. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015; 349:g7647.
40. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Inter Med 2009; 151:264-269.
41. Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Tierney JF et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. JAMA 2015; 313:1657-1665.
42. Adams CE, Polzmacher S, Wolff A. Systematic reviews: what needs to be done and not to be done. J Evid Based Med 2013; 6:232-235.
43. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007; 7:10.
44. Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am 2013; 95:e771-e777.
45. Clarke M, Williamson P. Core outcome sets and systematic reviews. Syst Rev 2016; 5:11.
46. Tunis SR, Clarke M, Gorst SL, Gargon E, Blazeby JM, Altman DG et al. Improving the relevance and consistency of outcomes in comparative effectiveness research. J Comp Eff Res 2016; 5:193-205.

Image by Pixabay