Alyami, A., Soares, M. J., Sherriff, J. L., Zhao, Y., Hallett, J., & Coombes, F. (2015). A systematic review protocol examining the effect of vitamin D supplementation on endothelial function. BMJ open, 5(6), e006835. https://doi.org/10.1136/bmjopen-2014-006835
Briggs, J. (2006). Appendix 1: Systematic review protocol example: smoking cessation interventions and strategies for hospitalised patients. In Evidence‐Based Clinical Practice in Nursing and Health Care (pp. 173–176). Blackwell Publishing Ltd. https://doi.org/10.1002/9781444316544.app1
Moon, D., Nichols, C. B., Sgourakis, A., Cruce, A., Zhang, Y., Haran, H., & Johnson-Motoyama, M. (2022, March 8). Engagement measures in maltreatment prevention studies: a scoping review. Retrieved from osf.io/7ybpk
Moore, E., Howson, P., Grainger, M., Teh, Y.A., & Pfeifer, M. (2022). The role of participatory scenarios in ecological restoration: a systematic map protocol. Environmental Evidence, 11(1), 1-9. https://doi.org/10.1186/s13750-022-00276-w
Nkangu, M., Obegu, P., Asahngwa, C., Shiroya, V., Gobina, R., Agbaw-Ebai, F. P., Keboa, M., & Foretia, D. (2021). Scoping review protocol to understand the conceptualisation, implementation and practices of health promotion within the context of primary healthcare in Africa. BMJ open, 11(12), e049084. https://doi.org/10.1136/bmjopen-2021-049084
O'Connor, T., Moore, Z. E., & Patton, D. (2021). Patient and lay career education for preventing pressure ulceration in at-risk populations. Cochrane database of systematic reviews, 2(2), CD012006. https://doi.org/10.1002/14651858.CD012006.pub2
Welsh, B. C., & Farrington, D. P. (2003). Protocol: Effects of closed circuit television surveillance on crime: Protocol for a systematic review. Campbell Systematic Reviews, 1(1), 1–12. https://doi.org/10.1002/cl2.15
Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1), e001107. https://doi.org/10.1136/bmjgh-2018-001107
Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), 579. https://doi.org/10.1186/s12913-014-0579-0
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A.-M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: A guide to information retrieval for Campbell systematic reviews. Campbell Systematic Reviews, 13(1), 1–73. https://doi.org/10.4073/cmg.2016.1
Lefebvre, C., Glanville, J., Briscoe, S., Featherstone, R., Littlewood, A., Marshall, C., Metzendorf, M.I., Noel-Storr, A., Paynter, R., Rader, T., Thomas, J., & Wieland, L.S. (2022). Chapter 4: Searching for and selecting studies. In Higgins, J.P.T et al. Eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). https://training.cochrane.org/handbook/current/chapter-04#section-4-3
Vassar, M., Carr, B., Kash-Holley, M., DeWitt, E., Koller, C., Day, J., Day, K., Herrmann, D., & Holzmann, M. (2015). Database choices in endocrine systematic reviews. Journal of the Medical Library Association, 103(4), 189–192. https://doi.org/10.3163/1536-5050.103.4.005
Stansfield C, Brunton J (2023) Crowdsourced information resources relating to low-resource settings. London: EPPI Centre, UCL Social Research Institute, University College London. https://eppi.ioe.ac.uk/cms/
Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017). Optimal database combinations for literature searches in systematic reviews: A prospective exploratory study. Systematic Reviews, 6(1), 245. https://doi.org/10.1186/s13643-017-0644-y
Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2016). The contribution of databases to the results of systematic reviews: A cross-sectional study. BMC Medical Research Methodology, 16(1), 127. https://doi.org/10.1186/s12874-016-0232-1
Frandsen, T. F., Gildberg, F. A., & Tingleff, E. B. (2019). Searching for qualitative health research required several databases and alternative search strategies: A study of coverage in bibliographic databases. Journal of Clinical Epidemiology, 114, 118–124. https://doi.org/10.1016/j.jclinepi.2019.06.013
Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177. https://doi.org/10.1016/j.joi.2018.09.002
Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2016). A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Systematic Reviews, 5(1), 27. https://doi.org/10.1186/s13643-016-0197-5
Ritchie, S. M., Young, L. M., & Sigman, J. (2018). A comparison of selected bibliographic database subject overlap for agricultural information. Issues in Science and Technology Librarianship, 89. https://doi.org/10.5062/F49Z9340
Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people’s views: A case study in public health. BMC Medical Research Methodology, 12, 55. https://doi.org/10.1186/1471-2288-12-55
Stevinson, C., & Lawlor, D. A. (2004). Searching multiple databases for systematic reviews: Added value or diminishing returns? Complementary Therapies in Medicine, 12(4), 228–232. https://doi.org/10.1016/j.ctim.2004.09.003
Whiting, P., Westwood, M., Burke, M., Sterne, J., & Glanville, J. (2008). Systematic reviews of test accuracy should search a range of databases to identify primary studies. Journal of Clinical Epidemiology, 61(4), 357–364. https://doi.org/10.1016/j.jclinepi.2007.05.013
Pieper, D., & Puljak, L. (2021). Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. Journal of Clinical Epidemiology, 132, 146–147. https://doi.org/10.1016/j.jclinepi.2020.12.027
Rockliffe, L. (2022). Including non-English language articles in systematic reviews: A reflection on processes for identifying low-cost sources of translation support. Research Synthesis Methods, 13(1), 2–5. https://doi.org/10.1002/jrsm.1508
Walpole, S. C. (2019). Including papers in languages other than English in systematic reviews: Important, feasible, yet often omitted. Journal of Clinical Epidemiology, 111, 127–134. https://doi.org/10.1016/j.jclinepi.2019.03.004
Wu, X.-Y., Tang, J.-L., Mao, C., Yuan, J.-Q., Qin, Y., & Chung, V. C. H. (2013). Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-Based Complementary and Alternative Medicine: ECAM, 2013, 812179. https://doi.org/10.1155/2013/812179
Bramer, W. M., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). Evaluation of a new method for librarian-mediated literature searches for systematic reviews. Research Synthesis Methods, 9(4), 510–520. https://doi.org/10.1002/jrsm.1279
Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378
Gusenbauer, M., & Haddaway, N. R. (2021). What every researcher should know about searching – clarified concepts, search advice, and an agenda to improve finding in academia. Research Synthesis Methods, 12(2), 136–147. https://doi.org/10.1002/jrsm.1457
Jenuwine, E. S., & Floyd, J. A. (2004). Comparison of Medical Subject Headings and text-word searches in MEDLINE to retrieve studies on sleep in healthy individuals. Journal of the Medical Library Association, 92(3), 349–353.
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews, 4(1), 5. https://doi.org/10.1186/2046-4053-4-5
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of Clinical Epidemiology, 75, 40–46. https://doi.org/10.1016/j.jclinepi.2016.01.021
Reliance on select common databases or published sources leads to biases in searching. A complete and systematic search includes searching grey literature sources. Standard 3.2.1 states: "Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies" from "Finding What Works in Health Care: Standards for Systematic Reviews".
Often defined as information that is published outside of the traditional publishing streams such as journal articles that get indexed in large comprehensive bibligraphic databases, examples of grey literature include:
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4(1), 138–138. https://doi.org/10.1186/s13643-015-0125-0
Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378
Haddaway, N.R., Collins, A. M., Coughlin, D., & Kirk, S. (2017). A rapid method to increase transparency and efficiency in web-based searches. Environmental Evidence, 6(1). https://doi.org/10.1186/s13750-016-0079-2
Haddaway, N.R., Collins, A. M., Coughlin, D., & Kirk, S. (2015). The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PloS One, 10(9), e0138237–e0138237. https://doi.org/10.1371/journal.pone.0138237
Preprints is a free preprint server for various disciplines supported by MDPI in Basel, Switzerland. Additional disciplinary preprint services are listed below.
Some of the more commonly used reporting standards related to Evidence Synthesis include:
Protocols: Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & PRISMA-P Group (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews, 4(1), 1. https://doi.org/10.1186/2046-4053-4-1
Searching: Rethlefsen, M.L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews, 10(1), 39 (2021). https://doi.org/10.1186/s13643-020-01542-z
Systematic Reviews: Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS medicine, 18(3), e1003583. https://doi.org/10.1371/journal.pmed.1003583
Scoping Reviews: Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., Lewin, S., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of internal medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
APA JARS-Quant: American Psychological Association. (2020). Quantitative Meta-Analysis Article Reporting Standards. Retrieved from https://apastyle.apa.org/jars/quant-table-9.pdf
MOOSE: Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., & Thacker, S. B. (2000). Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA: the Journal of the American Medical Association, 283(15), 2008–2012. https://doi.org/10.1001/jama.283.15.2008
MECCIR (Campbell Collaboration): The Methods Group of the Campbell Collaboration. (2019). Methodological expectations of Campbell Collaboration intervention reviews: Reporting standards. Campbell Policies and Guidelines Series No. 4. Retrieved from Link
ROSES: Haddaway, N.R., Macura, B., Whaley, P., & Pullin, A. S. (2018). ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environmental Evidence, 7(1), 1-8. https://doi.org/10.1186/s13750-018-0121-7
Screening for eligibility typically involves the following steps:
1. Developing and testing eligibility criteria (this may already have been done during the protocol stage).
2. Piloting and testing the screening process (this involves testing the application of the eligibility criteria, and establishing an initial agreement level; it also allows for an estimation of time).
3. Screening using title and abstracts.
4. Screening using the full-text of the articles.
For further information, see the relevant chapter of the conducting guide you may be following such as the Cochrane handbook, JBI evidence synthesis manual, and CEE guidelines.
Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020). Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology, 121, 20-28. https://doi.org/10.1016/j.jclinepi.2020.01.005
Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019). Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods, 10(3), 330-342. https://doi.org/10.1002/jrsm.1354
Porritt, K., Gomersall, J., & Lockwood, C. (2014). JBI's Systematic Reviews: Study selection and critical appraisal. The American journal of nursing, 114(6), 47–52. https://doi.org/10.1097/01.NAJ.0000450430.97383.64
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology, 19(1), 132. https://doi.org/10.1186/s12874-019-0782-0
Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022). Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis. JMIR Medical Informatics, 10(5), e33219. https://doi.org/10.2196/33219
Harrison, H., Griffin, S.J., Kuhn, I., & Usher-Smith, J. A. (2020) Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Medical Research Methodology, 20(1), 7 (2020). https://doi.org/10.1186/s12874-020-0897-3
Van der Mierden, S., Tsaioun, K., Bleich, A., & Leenaars, C. H. C. (2019). Software tools for literature screening in systematic reviews in biomedical research. ALTEX - Alternatives to Animal Experimentation, 36(3), 508–517. https://doi.org/10.14573/altex.1902131
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019). Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Systematic Reviews, 8(1), 278. https://doi.org/10.1186/s13643-019-1222-2
Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Systematic reviews, 7(1), 45. https://doi.org/10.1186/s13643-018-0707-8
Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42. https://doi.org/10/gn6pqk
Waffenschmidt, S., Hausner, E., Sieben, W., Jaschinski, T., Knelangen, M., & Overesch, I.(2018). Effective study selection using text mining or a single-screening approach: a study protocol. Systematic Reviews, 7(1), 166. https://doi.org/10.1186/s13643-018-0839-x
Büchter, R.B., Weise, A., & Pieper, D. (2020). Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance. Research Methodology, 20(1), 259. https://doi.org/10.1186/s12874-020-01143-3
Data Extraction. JBI Manual for Evidence Synthesis (2022). 11, 2.7. https://jbi-global-wiki.refined.site/space/MANUAL/4687700
Li, T., Higgins, J.P.T., Deeks, J.J. (editors). Chapter 5: Collecting data. In: Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook
Taylor, K. S., Mahtani, K. R., & Aronson, J. K. (2021). Summarising good practice guidelines for data extraction for systematic reviews and meta-analysis. BMJ Evidence-Based Medicine, 26(3), 88-90. http://dx.doi.org/10.1136/bmjebm-2020-111651
