Skip to Main Content

ESMIG's Evidence Synthesis Resources Guide

Protocol Development

Evidence Synthesis Protocol Templates

Evidence Synthesis Protocol Examples

Alyami, A., Soares, M. J., Sherriff, J. L., Zhao, Y., Hallett, J., & Coombes, F. (2015). A systematic review protocol examining the effect of vitamin D supplementation on endothelial function. BMJ open, 5(6), e006835.

Briggs, J. (2006). Appendix 1: Systematic review protocol example: smoking cessation interventions and strategies for hospitalised patients. In Evidence‐Based Clinical Practice in Nursing and Health Care (pp. 173–176). Blackwell Publishing Ltd.

Moon, D., Nichols, C. B., Sgourakis, A., Cruce, A., Zhang, Y., Haran, H., & Johnson-Motoyama, M. (2022, March 8). Engagement measures in maltreatment prevention studies: a scoping review. Retrieved from

Moore, E., Howson, P., Grainger, M., Teh, Y.A., & Pfeifer, M. (2022). The role of participatory scenarios in ecological restoration: a systematic map protocol. Environmental Evidence, 11(1), 1-9.

Nkangu, M., Obegu, P., Asahngwa, C., Shiroya, V., Gobina, R., Agbaw-Ebai, F. P., Keboa, M., & Foretia, D. (2021). Scoping review protocol to understand the conceptualisation, implementation and practices of health promotion within the context of primary healthcare in Africa. BMJ open, 11(12), e049084.

O'Connor, T., Moore, Z. E., & Patton, D. (2021). Patient and lay career education for preventing pressure ulceration in at-risk populations. Cochrane database of systematic reviews, 2(2), CD012006.

Welsh, B. C., & Farrington, D. P. (2003). Protocol: Effects of closed circuit television surveillance on crime: Protocol for a systematic review. Campbell Systematic Reviews, 1(1), 1–12.


Framing your Research Question

Useful Framework Lists
Further Reading

Booth, A., Noyes, J., Flemming, K., Moore, G., Tunçalp, Ö., & Shakibazadeh, E. (2019). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Global Health, 4(Suppl 1), e001107.

Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), 579.

Choosing Sources

Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A.-M. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: A guide to information retrieval for Campbell systematic reviews. Campbell Systematic Reviews, 13(1), 1–73.

Lefebvre, C., Glanville, J., Briscoe, S., Featherstone, R., Littlewood, A., Marshall, C., Metzendorf, M.I., Noel-Storr, A., Paynter, R., Rader, T., Thomas, J., & Wieland, L.S. (2022). Chapter 4: Searching for and selecting studies. In Higgins, J.P.T et al. Eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022).

Vassar, M., Carr, B., Kash-Holley, M., DeWitt, E., Koller, C., Day, J., Day, K., Herrmann, D., & Holzmann, M. (2015). Database choices in endocrine systematic reviews. Journal of the Medical Library Association, 103(4), 189–192.

Stansfield C, Brunton J (2023) Crowdsourced information resources relating to low-resource settings. London: EPPI Centre, UCL Social Research Institute, University College London.

Bramer, W. M., Rethlefsen, M. L., Kleijnen, J., & Franco, O. H. (2017). Optimal database combinations for literature searches in systematic reviews: A prospective exploratory study. Systematic Reviews, 6(1), 245.

Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2016). The contribution of databases to the results of systematic reviews: A cross-sectional study. BMC Medical Research Methodology, 16(1), 127.

Frandsen, T. F., Gildberg, F. A., & Tingleff, E. B. (2019). Searching for qualitative health research required several databases and alternative search strategies: A study of coverage in bibliographic databases. Journal of Clinical Epidemiology, 114, 118–124.

Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177.

Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2016). A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Systematic Reviews, 5(1), 27.

Ritchie, S. M., Young, L. M., & Sigman, J. (2018). A comparison of selected bibliographic database subject overlap for agricultural information. Issues in Science and Technology Librarianship, 89.

Choosing multiple sources

Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people’s views: A case study in public health. BMC Medical Research Methodology, 12, 55.

Stevinson, C., & Lawlor, D. A. (2004). Searching multiple databases for systematic reviews: Added value or diminishing returns? Complementary Therapies in Medicine, 12(4), 228–232.

Whiting, P., Westwood, M., Burke, M., Sterne, J., & Glanville, J. (2008). Systematic reviews of test accuracy should search a range of databases to identify primary studies. Journal of Clinical Epidemiology, 61(4), 357–364.


Mitigating Language Bias

Pieper, D., & Puljak, L. (2021). Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. Journal of Clinical Epidemiology, 132, 146–147.

Rockliffe, L. (2022). Including non-English language articles in systematic reviews: A reflection on processes for identifying low-cost sources of translation support. Research Synthesis Methods, 13(1), 2–5.

Walpole, S. C. (2019). Including papers in languages other than English in systematic reviews: Important, feasible, yet often omitted. Journal of Clinical Epidemiology, 111, 127–134.

Wu, X.-Y., Tang, J.-L., Mao, C., Yuan, J.-Q., Qin, Y., & Chung, V. C. H. (2013). Systematic reviews and meta-analyses of traditional Chinese medicine must search Chinese databases to reduce language bias. Evidence-Based Complementary and Alternative Medicine: ECAM, 2013, 812179.

Database Searching

Search Hedges

Search Strategy Repositories

Other search strategy development resources

Bramer, W. M., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). Evaluation of a new method for librarian-mediated literature searches for systematic reviews. Research Synthesis Methods, 9(4), 510–520.

Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217.

Gusenbauer, M., & Haddaway, N. R. (2021). What every researcher should know about searching – clarified concepts, search advice, and an agenda to improve finding in academia. Research Synthesis Methods, 12(2), 136–147.

Jenuwine, E. S., & Floyd, J. A. (2004). Comparison of Medical Subject Headings and text-word searches in MEDLINE to retrieve studies on sleep in healthy individuals. Journal of the Medical Library Association, 92(3), 349–353.

O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: A systematic review of current approaches. Systematic Reviews, 4(1), 5.

McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of Clinical Epidemiology, 75, 40–46.

Grey Literature Searching

Reliance on select common databases or published sources leads to biases in searching.  A complete and systematic search includes searching grey literature sources. Standard 3.2.1 states: "Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies" from "Finding What Works in Health Care: Standards for Systematic Reviews". 

Often defined as information that is published outside of the traditional publishing streams such as journal articles that get indexed in large comprehensive bibligraphic databases, examples of grey literature include:

  • Technical reports
  • Conferences and proceedings
  • Preprints
  • Patents
  • Clinical trials
  • Policy briefs
  • Working papers
  • Theses and dissertations
  • Unpublished & on-going studies
Further Reading

Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4(1), 138–138.

Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217.

Haddaway, N.R., Collins, A. M., Coughlin, D., & Kirk, S. (2017). A rapid method to increase transparency and efficiency in web-based searches. Environmental Evidence, 6(1).

Haddaway, N.R., Collins, A. M., Coughlin, D., & Kirk, S. (2015). The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PloS One, 10(9), e0138237–e0138237.

Grey literature source lists

Conference Proceedings

Theses and dissertations


Preprints is a free preprint server for various disciplines supported by MDPI in Basel, Switzerland. Additional disciplinary preprint services are listed below.

Technical Reports

Documentation and Reporting

Some of the more commonly used reporting standards related to Evidence Synthesis include:

Protocols: Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & PRISMA-P Group (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews, 4(1), 1.

Searching: Rethlefsen, M.L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B.  PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews, 10(1), 39 (2021).

Systematic Reviews: Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS medicine, 18(3), e1003583.

Scoping Reviews: Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., Lewin, S., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of internal medicine, 169(7), 467–473.

APA JARS-Quant: American Psychological Association. (2020). Quantitative Meta-Analysis Article Reporting Standards. Retrieved from

MOOSE: Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., & Thacker, S. B. (2000). Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA: the Journal of the American Medical Association, 283(15), 2008–2012.

MECCIR (Campbell Collaboration): The Methods Group of the Campbell Collaboration. (2019). Methodological expectations of Campbell Collaboration intervention reviews: Reporting standards. Campbell Policies and Guidelines Series No. 4. Retrieved from Link

ROSES: Haddaway, N.R., Macura, B., Whaley, P., & Pullin, A. S. (2018). ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environmental Evidence, 7(1), 1-8.

Screening Studies

Screening for eligibility typically involves the following steps:

1. Developing and testing eligibility criteria (this may already have been done during the protocol stage).

2. Piloting and testing the screening process (this involves testing the application of the eligibility criteria, and establishing an initial agreement level; it also allows for an estimation of time).

3. Screening using title and abstracts.

4. Screening using the full-text of the articles.

For further information, see the relevant chapter of the conducting guide you may be following such as the Cochrane handbook, JBI evidence synthesis manual, and CEE guidelines.

Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020). Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology, 121, 20-28.

Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019). Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods, 10(3), 330-342.

Porritt, K., Gomersall, J., & Lockwood, C. (2014). JBI's Systematic Reviews: Study selection and critical appraisal. The American journal of nursing, 114(6), 47–52.

Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology, 19(1), 132.


Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022). Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis. JMIR Medical Informatics, 10(5), e33219.

Harrison, H., Griffin, S.J., Kuhn, I., & Usher-Smith, J. A. (2020) Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Medical Research Methodology, 20(1), 7 (2020).

Van der Mierden, S., Tsaioun, K., Bleich, A., & Leenaars, C. H. C. (2019). Software tools for literature screening in systematic reviews in biomedical research. ALTEX - Alternatives to Animal Experimentation, 36(3), 508–517.


Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019). Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Systematic Reviews, 8(1), 278.

Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Systematic reviews, 7(1), 45.

Khalil, H., Ameen, D., & Zarnegar, A. (2022). Tools to support the automation of systematic reviews: A scoping review. Journal of Clinical Epidemiology, 144, 22–42.

Waffenschmidt, S., Hausner, E., Sieben, W., Jaschinski, T., Knelangen, M., & Overesch, I.(2018). Effective study selection using text mining or a single-screening approach: a study protocol. Systematic Reviews, 7(1), 166.

Critical Appraisal

Many libraries have curated lists of various critical appraisal tools that can be used to evaluate and assess the methodological rigor of included studies. Critical appraisal usually happens before data extraction. Studies scoring poorly are typically excluded.
The following organizations have created various appraisal tools.

Data Extraction and Synthesis

Büchter, R.B., Weise, A., & Pieper, D. (2020).  Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance. Research Methodology, 20(1), 259.

Data Extraction. JBI Manual for Evidence Synthesis (2022). 11, 2.7.

Li, T., Higgins, J.P.T., Deeks, J.J. (editors). Chapter 5: Collecting data. In: Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., & Welch, V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from

Taylor, K. S., Mahtani, K. R., & Aronson, J. K. (2021). Summarising good practice guidelines for data extraction for systematic reviews and meta-analysis. BMJ Evidence-Based Medicine, 26(3), 88-90.