Skip to main content

Scholarly Communication Toolkit: Measuring Impact

Measuring the Impact of Research

journal citation exampleOne of the many challenges faced by scholars is to identify the impact their scholarship is having in their respective disciplines. This need is frequently driven by promotion and tenure. Alternatively, they may be interested in locating the most important research being done in their field. What are the important ideas now shaping opinions, methods, and research? The impact of published scholarship is also important for institution-level evaluation conducted by university and college administrations and for collection building and maintenance projects conducted by the library.

Traditionally, journal impact factors have been utilized to calculate the impact of scholarship. The impact factor came into use during the 1970s through the work of Eugene Garfield. Calculated as the average number of cited articles divided by the number of citable items in a journal in the past two years, the impact factor illuminates which journals are most influential in a scholar’s given field. In other words, it is a measure of how often a journal is cited by other journals in a field. 

The journal impact factor is still used today as a measure of the relative importance of a journal within its field. However, there has been growing dissatisfaction with reliance upon the traditional journal impact factors due to the time lag in assessment that they impose since journals take time to accrue citations, the prospect of gaming the system, and the risk of elevating the rank of an article that is cited as a cautionary or corrective tale to the scholarly community. As a result different metrics have emerged including author and article level metrics and alternative metrics, which rely upon social media, reference/bibliography creation software and other non-traditional citations. Additionally, tools are available to researchers to create profiles that track their individual impact and correctly and consistently identify what research outputs are the product of their efforts.

This section of the Toolkit does not provide guides to finding and using the variety of traditional journal, article and author level metrics. Rather this section provides librarians with resources to assist them in thinking critically about traditional metrics to help facilitate conversations with the scholars they serve about the role of impact factors in evaluating scholarship, whether for tenure and promotion or for finding and evaluating research in one's field. This section will also provide tools and resources on the growing field of altmetrics as well as tips and resources for assisting scholars with maximizing the identification and impact of their research.

Impact Factors - A Critical Review of a Traditional System

The following resources offer a critical look at traditional measurements of impact:

Peer Review

In the past, peer reviewed journals have been the standard for measuring the quality and importance of scholarly literature. However, the growing number of journals and articles being published has placed significant strain on this process. Part of the problem is attributable to a rate of publication that is far outpacing the pool of available referees. With scant incentive available for referees in the form of credit in the tenure and promotion process, this shortage is expected to continue.

Despite its strong track record in filtering research, the peer review system has come to show signs of weakness.  The authors of the altmetrics manifesto point out that “peer review…is slow, encourages conventionality, and fails to hold reviewers accountable…[and] given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.” This prognosis is not good and is unsustainable over the long term. In response, new alternative forms of peer review have begun to develop that take a more open and transparent approach to the traditional peer review model:

Open Peer Review: Open peer review begins with an author or editor posting an unpublished manuscript online in a comment-enabled web environment, and then inviting peers to contribute comments and criticisms. The argument is that open peer review not only allows for transparency in the peer review process, but also enables a wider variety of input, including cross-discipline critique and more technical, "non-scholarly" input.

Post-Publication Peer Review:

  • Altmetrics are a form of post-publication peer review -- see the Altmetrics box on the right for more info
  • Faculty 1000 - Identifies and evaluates the most important articles in biology and medical research publications
  • PubPeer - tool for commenting on journal articles
  • RetractionWatch - blog on retractions of scholarly articles, which is a function of post publication review

Altmetrics

As more information is disseminated electronically, researchers will come to interact with that information on the open web in a variety of ways and through many different platforms and media types. Altmetrics measures how many times a journal article is downloaded, shared, commented on, and cited in social media outlets and can provide a meaningful indicator of the impact an article has among different user populations.  One of the founders of the study of altmetrics Jason Priem and Bradley Hemminger have compiled a list of sources from which data can be collected and relayed back to scholars as meaningful impact data.  These sources fall into seven categories:

  1. Social bookmarking sites (Delicious, CitULike, Connotea)
  2. Reference managers (Mendeley, Zotero)
  3. Recommendation sites (Dig, Reddit, Friendfeed, Faculty of 1000)
  4. Publisher hosted comment spaces (PloS, British Medical Journal)
  5. Microblogging (Twitter)
  6. Blogs (WordPress, Blogger, Researchblogger)
  7. Social networks (Facebook, Nature Networks, Orkut)

Culling usage data from various social sites has several advantages.  First, sites which have open APIs (application programming interface) can be accessed immediately for up-to-date usage statistics – an advantage over traditional citations.  Second, a growing number of commercial platforms such as ImpactStory, Altmetric.com, and Plum Analytics allow scholars to track usage of their works across several research blogs, journals, and user populations. This allows a granularity of data and presents a broader perspective of overall impact. It is important to note that altmetrics is not meant to replace traditional citation, but is best used in conjunction with citations for an overall picture of scholarly impact.

Additional Readings, Resources and Presentations on Altmetrics:

Maximizing Author Impact - Tools & Resources

There are a growing number of tools available to scholars to maximize their impact, whether through proper identification of themselves as the creator of a product of research or through social media like profile pages highlighting their expertise and output. Libraries can assist scholars in maximizing their impact by promoting author identification tools such as ORCID, sharing data on open access publishing and citation rates, and directing scholars to sites that allow them to develop there own social media like researcher profile pages. 

Tips and Tools for Maximizing Impact

  • ORCID - ORCID (Open Researcher and Contribution ID) aims to solve the name ambiguity problem in scholarly communications by creating a registry of persistent unique identifiers for individual researchers and an open and transparent linking mechanism between ORCID, other ID schemes, and research objects such as publications, grants, and patents. See also ORCID Author Identifiers: A Primer for LibrariansMedical Reference Services Quarterly (2016).
  • The Becker Medical Library at Washington University of St. Louis has put together a very extensive and comprehensive list of tips and tools to help scholars to maximize their impact. 
  • London School of Economics Handbook for Social Scientists on Maximizing Impact of Research
  • FigShare - Figshare allows researchers to publish all of their research outputs in seconds in an easily citable, sharable and discoverable manner. All file formats can be published, including videos and datasets that are often demoted to the supplemental materials section in current publishing models. By opening up the peer review process, researchers can easily publish null results.
  • Google Scholar Citations -researchers can also create a public profile with their articles and citation metrics. If the profile is public, it can appear in Google Scholar search results when someone searches for your name.
  • Article covering the variety of researcher services that are available: Making Sense of Researcher ServicesJournal of Library Administration (2016)

Using Open Access to Increase Impact

Credit

Portions of this section of the Toolkit are adapted from an essay that was originally written by Bruce Runnels for a course in scholarly communications at the School of Library & Information Science at IUPUI with subsequent amendment by Jen Waller of Miami University (Ohio), Kevin Smith from University of Kansas, Steven R. Harris from the University of Nevada, Reno, and Christine Fruin from the University of Florida.