Research metrics

Welcome to this module about the ways in which research can be measured.

Data, Metrics, Key Performance Indicators... these terms are everywhere these days, as we increasingly seek hard data to monitor and improve the quality of many of the things we do. Research metrics can be very useful, but they also come with important caveats, so we need to be responsible in how we use these tools.

In this module, you will learn:

• The meaning of common metrics such as  Journal Impact Factor and H-index
• What are the main limitations of metrics
• A better, responsible approach to using metrics

Types of metric

Let’s start with the most commonly used research metrics. This video by Clair Castle at the Chemistry Library explains what they are and how they can be used.

There are many more types of research metrics, beyond the common ones discussed so far. If you are interested to know more Taylor and Francis have produced this quick and informative one-page guide.

If you want to dig deeper into metrics, the Metrics Toolkit has pages about a variety of metrics, with detailed explanations of how they are calculated and the appropriate ways of using them.

Uses and limitations of metrics

Metrics can be used to make comparisons at different levels. The diagram below shows some of these uses:

In her blog post clarifying conversations about metrics, Lizzie Gadd further explores some possible uses for metrics:

• Measure to understand: "science of science" activities that study publication patterns and trends for the sole purpose of understanding them better.
• Measure to show off: "pick me!" activities. The use of metrics to market an individual, group or university on promotional materials or grant applications.
• Measure to monitor: plotting progress against an objective whether internally or externally set. This may include some comparison activity.
• Measure to compare: the use of indicators to compare one entity with another. University rankings are an example of this.
• Measure to incentivise: the use of indicators to incentivise certain behaviours. The measurement of open access content submitted to REF is an example of this.
• Measure to reward: any activity that results in some kind of reward for the entity being measured, be this a job, promotion, grant, prize or reward of any description.

(Taken from The Blind and the Elephant: Bringing Clarity to our Conversations about Responsible Metrics  by Lizzie Gadd)

However, metrics have been shown to have significant limitations.   

• Focus on numbers: metrics are numerical measures which don't always take the wider impact of the research into account.
• Lack of consistency: each metrics is calculated in a different way and often uses proprietary sources of information. This means that researchers are often given different metrics on the same piece of work.
• Potential bias: there is a lack of consistency across disciplines and career stages which may unfairly disadvantage some researchers. For example, a metric which calculates the number of publications will reflect better on a more established researcher than a junior colleague.
• Potential for gaming: practices such as self or gift citation make it easy to manipulate the numbers. 
• Fit for purpose?: many of the most popular metrics in use today were developed for other purposes such as helping librarians to select stock, Is it really fair that they are now being used for career decisions?

The risks associated with using metrics irresponsibly increase as one moves from large to small scale and from understanding to incentivising or rewarding. For instance, it is probably acceptable to use a simple metric to investigate how patterns of research vary between countries, but not acceptable to use a single metric to determine the hiring of individual researchers.

So what is a better alternative?

Responsible metrics

Using a single, or a few measures to assess research quality would be unfair and create skewed incentives that could perversely decrease the quality of the research. However, metrics still have value when understood and used appropriately. This video by Claire Sewell at the Moore Library will give you an overview of what we mean by the term ‘responsible metrics’. If you prefer, you could find the same information in this handy guide.

If you are interested in knowing more about the potential impact of irresponsible metrics and the recommendations made by the Independent Review of the Role of Metrics in Research Assessment and Management, you can read the Metric Tide report 

In 2019 the University of Cambridge signed the San Francisco Declaration On Research Assessment (DORA), committing to a responsible use of metrics in assessing research. Find out more about what this means through the slides below and on the DORA website.  

Further resources

There is much to explore in the wonderful world of metrics. We have collected some of the most informative links below if you want to read further on any aspect of bibliometics, Altmetrics or responsible metrics.

The Metrics Toolkit contains lots of useful information on how traditional bibliometrics are calculated and how they can be used appropriately. 

Learn more about Altmetrics (alternative metrics) including how they are calculated and how to see the results on the Altmetric website. You can also see the metrics for your own research using the University of Cambridge login option.

The University of Cambridge has committed to the principles of DORA - the San Francisco Declaration on Research Assessment. You can read the document for yourself on the DORA website.