Limitations of metrics

Metrics and measurement are part of academic life and are involved at almost every stage of the research lifecycle but this does not mean that they are above criticism. As metrics are used in an ever increasing range of decisions, some people have begun to question whether they are in fact the best measures to use. Each individual measure has its own advantages and disadvantages but we also need to explore the broader picture.

The most obvious limitation of metrics is that they only tell part of a larger story - the numerical part. Although it is easier to measure things in terms of numbers relying solely on this can be misleading. Other, qualitative factors also need to be included in order to get a complete understanding of what is happening and perhaps more importantly why.

This becomes a problem when we consider the types of decisions which are based on these metrics. With the tightening of budgets there are only so many academic roles to go around and competition is fierce. It is a fact of academic life that those researchers who can demonstrate an impact (usually in numerical form) are more likely to be sought after for positions or any promotions that are available, If a researcher has a lower metric score in one area than another they are placed at a disadvantage which could have a long-lasting impact on their career. Given that most of the common metrics used were designed for library stock selection is it right that such major decisions are being based on their result?

There have also been criticisms of a lack of consistency. All metrics uses slightly different calculations to reach a final figure and it can be hard to compare like with like. This is especially true when comparing individual researchers across disciplines or institutions where practices and the corresponding numbers may differ. If, for example, the number of citations is being taken as a measurement of career success does this not unfairly bias the results against an early career researcher who may ot yet have had a chance to publish as much work? Citation counts in particular are also part of another problem as simply recording the number of times something has been cited does not equal quality. A famous example of this is the Andrew Wakefield paper published in 1998 which linked the MMR vaccine to the development of autism. The research was retracted and widely discredited but the metrics of the paper are sky high as everyone cites it as an example of poor research (including this resource!). In a similar way there have been claims that simple counts bias different researchers as women are cited significantly less than their male colleagues.

Numerical metrics are also open to gaming where authors can try to increase the amount of times their work is viewed or cited by citing themselves or persuading their colleagues/students to reference them in their own work. There are also problems such as gift authorship where the names of prestigious academics are added to work to give it more appeal. This may seem harmless but it is a form of manipulation as their metric scores are artificially inflated.