An empirical investigation of the Goodhart effect for the widespread introduction of (alt)metrics

Indiana University, Bloomington IN

Investigators

Abstract

Policy-makers, and the scholarly community at large, increasingly rely on quantitative metrics to inform decisions on matters of science policy and the allocation of resources. However, there is a danger of the so-called Goodhart effect -- once a measure becomes a target, it ceases to be a good measure. Incentives may change the very behaviors they purport to measure, and consequently threaten the validity and reliability of the measure itself. The widespread availability of indicators and metrics can exert an even more pernicious social influence effect; when members of a community become aware of the evolving group consensus this affects their ability to collectively make accurate assessments. Both effects can be exacerbated by goal displacement when metrics evolve from measurement tools to objectives unto themselves. The project focuses in particular on altmetrics, which are specifically engineered for real-time public consumption, but the results generalize to all science evaluation metrics that are intended to be widely available. The project conducts a hypothesis-driven investigation into the Goodhart effect by analyzing whether the introduction of metrics indeed changes the behavior of scholars and policy-makers and whether this affects the validity of certain indicators. This investigation is aimed at determining under which conditions the Goodhart effect occurs and whether its effects are manifested at the level of policy-making and long-term trends in scholarly behavior. The project studies the causal pathways for the Goodhart effect by means of an ethnographic study of how metrics and indicators such as altmetrics are factored into the decision-making process of policy-makers which will illuminate several decision-making processes in scholarly communication; a data-driven analysis of how the recent introduction of altmetrics changed scholarly behavior by examining the longitudinal evolution of a variety of scholarly indicators to identify the quantitative longitudinal markers that are associated with the introduction of (alt)metrics, and; a double-blind, controlled experiment to determine the presence and magnitude of the social influence effect in scholarly communication using artificial online scholarly markets.