Among the books I have recently borrowed from the library, Jerry Muller’s (2018) book, the Tyranny of Metrics, has been the one I’d like to purchase a copy of and keep at hand for future reference. Muller is a historian who has written books on Adam Smith, various aspects of capitalism, and the history of conservative political thought. The initial seed for the Tyranny of Metrics, he writes in the introduction, was a series of minor, personal annoyances he experienced as Head of a university department, which upon further inspection turned out to be neither minor nor personal.
Like many academics who take on administration duties within their departments or faculties, Muller felt the pressure to conduct regular, extensive quantitative reviews and provide frequent reports for the purpose of tracking and comparing academic performances.
Why would such a task be unwelcome, annoying, or counter-productive? Aside from the sheer time and effort that continually goes into building such reports, the metrics themselves can be misleading. As one example, imagine a comparison between a History Department and a Psychology Department within a university. While the Psychology staff tend to publish numerous articles each year, forming rather large groups of “co-authors,” the History staff tend to write single-authored books, each of which requires at least a couple of years to complete. The Psychology staff, furthermore, might belong to scholarly networks–or more aptly publishing “gangs”–who not only write together and “review” each others’ works, but also generously cite the publications within their own network, making sure everyone looks good with respect to the relevant metrics. When the senior admin of the university put the annual performances of two departments side by side, they might draw the wrong conclusions from the comparison: “Why can’t the historians be as hard-working and high-impact as the psychologists?!”
Expecting the History department, in our imagined university, to be more like the Psychology department, presupposes that the metrics are giving us a fair comparison based on a common frame of reference. We are also accepting that the high number of publications and citations in the Psychology group reflects advancement in scholarship. We might forget that the behavior of the highly “productive” and highly cited group results precisely from their attempt at directly satisfy the metrics, which is far easier than producing scholarly works of considerable merit.
Taking the investigation beyond the context of universities, Muller develops a broad sociological-historical study of metrics with reference to schooling, medicine, policing, the military, business and finance, and philanthropic organizations. With the help of examples, he shows the same patterns across different domains. First and foremost, our metrics often miss important aspects of what we try to measure. Grades in school, for instance, don’t measure the teacher performance or the student’s ability. There are many other factors more strongly linked to grades, including the socioeconomic status of the students’ families. To pick one measure and then ask one set of agents (e.g., teachers) to fix/increase that measure would be to ignore the complexity of the situation.
In addition, the use metrics can, and often does, encourage gaming the system. People who are being evaluated might say, If metrics are the real thing, then fine! We will give you the metrics. Teachers might end up teaching for the test, specifically, or helping the students cheat; law enforcers might decide to take on easy and relatively inconsequential cases, instead of tackling more difficult (time-consuming) cases with larger social impact, because the difficult cases would bring down their scores.
Performing to satisfy the metrics, involves several types of sacrifice: The first is sacrificing the real target in favor of appearances. The second is sacrificing the long-term in favor of the short-term. A helpful example of the second sacrifice happens in non-profit organizations, who want to show their funders that they are keeping overheads as low as possible. Doing so, however, would require not spending on, say, training their employees or upgrading their computers, both of which are resources required for the long-term. From the standpoint of a fixed metric, a long-term investment would, at best, seem invisible, and at worst, be flagged as wasteful.
The third type of sacrifice is the sacrifice of innovation, in favor of repeating the same methods proven to satisfy the metrics. Innovation involves taking risk. It also requires suspending the assumptions of the established order, and the metrics are–it is worth remembering–the products of those established orders and their built-in assumptions.
What is the alternative to the use of metrics? That is not the question we ought to ask. Instead, we ought to ask, How should the metrics be complemented, guided, and constrained such that their damage is minimized? We can try to complement metrics with other ways of knowing that get pushed aside when we place too much confidence in the measurements. The problem with metrics is exemplified in the case of a new senior manager, who enters an organization as an “external hire.” The new person knows little about the organization, but can quickly acquire a feeling of knowing by looking at the reports and analytics. Not only that, but they might even get a feeling that they know how to improve the organization by putting pressure on the right people. They might set up a reward/punishment system tied to measured performances. They can do all of that without getting to know the organization first-hand. Our first task is to identify the misplaced confidence attached to that kind of outlook.
The other kind of knowing, contrasted with the knowledge based on metrics, shows up in single-case studies, in human relationships, over time, when we pay attention to multiple aspects of a single case. The other kind of knowing is sometimes (quite unfairly) called “subjective.” The TV, Wire, to which Muller makes continuing reference throughout the book, demonstrates this type of knowing in fictional form. With Wire, we are shown one case, the city of Baltimore, through different perspectives: crime, policing, governance, schooling, etc. The multiplicity of perspectives might be clue about the nature of that “other way of knowing,” as is the tracking of a single case. There is a point of view that would be interested in putting those different perspectives together, recognizing their connections. That point of view isn’t anti-measurement, it isn’t anti-scientific or anti-objective; it isn’t interested in replacing one form of tyranny with another; it might simply be a persistent, sympathetic interest, a curiosity, or a belief that there is still more to find out.