Can we stop treating cost-benefit analysis as if it’s “objective”?

CBA is an important evaluation tool, and its use involves as much subjectivity as any other

Economic methods of evaluation were the first evaluation methods I learnt and used, and I remain a proponent of them. Systematically and transparently identifying, quantifying, valuing and comparing costs and consequences of alternative courses of action is critical to making good resource allocation decisions. I don’t think evaluators do it enough.

On the other hand, we need to be cognisant of the limitations of any methods we use. It vexes me that economic analysis is sometimes regarded as if it provides an “objective” answer to an evaluative question, revealing a program’s “true value” or overriding other value claims. We have a professional duty to challenge such misconceptions.

Cost-benefit analysis (CBA), for example, seeks to determine whether, and by what margin an intervention creates enough value to justify the resources it consumes, by valuing resources and impacts in monetary terms. A benefit-cost ratio (BCR - like, “the program’s impacts represent $2.50 of social value for every $1 spent”) can carry a lot of authority in the corridors of power, even when we know it didn’t count everything of value.

Know Your Meme - For the Better, Right?

So, where did CBA come from, and what assumptions does it lean on? Let’s take a look.

CBA was born out of a positivist perspective of knowledge creation, which views truth as an objective reality that can be determined empirically if we have the techniques to measure it. After all, if we can have universal laws of gravitation and thermodynamics, why not social welfare too? 😊

In CBA, this model of ‘truth’ was based on principles from New Welfare Economics (new in the 1930s) which modified assumptions of welfare economics from the late 1700s and early 1800s, and attempted to establish ‘objective’ standards by which policies could be deemed economically desirable (or not) and optimised for a single, scientifically ‘ethics-free’ criterion of maximising welfare - neatly unburdening policy decisions from the messiness of human judgement (what could go wrong?)

The welfare-maximisation criterion, called Kaldor-Hicks efficiency, asserted that any change producing benefits to society greater than its costs should be deemed worthwhile, regardless of who is made better or worse off.¹

Wow… that’s a values stance and a half, isn’t it, for something that started out as an attempt to be values-free?

When we select CBA as our method of evaluation, we’re adopting a normative position whether we declare it (or know it) or not.

This position was underpinned by a set of rationale including:

⚖️ Consequentialism (the idea that a course of action is only as good as its consequences; the ends justify the means; impact trumps intent)

🚜 Utilitarianism (the idea that consequences can be ranked from best to worst based on their utility; more is better)

💰 A particular model of utility that assumed, among other things: that individuals are rational, self-interested actors with well-defined preferences that don’t change over time; that individuals want to maximise their own utility; that a person’s utility increases when their individualistic, self-interested preferences are satisfied; that people have the information and ability to accurately assess what will make them better off; that money can be used as a proxy for utility based on the amount that individuals would be willing to pay for a gain in utility (or receive in compensation for a loss of utility); that societal utility should be the sum of individual utilities (implying that a dollar gain for one person can offset a dollar loss to another regardless of their relative economic positions); and that the overall change in societal utility brought about by a policy or program represents its impact on social welfare (thus allowing analysts to compare different policy options based on their net benefits to society as a whole). This traditional model of utility didn't include altruistic preferences, collective utility, or equity.

As oft observed, models can simultaneously be wrong and useful. CBA is useful, providing an approximate answer to an important question on a systematic, transparent, disciplined basis, and uniquely summarising the impact of an intervention on overall social welfare in a single number. Equally, the “wrongness” of the underlying model of utility is self-evident - e.g., I for one have been known to have different preferences at different times, be an inconsistent judge of what is in my best interests, behave altruistically, be influenced by group deliberations, be willing to pay for things that don’t make me personally better off, and various other irrational decisions. CBA’s traditional model of valuing might represent one notion of what ‘should’ matter to us, but it imperfectly represents what actually does matter to us.²

That being said, CBA continues to evolve and develop.

There are alternatives to utility in modern CBA, such as subjective wellbeing. CBA can be adapted to accommodate altruistic and collective values, and can be used to provide some insights into distributional effects.

I would classify contemporary CBA as post-positivist because we all get (…right?) that CBA isn’t objective or values-free. It involves analyst judgements about which costs and benefits are relevant and material to count, which methods and data sources should be used to estimate monetary values, the time horizon over which costs and benefits are expected to last or matter, the degree to which society should value the present over the future, which variables should be subjected to sensitivity analysis, how to interpret and report findings, whether and how to weigh up considerations not reflected in the benefit-cost ratio such as non-monetised benefits and costs, equity considerations, uncertainty, and so on.

Know Your Meme - Moe Tossing Barney Out

We also understand (…right?) that CBA doesn’t “measure” social value. Social phenomena play out through the reasoning, values and experiences of individuals and groups, which are complex, unstable, not all that predictable, and just a little bit resistant to universal truths. Economists also understand that wider, messier aspects of public policymaking should be combined with the outputs of CBA to inform decisions. As I’ve argued, this principle also applies to evaluation; CBA can enhance mixed methods, and mixed methods can enhance CBA.

So, CBA is an important evaluation tool and its use involves as much subjectivity as any of the tools at our disposal.

That’s why evaluative thinking is so important. Let’s welcome CBA into the field of program evaluation - but let’s remember that a benefit-cost ratio isn’t an objective measure. Nor is it an evaluative judgement. No matter what methods and metrics we use, humans are still responsible for doing the evaluative reasoning required to make judgements from the selected evidence.

Know Your Meme - Khaby Lame

Adapted from this long article👇

CBA and SROI in the Garden of Eval

First subscriber to correctly identify this lake wins a 12-month paid subscription with access to the full archive.

1. Kaldor-Hicks efficiency is an elaboration on Pareto efficiency. An allocation of resources is said to be Pareto-efficient if there is no alternative allocation in which one person can be made better off without making somebody else worse off. The Pareto criterion is too restrictive to be practical for evaluating real-world policy proposals, which usually produce winners and losers. Kaldor (1939) modified the Pareto criterion by arguing that for an action to be in the public interest, those who gain from it should be able (in principle) to compensate those who lose from it, and still find the action worthwhile. However, the compensation does not actually have to take place. Hicks (1939) added that the losers must not be able to bribe the winners to forego the action. For a lucid explanation of how Kaldor-Hicks efficiency came about, see this.

2. Also see this excellent post from Anthony Clairmont this week:

What is value?