VfI backstory and overview

A quick recap to start the new year

Welcome

Hi! I’m Julian. If you’ve recently subscribed, welcome to Evaluation and Value for Investment, where I share principles and practices for getting clear answers to value-for-money questions - and hopefully start some conversations. I’m on holiday currently and taking the opportunity to look back at where we’ve come so far…

The quest for better VfM assessment

There’s increasing scrutiny on public policies and social programs to determine whether they deliver value for money (VfM), but traditional approaches to assessing VfM are incomplete. The disciplines of economics and evaluation share an interest in valuing resource use, but tend to operate as complementary or competing disciplines rather than being integrated within an overarching logic.

Cost-benefit analysis (CBA) is sometimes said to be the “gold standard” for evaluating VfM - but there are no gold standards in evaluation, and CBA has recognised limitations.

On the other hand, the field of program evaluation offers diverse approaches to determining value but rarely includes costs, let alone reconciling value added with value consumed.

The disciplinary divide between evaluation and economics holds us back from making better resource allocation decisions. What if we could integrate insights from both disciplines?

To address this question, I undertook some theory-building research.

First, I took a deep dive into the literature and developed a conceptual model proposing requirements for good VfM evaluation. I published it in the American Journal of Evaluation (I’m working on a summary of this article for a future Substack post).

Next, I assessed the extent to which CBA could meet the conceptual model’s requirements. Spoiler: gaps were found. Building on the conceptual model and gap analysis, I developed a series of theoretical propositions about better VfM assessment, for real-world road-testing.

Then I got practical. I developed a process model (you may have seen it), setting out a series of steps to operationalise the conceptual model. Good fortune brought opportunities to put the model through its paces in my consulting work. Theory fed practice, and practice fed theory, putting experiential flesh on the bones of the conceptual model. My research took longer than originally planned, because each cycle of pragmatic empirical theorising delivered rich learning and I didn’t want to rush it. The theory-building research might have been quicker without the interconnected consulting work, but it wouldn’t have been as good.

I unpacked two of the VfM assessments of real programs in case studies, which I analysed to assess the conceptual quality of the theoretical propositions. Then I took two months out from consulting, plus most of a summer break, to write it all up.

The research provided proof-of-concept for a practical theory to guide evaluation of VfM in public policies and social programs.

The research made seven significant and novel contributions to the field of evaluation:

1️⃣ VfM is an evaluative question (“good resource use?”), demanding a judgement based on evidence and logical argument.

2️⃣ VfM is a shared domain of two disciplines, because it’s concerned with merit, worth and significance (the domain of evaluation) and resource allocation (the domain of economics).

3️⃣ CBA isn’t a rival to evaluation; it is evaluation in the sense that it’s one way of implementing the general logic of evaluation. It evaluates an important dimension of VfM. Using CBA can strengthen an evaluation.

4️⃣ CBA isn’t the whole evaluation; it’s usually insufficient on its own because of limitations in its scope and warrants.

5️⃣ A stronger approach to answering VfM questions involves evaluative reasoning (using explicit criteria and standards) to make transparent judgements from evidence (gathered and analysed using a mix of qualitative, quantitative and economic methods according to context).

6️⃣ Program evaluation standards should guide economic evaluation, and this has implications for the way CBA is used including the nature and extent of stakeholder involvement, the use of CBA in conjunction with other methods, and decisions about when not to use CBA.

7️⃣ The approach to the case studies modelled the use of probative inference to corroborate the propositions of the conceptual model.

Well, that’s the short version.

If, for whatever reason, you wanted to read a 200-page version of the story above, my PhD dissertation is available here. For a more personal account of my backstory, see here. To cut to the practical chase, check out the guidance documents written with Oxford Policy Management (for the international development sector), Verian Group (for combining CBA with other methods in domestic public policy), Dovetail Consulting (for health and community services including contexts where economic methods of evaluation are out of scope), and over 100 articles right here on Substack.

To discover…

🐸 🐷 What the approach looks like in practice, illustrated in a fable with a star-studded cast of indigenous Muppet Frogs, colonial Muppet Pigs, Sam Eagle as an economist and Animal as an evaluator, see here.

🌍 What the approach actually looks like in practice, in real-world evaluations from around the globe, see here and here.

💎 A thematic directory of the topics I’ve explored on Substack so far, see here.

Thanks for reading!

☎️ Get in touch if you’d like to find out about training, mentoring, joining forces to tackle a project together, or developing your organisation’s evaluation systems and capabilities. It’s what I do for a living.

🧩 I’m looking forward to another year of sharing the Value for Investment approach on Substack, one puzzle piece at a time. Next week I’ll give you a heads-up on some of the future topics I have brewing…