Realist economic evaluation

Not just “is the intervention worth it?” but how, for whom, when, and why

Imagine a local health service pilots a diabetes management program.

The program improves health outcomes and reduces long-term healthcare costs by supporting patients with education, peer support, and regular monitoring. The pilot shows encouraging results, and now policymakers need to decide whether to invest in scaling it up across more neighbourhoods.

The program works very effectively for some people but less well for others. Costs vary too. The program could appear only marginally more cost-effective than alternatives if results are averaged across all participants.

But if we can understand for whom it is most cost-effective and how, we may be able to target resources where they will make the greatest difference - and perhaps trial different ways of working with people in different circumstances to make it more effective and cost-effective for them too.

This is an example of the kind of problem we might be better able to address if we use realist evaluation and economic evaluation together.

At face value, these approaches are as different as chalk and cheese. Can we integrate them? Should we integrate them? I think so. For some evaluations, under some circumstances (see what I did there?), it might help us get more useful answers to cost-effectiveness questions, and in turn help people make better resource allocation decisions.

Evaluations generally involve answering both causal questions and value questions. Part of the intrigue of mixing realist and economic evaluation approaches arises out of their complementarity: Realist evaluation is concerned with the causal part, and economic evaluation is concerned with the valuing part.

Realist evaluation is an approach to understanding causation.

Realist Evaluation is a distinctive approach to understanding how actions generate impacts in different contexts. The hallmark realist question is: For whom, under what circumstances, and why is the intervention effective?¹

Realist evaluation digs beneath the surface of whether an intervention “works” to explore how and why it works (or doesn’t) to varying degrees for different people in different contexts. Rather than assuming a simple cause-and-effect relationship, realist evaluation recognises that outcomes are shaped by complex interactions between the intervention, the people involved, and the context in which events unfold. Central to this approach is the idea of generative causation: outcomes arise when specific mechanisms are triggered in particular contexts, not just because an action was taken.

This notion is captured in the core analytic framework of realist evaluation: the context-mechanism-outcome (CMO) configuration. Evaluators develop and test program theories that specify which mechanisms (such as changes in reasoning, motivation, or resources) are expected to produce certain outcomes, and under what contextual conditions these mechanisms are likely to be activated. For example, a training program might boost skills (mechanism) and improve job performance (outcome) only in workplaces with supportive management (context). By systematically unpacking these CMO patterns, realist evaluation generates nuanced explanations that help decision-makers understand not just what works, but for whom, in what circumstances, and why. This richer understanding of causality can support better adaptation, scaling, and transfer of interventions in real-world settings.

Economic methods of evaluation are approaches to determining value.

They’re a distinctive set of approaches to valuing and comparing the costs and consequences of interventions. The archetypal economic evaluation question is: Is this intervention worth doing, bearing in mind its costs, its consequences, and the next-best alternative use of resources?²

Economic evaluations systematically assess and compare the value of interventions by quantifying both the resources required (costs) and the outcomes achieved (consequences), as well as the value that people or society place on those outcomes. At its core, economic evaluation asks whether an intervention is worth pursuing, taking into account not only its costs and effects but also what could be achieved with those same resources if used differently - the opportunity cost.

There are different types of economic evaluation, each offering a different lens for valuing interventions. For example, as I have detailed in previous posts:

Cost-Benefit Analysis (CBA) expresses both value consumed (costs) and value created (benefits) in monetary units, allowing direct calculation of net economic and social gain or loss. This method is useful when costs and benefits can be reliably attributed and monetised.

Cost-Effectiveness Analysis (CEA) calculates the ratio of costs (measured in money) to an outcome (measured in natural or physical units, such as life-years gained or cases averted), and compares this ratio with the costs and outcomes of the next-best alternative intervention. CEA is often used for health interventions where outcomes are not easily monetised.

Cost-Utility Analysis (CUA) is a specific form of CEA that incorporates measures of both the quantity and quality of life, such as quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs), enabling comparisons across interventions that affect both morbidity and mortality.

Causality and valuing are both important in evaluation.

Understanding causal relationships and determining the value of something are distinct and separate tasks, each with their own methodologies and logics. Evaluation needs both. Note, however, that realist evaluation is just one way of approaching causality, and economic evaluation is just one set of approaches to valuing.

While economic evaluation focuses on determining value, it also needs to address causality along the way. Realist evaluation has the potential to support a more nuanced approach to understanding causality in economic evaluation than the approaches typically used.

Conversely, realist evaluation focuses on its unique approach to causality and doesn’t concern itself with value - but it would be novel and useful if it did address value too (and if I was extra cheeky I might even venture to argue that it isn’t really evaluation unless it does address value).

Asking - and answering - novel questions

Harvey Specter, the irrepressible lawyer in the Netflix series Suits, said, never ask a question unless you already know the answer. Probably good advice for courtroom tactics, but useless for evaluation. For evaluation I prefer, don’t ask a question unless you’re going to actually use the answer.

I think realist economic evaluation potentially can provide useful answers to novel questions. In principle, if we combine realist evaluation with economic methods, we may be able to answer questions like:

How, for whom, and under what circumstances do different costs and consequences arise?

How, for whom, and under what circumstances is the investment worth doing on the basis of its costs, consequences and the next-best thing we could do with the resources?

Good answers to these questions could inform better allocation of resources and contribute to better value for money.

It’s not straightforward though

As you know, sometimes words have two meanings (Plant & Page, 1971) - and this turns out to be especially the case when we bring realist evaluators and economic evaluators together in the same room. To share just one example, to an economist, resources are the raw ingredients (monetary and non-monetary) that fuel an intervention. But to a realist, resources are elements provided by an intervention that actors (such as participants or stakeholders) can use or respond to, which then interact with their reasoning and context to produce outcomes.

There are significant philosophical and methodological differences between economic and realist evaluation too, which make genuine integration a complex challenge. The positivist roots of economic evaluation and the critical realist foundations of realist evaluation are not natural bedfellows.

For example, economic evaluations are typically quantitative, comparative, and summative - they focus on measuring or estimating costs and outcomes, relative to alternatives. In contrast, realist evaluations are theory-driven and explanatory, aiming to unpack the underlying mechanisms and contextual factors that shape how and why an intervention works (or doesn’t).

These differences play out in their preferred study designs. Economic evaluations, especially ex-post analyses, are a natural fit with experimental designs like randomised controlled trials (RCTs), which provide the kinds of data needed for quantifying outcomes. For many realist evaluators, however, RCTs are seen as unduly rigid or reductionist, unable to capture the rich, context-dependent causal processes that realist evaluation seeks to understand. That said, this view isn’t universal; there are even Realist RCTs that aim to provide nuanced evidence to inform policy and practice.

We evaluators take our pluralism seriously³ - and as far as pushing the boundaries of eclectic methodological pluralism goes, realist economic evaluation is an interesting test case.

There is a Realist Economic Evaluation Methods (REEM) Project.

The REEM Project is a significant initiative aimed at combining realist and economic evaluation approaches. The project seeks to develop practical frameworks and guidance that genuinely integrate these two traditions. The project has three phases:

• Phase 1: Mapping theoretical, methodological, and practical similarities and differences between realist and economic evaluation, and developing provisional guidance for integration.

• Phase 2: Piloting the guidance in real-world evaluations of complex interventions to test feasibility, strengths, and weaknesses.

• Phase 3: Synthesising lessons learned to refine and disseminate guidance for future use.

REEM aims to enable evaluations that are both context-sensitive and explanatory (hallmarks of realist evaluation) while also systematically capturing costs and consequences (the domain of economic evaluation). Rather than imposing a single, rigid protocol, REEM is conceived as a flexible framework that integrates theories of causality (realist) and value (economic), providing richer, more actionable evidence for policymakers and funders facing complex decisions in real-world settings.

You can check out the REEM project and follow its work here, which includes a glossary and interim guidance paper, with more publications to come. The REEM project is funded by the UK National Institute for Health and Social Care Research (NIHR) Health and Social Care Delivery Research (HSDR) programme.

Professor Angela Bate of Northumbria University recently presented on the REEM project at the 2025 UK Evaluation Society Conference in Glasgow and her slides are available here.

In summary

Realist economic evaluation is an idea with serious potential, now getting lift-off through the REEM project to answer not just whether an intervention is worth the investment, but how, for whom, when, and why enough value is (or isn’t) created to justify the costs. I’m excited to see where this leads and look forward to sharing future publications here as they come out.

Acknowlegements

Many thanks to Prof. Angela Bate for peer review. Errors and omissions are my responsibility.

I’m a long-time supporter of the REEM project, having participated in early collaborative work that subsequently led to its establishment. I am a member of the REEM International Interdisciplinary Advisory Group; however, due to geographical constraints, I have not been able to participate as actively as I had intended.

This post represents my opinion and not that of any organisation or group I work with.

Also see:

Eco-systemic frames in medical research (and evaluation)

Thanks for reading!

1. Pawson, R. (2013). The Science of Evaluation: A Realist Manifesto. Sage.

2. Drummond, M. F., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddard, G. L. (2005). Methods for the economic evaluation of health care programs. Oxford University Press.

3. As Michael Quinn Patton has noted, there are over 100 distinct evaluation approaches, each with their own theoretical foundations, methodologies, and priorities.* This pluralism means evaluators rarely agree on every aspect of practice or theory, and the profession continues to debate what constitutes valid evaluation in different contexts. While I value this diversity and debate, and I think it enriches the field’s intellectual rigour, I also think that if we spent as much time communicating the importance of evaluation to external audiences as we do to internally debating its definition and methods, we wouldn’t be “the largest profession no-one has heard of” (as John Gargani put it).

*Incidentally, I am highly chuffed to see that MQP included me in his video at 6:49 and a little disappointed that he associated me with cost-benefit analysis rather than Value for Investment more broadly. Still, no such thing as bad press, right?