This is a follow-up post to one I wrote last year about using ratios in value-for-money (VfM) assessment. It’s often assumed that ratios are easy to produce, and that they’ll tell us a lot about VfM - but in reality, there are multiple hurdles to meet, one of which is the comparability challenge. A ratio is just a number unless we have something to compare it with. Benchmarking or comparison with similar programs requires outputs or outcomes that are relatively homogeneous, access to comparator data, ratios calculated on a consistent basis for all comparators, and contextual differences accounted for. As I said in that earlier post:

That’s a stringent set of conditions to meet. But without them, ratios risk becoming an exercise in wishful thinking. The numbers may look objective, but without careful attention to context, comparability, and the underlying assumptions, we could be fooling ourselves.

In this post, I look at another tool that can support comparisons between programs: rubrics. In previous posts, I’ve explored different kinds of rubrics we can use for evaluation, ranging from generic to highly context-specific. Importantly, these approaches aren’t mutually exclusive. My go-to approach for assessing value for money (VfM) blends aspects of generic, analytic and holistic rubrics, as explained here. Today, I want to build on that foundation and tackle a thorny issue: can we use rubrics to compare different programs - and if so, how?

The trade-off between specificity and generality

Rubrics support transparent evaluative judgements by setting out explicit criteria (aspects of performance) and standards (levels of performance) that help us to make sense of evidence and support the validity of evaluative claims. One of the ways to enhance validity is to involve stakeholders in rubric development. Another is to make rubrics context-specific so they relate to the unique features of performance and value that matter in a particular program, at a particular time and place.

Rubrics are powerful in part because they reflect the values and priorities of stakeholders in a particular setting. Deeply contextualised, program-specific rubrics are good for clarity and support meaningful evaluation, but the flipside is that this strength can become a limitation when it comes to comparing performance between programs.

For example, the 2026 Magenta Book states that while rubrics offer flexibility to draw on a range of quantitative and qualitative evidence, “they will generally not produce a numerical value summarising the value for money of the intervention. This can limit the comparability of findings across different interventions”. This post represents a Cubist Evaluation of that claim: in my view, the Magenta Book’s statement isn’t completely wrong, but it isn’t completely right either.

The first thing to acknowledge is that such comparisons are inherently challenging, with or without rubrics.

Different programs operate in varied contexts, including geographic, demographic, cultural, and socio-economic factors, population needs, and local responses. Whether we’re comparing numerical or narrative data (or both), we need to identify contextual differences and take them into account.

Quantitative indicators like cost-efficiency, cost-effectiveness, and benefit-cost ratios can also struggle with comparability, as I’ve discussed before. Each cost-benefit analysis (CBA) design, for example, involves many analyst decisions (e.g., scope, perspective, time horizon, discount rate) and contextual factors that influence results. There are many ways to estimate monetary values of different impacts, leading to differences between studies. Differences in methods, data, study populations, and the types of impacts included (or omitted) mean that direct comparisons between studies require careful scrutiny.

The upshot is that benefit-cost ratios of different programs are often not as comparable as they may appear. Comparisons are most reliable when the analyses are based on equivalent methodological decisions like those noted above, occurred at the same time and place, and examined programs of a similar nature. The more they depart from these conditions, the more careful we have to be in making comparisons.

So, the comparability challenge isn’t new - and it certainly doesn’t disappear when we use rubrics.

Can comparisons still be useful?

Despite these challenges, comparisons (numerical or qualitative) can be valuable if we approach them thoughtfully. Instead of relying solely on tables of indicators, we need to engage with mixed-methods evidence and treat context as a crucial factor. This isn’t some sort of optional add-on to economic analysis - it’s absolutely crucial to make meaningful comparisons.

This is where rubrics can shine. They provide structured frameworks for deliberation, guiding evaluators to consider evidence carefully without forcing simplistic or inappropriate conclusions.

Designing rubrics for comparison: finding the right balance

Rubrics are tailored to fit the purpose of an evaluation. For example, we can design generic criteria and standards that can be used for any program, bespoke rubrics tuned to one program, or we can meet in the middle and develop rubrics that support comparisons in a specific context. Let’s unpack these three options a bit.

Option A: Generic criteria and standards that can be used for any program.

A generic set of standards can be as simple as:

• Excellent: exceeding expectations

• Good: generally meeting reasonable expectations

• Adequate: meeting minimum requirements and showing acceptable progress

• Poor: falling short of minimum requirements or acceptable progress.

These standards can be used with any set of criteria. To support like-with-like comparisons, a common set of criteria could be defined and used for all programs, such as:

• Economy: good stewardship of resources

• Efficiency: productivity and ways of working to maximise value

• Effectiveness: contributing to positive outcomes that meet the needs of the target group

• Equity: addressing inequities through fair and equitable allocation of resources, actions, and outcomes

• Cost-effectiveness: meeting its value proposition to key stakeholder groups.

These criteria can be applied to any program. They take relatively little effort to develop. However, they don’t offer a lot of guidance for making evaluative judgements, increasing the cognitive burden and the potential for wide variation at the synthesis stage, when deliberating on the evidence.

As I have argued previously, this approach may be appropriate and provide just-enough guidance when working in highly complex contexts where processes and outcomes are more emergent. Generic rubrics can also be suited to making comparisons across multiple programs. However, a strong moderation process is needed to ensure judgements are as consistent as they appear, and to ensure important contextual differences are properly considered.

Overall, any up-front specificity we can reasonably add to make these rubrics slightly less generic will be helpful when it comes to making well-formed judgements.

Option B: Bespoke rubrics for each program

If a rubric is designed for a single program at a specific place and time, it’s well-attuned to that context but not intended for any other context. However, if we know in advance that we want to compare programs or make judgements on a consistent basis, we can align each program’s rubrics with a set of generic criteria and standards like the ones above. This way, even when each program has its own bespoke rubrics, they still reflect the same underlying definitions for each aspect and level of performance.

This is the approach we described in Oxford Policy Management’s guide to assessing value for money: terms like excellent and good are used consistently across multiple VfM assessments.

Option C: Meeting in the middle

To facilitate comparisons across a portfolio of programs with shared objectives and features, we can strike a balance between the generic rubrics of Option A and the bespoke rubrics of Option B. Here, the rubric is tailored enough to be meaningful, and general enough to support valid comparisons across multiple investments within a portfolio or sector. Here’s an example developed for a set of market development programs.

The technocratic-to-deliberative continuum

As I’ve said before, there’s more to evaluative reasoning than rubrics. Rubrics don’t make evaluative judgements - people do. Technocratic approaches to synthesising evidence with explicit criteria and standards (such as multi-criteria decision analysis, if-then statements, rubrics, etc) can provide structure for making transparent evaluative judgements, but evaluation also involves a set of cognitive and social processes - including mixed reasoning approaches that combine deliberative reasoning for inclusive rigour, all-things-considered approaches to weigh alternative conclusions, and tacit judgement to surface considerations that may not have been made explicit. The balance of reasoning strategies will vary.

Similarly, we can conceptualise potential approaches to comparison along a continuum from technocratic to deliberative:

• Cost-effectiveness, cost-utility, and cost-efficiency ratios: These are highly technocratic - comparison “on rails.” They provide a ratio expressed as the average cost per unit of output or outcome (e.g., cost per student or per graduate), giving us a number we can use to compare options. Careful though: the scope and definitions of costs and outputs/outcomes need to be consistent across every intervention being compared. These methods use measurement and independent observation but still require judgement in design and interpretation, including attention to contextual differences. They can be used to compare across different interventions and health conditions, but work best when programs are similar in their objectives, populations, and contexts. This reduces the risk that observed differences in ratios are due to contextual or methodological factors rather than true differences in program value.

• Benefit-cost ratios: Not quite on rails to the extent of cost-effectiveness analysis, but still following a paved highway. Still fairly technocratic, but with extra room for analyst judgement because the benefit measure is more flexible - for example, in determining which benefits to value in monetary units, and how to place monetary values on them. Benefit-cost ratios may look objective, but they’re shaped by subjective choices. This can make comparisons problematic. Moreover, rather than aiming for a single ratio, cost-benefit analysis is often better viewed as a “voyage of discovery, promoting systematic exploration of the evidence on policy impacts” - i.e., scenario analysis, producing a range of results under different assumptions. Using CBA in this exploratory way is a key strength of the method, but does place further limits on comparability. It is not only rubrics that face such trade-offs!

• Tight rubrics (like Options B and C): Comparison on a well-marked offroad trail. The logic is clear, the scaffolding for evaluative claims is detailed, and the approach is transparent. People are making judgements, but those judgements can be decently comparable when the process is well designed and contextual differences are explicit.

• Loose rubrics (like Option A): This is more like bush-bashing - harder to find your way, but generic criteria and standards serve as landmarks. More deliberation and effort are needed to build a logical argument from evidence to judgement. Comparisons may still be possible, but as the scaffolding gets looser, the reasoning has to compensate with meticulous attention to how the argument is constructed and warranted.

Everywhere on this continuum, context is important, and a strong moderation process is needed to ensure comparisons are valid, judgements are made in a consistent way, and the logic underpinning the arguments is well documented. No method removes the necessity of careful deliberation and judgement.

Importantly, these options are not mutually exclusive. For example, rubrics can be combined with economic ratios, providing an explicit basis for taking wider contextual factors into account.

Conclusion: meeting in the middle

The primary challenge isn’t in using rubrics to compare programs, but in using any method or approach (whether numerical or qualitative) to make valid, meaningful comparisons, using a framework that supports like-with-like analysis while recognising contextual differences.

No matter how you slice it, comparisons are fraught. Numbers may help, but blind faith in indicators won’t. Ratios may look objective, but they require just as much care and attention to context as narrative evidence. The beauty of rubrics is that they help to make this deliberation intentional and transparent.

In the end, the best comparisons are those that are both rigorous and reflective, embracing complexity rather than glossing over it. That’s the value rubrics and collaborative deliberation can bring to the table. And that’s why rubrics don’t “limit the comparability of findings across different interventions” as the Magenta Book claims: on the contrary, they can support better comparisons when designed with this purpose in mind.

Thanks for reading!

Substack shifting into 5th gear

You might notice a small change in rhythm over the next while. For the past three-and-a-half years, I’ve shown up here almost every week with a longform post. I set myself that challenge as a way of writing a book, non-sequentially, one puzzle piece at a time. Now that the book is nearly complete, I’m shifting Substack to a slightly roomier cadence of roughly one substantial post every two weeks.

My existing and future Substack posts will continue to fill a niche in the VfI resource library (alongside my web resources). While the book will provide a stand‑alone foundation for VfI, Substack will still be the home for deep dives, case examples, ongoing developments and exploring new questions.

This change in output will give each new piece more space - more time for you to read, use and share it, and more time for me to make it worth landing in your inbox. There will still be the occasional extra note when something timely or curious pops up. But if you notice a little more breathing space between posts, that’s intentional - and, I hope, useful.

What struck me most wasn’t just the range of methods and case studies, but the shared feelings running underneath them. Evaluators seemed energised and hopeful about the field, yet also uneasy about the pressures we’re operating under and the limits of our current tools in the face of complexity and political volatility. Many described the conference as a much‑needed chance to reconnect with their people – a reminder that evaluation is a community of practice, not just a set of methods.

A big emotional thread was around identity and professionalisation. People were proud of evaluation as “definitely a thing”, but sceptical about narrow gatekeeping – more excited by “expanding the tent” and spreading evaluative thinking than by building walls around who counts as an evaluator. There was also real anxiety about quality and reputation in an unregulated market, especially for smaller organisations without specialist commissioning or analytical capacity.

Another feeling was restlessness about evidence use. Over and over, we heard that “the report isn’t the product” and that evidence doesn’t travel by itself; it only makes a difference when it’s connected to the right questions, people, processes and decision windows. That brought a mix of frustration (at shelved evaluations and performative reporting) and determination to design learning architectures – governance hooks, feedback loops, portfolio learning, knowledge management – that make evaluation part of the system’s operating model, not an afterthought.

AI was everywhere – and the emotional tone was “curious but cautious”. Sessions on AI‑assisted evaluations cut through both hype and doom, leaning towards “human‑in‑the‑loop” rather than AI‑run evaluations. People were intrigued by ways AI can support workflow and volume, but insistent that human judgement, framing, ethics and relationships remain non‑negotiable; hallucination/confabulation risks, governance, and proportionate, transparent use came up repeatedly.

Perhaps the deepest pondering was about what evaluation is for in complex, relational systems. Many challenged the idea of evaluation as neutral, external, purely evidence‑generating. Instead, they talked about evaluators as participants in systems, shaping what counts as value through what we measure, whose voices we centre and how we facilitate sense-making. There was strong resonance with:

• measurement as something that shapes reality, doesn’t just describe it

• value as socially constructed and contested, negotiated in relationship rather than settled by a hierarchy of methods

• the need to surface tacit forms of value – trust, belonging, psychological safety, collective agency – that standard metrics tend to flatten or ignore.

Closely linked was a hunger for more human‑centred, participatory and creative evaluation. Stories about peer support, lived experience, and community‑based work highlighted how top‑down, misaligned metrics can feel extractive, traumatising or simply pointless when imposed after the fact. Evaluators were pondering how to design proportionate, trauma‑informed, co‑designed approaches that fit the realities of practice, use creative methods, and make space for reflection and learning rather than just compliance.

Across sectors, people were also wrestling with uncertainty and politics. There was a sober recognition that institutions and guardrails can be weakened, that misinformation is rising, and that policy timelines often clash with the time needed for robust evaluation. Instead of wishing politics would slow down, many asked how evaluation can adapt – by focusing on timeliness and accessibility, building data and evidence infrastructure, working with empathy, and supporting decision‑makers to stay in relationship with uncertainty rather than collapsing into false simplicity.

Finally, there was a very practical concern with embedding evaluation cultures. From research culture to integrated care, people were asking: how do we move from one‑off projects to evaluation as a routine, relational, trusted practice that builds system‑level learning? Themes like co‑production, evaluative thinking, stakeholder buy‑in, proportionate methods, and capacity strengthening that outlasts individuals kept surfacing. Underneath the technicalities was something quite simple: a desire for evaluation that feels useful, dignifying and reparative – for practitioners, communities and decision‑makers alike.

So, how did my little AI friend do at summarising your sentiment? What did it miss?

Many thanks to the UKES Conference team and sponsors for a fantastic event.

And thanks to my excellent co-presenters:

• Jonathan Kuhn-Patrick on AI in evaluation; and

• Daniel Wate, Marion Guillaume, Patrick Ward, James Collis and Alex Hurrell on balancing fidelity and flexibility as VfI work scales.

See you next year!

Photo: Laura Hayward introducing the State of Evaluation Report 2026 at UKES 2026

Evaluative Practice, Spring 2026

The latest edition of the UK Evaluation Society’s publication, Evaluative Practice, focuses on the theme of evaluation for action. It includes my article on human‑led, AI‑enhanced evaluation, thinking through how we can draw on AI tools without outsourcing the human judgement that sits at the heart of good evaluation.

Also included are pieces on influencing change, evaluation design for use, team-based evaluation planning, working with young people and platforms like Snapchat, decolonial and systemic storytelling, AI and evaluation more broadly, and evaluation in local government. It’s a rich, very readable set of papers, and you can download the full issue here.

Thanks for reading

The refreshed Green Book and Magenta Book provide clearer, more consistent guidance on making balanced value-for-money judgements across the policy cycle. The Value for Investment approach offers a practical way to implement those principles, using transparent reasoning, mixed methods, and stakeholder engagement.

The UK government has updated two key guidance books for policy development that are influential internationally: the Green Book for appraisal and the Magenta Book for evaluation. Together, they provide a clearer and more consistent framework for developing, testing and reviewing public spending proposals across the policy cycle.

One of the most useful features of the refresh is the shared emphasis on balanced, evidence-based judgement. The 2026 Green Book defines value for money (VfM) as a “balanced judgement about the optimal use of public resources to achieve the objectives of a proposal”. It makes clear that this judgement must consider costs and benefits that cannot be easily valued in monetary terms as well as those that can, together with public sector financial impacts, distributional effects, risk, and uncertainty. The updated Magenta Book carries that logic into evaluation, where VfM evaluation asks whether an intervention has been a good use of resources, in light of what was achieved, for whom, how, why and at what cost.

The scale of this agenda is increasingly visible in practice: when the Government’s Evaluation Registry was launched publicly in March 2025, it already included information on over 200 VfM evaluations. In a little over a year, this has doubled to 450. The trend is even more striking looking at the pipeline. Just 15% of previously completed evaluations on the Registry include a VfM evaluation. That figure now stands at 40% with VfM evaluation for ongoing evaluations and 60% for planned evaluations – a fourfold increase compared to the historic figure.

This is a welcome development. The emphasis on balanced value judgements recognises that good public decisions cannot be reduced to a single number, however useful summary metrics may be. Benefit-cost ratios, cost-effectiveness ratios and other quantitative indicators remain important, but they need to sit within a broader account of public value, delivery, evidence quality and uncertainty.

This is where Value for Investment (VfI) can make a practical contribution. The VfI approach complements the Green Book and Magenta Book by offering a structured way of helping practitioners apply the spirit of the guidance in real-world settings, especially where evidence is mixed, outcomes are diverse, and some important dimensions of value cannot readily be valued monetarily.

A more joined-up view of value

The Green Book is primarily concerned with appraisal: assessing the costs, benefits and risks of different options before decisions are made. The Magenta Book is concerned with evaluating the processes, outcomes and value of policies, programmes and projects once they are implemented, and what can be learned for future decisions.

In practice, these tasks are closely connected. Appraisal sets out the case for action. Evaluation then tests how far those expectations were realised. That connection is especially important for value for money. A VfM appraisal asks whether a proposal is expected to represent a good use of public resources. A VfM evaluation asks whether it actually turned out to be a good use of resources, and what can be learned for future choices. The VfI approach helps join these questions together by making the evaluative logic explicit from the start.

The key idea is simple: if VfM is a balanced judgement, then practitioners need a transparent way to make that judgement. They need to define what matters, agree how good performance will be recognised, gather fit-for-purpose evidence, and show how the evidence supports the conclusion reached.

Interdisciplinary work, integrated work streams

Both the Green Book and the Magenta Book now point towards a more interdisciplinary way of working. Appraisal and evaluation require economic analysis, evaluation design, delivery insight, policy knowledge, financial understanding, data expertise and, where appropriate, the perspectives of users, communities and partners.

VfI is designed for exactly this kind of work. It brings together economic and evaluative thinking, rather than treating them as separate or competing traditions. Cost-benefit analysis, cost-effectiveness analysis and other economic methods can provide crucial evidence. VfI places them within a wider evaluative frame that also considers stewardship of resources, productive delivery, equity, context, uncertainty, and the values of different stakeholders.

This matters because many public interventions create value in ways that are only partly captured in monetary terms. Some benefits are intangible, long term, preventative, relational or unevenly distributed across places and groups. A purely monetised account may still be informative, but it may not be sufficient for a balanced judgement.

The updated Magenta Book recognises this in its set of principles for VfM evaluation. These centre on strong links between VfM evaluation and appraisal; coherence between VfM, process and impact evaluation; and building from empirical data to ensure that monetisable findings are interpreted in the context of a broader set of evidence. VfI provides a practical structure for bringing those strands together, so that VfM is informed by evidence from across the evaluation, rather than bolted on at the end.

Mixed methods as a strength, not a compromise

The new guidance is also clear that different questions require different methods. The Magenta Book and its extra methodological guidance (Annex A) set out a broad range of approaches. These include theory-based impact evaluation, experimental and quasi-experimental methods, cost-benefit analysis, cost-effectiveness analysis, social return on investment, Four Es frameworks, rubric-based approaches, synthesis methods, interviews, focus groups, case studies, surveys, monitoring data and consultative or deliberative methods.

This breadth is important. It avoids the unhelpful idea that there is one “best” method for all evaluation questions. Instead, it invites practitioners to choose methods that are fit for purpose: appropriate to the intervention, the decision context, the available data, the maturity of delivery and the kinds of value at stake.

VfI works in the same way. Rather than prescribing a single method or a fixed hierarchy of evidence, it asks what evidence is needed to make a credible judgement against agreed criteria, and then uses the most appropriate mix of methods to answer that question.

That might include a cost-benefit analysis where benefits and costs can be reasonably monetised. It might include a cost-effectiveness analysis where the objective is clear but monetisation is not appropriate. It might include process evidence to understand delivery, qualitative research to understand experience and context, impact evidence to test contribution or attribution, and distributional analysis to understand who benefited and who did not.

The point isn’t to collect every possible form of evidence, but rather to collect the evidence needed to support a defensible judgement. Mixed methods are valuable because they allow practitioners to examine a proposal or intervention from more than one angle, and to explain the story behind the numbers.

Making judgement visible

Perhaps the most important shared theme is transparency in judgement. The Green Book emphasises balanced appraisal drawn from clear evidence and assumptions. The Magenta Book emphasises useful, credible, robust and proportionate evaluation, with openness and transparency across planning, analysis and reporting.

This is an area where VfI is particularly well aligned. The backbone of the approach is the use of explicit criteria and performance standards, often expressed through rubrics. A rubric sets out what matters and what different levels of performance look like, for example what would count as excellent, good, adequate or poor value in a particular context.

Explicit criteria and standards provide a bridge from the evaluation to its practical use in policy making. They push the evaluation findings off the fence. Rather than a busy policy maker wading through an evaluation to reach their own conclusions in a hurry, evaluative judgements are shown upfront with transparent reasoning – including the strengths and limitations of the evidence.

Rubrics do not replace data. They sit alongside, providing a transparent framework to guide the selection of evidence and methods, interpretation of evidence, and making evaluative judgements. This is especially useful when evidence is mixed, when some outcomes can be valued monetarily and others cannot, or when different stakeholders place value on different things.

The Magenta Book Annex A now explicitly includes rubrics as one of the tools for VfM evaluation. It describes rubrics as a structured set of criteria and performance standards, usually co-designed early with evaluators and stakeholders, to provide a clear basis for integrating quantitative and qualitative evidence into an overall value for money judgement. That is very close to the practical logic of VfI, which treats the heart of evaluation as reasoning carefully from agreed criteria and standards to reach conclusions from credible evidence that are transparent and open to scrutiny.

Stakeholder engagement that strengthens judgements

The refreshed guidance also places appropriate weight on stakeholder needs and use. The Magenta Book describes good evaluation as tailored around the needs of stakeholders, including decision makers, users, implementers and the public. It also recognises that stakeholder engagement can improve understanding, build stronger relationships and increase the likelihood that evaluation findings are used.

VfI treats stakeholder engagement as an integral part of the evaluative endeavour, not as a separate communications exercise. Stakeholders can help define what value means in context, identify important forms of evidence, test whether the theory of change is plausible, and interpret findings in light of delivery realities.

This doesn’t mean that every judgement becomes a matter of preference or negotiation. On the contrary, well-designed participation can make judgement more disciplined. It helps surface assumptions, clarify trade-offs and make explicit why some dimensions of value matter more in a particular context.

This is particularly relevant for place-based, relational or complex interventions. In these cases, the people closest to delivery and experience may understand mechanisms, constraints and unintended effects that are not visible in administrative data alone. Their perspectives can strengthen both appraisal and evaluation, provided they are gathered and used systematically.

A practical scaffold for implementation

Many practitioners already know that balanced judgement, mixed methods and stakeholder engagement are desirable. The challenge is how to do these things in a way that is proportionate, systematic and useful for decision making. VfI offers a practical scaffold for implementing these principles. In broad terms, it involves the following eight steps:

This sequence aligns naturally with the emphasis on balanced value for money judgements set out in both the Green Book and Magenta Book updates.

Using this process, VfI can be implemented across whole organisations to navigate different contexts and requirements. The framework is designed to be applied at different levels of scale, from individual interventions to organisation-wide portfolios. A major national programme may need a substantial VfI framework, multiple methods and extensive engagement. A smaller local intervention may need a lighter version, with a narrower set of criteria, a proportionate evidence plan and a simpler rubric. Rubrics can be developed once at portfolio level and applied across multiple similar programmes, enabling like-with-like comparison while maintaining contextual responsiveness. The same principles apply in each case: the judgement should be explicit, based on credible evidence, proportionate, and useful.

Complementing the wider methods set out in the Magenta Book

It is important to be clear about what VfI is, and what it is not. VfI is not a single analytical method in the same sense as cost-benefit analysis, difference-in-differences or contribution analysis. Nor is it a substitute for good evaluation design, good data or appropriate economic analysis.

Rather, VfI is a wrap-around framework for bringing evidence and values together to answer value for money questions. It can be used alongside any of the methods set out in the Magenta Book. It helps practitioners decide how those methods contribute to an overall judgement about resource use, and it helps them combine multiple pieces of evidence:

This combination is often what rigour requires when decisions are complex and the evidence base is necessarily varied. The discipline lies in being explicit about the judgement being made, the standards being applied, the evidence being used and the uncertainty that remains.

Supporting test, learn and adapt

The refreshed Magenta Book also gives more prominence to evaluation as part of policy design, development and delivery. Its inclusion of Test and Learn guidance is consistent with a view of evaluation as something that supports improvement as well as accountability.

VfI can support this way of working. When criteria and standards are developed early, they can inform design choices, monitoring systems and learning questions during implementation. They can help teams ask not only whether an intervention is on track, but whether it is still likely to represent a good use of resources as circumstances change.

This is valuable in uncertain environments. Public programmes often operate in changing contexts, with evolving delivery conditions, emerging evidence and shifting needs. A transparent VfI framework can help teams revisit assumptions, test whether the value proposition still holds, and adapt without losing sight of the original objectives.

In this sense, VfI is more than a reporting tool. It is also a learning tool, helping teams connect appraisal assumptions, implementation evidence and evaluation findings over time.

From guidance to good practice

The refreshed Green Book and Magenta Book provide a clear opportunity to strengthen the way VfM is understood and assessed in UK public policy. They support a view of VfM as a balanced judgement, grounded in evidence and attentive to objectives, costs, benefits, risks, uncertainty, distributional effects and learning.

Value for Investment offers one practical way to support that ambition, throughout the policy cycle. It helps practitioners move from principle to practice by making the judgement process explicit, combining methods appropriately, engaging stakeholders constructively and integrating evidence across appraisal and evaluation.

The result is not a more complicated version of value for money. It is a clearer one. It recognises that public value is often multi-dimensional, that evidence is often mixed, and that judgement is unavoidable. The task is therefore to make that judgement as transparent, credible and useful as possible.

That is the promise of the refreshed guidance. VfI can help realise it.

References

HM Treasury (2026) The Green Book: UK Government Guidance on Appraisal. London: HM Treasury.

HM Treasury (2026) The Magenta Book: Central Government Guidance on Evaluation. London: HM Treasury.

HM Treasury (2026) Magenta Book Annex A: Analytical Methods for Use within an Evaluation. London: HM Treasury.

King, J., Wate, D., Namukasa, E., Hurrell, A., Hansford, F., Ward, P., Faramarzifar, S. (2023) OPM’s Approach to Assessing Value for Money. Oxford: Oxford Policy Management.

King, J. and Hurrell, A. (2024) A Guide to Evaluation of Value for Money in UK Public Services. London: Verian Group.

Thanks for reading

There will soon be a third: I’ve signed a book contract with Routledge, with the working title of Value for Investment: An Evaluative System for Government, Public Services, and Social Investment. I’ll continue to share updates on the journey to publication.

Today though, I’m sharing some real-world VfI examples that have recently been added to my resources page. Together, these additions say something about where VfI is being adopted and adapted. These public-domain examples are diverse, ranging from humanitarian innovation, through global funds and multi-country gender-based violence prevention work, to radioactive waste, school-centred community hubs, and city-wide arts investment.

At its heart, VfI is about providing better answers to value‑for‑money questions by combining economic and evaluative thinking, with public value at the centre. All of the cases below are recognisably working on that problem, in their own ways.

Humanitarian innovation: backing worthwhile ideas

Humanitarian innovation is risky, long‑term, and deeply intangible. Portfolios of small bets, weak signals, uncertain pathways to impact, and strong ethical obligations all mean the usual cost‑per‑output routine doesn’t get you very far.

Elrha’s guide to value for money in humanitarian innovation is a rich example of VfI principles adapted for a new domain. Elrha is a UK‑based charity that funds and supports humanitarian research and innovation, sitting in the space between big global funders and frontline responders. They work across multiple crises and sectors, backing ideas that could improve how the humanitarian system serves people in extremely tough circumstances.

VfM is tricky in humanitarian innovation for many reasons, including:

• Resource scarcity. Needs are rising faster than budgets, and trade‑offs are stark; VfM is always in the shadow of finite or shrinking resources and very high stakes.

• High uncertainty. Contexts are fluid, evidence is imperfect, and not every promising idea will land.

• The innovation paradox. There is tension between urgency (people need help now) and experimentation (we need to try new things), with the risk that costly failures may be borne by people in crisis, even as “failed” ideas can be valuable as learning for the system.

• Power and accountability. Decisions about what to test and scale are often made far from the communities affected; who gets to decide what counts as “value” is not a neutral question.

• The ethical imperative. In this context, VfM is not just an economic question; it is directly about ethical responsibilities toward people in crisis.

Elrha’s framework is explicit about this. It frames VfM assessment as a tool to support decision‑making (not as the decision‑maker), and explicitly situates VfM alongside other decision considerations in high-stakes settings.​

This framework is about making good choices when supporting innovation. Elrha starts by asking what good value looks like for humanitarian innovation portfolios: supporting learning, surfacing and pruning lower-value ideas early, and scaling promising ones. Value for money, in their hands, is something that should help innovation teams feel supported rather than policed. Ethics and do‑no‑harm are bottom lines, not trade‑offs.​

This guide does not lean on monetisation as the primary way to judge value, and instead prioritises structured, evidence-informed judgements about costs, benefits, learning, and risk. The guide is clear that the job is to produce a transparent, reasoned judgement, grounded in evidence and argument, not an illusion of precision.​

Elrha’s framework models core VfI principles, from starting with a clear value proposition, to using multiple kinds of evidence, and ending with a transparent, deliberative judgement, assisted by explicit criteria and standards. If anybody’s mental model of VfM is still “cost‑benefit analysis plus some hand‑waving about intangibles”, Elrha’s work is a really helpful counter-example.

Global scale and contested terrain: the Spotlight Initiative

The Spotlight Initiative is a multi‑year, multi‑country programme focused on ending violence against women and girls - involving multiple UN agencies, country variations, and a theory of change that runs from services to laws to social norms.

In this value for money assessment, we see recognisable VfI features. The assessment uses four criteria - economy, efficiency, effectiveness, and equity - to frame 20 areas of inquiry. Judgements are made using explicit performance standards (based on OPM’s guide) to differentiate excellent, good, adequate, and poor performance, and a performance language anchored in a complex theory of change on gender-based violence (GBV) prevention, response, and norms change. The criteria and sub-criteria go well beyond counting activities, and cover issues such as implementation quality, adaptability and responsiveness, innovation and piloting new approaches, learning and knowledge management, catalytic effects/leverage, and the integration of human‑rights and gender‑equality approaches in design and implementation.

In a programme like Spotlight, it would not be very meaningful to assess VfM based on ratios like “cost per case of violence averted”. The assessment is informed by mixed methods including case studies, survey data, interviews, planning and monitoring data, accompanied by a realistic treatment of the limits of outcome data across dozens of contexts. Costs still matter, but they are held up against a rich picture of change and value, not a single scalar outcome.

Spotlight is working in contested political terrain, across multiple countries and systems. In these diverse settings, their VfM framework shows how much further you can get by moving beyond cost‑per‑X ratios towards an explicit evaluative framework that supports transparent and traceable value judgements.

Multi-lens portfolio assessment: the Global Challenges Research Fund

The Global Challenges Research Fund (GCRF) VfM assessment applies evaluative reasoning and mixed methods in a large research funding context.

The GCRF is a £1.5 billion UK government funding stream that, over several years, channelled resources into research partnerships between UK institutions and low‑ and middle‑income countries. The aim was to support excellent research that also contributed to the Sustainable Development Goals, with a strong emphasis on equitable partnerships and Southern leadership. Once the initial waves of funding were committed and many projects were underway, the Department for Science, Innovation & Technology (DSIT) commissioned an independent VfM assessment to take stock at portfolio level and to ask how well the fund’s actual investments and practices lined up with its stated public‑value ambitions.

The VfM assessment of the GCRF features multiple lenses on value. The evaluation team developed a rubric‑based approach using an adapted 4Es framework with 14 sub-dimensions. Under effectiveness, for example, they looked at research quality and use, sustainable and equitable partnerships, enhanced challenge‑oriented capabilities, and pathways to impact. Equity was treated as a cross-cutting criterion, looking at who benefits, who leads, and how far the work aligns with the fund’s public‑value aims.​ For this sample, the cost-effectiveness dimension could not be assessed because of data limitations.

The assessment is explicitly portfolio‑level. It asks, across diverse awards – large research centres, networks, fellowships, applied projects – whether the portfolio looks like a good way of delivering on the GCRF value proposition. That is a much better fit for complex and diverse research funds than attempting to meaningfully rank every grant on a common marginal cost‑effectiveness yardstick.​

VfI goes underground: the Geological Disposal Facility

Radioactive waste is not the most obvious place to look for thoughtful VfM work - but the UK’s Geological Disposal Facility (GDF) programme is a strong exception.

The evaluation plan is explicit about uncertainty and public value.​ It starts by acknowledging “known unknowns”, overlapping phases, difficulty finding counterfactuals, and very long timeframes.​ It structures the evaluation in three mutually‑informing strands – implementation/process, impact and VfM – with contribution analysis and quasi‑experimental comparisons in the mix, plus simulation to update business‑case assumptions as evidence evolves.​ It explicitly recommends a Value for Investment approach, alongside Green Book and National Audit Office (NAO) guidance, because it provides a framework where both monetised and non-monetised benefits contribute to explicit evaluative judgements.​

Three features are worth highlighting:

1. Phased, adaptive evaluation: The plan lays out different evaluation questions and methods for different phases: early community engagement and consent processes; site characterisation and the Test of Public Support; design and construction; initial operation; and, much later, closure. At each point, it asks “what can we really observe and judge now?” and treats the business case as a living hypothesis to be updated.​

2. Contribution, not attribution: Impact work is framed in contribution terms, combining multiple lines of quantitative and qualitative evidence (community indicators, institutional changes, perceptions, economic modelling), with an explicit acceptance that attribution will always be partial. That is a healthy stance in any complex programme; it is essential in one that will run for decades and interact with countless other influences.​

3. Bringing VfI into the Green Book world: The Green Book provides the UK government’s core guidance on appraisal and cost-benefit analysis, and is clear that VfM is ultimately a balanced judgement that must weigh wider (including non‑monetised) considerations alongside benefit–cost metrics. The GDF team proposed VfI because the plan notes that important parts of the value proposition – such as community consent, trust, intergenerational equity, and environmental safeguards named in the business case and programme documentation – could not plausibly be reduced to monetised flows. VfI here is a way of integrating those dimensions with cost and risk information, in a coherent, transparent judgement over time.​

When the people responsible for burying radioactive waste are grappling explicitly with uncertainty, public value and structured judgement, it tells you something about where the mainstream is moving regarding evaluating in complexity.

Place‑based public value: schools as community hubs

VfI is not just for global funds and big infrastructure - it’s equally at home in local place‑based work. Two chapters from the book, Schools as Community Hubs, are a neat illustration.​

• In “An Evaluation Framework for Schools as Community Hubs”, Janet Clinton, Ruth Aston and Hayley Paproth adapted the CDC evaluation framework to the realities of hubs. They emphasised engaging stakeholders, making the theory of change explicit, focusing the evaluation on the right questions, and using mixed qualitative and quantitative evidence against clear criteria and standards, with equity and contribution built in.​

• In “Are Schools as Community Hubs Worth It?”, Aston, Clinton and Paproth took on the “is it worth it?” question directly. They walked through the familiar economic tools – cost‑benefit analysis, cost‑effectiveness, cost‑utility, cost‑consequence, social return on investment – and argued why, on their own, each is too blunt for complex, multi‑goal, place‑based work. They concluded by arguing for a Value for Investment approach with an explicit equity lens, using rubrics to define performance levels, and bringing multiple sources of evidence into a transparent value judgement.​

Schools as community hubs are, by design, trying to address “wicked problems” – clustered disadvantage, fragmented services, weak social capital – using the universal platform of a local school. They typically involve multiple funders, multiple partners, and layered ambitions (learning, health, wellbeing, community connections). In that setting, I would contend that:

• Single benefit–cost ratios tend to obscure more than they reveal.

• Equity isn’t peripheral to VfM; it is core to the definition of success.

• Contribution stories, backed by numbers where they’re meaningful, are a powerful way to make sense of change.

VfI is a natural fit.

City-level public value: arts investment in Tāmaki Makaurau

Closer to my home, Auckland Council’s Arts Matter evaluation shows what VfI can look like in city‑level arts and culture investments. The evaluation uses six case studies of council arts investment in different initiatives and venues to explore the types of value generated, and what this implies for future council investment and support.

The evaluators worked within a VfI frame: they examined whether Auckland Council’s arts and culture investments are a good use of public resources to generate social, cultural, and economic outcomes for Aucklanders, and how more value could be created. The cases span forms of support beyond funding alone, including staff resourcing, venue provision and subsidisation, and capacity-building, and they show how these elements combine to create public value.

Rubrics were developed with stakeholders to describe what “good” looks like for different dimensions of value:

• Stewardship of resources

• Building artistic, social and cultural capital

• Generating social and economic value.

Multiple forms of evidence, from interviews, focus groups, survey, document review, and observation, were brought together and weighed against the rubrics to reach explicit evaluative judgements about performance and opportunity. Economic evaluation methods were not used, but the evaluation qualitatively examined the role of the cases in fostering economic activity.

Framed this way, VfI at city scale is about how a mix of resources, relationships, and culturally responsive approaches work together to generate cultural and social value, and economic potential, across Tāmaki Makaurau. The Arts matter evaluation shows how local government can apply evaluative thinking about scarce resources, through the use of rubrics and participatory sense-making, in a way that fits the complexity and diversity of a metropolitan arts ecosystem.

Threads that run through these examples

Across humanitarian innovation, global GBV work, research funds, radioactive waste, school‑centred hubs, and city-level arts investment, a few common threads emerge:

1. From projects to portfolios and systems: None of these examples is really about isolated projects. They span portfolios (Elrha, GCRF, Spotlight), systems (school hubs, city arts ecosystems), or whole lifecycles (GDF). They showcase applications of VfI as a portfolio and system framework.

2. Explicit evaluative judgements: All of them are moving away from the idea that a single number can bear the weight of judgement. Instead, they are using rubrics and mixed methods to guide deliberations and reach structured, defensible judgements that make values, evidence and reasoning visible.

3. From “neutral” technocracy to public value: Rights, gender equality, equitable partnerships, Southern leadership, community consent, intergenerational equity, and place‑based wellbeing take centre stage – they’re not being treated as “soft” add‑ons. They are prominent in the criteria. That is what a public‑value lens looks like in practice. Value for money is more than efficiency alone.

4. From one‑off assessments to ongoing sense‑making: There’s a shift from “doing a big evaluation at the end” to phased, repeated assessments and continuous learning: Elrha’s emphasis on portfolio learning; Spotlight’s iterative cycles; GDF’s planned VfM assessments at multiple milestones; the hub frameworks’ focus on feedback loops.

All of this is recognisably VfI‑shaped. Hats off to the teams who undertook these projects, building in VfI principles and processes alongside other frameworks and tools to develop contextualised, fit-for-purpose evaluations in diverse settings.

If you have an example to add, drop me a line

If you’d like to explore these reports and more, my VfI resources page pulls together guidance, case examples and tools. I’m keen to keep the page up to date and as useful as possible, so if you have a publicly available example you’d be happy to share, I’d very much like to hear from you.

Thanks for reading!

If you stand in front of a good Cubist painting, nothing quite lines up. The bottle is both sideways and front‑on, the guitar is opened out and folded in, and the background seems to cut through the objects rather than sit behind them. It is one scene and many scenes at once. Early critics thought this was a mess, but Cubists argued they were getting closer to the truth by showing several perspectives on the same subject in a single composition.

In my earlier piece, Cubist Evaluation: an avant‑garde manifesto, I argued that evaluation can borrow from this spirit. Sometimes, people can look at the same thing and reach different conclusions. Cubist Evaluation is for those times. Instead of chasing one polished, definitive story of “what works”, we can deliberately fracture and reassemble different reads of evidence and different value stakes, hold multiple perspectives in play, and use that tension to see familiar problems in new ways. Cubist Evaluation treats deconstruction and reassembly as tools for exploring new views and asks us to honour multiple perspectives without assuming they should necessarily add up to one consensus verdict.

The use of AI tools in general practice is exactly the kind of subject that looks like one problem from a distance and many problems up close, which is why Cubism earns its place in the debate. On the surface, it might look like a technical question about safety, efficiency, and regulation. Underneath, it is also about whose time matters, whose risks are tolerated, whose data and ways of speaking fit the system, and what we think a “good” consultation should feel like. Reasonable people can look at much of the same evidence and come away with quite different, equally well‑argued conclusions about whether AI in a GP clinic is a blessing, a necessary compromise, or a step too far. That makes it a natural candidate for a Cubist Evaluation, because insight can come not from picking a single winning argument, not from forcing a consensus story, but from understanding how multiple different value stories interact.

For evaluators and people commissioning evaluations, this matters far beyond AI in health care. Versions of the same pattern show up whenever experts and stakeholders look at the same program, policy, or dataset and reach different conclusions about what’s going on and what should be done – from nutrition and public health to climate action, from free speech versus censorship to car parks versus bike lanes. In each case, there are multiple “right” answers - you don’t have to invalidate one argument to recognise the validity of another, but they exist in tension and resist easy reconciliation. A Cubist lens gives us a structured way to surface those competing lenses, work with them transparently, and support better‑grounded decisions instead of boiling reports down to single narratives.

A GP room with an extra listener

On a busy Tuesday morning in a suburban general practice, a GP is running an hour late and a patient with a vague mix of fatigue, breathlessness, and worry comes in. On the desk is the usual computer screen, but there’s also something else in the room: an AI system “listening” to the consultation, turning the conversation into structured notes, and suggesting next steps in real time.

The patient had already filled in an online form the night before, describing symptoms in their own words and rating their level of concern. That information has been processed by an algorithm that assigned a risk category, suggested appointment length, and flagged a couple of questions for the GP to ask. As they talk, the AI is drafting the consultation note, a referral letter “just in case”, and a brief summary the patient can take away.

Nothing ground-shaking happens in this consult. The GP is thoughtful and cautious, the patient feels mostly heard, the AI makes a couple of sensible prompts and one slightly odd suggestion that the GP ignores. At first glance it may look like a small, unremarkable step in the ongoing transformation of primary care. But on closer inspection, this ordinary scene is full of tensions about safety and speed, equity and convenience, human contact and automation, responsibility and trust.

How a Cubist lens can help

This is the kind of place where Cubist Evaluation might have something to offer, because it focuses not on determining whether AI in general practice is “working”, but on bringing the overlapping planes of evidence, values, and experience into view. In Cubist Evaluation, the objective is not to squeeze everything down to one right answer; it is to make the tensions visible and usable, recognising that some of them may be irreconcilable yet still crucial to grapple with.

What exactly are we evaluating here?

Before we get tangled in arguments surrounding AI in health care, it helps to be precise about the evaluand. We’re looking at the use of a specific and increasingly common tool: AI systems that sit alongside GPs in everyday primary care. That includes tools that:

• Invite patients to describe their concerns online before the appointment, classify urgency and route people to self‑care, nurses, or GPs;

• Generate prompts or risk flags during the consultation; and

• Produce notes, letters, or summaries while the GP talks to the patient.

Different countries, vendors, and practices implement this in different ways, but the basic idea is an extra layer of digital support that influences who gets seen, when, for how long, and with what informational frame. Even within a single practice, there often isn’t one stable model so much as a shifting mix of tools, workarounds, and role-specific habits, which means any rubric will land differently in different rooms and on different days. A Cubist stance treats those local variations as part of what we are evaluating.

From a Cubist perspective, you can think of this evaluand as the common object placed on the table, like the bottle or guitar in a Cubist still life. The rest of the debate is about the different angles, planes, and lenses through which that same object is seen, modelled, valued, and turned into policy and practice arguments.

We’re not judging the merit, worth or significance of AI in general practice; we’re not going to conclude whether its performance is “excellent, good, adequate, or poor”. We are exploring how different ways of looking, and different commitments about what matters, generate multiple valid perspectives that might not fit neatly together. The point of the exercise is to make tensions visible rather than rush to resolve them.

Why this is not a single-verdict question

If this were simply a matter of “does it work?”, a good mixed‑methods evaluation might be enough: run trials, collect implementation data, listen to patients and clinicians, then weigh the findings and deliver a verdict. But AI in general practice quickly escapes that frame. Different studies point in different directions, effects vary with context, and many of the things that matter - trust, dignity, efficiency, equity, professional judgement - matter in different ways to different people.

Different kinds of evidence sit on these planes, and they do not all belong in a universal hierarchy. Randomised controlled trials, model benchmarks, incident reports, qualitative interviews, routine data, practitioner narratives and patient lived experiences each tell us something about how AI is working in the clinic, and different communities place them in different orders of importance. A Cubist approach emphasises fitness for purpose of evidence and resists “gold standards”. It takes seriously the idea that what counts as credible evidence can legitimately differ depending on the topic and stakeholders, and that sorting out those differences is part of the evaluative work.

Even if everyone agreed that we had perfect data on safety, access, and cost, reasonable people would still disagree about how much weight to give each dimension and which risks are acceptable in exchange for which gains. A patient who has struggled for years to be taken seriously may see any tool that standardises questions and raises red flags as a long‑overdue safeguard, even if it feels impersonal. A GP in an under‑resourced clinic may welcome anything that frees minutes and headspace, while a colleague in a well‑resourced practice might worry more about subtle shifts in the erosion of professional judgement and autonomy. A policymaker under pressure to increase access might privilege throughput and coverage; an equity advocate might prioritise how the system treats those with the least digital access and the most complex lives.

In other words, the disagreement here is not just about what is true; it is also about which combinations of risks and benefits are acceptable in policy and practice, for whom, and at whose expense - a tangle of contested “oughts”. That makes AI in general practice not yet ready for a single, authoritative evaluative judgement, and a good candidate for a Cubist Evaluation that keeps multiple warranted judgements in view at once.

The Cubist Evaluation manifesto

Cubist Evaluation starts from the idea that complex interventions and settings can be depicted more like cubist paintings than like photorealistic landscapes. Instead of searching for a single “correct” vantage point on merit or worth, we assemble multiple planes of evidence and value side by side - different ways of seeing the same scene - and resist the urge to make an overall evaluative judgement. This is not for every evaluation but for specific occasions that call for this treatment.

In the manifesto, I explained how Cubist Evaluation takes its cue from Cubist art: instead of aiming for one picture of something, it deliberately fractures and reassembles evidence, values, and mental models to reveal more of what is going on. Just as Cubist painters showed their subjects from several angles at once, Cubist Evaluation treats value, evidence, and rigour as multi‑perspectival, and invites us to compose with several legitimate viewpoints at once. Underneath that is an ethic of mixed reasoning - technocratic, deliberative, all-things-considered, and tacit modes of judgement (Schwandt, 2015) interacting rather than letting any one of them claim the whole canvas.

In the GP room with an AI assistant, one plane of interest might be clinical safety: how often risk flags are correct or incorrect, which harms are avoided, which new failure modes show up. Another might be workload and access: how many appointments are freed up, who gets seen sooner, what happens to staff stress. A third plane might be equity: whose data the model fits poorly, who struggles with digital forms, who is filtered out. Yet another might be the texture of care: how it feels to be listened to with, and through, a machine. Multiple perspectives like these can exist in any evaluation. The difference in Cubist Evaluation is what we do with them.

Four elements of the Cubist Evaluation manifesto feel especially relevant in this GP setting:

• Contributing new understanding through deconstruction and reassembly. Here, that means pulling apart familiar stories like “AI will make primary care more efficient and accessible” into distinct value stories about safety, access, efficiency, equity, relationships, and responsibility, then recomposing them so the tensions between them are in plain view.

• Honouring multiple perspectives. Patients, clinicians, equity advocates, system managers, and vendors “value the same things differently”; they may use overlapping criteria but weigh them differently, and a Cubist approach treats that variation as important information.

• Challenging dominant narratives. Digital‑first and AI‑enhanced primary care are often framed as inevitable progress. A Cubist stance asks whose experiences and risks are backgrounded by that story, and how alternative stories - for example, from those at the sharp end of inequities and power imbalances - can be brought into view and examined without being dismissed as irrational, technophobic or uninformed.

• Thinking in multiple dimensions. AI in general practice lives across several dimensions at once: model performance on benchmark data, system dynamics in real clinics, lived experiences of patients and staff, and the shared sense‑making that happens in communities and professional bodies. Cubist Evaluation asks us to work with all of these, rather than elevating any one as “the whole truth”.

Underneath this is an ethic. Cubist Evaluation is pluralist, not anything‑goes: it invites multiple perspectives into the composition, but still expects judgements to be argued for, tested, and owned - more like working with a disciplined, limited palette than throwing every available colour at the canvas. A Cubist stance does not suspend judgement about which planes are better warranted; it asks us to make those warrants visible and discussable, instead of collapsing them into a take‑it‑or‑leave‑it package deal. The evaluator’s role is less like that of a judge handing down a verdict and more like that of an artist‑cartographer, composing a canvas where these different planes can be seen, compared, and argued with.

What does it look like when we paint the general practice AI canvas according to these principles? Let’s take a look at some of the facets that contribute to the picture.

Facet 1: Safety and clinical effectiveness

One obvious plane is safety and clinical effectiveness: to what extent does adding AI to the mix help GPs make better decisions, or create new ways for things to go wrong? AI tools can spot some risks that humans miss, nudge clinicians toward guideline‑concordant care, and provide useful differential diagnoses or prompts, particularly for common conditions and well‑represented populations. At the same time, we know they confabulate, and some confabulations can be hard to spot.1 There are documented concerns about automation bias (clinicians over‑trusting suggestions), poorly calibrated risk scores, and the difficulty of detecting rare but serious errors in real‑world use.

The evidence base here is not static; “the science” isn’t “settled”. Experimental studies and pilots sometimes show more optimistic results than real-world practice, and tools that perform well on benchmark datasets do not always translate into better patient outcomes in diverse primary care settings. Safety also looks different depending on where you sit: a GP who has recently missed a critical diagnosis may see any extra prompt as a net gain, while another worries that an overload of generic alerts will lead to “warning fatigue” and more mindless clicking. It matters how these tools are used, and how GPs and AI assistants check each other’s work. A Cubist treatment of this facet does not try to settle whether AI is simply “safe enough”; it lays out how different AI setups, different kinds of evidence (from trials to incident reports to clinician narratives), and different risk appetites pull this plane into different shapes in different contexts.

Facet 2: Access, workload, and system pressure

A second plane is access and workload: what happens to who gets seen, how quickly, and at what cost to staff when AI steps into the front line of general practice? Proponents point to very tangible gains: online triage systems and decision‑support tools can sort routine from urgent requests, smooth peaks in demand, reduce telephone backlogs, and help practices match appointment length to need, all of which promise shorter waits and more capacity. Clinicians and managers also report that well‑designed tools can take over some low‑value administrative work, freeing up scarce human attention for more complex patients.

At the same time, these tools can shift work rather than remove it, and not always in ways that feel fair or sustainable. Online forms can generate longer, more complex pre‑visit information that someone has to read; triage systems can surface latent demand and increase overall contact volume; and digital‑first pathways can leave receptionists and nurses managing a new layer of exceptions and fall‑throughs. For a GP in a chronically under‑staffed clinic, “increased access” can feel like more to do, differently distributed. For nurses and reception staff, AI‑mediated triage may mean a different pattern of interruptions and emotional labour than for GPs.

A Cubist evaluation of this facet resists simple binaries - for example, that AI either “eases pressure” or “burns people out”; it traces how access and workload effects play out for different roles and patient groups, and how those experience‑based readings sit alongside aggregate metrics like appointment counts and response times. It might borrow some mental models from Realist Evaluation by contrasting different context-mechanism-outcome configurations but it goes beyond Realist frames by focusing as well on how “what matters” can vary.

Facet 3: Equity and the inverse care law, rebooted

A third plane is equity. Digital and AI‑enabled pathways are often sold as a way to widen access, but in primary care they also risk replaying the inverse care law in a new key: those who most need care are often least able to navigate the systems designed to provide it. Online triage can be a gift for people with good devices, fast internet, high literacy, and confidence expressing themselves in writing; it can be a barrier for those with limited digital access, language barriers, cognitive challenges, or chaotic lives.

Bias can also creep in through the models themselves. AI tools trained on data that under‑represent certain groups may perform less well for them, misclassifying symptoms or assigning systematically lower risk scores. Even when algorithms are technically well‑calibrated, the way practices configure and use them can concentrate disadvantages - for example, defaulting certain types of patients to self‑care advice rather than live contact, or making it harder to bypass digital entry points. A Cubist view of this facet asks who gains, who loses, and how different communities, advocates, and clinicians interpret the same patterns of benefit and harm through their own histories and priorities. The question of who is actually benefiting, and on whose terms, echoes concerns from Critical Systems Heuristics about whose interests are being privileged or backgrounded when we design and legitimate a system.

Facet 4: Relationships, trust, and the feel of care

Primary care is not only about getting the right answer; it is also about how it feels to seek and receive care. AI changes the texture of that experience in ways that are hard to capture in metrics but matter deeply to patients and clinicians. Some patients report appreciating the structure and thoroughness of online questions, the sense that “nothing will be forgotten,” or the way written forms let them organise their thoughts before seeing the GP. Others may experience the same tools as cold, transactional, or even hostile - especially when they are distressed, fearful, or not sure how to fit their story into pre‑set boxes.

For clinicians, AI can feel like a supportive colleague or an unwelcome intruder. Ambient scribes and decision‑support prompts may free GPs to look up from the keyboard and make more eye contact, or they may introduce a constant sideways glance at the screen, a subtle sense that the “real” story is the one being written into the record. Trust is at stake in multiple directions: patients must decide whether they are comfortable with machines listening to intimate details; clinicians must decide how far to lean towards or away from AI suggestions; systems must decide how much to rely on AI‑generated artefacts when auditing performance or investigating complaints. A Cubist treatment of this facet brings into view the different ways patients, clinicians, and managers describe how AI affects the emotional and relational fabric of care, accepting that these accounts may remain in tension even when the technical performance looks satisfactory.

Facet 5: Responsibility, consent, and governance

A fifth plane concerns responsibility: when AI changes what happens in a consultation, who is answerable for the outcomes, and on what basis? In most pilots and deployments, the formal line is clear that clinicians remain responsible for clinical decisions, and AI tools are framed as decision support rather than decision makers. But in practice, the boundaries blur. If an AI‑driven triage system directs a patient to self‑care and their condition deteriorates, responsibility is distributed across designers, commissioners, practice staff, and the clinician who may never have seen the case.

Consent and transparency add further layers. Patients may be asked to agree to digital triage and ambient recording under time pressure, in unfamiliar language, or with limited understanding of how their data will train future models. Clinicians must decide how and when to disclose AI involvement in their reasoning, and organisations must choose how much to rely on AI‑generated notes and risk scores in governance, complaints, and performance management. Vendors also have stake in how responsibility is framed, since it shapes liability, product design, and marketing claims. A Cubist evaluation does not try to settle, once and for all, what the “right” allocation of responsibility should be. Instead, it lays out the different accountability narratives of regulators, professional bodies, clinics, vendors, and patients, and how each story affects the use of AI in everyday general practice.

So far we have sketched some of the planes in the picture; now we turn to the lenses people use to look at those planes.

Three value rubrics, three legitimate readings

Once we have these facets on the table, the signature move in a Cubist Evaluation is not to average them, or try to reconcile them or arbitrate them into an “overall judgement” but to study differences and tensions - to recognise and acknowledge that different value rubrics are at play, to surface those rubrics, lay different rubrics over the same canvas and understand how they re‑compose the scene. In practice, that also means recognising that the same tool can look very different at different points in the journey: an online intake form might be a net gain for some patients, while the in-consultation prompts may feel intrusive, and the auto-generated summary letter may land somewhere in between.

Here are three examples of stylised rubrics that people already use - often implicitly - when they talk about AI in general practice, and three different readings they could produce from the same underlying evidence and experiences.2

1. Efficiency and coverage rubric

Under this rubric, the primary question is whether AI helps stretched primary care systems do more good for more people with the resources available, with an emphasis on throughput, timeliness, and clinician workload. Gains in access, reduced backlogs, and reclaimed clinician time pull heavily in favour of adoption, while equity and relational concerns are noted but treated as issues to be mitigated through design tweaks and support. From this vantage point, a reasonable conclusion is that AI triage and decision support have high merit, provided safety is monitored and tools are iteratively improved.

2. Safety and equity rubric

Here, the priorities flip: the central question is whether AI reduces avoidable harm and narrows, rather than widens, unfair gaps in who gets timely, appropriate care. Evidence of mis‑triage, model bias, digital exclusion, and degraded experiences for already marginalised groups weighs heavily, while efficiency gains count only insofar as they do not exacerbate these patterns. From this vantage point, a reasonable conclusion might be that AI tools have at best mixed merit, acceptable only in tightly bounded use‑cases and in settings that can demonstrate robust equity‑by‑design and real‑world monitoring.

3. Relational primary care rubric

A third rubric puts the continuity, trust, and human feel of general practice at the centre. Here, the key questions are: does AI help or hinder deep listening; does it support or erode professional judgement; does it make patients feel more or less seen as whole people? Efficiency and safety gains are relevant, but they are not decisive if the technology systematically transactionalises encounters or undermines trust. From this vantage point, the same evidence might support a conclusion that AI in general practice has context‑dependent merit: potentially valuable where it clearly frees up attention for relationship‑building, but of low or negative value where it inserts itself between clinician and patient.

All three rubrics are intelligible, and all three can be applied in good faith to the same body of evidence. None of these perspectives are “Drunk Uncle Mike at the family barbecue”. They all have valid warrant claims, and the Cubist treatment has made those claims explicit and traceable, so that debates about AI in general practice can shift from “who is correct?” to “how does this look through different lenses?” and “whose values and risks have we been prioritising here, and why?”.

Of course, there are more than three rubrics, and these three examples are only sketches. In practice, people who broadly care about the same things still value them differently: they set different thresholds for “safe enough”, tolerate different trade‑offs between access and depth, and draw the line in different places on what counts as a meaningful erosion of trust. So instead of just three evaluative claims, a Cubist Evaluation could usefully surface a range and distribution of warranted judgements, and their associations with particular roles, contexts, and value priorities.

Staying with the tensions, on purpose

At this point, a regular evaluative reflex is to ask: so, adding it all up, is AI in general practice worth it or not? A Cubist stance resists that reflex. It treats the unsettled co‑existence of these different readings as data, and as an important feature of the reality we are evaluating, especially in domains where decisions affect many people in different ways and value commitments are plural.

Staying with the tensions is a choice. It isn’t an abdication of judgement when it is chosen intentionally, as a way of making judgement more honest and more shareable. Instead of pretending there is a single correct weighting of safety, equity, efficiency, and relational care, we surface the chains of reasoning that lead different actors to privilege one rubric over another and show how those chains sit against the same underlying facets. That, in turn, creates space for decision‑makers, practitioners, and communities to see where they align or disagree, where compromise or redesign might be possible, and where context‑specific choices (for this clinic, this population, this use‑case) are more appropriate than a universal verdict.

Let me be crystal clear here. Social media discourse often tilts toward false dichotomies where someone is “right” (and virtuous) and someone else is “wrong” (and malevolent), or toward a post-truth ambivalence where all opinions are treated as equally valid. At the other extreme, bad‑faith actors sometimes weaponise any disagreement to suggest that nothing is knowable and no action is justified. Cubist Evaluation is meant as a counter to these tendencies. It does not say that every opinion is just as good as every other; it pushes us to show our working - how different evidentiary criteria, value rubrics, risk logics, and warrant claims rotate the same object into different shapes - and to sense‑check those narratives in daylight. By fracturing a debate into multiple planes and then carefully reassembling them in full view, the aim is to turn heat into light: to make it easier for people who disagree to see how their reasoning fits and doesn’t, and where there is room for productive movement, rather than simply trading accusations or retreating into stalemate.

What a Cubist Evaluation adds, beyond “good evaluation”

Cubism was never just a visual gimmick. It was a revolt against the idea that a single, fixed viewpoint gives the most truthful account of reality. By breaking objects into geometric planes and showing them from several angles at once, Cubist painters asked viewers to work harder, to piece together a more complex, time‑infused sense of the subject from fragments that did not quite line up. The payoff, in their eyes, was a kind of higher accuracy: a representation that admitted the messiness of perception and perspective instead of pretending it wasn’t there.

Cubist Evaluation borrows that logic for value and evidence. Someone does not necessarily have to be wrong for someone else to be right. In complex, uncertain settings, different, partially overlapping ways of knowing and valuing are always in play, and attempts to compress them into a single seamless story risk hiding tensions that actually matter for decisions. Social choice theory makes a similar point in more formal terms: Arrow’s impossibility theorem shows that if we insist on a seemingly reasonable set of fairness conditions, no decision rule can aggregate individual rankings into a single collective ordering without violating at least one of them. There is, in other words, no friction‑free way to land on one uniquely right collective answer when values and perspectives diverge, whether in elections or in health policy. Cubist Evaluation offers a set of lenses for mapping that friction.

Three differences matter here. First, a Cubist approach refuses to hide the plurality of rubrics behind phrases like “on balance” or “overall, the benefits outweigh the risks”. Instead, it lays out several legitimate “on balances” side by side and makes clear who each one serves. In doing so, it also draws attention to the different evidentiary rubrics at work - what counts as “good enough” evidence for one group may not for another - and treats those differences as part of the story rather than something to be standardised away.

Second, it treats disagreement between stakeholders as a central part of the subject matter: the evaluation’s output is a structured map of tensions and varying judgements, not a single answer.

Third, by making chains of reasoning explicit, it gives decision‑makers and communities something they can actually argue with, adapt, and reuse in their own contexts, instead of presenting them with a verdict that appears more authoritative and settled than the underlying evidence and values really allow.

In complex and contested areas like this, those abstract points translate into some concrete design prompts for Cubist Evaluation:

• Treat multiple perspectives as a feature. Expect different stakeholders and experts to read the same evidence differently, and plan to elicit and work with those differences rather than smoothing them away.

• Make the lenses explicit. Surface the value criteria, evidentiary standards, and risk rules people are using, so disagreements are about planes in the composition - safety, equity, efficiency, relationship - rather than about what is the correct conclusion.

• Hold the tension long enough to learn from it. Triangulate across planes, and support stakeholders to deliberate and own whatever canvas of judgements they reach, instead of rushing to a single “overall” answer.

That is what it means, in practice, to move from Cubism as art history to Cubism as an ethic and approach for evaluation.

How this post enacts the Cubist manifesto

In my earlier Cubist Evaluation manifesto, I made a few bets about what rigorous evaluation in complexity requires. These are working hypotheses, not settled doctrine. This AI‑in‑general‑practice case gives those ideas a concrete case to work on:

• Multi‑layered evaluative reality. The “shared object” here, AI‑enabled triage and decision support in GP clinics, remains constant while our analysis fans out into several planes of disagreement about safety, access, equity, relationships, and responsibility.

• Multiple ways of knowing and reasoning. We see quantitative performance metrics, implementation studies, lived experience, professional judgement, and public narratives interacting, rather than any one of them owning the truth. Different stakeholders may place more trust in different pieces of evidence and if so, we map that too.

• Rigour as working through tension. Instead of picking a winner or averaging views, this post holds disagreement in view and makes the underlying lenses visible, so readers can see what each stance is doing with the evidence and values in play. Rigour comes from disciplined deliberation around those tensions. Conventional (e.g., post-positivist) notions of rigour are not shunned here, they may be part of the picture but not all of it.

• Design as choreography. The facets and rubrics structure is deliberate: a worked example of fracturing a familiar story and recomposing it as a kind of evaluative Cubist still life.

Philosophically, the stance is pluralist rather than anything‑goes. Different perspectives are invited into the composition, but they still need warrants. By unpacking those warrants, they become examinable and arguable; some interpretations may turn out to be better supported than others.

Questions for your own Cubist seeing

In Cubist painting, the work is not finished when the artist puts down the brush. Meaning emerges in the interaction between the fractured image and the viewer’s interpretive work. Cubist Evaluation treats judgement the same way. This post, therefore, isn’t a finished story about AI in general practice; it is just one arrangement of planes to invite your own sense-making.

Yet judgements still have to be made. Decision-making will go ahead, with or without the benefit of our judgements, and Cubist Evaluation will be nothing but a frivolous metaphor if it doesn’t land clear judgements that help people make better decisions in complex, contested spaces. A Cubist stance is particular about the form those judgements take (foregrounding incommensurable tensions) and how they are made (through collective interaction with the canvas).

With these aims in mind, here are some prompts to ponder how you see this case and what it does to your own evaluative practice:

• What do you notice first? Which facet feels most salient to you - safety and clinical effectiveness, access and workload, equity, relationships and trust, or responsibility and governance - and what does that reveal about your default lens on contested technologies?

• Where do you feel tension? Are there places where you find yourself wanting to collapse the planes, or dismiss one view as obviously wrong? What would it look like to stay with that tension a bit longer without abandoning your critical judgement?

• Whose knowledge, values, and interests are still outside this frame - for example, Indigenous peoples, rural communities, people with specific conditions or needs (e.g., mental health, physical disabilities), future generations? What might it take to bring them in, and how would bringing them explicitly into the composition shift what we come to see as credible, important, or just?

• How might this change how you work? Thinking about your own evaluations and the decisions they are meant to inform, where could you treat disagreement, discomfort, or competing narratives less as problems to be resolved and more as subjects to be deconstructed, mapped, grappled with and presented in new light?

• What would a “layered judgement” look like in your context? Instead of a single verdict, how might you describe “good for whom, on which dimensions, under what assumptions, and with which remaining points of contestation”?

• If you had to turn your current seeing into one practical move - a choice, design change, conversation, or tool - that would help real people navigate contested decisions in your own context, what would you do differently next week, and why?

The Cubist twist is that your reactions are part of this evaluation. The way you (and others) interpret, weigh, and argue over different planes is inextricably connected with the mixed‑reasoning process that Cubist Evaluation wants to put at the centre.

You may find parts of this piece uncomfortable. If so, I chose a good topic to illustrate Cubist Evaluation at work. The unsettling is part of the point. A Cubist Evaluation doesn’t pretend that every opinion is just as valid as every other, but it does commit to unpacking and stress-testing narratives (whether they are narratives one agrees or disagrees with), slowing down the leap from “I disagree with that conclusion” to “there is nothing to learn from that perspective,” and helping to make the lenses and value judgements doing the work more visible and discussable.

You do not need to come away from this deciding that a particular argument is right or wrong, or that nobody really knows enough to claim anything. In fact, if you did either of those, I would feel you’d missed the point. A pluralist Cubist stance asks for something more modest and more demanding: to notice how your own judgement is also a perspective, grounded in particular evidence choices and value priorities, unsettled and provisional material for deliberative, collective sense-making.

If you want to do this in your own system…

Not everything needs a Cubist Evaluation; it’s for those dilemmas, trilemmas or multilemmas where making sense of different perspectives in tension is the job, and where you make a deliberate choice not to arbitrate between them or resolve them into a single story.

Some evaluative questions need overall judgements, and some evaluations may benefit from adversarial, deliberative, or all‑things‑considered approaches. In those cases, Cubism is the wrong frame.

Where the Cubist treatment may come into its own is in those situations where people can look at the same thing and, working from the same body of credible evidence, reach different warrantable judgements. In that space, I am excited about the potential for Cubist Evaluation to help us navigate rigour in contested and complex spaces.

The work ahead is to design, test and learn together what Cubist Evaluation looks like in practice. That includes, for example, recognising when multiple epistemological and axiological perspectives are in play, clarifying specific Cubist use‑cases, purposefully designing for fragmented judgements, and presenting clear maps of the range and diversity of evaluative claims. Done well, those maps should leave room for end‑users to make context‑specific choices, with value trade‑offs out in the open.

I don’t have a finished Cubist method to offer; I have a manifesto, some prototypes, and a hunch that this is worth pursuing. I want to keep developing Cubist Evaluation as a transparadigmatic, mixed‑reasoning practice in real‑world settings, alongside others who are interested in pluralistic, power‑sharing approaches to evaluative judgement.

If you have a live project where people are likely to see the same evidence differently, and where there is value in making those divergent, yet defensible, judgements visible rather than smoothing them away, I’d love to hear from you.

Thanks for reading!

And thanks to Matt Healey for helpful peer review. Errors and omissions are my responsibility. Opinions are mine alone and don’t represent the people and organisations I work with.

E-bikes are a case study in strong value for investment, delivering both tangible (financial, time, health, environment) and intangible benefits (enjoyment, freedom) for the time, effort, and money you put in. Right now, with higher fuel prices and supply uncertainties, that value looks better than ever.

Backstory

An early adopter, I bought my first e-bike in 2015 with no business case. I just wanted it.

I loved it; it was powerful, well-engineered, and its rear hub motor provided an always-on, silent tailwind. But like a lot of e-bikes at the time, it only had a small battery and not enough range. Eventually, I realised this was just my gateway e-bike. It now has a happy new owner.

One fine day in Vancouver I rented a Specialized Turbo, and from that moment on I wanted one of my own.

Not long after, they released a new model and I was there with the readies. Here’s a photo when it was brand new and the rear tyre had just blown out when I was seeing how fast it would go 😶

This bike is now eight years old and still feels new. Lesson: quality remains long after price is forgotten. Value for money does not come from buying the cheapest.

A wise friend taught me that the right number of bikes to own is N+1 (he is not a VfM expert, but he does know a lot about bikes). In a weak moment, I took his advice to heart when I realised an e-gravel bike would make my life complete. To make room for it, I sold my regular pushbike, which I was no longer using anyway; e-bikes are simply more satisfying.

In case you need to justify your purchase on more rational grounds, I can offer the following six cases for e-bikes offering good value for investment. After eleven years and three e-bikes, here’s how the value stacks up.

1. Return on effort

A simple qualitative cost-benefit analysis (CBA) explains the first VfI case: On an e-bike, the benefit (speed and distance) for a given cost (physical exertion and time) is greater than that of a regular bicycle.

You still have to pedal. The bike multiplies the effort you put in, providing a positive return on exertion.1 The more you put in, the more you get out. It responds instantly and proportionately - just as Specialized used to advertise, “it’s you, only faster”. It incentivises and rewards effort, increasing enjoyment and encouraging you to ride more, which is at the heart of the value proposition: more reward for a given level of effort.

2. More exercise, better health

In case you haven’t heard, cycling is exercise, and exercise is good for you. For example, this systematic review found that cycling has benefits for cardiovascular health and all-cause mortality. It’s a magic pill. It turns out that e-bikes may even lead to more net exercise than the old pushie (or “acoustic”), making them an extra-strength magic pill.

According to this research, e-cyclists put in less effort per kilometre than acoustic riders, but still maintain a moderate intensity heart rate and oxygen consumption, consistent with beneficial exercise.

Meanwhile, the increased enjoyment of more speed for less effort may nudge you into getting more net exercise. I can attest that I’m spending more time on a bike than I used to. What’s more, a good hard sprint on an e-bike is sensational and depletes glycogen just as well as a pushbike.

Incidentally, exercise is good for the brain too. Some of my best problem-solving moments have happened on rides. I thought up this article while hurtling along on two wheels and thumbed the first draft into my phone when I got home, under the influence of dopamine (does it show?).

If e-bikes incentivise enough of us to ride more, there’s a population health benefit.

3. The financial case

According to this calculator, the ride I do most often instead of driving saves me about $30 per trip (NZD). It’s only a rough estimate based on averages, but the underlying point is valid: the savings can stack up. You really need to run your own numbers because everyone’s case is different. Here are a couple of ways to approach it:

1. Having an e-bike instead of a car: This can save significant money from day 1. Both e-bikes and cars have a wide range of prices, and those ranges overlap - but on the whole it’s fair to say you can get a good e-bike or cargo bike for less than a car of equivalent quality. If you can switch out your car (or second car) for a bike it’s a financial no-brainer: lower purchase/finance cost, lower maintenance costs, lower running costs.

2. Having an e-bike and keeping the car: Even if you still need a car, an e-bike can pay for itself over time by reducing how much you drive. It depends on individual factors like your choice of vehicles and how you use them.

Let me give you a case example. I live in Auckland, a sprawling city with the beginnings of decent cycling infrastructure. The bike paths don’t always go where I need to go (neither does public transport), and there are times when it doesn’t feel safe or convenient to cycle. As much as I’d love to ride everywhere, the fact is I have chosen to keep my car, so the bike costs are extra.

Still, every time I use the bike instead of the car, it saves on running costs. What would it take for those savings to cover the full cost of the bike?

To work that out, we first need to estimate the marginal cost of driving versus cycling: the amount I pay to drive one extra km or save by cycling that km instead. The marginal cost ignores the fixed costs of ownership and counts only the extra bit I pay to use the vehicles, given that I already have them - mainly fuel/electricity, plus that little bit closer it brings me to needing new tyres, brake pads, etc.

I have estimated that the marginal cost of driving my car is about 38 cents per km, and the marginal cost of taking the e-bike is 0.1 cent per km. So every km I travel on the bike instead of the car saves me a net 37.9 cents, gradually (very gradually) offsetting the cost of the bike.

On that basis, to fully recover the costs of owning and running an e-bike, I would need to reduce car usage by around 60km per week for ten years (note: it’s not enough just to ride that distance - these have to be km that would otherwise have been travelled by car).

That seems achievable. It is a satisfying mind-game and a mild extra incentive to ride more, because every time I ride instead of driving it makes the bike a little bit cheaper and brings me closer to breaking even (getting the bike ‘for free’ in the sense that I would otherwise have spent the same amount of money on petrol).2 There is also comfort in knowing that a chunk of my local travel runs on a small battery and a standard power socket, whatever is happening in global oil markets.

4. The environmental case

When you swap out a car for an e-bike trip, you’re incrementally reducing both your CO₂ emissions and the fine particles that end up in everyone’s lungs and our waterways. Road traffic particulate pollution comes from two main sources: exhaust from internal combustion engines, and non-exhaust sources like small pieces of tyres and brake pads that wear off as we drive any vehicle. Those non-exhaust sources are becoming an increasing share of the problem as vehicle weights increase, and recent studies in big cities suggest they already account for most traffic-related PM10 and a large share of PM2.5.

E-bikes help on all of these fronts. When your car stays home, so do the pollutants. A bike plus rider weighs a tiny fraction of a car, so there is far less energy to shed every time you slow down, which means far fewer brake particles and less road and tyre wear. Multiplied across many trips and many riders, there’s a meaningful reduction in the fine dust that worsens respiratory and cardiovascular disease, especially in dense urban areas.

The more you ride, the more you amortise the one-off emissions from producing the bike and its battery, so each substituted car trip makes the environmental return on your e-bike a little bit better.

We can estimate monetary values for these environmental benefits. Add them to the population health and financial benefits, and the bike is even more likely to pay for itself.

5. The congestion case

E-biking is often quicker than driving for the routes I choose to cycle. I can say that with confidence because I compared stats back in 2016, tracking my average speeds over a few weeks on comparable trips. The bike computer showed an average speed of 28kph while the car computer showed 16kph for trips with the same origin and destination. If I updated the comparison, I would expect to see an even greater difference because the newer bikes are faster, while Auckland traffic is more congested than ever.

Time is money, and we can work out a monetary value of getting places quicker by e-bike. Add that to the financial and environmental case and the bike’s benefits just keep on growing. Our transport planners do this all the time when justifying new roads.

But travel time is more than just money, as I explored in an earlier article: a study found each hour on a bike to be more worthwhile travel time than an hour in a car, in terms of the average value people place on it. Nice to know it’s not just me.

There’s a mild altruistic argument too: e-bikes aren’t practical for all people’s needs, but each person on a bike instead of in a car slightly reduces traffic congestion for everybody else.

6. The happiness case

Happiness is partially factored into each of the cases above. That surge of electric torque always brings a smile. Riding makes me healthier, and a healthier me is a happier me. I’m also happy that I incrementally reduce the cost of the bike every time I take it out instead of the car. Sitting in congested traffic is no fun, but I must confess that riding alongside a motorway during rush hour and overtaking lines of cars brings a unique kind of happiness.

But if I’m brutally honest, the five cases above are rationalisations. In VfI terms, they are valid, but not decisive. I could use those arguments to justify my decisions to others, but they don’t really explain why I have an e-bike. To understand that, there’s a sixth, more intangible and more important case we need to take into account. Above all of the rational analysis is the happiness factor. An e-bike just is a happy place to be, even if (especially if) I have no particular place to go. That everyday joy gets to the heart of the e-bike’s value to me, hard though it may be to include in a spreadsheet.

In summary

E-bikes give an instantaneous positive return on effort, make you smile more, live longer, get places faster, and (if they substitute for car trips) their cost decreases with each pedal revolution. How’s that for good value for investment?

Thanks for reading!

This is a refreshed version of a post from two years ago. If you’re a long-time reader, thank you for sticking around, and I hope the second ride was still worthwhile.

There’s a huge variety of e-bikes to meet different needs and I recommend taking your time to understand the different options, go for lots of test rides, and choose one that meets your needs and budget. You don’t have to spend a fortune on an e-bike but it’s worth paying for quality - cheapies can be unreliable with uncertain supply of parts and service. Here’s one website to get you started.

A brief history

I’ve learned a lot from John Gargani over the years, including comments he shared in jest that stuck with me. Among these, that the best ideas are stolen ones, and that there are many different kinds of evaluation - realist, constructivist, feminist, pragmatist… but not yet Cubist. So it felt only fitting that I should steal John’s idea and try to develop Cubist Evaluation into a genuinely novel and useful approach.

The first opportunity arose at the 2023 Australian Evaluation Society Conference in Brisbane, when Kate McKegg invited me to join a panel on Adapting and embracing new conceptions of rigour for evaluation practice in complexity. The idea for the panel came out of a blog post that Kate had co-authored with Keira Lowther, which you will find here.

In my contribution to this panel (slides included here), I floated Cubist Evaluation - noting that the early 20th-Century Cubist art movement represented a major shift that had enduring influences on art, architecture and film, and arguing that evaluation can draw inspiration from Cubism too. In a subsequent post, which you can read here, I developed the idea some more. This post distils and deepens the key principles from both and responds to a question I was asked more recently: how is it different from bricolage?

What is Cubist art?

Emerging in the early 1900s, Cubism, pioneered by artists like Pablo Picasso and Georges Braque, broke radically from traditional perspectives. Instead of depicting a scene from one fixed viewpoint, Cubist painters fractured objects into geometric forms and reassembled them to show multiple angles at once. In the early Analytic Cubism phase, they often did this using a deliberately limited, almost monochrome palette, so that attention stayed on the structure of planes and relationships rather than colour. The Cubist approach challenged the assumption that reality is singular or static, opening new possibilities for how reality can be seen, represented, and interpreted. See here and here for examples.

Beyond technique, Cubism was a philosophical shift: it invited viewers to engage in active interpretation, recognising that meaning arises through perception and context. This multi-perspectival philosophy of seeing and interacting with the work is what Cubist Evaluation seeks to evoke - an evaluative practice that deconstructs, reassembles, and collaboratively reimagines value while holding onto important nuance.

What is Cubist Evaluation?

Cubist Evaluation is a manifesto for rigorous evaluation in complexity, setting out a provocative (I hope) set of principles for evaluators to argue with, adapt, and extend.1 It is not a prescribed method, tool, or framework, and at this stage is intentionally incomplete – a nascent idea with room for development.

At heart, Cubist Evaluation makes four bets. First, in complex settings, evaluative reality is multi-layered and not well captured from a single vantage point. Second, evaluative judgement is stronger when it draws on multiple ways of knowing and reasoning, rather than hoping one is enough. Third, rigour is not the tidying away of tension, but the disciplined working-through of it with the people whose lives and interests are at stake. Fourth, Cubist Evaluation is a way of designing evaluations so those commitments show up in practice, through a purposeful choreography of methods, perspectives, and reasoning.

Cubist Evaluation argues that rigorous evaluation in complex environments needs more than any one perspective or technique. It calls for treating a diversity of viewpoints as epistemically and axiologically important, and aims to contribute new insight through deconstruction and reassembly so that subjects which may have been taken for granted can be usefully viewed from new angles. Just as Cubism opened up new possibilities in art, Cubist Evaluation aims to stretch evaluative practice - in particular, by blending mixed reasoning, eclectic methodological approaches, and collective sense-making, and moving beyond mechanistic evaluation frameworks to make sense of impact and value in richer ways.

How is it different from bricolage?

I’ve been pondering that since Matt Healey and Tenille Moselen put that question to me in their podcast. Bricolage and Cubist Evaluation both seek to disrupt narrow, technocratic forms of evaluation that can marginalise lived experience, minority viewpoints, and methods that, for whatever reason, have lower status in mainstream policy discourse. They share a commitment to methodological and paradigmatic pluralism, flexibility, reflexivity and attention to context. Both approaches draw on a variety of methods to create richer, more nuanced understandings.

However, where bricolage often emphasises the creative, context-specific assembly of what is at hand, Cubist Evaluation builds more explicitly on a set of principles that echo Cubist art – deconstruction, multiple perspectives, unsettling dominant narratives, and rethinking how meaning is made. Put another way, bricolage tells us something important about how evaluators can work; Cubist Evaluation is additionally trying to say something about how evaluative reality is depicted and how judgements are composed. Its added claim beyond methodological flexibility is the deliberate refusal of a single vanishing point on value.

Cubist Evaluation principles

Cubist Evaluation translates the central themes of Cubist art into a set of working principles for evaluative practice. Just as Cubist painters showed their subjects from several angles at once, these principles approach value, evidence, and rigour from different but connected perspectives, each revealing something distinct. Together, they invite evaluators and stakeholders to purposefully deconstruct and reassemble value stories, hold multiple viewpoints in play, and experiment with new ways of making sense of impact, merit and worth in complex, contested settings.

1. Contributing new meaning through deconstruction and reassembly

Cubist art fractured and reassembled objects to offer new ways of viewing things. Cubist Evaluation applies the everyday practices of analysis and synthesis as tools for fracturing and reassembly, deliberately using them to disrupt how value is depicted and communicated, and inviting people to see familiar things in new ways.

For example, instead of simply reporting who benefits and who does not, a Cubist Evaluation of a national employment program might pull apart its core assumptions – the efficiency case, the equity case, the political narrative, and the diverse lived experiences of participants – and treat each as a distinct, and sometimes competing, value story, rather than aiming for a unified verdict. In a sense-making workshop, stakeholders might then reassemble these fractured pieces into an account that openly acknowledges how the program can both entrench precarity for some groups and create opportunity for others, forcing a redefinition of what “success” should mean and what must change in the next policy iteration.

2. Honouring multiple perspectives

Rather than limiting depiction to a single viewpoint, Cubist Evaluation, like Cubist art, brings together diverse vantage points and treats each as a legitimate lens, revealing the subject in a broader context. In evaluation, this is about multiple voices contributing to the evaluation discourse. Different people experience and “value the same things differently”. Cubist Evaluation recognises that variation as important information, and asks us not to assume any single perspective is sufficient to disclose the full evaluative picture.

Imagine an emergency department initiative where the main visible success story is a waiting‑time target. The headline objective is to bring average wait times under four hours. A Cubist Evaluation would bring together the perspectives of patients and families, clinicians in different roles, managers, and system‑level data on flow, safety, and equity, and treat all of these as legitimate lenses on value. In a shared sense‑making process, people might surface that shorter waits have come at the cost of rushed conversations, poorer continuity of care for some groups, and increased burnout in parts of the workforce, even as headline targets are achieved. Honouring multiple perspectives here means noticing how easily any single one – whether it is the target, the clinician view, or the patient story – can stand in for the whole, and instead working with stakeholders to craft a judgement that does justice to all of them.

3. Challenging dominant narratives

Cubists bravely challenged prevailing approaches to painting. When they sliced and rearranged reality, they were intentionally unsettling what counted as a “correct” way of seeing. Instead of aiming for idealised or photorealistic paintings of nature, they showed things from multiple angles at once, abandoning Renaissance rules of perspective to reveal more complex views.

In evaluation terms, this principle is about how we work with power in valuing - whose definitions of “good" set the terms of debate, and whose experiences are sidelined when one view dominates. The deconstruction and reassembly of evidence and values described in Principle 1, and honouring multiple perspectives described in Principle 2, are enacted in ways that deliberately trouble the taken-for-granted story about what “good” looks like. Challenging dominant narratives isn’t just for fun; it pushes back against unquestioned assumptions that can marginalise people, ideas, or inconvenient pieces of evidence in complex settings. In that sense, this principle is not only interpretive but ethical, because it asks who gets to define what counts as good, successful, or valuable in the first place.

For example, imagine a national digital inclusion initiative, described and funded under a simple banner of “more devices and connections = success”. A Cubist Evaluation would name that as the dominant narrative and then pull it apart. It might distinguish the infrastructure story (households connected), the capability story (who can actually use the technology with confidence), the equity story (who is still excluded or harmed), and the lived‑experience story (what online access changes, whether intended or unintended, and what it doesn’t change in people’s daily lives), treating each as a distinct, and sometimes competing, value story rather than privileging one headline about coverage.

The evaluative work is to keep those tensions visible and renegotiate what counts as success – for example, by adopting a multi‑dimensional theory of value that balances coverage, affordability, accessibility, safety, data sovereignty, and cultural fit – rather than allowing a single performance story to dominate.

4. Thinking in multiple dimensions

Early Cubist painters experimented with more than three dimensions of reality on a flat canvas – fusing different angles, moments, and even the “in‑between” spaces into a single, simultaneous image, suggesting a world that is spatial, temporal, and psychological all at once.

If objectivity means observing individual, external phenomena, then Cubist Evaluation says we need more dimensions than that. It can be useful to distinguish at least four dimensions, including: inter-objectivity (conceptualising systems, networks, technology, government, and the natural environment as collective, external phenomena); subjectivity (individual, internal observation of thoughts, emotions, memories, perceptions, and states of mind); and inter-subjectivity (shared values, meanings, language, relationships, cultural backgrounds, ways of knowing, and collective-sense making).

For example, a Cubist Evaluation of a youth justice initiative might bring together rearrest rates and cost data (evidence that fits the above definition of “objective”, while acknowledging that measurement systems are inescapably value laden), alongside patterns in how different agencies and systems interact (“inter‑objective”), young people’s own accounts of safety and dignity (“subjective” lived experience), and a facilitated dialogue about what “justice” should mean from the whole community’s perspectives (“inter‑subjective” development of value). Instead of treating any one of these as the definitive reality, it would deliberately use all four dimensions together - like a Cubist composition that only comes into focus when you take in all its planes - to inform decisions about which responses are preferred overall.

The four dimensions used in this example are offered not as a definitive typology, but as a practical illustration of the broader claim that evaluative reality is multi-dimensional, the mental models we use to make sense of the world are choices, and using several models instead of just one can support situationally appropriate and rigorous evaluation.

5. Opening up new possibilities

Where the first four principles focus on how we could approach an individual evaluation (how we analyse and synthesise evidence and values, whose perspectives we bring into view, which stories we unsettle, and how we mix conceptual models), this one is about how that thinking can diffuse into wider evaluation practice.

Cubism revolutionised artistic expression. It offered radically new ways of depicting reality, breaking from earlier artistic conventions and influencing painting, architecture, film, and later experimentation with abstraction. In a similar spirit, combining varied approaches to evaluation can widen the palette of options available to evaluators and evaluation users, helping them imagine new possibilities, make richer sense of what is happening, and communicate value in fresh ways.

6. Disrupting how meaning is made

In Cubist painting, the artist does not offer a single, polished view of reality; they rearrange the subject so that the viewer has to make sense of it, actively piecing together multiple angles into their own account of what is going on. Meaning is no longer delivered as a finished product from canvas to audience; it emerges in the interaction between the painter’s choices and the viewer’s interpretive work.2

Cubist Evaluation treats evaluative judgement in a similar way. Instead of the evaluation team retreating to produce a closed verdict, they involve stakeholders as co-creators of meaning - interpreting what the evidence means together, weighing different value stakes, testing whether judgements feel right and debating those that do not, rather than having them handed down as a single, authoritative conclusion. This shared interpretive space is where the real evaluation happens.3

The evaluative judgements that emerge are collectively reasoned and contestable. If there are genuine tensions, ambiguities, or contradictions, the judgements will be brutally honest about them, treating them as legitimate features of the landscape that the evaluation must make room for, document, and sometimes leave partly unresolved. Rather than concluding "good value overall” you’re more likely to get a layered account such as “good value for some groups on some dimensions, and contested or uncertain on others - and here is how different stakeholders weigh them.”

Just as Cubist paintings refuse a central vanishing point and ask viewers to navigate a field of intersecting planes, Cubist Evaluation refuses a single vantage point on value. The work of evaluation is to host and provoke interpretive labour, designing deliberative processes to make sense of mixed-methods evidence through mixed reasoning - intentionally combining multiple ways of thinking (systematic, social, intuitive) to turn evidence into shared, defensible judgements.

Does this add anything new?

None of this is entirely novel. Many evaluators already work hard to honour multiple perspectives, think in multiple dimensions, challenge dominant narratives, and engage stakeholders in deliberative sense-making. Cubist Evaluation bundles and names these habits. It also pushes against the way evaluations can too easily tidy away conflicting impact stories and value stories into a neat, single narrative (like “performing strongly overall”) that feels rigorous but privileges one account and hides nuance, diversity, and uncertainty. Cubist Evaluation adds an explicit invitation to design for those tensions, holding different value stories side by side and working through them with stakeholders and rights-holders, so that multi-perspectival judgement and collective meaning-making become the core work of evaluation. Over time, its ambition is for these practices to move from the margins to the mainstream.

What Cubist Evaluation adds, beyond existing traditions, is a shared metaphor and a set of design prompts for putting that work at the centre of how evaluations in complexity are conceived, conducted, and defended.

Philosophically, Cubist Evaluation sits in a pluralist space. It draws energy from constructivist and interpretivist ideas about meaning-making, from realist traditions that emphasise how outcomes emerge through context–mechanism configurations, from critical traditions that pay attention to power and marginalisation, and from pragmatic mixed-methods thinking that values fitness for purpose over paradigm purity. But it is not reducible to any one of these. Its distinctive feature is treating evaluation as the deliberate composition of multiple viewpoints, mental models and reasoning modes in order to develop richer and more defensible judgements in complexity. This is a pluralist stance, not an anything-goes relativism: different perspectives are invited to the composition, but judgements still have to be argued for, tested, and owned - more like the disciplined, limited palette of Analytic Cubism than an indiscriminate splash of colours.

Who is Cubist Evaluation for?

Cubist Evaluation is for people working in and around evaluation who are wrestling with complexity, plurality, and power, and dissatisfied with single-method, technocratic answers.​

Primary audiences include:

• Evaluators who work in complex, uncertain, and contested settings, where mixed methods and mixed reasoning already feel necessary, but existing frameworks do not fully accommodate the diversity of perspectives and values in play.​

• Commissioners and funders who want rigour without reductionism, and who are open to evaluative processes that explicitly surface different value judgements and support collective sense-making.​

• Practitioners and learning partners (for example, in NGOs, public agencies, philanthropy, and systems-change initiatives) who see evaluation as a tool for reflection, adaptation, and innovation, and who are ready to experiment with co-designed, multi-perspective approaches.​

I don’t want to limit who Cubist Evaluation might be for, but I think the people and contexts it might particularly suit include:

• Teams working on initiatives that cut across multiple systems or lifeworlds, where inter-objective and inter-subjective perspectives matter as much as conventional “objective” indicators - e.g., a cross-agency initiative on youth wellbeing working in multiple cultural contexts and spanning health, education and justice.

• Evaluators influenced by realist, developmental, feminist, Indigenous, or critical traditions, who want an integrative metaphor that legitimises combining technocratic, deliberative, tacit, and narrative forms of evaluative reasoning.​ Cubist Evaluation isn’t meant to replace these traditions; it offers a metaphor and rationale for combining different ways of reasoning.

• Organisations that want evaluation to question dominant stories about performance and value, bring marginalised voices into judgement processes, and generate new meaning from evidence rather than simply reconfirming existing frames.​

• Critical thinkers who are comfortable with the idea that not everything is a consensus and that multiple perspectives can coexist in tension, each having validity without requiring another to be wrong.

• People who would rather confront questions that are hard to answer than accept answers that may not be questioned, for example by recognising both the benefits and the harms of major social, technological, environmental, energy, or public health policies, positions, and interventions - rejecting the Manichean dualism of everyday political narratives and instead surfacing tensions and trade-offs, and seeking to understand what is gained and what is lost in context.

What next?

Cubist Evaluation isn’t just for fun; it’s a call for evaluation that stays rigorous in complexity, making sense of values, evidence and uncertainty in ways that are inclusive and richly informed by diverse forms of knowledge and experience. Its value proposition is about generating new, useful, collectively owned insight that can change lives for the better. It’s just getting started - there is much more to explore, both theoretically and in practice.

For example:

• How would we design a Cubist Evaluation from scratch? What would it look like to intentionally build in mixed reasoning, multiple perspectives, and collective sense-making from the very first conversation with commissioners and stakeholders?

• What practical operating principles might guide a Cubist Evaluation team day to day - e.g., in how they fracture and reassemble evidence and values to invite people to see familiar issues in new ways?

• How do we know when we’re really honouring multiple perspectives, as distinct from just adding more methods or voices without shifting who materially contributes to reaching conclusions?

• When different value stories point to different preferred options, what does a Cubist judgement (or set of judgements) look like in practice, and how can we make the path from plural perspectives to a concrete decision transparent and accountable?

• What kinds of visual, narrative, or dialogic projects would feel “Cubist” in evaluation terms - not just in the report, but in how we facilitate workshops, sense-making processes, or board conversations about what the evidence means?

• How might we document and defend rigour in a Cubist Evaluation in ways that satisfy more conventional audiences, while still challenging the very ideas of rigour that don’t fit complex, contested settings?

• How will we know if any of this is useful and makes a real difference - meeting Cubist Evaluation’s value proposition of making rigorous, pluralistic evaluative judgements in complex settings and generating new, collectively owned insight from evidence?

If this resonates, stay tuned - and get in touch if you have a live project where you want to experiment with Cubist Evaluation in practice.

Thanks for reading!

If this was interesting, useful, and enjoyable, a quick tap on the ❤️ helps me know it landed - and it also nudges Substack to show the piece to more people who might benefit from it. Thank you.

3

Just as I was putting the finishing touches on this post, Chris Lysy dropped this and it got me thinking: instead of a sense-making process followed by a report, what if, at least some of the time, the process is the report?

A scenario

Imagine we’re evaluating a large research fund. The fund has provided grants for projects across many years, sectors, places, and sizes - everything from biomedicine to social policy, from small grants for local pilots to massive investments in multi-site initiatives run by national consortia.

To help evaluate the fund and understand how it works in practice, we’ve selected a sample of grants to use as case studies. Our sample broadly reflects the diversity of grants. It isn’t large enough to provide statistically generalisable findings - but that isn’t our aim. Rather, we plan to dig deep, build understanding by comparing different cases, and draw out lessons about how the fund delivers on its value proposition.

To some stakeholders, an approach that doesn’t produce statistically generalisable findings feels at odds with their understanding of good research. They raise concerns about our methodology, expressing scepticism and challenging us to defend our design.

Five misunderstandings

One of the most-cited papers in the journal Qualitative Inquiry, ”Five Misunderstandings About Case Study Research” (Flyvbjerg, 2006) challenged common criticisms of case studies, systematically pulling apart five prevailing myths - that case studies can't offer theoretical insight, can't be generalised from, only generate hypotheses rather than test them, are prone to researcher bias, and are too hard to summarise or use to inform broader lessons. Through practical examples and philosophical reflection, the paper argued that these beliefs are either oversimplified or just plain wrong. Case studies, when carefully selected and rigorously conducted, are a powerful way to build context-rich knowledge. This post closely follows Flyvbjerg’s “five misunderstandings”, with my gloss on what they mean for evaluation.

Myth 1. Abstract theory is more valuable than context-dependent, practice-based knowledge

The misunderstanding: In practice, people may default to treating high-level theoretical knowledge about ‘the general case’ as more valuable than concrete, practical knowledge from specific cases, especially when they are under pressure to show scalable, comparable results. For example, in our grant fund evaluation, some stakeholders may worry that lessons from individual projects will be too tied to their unique settings to inform broader program strategy.

Flyvbjerg’s argument: Context-dependent, practical knowledge is essential for developing real expertise; theory matters, and it becomes more meaningful when understood in light of specific cases. In complex real-world interventions, lessons from the field are needed to drive learning and improvement, and people responsible for designing, delivering and governing programs move from rule-based “beginners” to genuine experts by engaging deeply with many concrete projects and contexts over time.

Implications for our grant fund evaluation: Case studies will help us learn how things worked in different settings (such as urban academic centres, regional hospitals, and small-town community-led initiatives). They’ll contribute a nuanced understanding that’s hard to get from surveys or large datasets alone. Broad theories also have their place, but the practical lessons from the case studies will be very useful, capturing rich details that are essential for informing practice and policy.

Myth 2. You can’t generalise from a single case

The misunderstanding: A common view is that you can’t responsibly generalise from one (or a few) cases, so case studies have limited value for program-wide decisions. In our grant fund evaluation, some stakeholders are not completely against case studies, but they express doubts about whether in-depth studies of a few projects can say anything useful about the fund as a whole.

Flyvbjerg’s argument: Generalisation is possible, especially when cases are well chosen. For example, a carefully selected typical, or “paradigmatic” case can teach us a lot, by highlighting characteristic features of a system and serving as a practical exemplar of how its mechanisms play out in everyday conditions. A critical or extreme case can help test or challenge broader assumptions. With thoughtful selection, cases can inform wider theory and practice, not by representing a statistical population, but through “the force of example”.

Implications for our grant fund evaluation: Our sample of case studies doesn’t claim to represent all grants in a statistical sense. Instead, it is a route to analytic generalisation - using carefully chosen cases to refine ideas about how the fund works, for whom, and under what conditions (Yin, 2010). By focusing on how individual projects reflect or challenge the fund’s overall value proposition and theory of change, we can spot patterns and principles that matter beyond each case.

To make this work in practice, we will be deliberate about case selection. Rather than a convenience sample, we’ll purposefully choose cases for their information value, including cases that illustrate “typical” conditions, plus a small number of critical or extreme projects (e.g., very small local pilots and very large multi-partner collaborations). This kind of information-oriented selection supports analytic generalisation, because it allows us to see how core mechanisms play out across contrasting conditions, and to identify where the fund’s value proposition holds, where it wobbles, and where it breaks.

Myth 3. Case studies can’t build or test theory - they only generate hypotheses

The misunderstanding: Case studies are often seen as just a starting point, useful for generating ideas but not strong enough to test or develop theories. For example, our stakeholders would be happy if we used case studies as a preliminary exploratory phase, then relegate them to the background once surveys and quantitative methods begin. However, we don’t want to do that because it would miss the opportunity for case-based insights to reinforce or challenge the program’s theory of change.

Flyvbjerg’s argument: Case studies can generate, test, and refine theories. They are powerful tools for both developing new ideas and putting existing theories or assumptions to the test, not only by suggesting new hypotheses but also by subjecting them to tough, context-rich examination where they can be falsified or forced to evolve. This brings us back to analytic generalisation: as Yin (2014) emphasised, when case studies are designed around explicit theoretical propositions and replicated across different contexts, they can advance or revise theory.

Implications for our grant fund evaluation: Case studies are more than just ‘early-stage’ exploration; by examining how the theory of change holds up in specific contexts, they can help refine, validate, or challenge core assumptions. For example, suppose the fund’s theory assumes that providing seed funding plus light-touch mentoring is enough to scale promising pilots. In one case study, we might see that this works well in a well-connected university setting but fails in a small community organisation with limited infrastructure. In another, we might learn that intensive brokering between partners, not the size of the grant, was a decisive mechanism.

Taken together, these cases would not only generate new hypotheses about what drives success, they would also put the fund’s theory of change under pressure in real-world settings. This is genuine theory testing - revising or retaining theoretical propositions based on how well they explain contrasting cases and contexts.

Myth 4. Case studies have a bias toward verification

The misunderstanding: Critics sometimes argue that case studies just confirm what the evaluator already thinks, because you can find whatever you look for in a messy real-world environment. For example, a couple of our evaluation governance group members are sceptical, suggesting that we will cherry-pick stories or interpret findings to support our personal views.

Flyvbjerg’s argument: Actually, in-depth case work often challenges initial assumptions. When researchers get close to the real action, they’re more likely to be confronted by surprises that upend their preconceptions and prompt new lines of thinking. Far from being merely affirming, case studies can and do falsify assumptions.

Implications for our grant fund evaluation: We will go into our case studies expecting our thinking to be challenged. Since case study work is so closely tied to front-line experiences, it’s more likely to expose flaws in the logic or implementation of the program than to simply confirm standard narratives. For us, this is an opportunity to spur learning and honest reflection. To guard against cherry-picking or confirmation bias, we will be transparent about how we select cases and what we are comparing with each case. We’ll look deliberately for rival explanations and disconfirming evidence, and use explicit propositions to link case evidence to findings.

For example, if a case seems to show strong outcomes in a particular sector, we will explicitly look at whether this is because of the fund’s design or because that sector already had highly capable organisations and pre-existing networks. We can also explore whether it was a bit of both, or whether some other factor was behind the patterns observed. Examining these rival explanations helps keep us honest and makes our reasoning easier for others to scrutinise.

Myth 5. Case studies are too detailed to summarise or turn into usable lessons

The misunderstanding: Because case studies are specific and detailed, some think they’re impossible to summarise or turn into clear, usable lessons for wider decision-making. In our evaluation, stakeholders express concern that case study findings will be so context-dependent and narrative-rich that they can’t be organised into useful recommendations or transferrable lessons for the whole fund.

Flyvbjerg’s argument: While it’s true that the richness of a case can’t always be boiled down to one-sentence conclusions, this is a valuable feature. Detailed narratives may resist reductive summary, but they offer deep insights that matter for practice and theory development; the difficulty of neat summarisation reflects the complexity of real social situations, not a flaw in case study methods.

Implications for our grant fund evaluation: We will embrace the detail. Not everything needs to end in bullet points, summary statistics, or simple conclusions. The stories themselves are important, especially when they shed light on how the fund’s aims play out in the day-to-day realities of funded projects. We will summarise what we can, but we’re not worried if some findings resist being squeezed into a brief ‘takeaway’ - because that’s the point. Sometimes, it’s the complexity and specificity that make the case meaningful for future policy or learning.

Discussion

When we draw on case studies in evaluation, many of us run into the demand for “generalisable” findings. The traditional expectation, especially from those more familiar with large-scale surveys or experimental methods, is that you should be able to make inferences from your sample to a wider population. In other words, conclusions would, in theory, hold for all grants or projects, not just the ones you looked at. That’s statistical generalisation.

This is where that familiar question pops up: “But is it representative?”, as if anything short of a large survey automatically fails the quality test.

In our fund evaluation, with its diverse mix of sectors, scales, and settings, statistical generalisation is not the goal. Polit and Beck (2010) described statistical generalisation as just one of three models of generalisation, alongside analytic and case-to-case forms, which can be appropriate for qualitative or case-based work. Here, we are pursuing what Yin (2010) called analytic generalisation: we use case studies to build and test ideas about patterns and mechanisms, then consider where those ideas are likely to hold across different kinds of projects and contexts. Rather than claiming that something is true of all projects, we are asking how the mechanisms and lessons we see in these cases strengthen our understanding of the fund.

In practice, this means stepping back from individual cases to look across them and identify patterns, lessons, and mechanisms that speak to the fund’s theory of change and value proposition. ‘Theory’ here does not have to mean grand, abstract models. It includes mid-range mechanisms, practice-based frameworks, or simple ‘if-then-because’ propositions about how change is expected to happen. We should state up front which ideas or assumptions we’re exploring, so the path from case to conclusion is clear and open to scrutiny.

Typically, analytic generalisation involves comparing the results of case studies to existing theories or frameworks to see how the cases support, refine, or challenge them. We can then argue, with reference to both theory and context, how and where insights may be transferable or relevant to other cases, without pretending they apply everywhere.

By anchoring our findings in the fund’s theory (in this broad sense), we can speak to program-level mechanisms even if our sample is small. We’re aiming for transferable learning, not universal truths: the aim is not to claim that all projects will follow the same path, but to make a reasoned case for when and why certain dynamics observed in our cases might play out elsewhere, in line with Polit and Beck’s (2010) notion of case-to-case transfer.

Analytic generalisation gives us a disciplined way to learn from a small, carefully chosen set of case studies, generalising findings to theoretical propositions, not populations (Yin, 2010; 2014). It provides a bridge from the rich detail of individual projects to broader program learning, supporting strategic thinking and evidence-informed decision-making, even when statistical generalisability isn’t possible or meaningful.

As an aside, this is the same logic I used in my doctoral research. I developed a conceptual model for evaluating value for money, translated it into a series of theoretical propositions, and tested it through two international development case studies, abstracting from the cases back to the propositions through analytic generalisation and replication.

Of course, the case studies may just be one component

If we also want some statistically generalisable findings - i.e., findings that can confidently be extended from our sample to the broader population of projects in the grant fund, we can do that too - for example, we could run a survey across a larger sample of grants.

Using both strategies together (case studies for depth, survey for breadth) is an example of a mixed methods design that gives us a richer and more robust evidence base than we would get from either study alone. While the case studies help us learn how and why things work (or don’t) in real-world contexts, surveys provide a less-detailed picture of a few key areas of enquiry across the fund.

For instance, a short survey could ask all grantees about their perceived level of support from the funder, the extent and quality of collaboration with partners, and whether key milestones were achieved on time and within budget. These items won’t match the nuance of case narratives, but they can show how widespread certain challenges or successes are, and where the in-depth cases are typical or unusual. Each method helps compensate for the other’s blind spots.

Rather than simply run each method in parallel, their findings can interact and sharpen each other. For example, insights from case studies can help prioritise and refine survey questions. Case narratives can provide exemplars that bring survey-based findings to life. Quantitative findings can help us gauge the magnitude of impacts and challenges and determine whether patterns seen in case studies are common or exceptional.

Bottom-line

Case study-based evaluations of complex, diverse programs are not ‘second-best’ apologies. They generate their own insights for real-world learning and analytic generalisation, showing how and why change happens in practice and helping to refine the theories and strategies that guide future decisions.

Further reading

Flyvbjerg, B. (2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245. https://doi.org/10.1177/1077800405284363

Polit, D. F., & Beck, C. T. (2010). Generalization in quantitative and qualitative research: Myths and strategies. International Journal of Nursing Studies, 47(11), 1451–1458. https://doi.org/10.1016/j.ijnurstu.2010.06.004.

Yin, R. K. (2014). Case study research: Design and methods (5th ed.). Sage.

Yin, R. K. (2010). Analytic generalization. In A. J. Mills, G. Durepos, & E. Wiebe (Eds.), Encyclopedia of case study research (Vol. 1, pp. 21–23). Sage.

Thanks for reading!

If this was useful, a quick tap on the ❤️ helps me know it landed - and it also nudges Substack to show the piece to more people who might benefit from it. Thank you.

Read more

Read more

Heidi is pushing VfI to do more of what it was intended for in complex settings: make evaluative reasoning explicit, bring mixed methods together, and involve stakeholders in judging what “good” looks like. In the rest of this post, she explores four things:

• Why democratic deliberation is her north star for this work

• How rubrics can scaffold that deliberation without collapsing complexity

• How we can expand the types of value we can bring into a VfI assessment of systems-change work

• What this means for the evaluator’s role, given real-world constraints on time, people and power.

Over to you, Heidi - starting with a brief introduction to your background and how you came into this work.

A quick introduction

I first came across VfM when I moved to the UK to work on some of the country’s largest research-for-development funds – £2.2 billion of public investment that was under a lot of scrutiny for not having an approach to VfM. This was also how I first met Julian, when I desperately emailed him for help on how to assess the VfM of such large funds. This challenge set me on a path to grappling with how we can assess value meaningfully in complex initiatives. I started a PhD (with Julian as one of my supervisors) while still in the UK to give myself the time and space to really wrestle with it, which I have just finished!

Over the course of my PhD, the pressure on governments to demonstrate good and efficient use of public funds has continued to increase. But at the same time, there has been a growing recognition of the complex and wicked problems facing society, leading to more and more systems-change initiatives that seek to change the conditions holding these problems in place. Evaluators asked to assess the VfM of such initiatives can feel caught between a rock and a hard place of efficiency and complexity (certainly I have!). The usual methods for VfM struggle with the unpredictable, emergent and non-linear outcomes of messy systems-change work, where attribution is impossible and time horizons stretch long.

However, the VfI approach, grounded in the principles of deliberative democratic evaluation, can provide a way forward – so let’s jump into the rest of the post.

Why democratic deliberation is my north star

When we evaluate systems-change work, we step into contested terrain. Beneath disagreements about the VfM of an initiative lie deeper debates: what are the problems in the first place? What types of systems change should we be striving for? And whose perspectives – of the system, and of what counts as valuable – actually matter?

Participatory evaluation is often put forwards, and rightly so, as a way to work through the different perspectives that underpin systems-change work. In this context, however, it is not enough to say that our VfM evaluation is participatory because we interviewed stakeholders or ran a workshop. Participation can still leave the underlying valuing work untouched.

Democratic deliberation, on the other hand, asks more of us. In deliberative democratic evaluation (House and Howe, 2003), three principles are central:

• Inclusion: all relevant perspectives, especially those most affected, should have a meaningful opportunity to shape how value is understood and judged.

• Dialogue: stakeholders should have space to articulate their views, hear others, and interrogate assumptions.

• Deliberation: the group works towards shared judgements about merit, worth or value through reason-giving and negotiation, not by aggregating individual scores.

For systems-change VfM, these principles are my north star – the principles I strive for. That doesn’t necessarily mean that I put them into practice in every project – time, budget and politics almost always get in the way – but they orient the decisions I make throughout an evaluation. They push me to ask, for example:

• Whose perspectives must be in the room if this judgement is to be legitimate?

• What power dynamics might get in the way of meaningful involvement in criteria-setting and judgement-making?

• What kinds of evidence and lived experience will we treat as credible, and why?

• How will we handle situations where commissioners, implementers and communities hold genuinely different ideas of value?

Crucially, democratic deliberation changes how we think about “who decides”. Instead of an evaluator or a small group of technical experts determining whether an initiative offers VfM, judgement making becomes something we do alongside those who have a stake in the answer.

In systems-change work, where no single actor can see or understand the messiness and complexity of the whole system, I don’t believe there is any better way to proceed.

A rubric as the scaffolding for democratic deliberation

Rubrics are sometimes dismissed as glorified rating scales. I see them differently. For me, a rubric is the scaffolding that allows democratic deliberation about value to happen in an explicit and transparent way.

Others have written elsewhere about what rubrics are and how we build them, so I won’t go into definitions here. But I want to highlight three things that rubrics enable when we are trying to reason together about VfM in complex systems.

First, rubrics make our thinking visible. When we articulate criteria (what matters) and standards (what “good” looks like), we’re putting our ideas about value on the table where others can see, question, and refine them. That’s a precondition for any meaningful deliberation and allows the rubric to become the shared artefact that anchors the group’s evaluative reasoning.

Second, rubrics give structure to the work of reaching judgement. Rather than an unbounded discussion about whether an initiative is “good value”, the group moves criterion by criterion, standard by standard, considering the evidence and reasoning in a disciplined way towards a final judgement.

Third, rubrics create a shared language for disagreement. In a deliberative panel, it is much easier – and safer – to say “I’m not convinced we’ve met this standard under the criterion about contribution to systems change” than to say “I just don’t think this program is very good”. The rubric focuses contention on the criteria and standards, not on the people in the room.

How this plays out in practice: WorkWell

The WorkWell program in Victoria, Australia, provides a concrete example of this approach in action. WorkWell was a large, multi-year systems-change initiative aimed at improving workplace mental health, with a focus on shifting employers from individual resilience strategies towards primary prevention.

In the final evaluation, we used a VfI-inspired, rubric-based process anchored in five types of value (more on that below!) and supported by a deliberative panel of stakeholders. Because there is a journal article that goes into WorkWell in detail, I won’t unpack it here, except to use it to illustrate the value types below. If you are interested in how the framework and deliberative process played out in practice, you can read that case study in full here.

Five types of value: reframing what counts as “good use of resources”

One of the biggest challenges in assessing the VfM of systems-change initiatives is that much of the value they create isn’t visible in typical funding cycles. If we only look for short-term, attributable outcomes, we risk seriously underestimating – or even mischaracterising – what an initiative has actually achieved.

To address this, I developed a schema that combines the Water of Systems Change framework (Kania et al. 2018) with the Cycles of Value Creation from Wenger and colleagues. The Water of Systems Change argues that systems change occurs when the conditions holding a problem in place shift – moving from explicit conditions like policies and resource flows, through relational conditions like relationships and power dynamics, down to the most implicit and transformative: mental models.

The Cycles of Value Creation was originally developed for social learning networks – not VfM at all – but when I came across it, I realised it could be overlaid with the systems-change framework to give us something genuinely useful. Together, they produce five types of value for systems-change work:

• Inherent value – value in the activities themselves.
  This is the immediate value that comes from doing the work – evident even before anything else shifts. For WorkWell, it included the community and connection created for the 30,000 workplace leaders who engaged with the program, and measurable improvements to worker mental wellbeing in several funded projects.

• Potential value – enablers for future systems change.
  This is the type I find most critical – and most often overlooked. It’s what happens when you change relationships, connections, and power dynamics: you’re not yet seeing outcomes, but you’re creating the conditions for them. In systems change, this loading of potential can be a long, slow process until a tipping point is reached. For WorkWell, it looked like bringing together stakeholders across Victoria who had never worked together on prevention before – and those partnerships became increasingly mature and collaborative over the course of the program.

• Applied value – the system starts to shift.
  This is when potential turns into action: changes in practice, policy, and resource flows. For WorkWell, this included changes to employer practices and policies, including some industry-wide ones, and the Victorian Government’s decision to continue funding WorkWell as a business-as-usual function of WorkSafe.

• Realised value – outcomes for people and systems.
  This is the type funders most often go straight to, and that traditional VfM methods were built to measure. For WorkWell, this would have meant a reduction in mental injury claims – and we knew, going in, that this hadn’t happened yet. This is exactly why the broader framework was needed.

• Transformative value – systems transformed.
  Transformative value is about shifts in mental models and underlying purposes: when deeply held beliefs about “how things are done around here” move. For WorkWell, this was arguably its most significant contribution: a paradigm shift in how employers, regulators and industry understood their role in preventing mental injury.

In practice, this typology gives stakeholders a wider vocabulary for talking about, evidencing, and valuing what systems-change work creates. We can even build rubrics around these types of value! It allows a deliberative group to ask not “have headline outcomes changed yet, yes or no?” but “given the nature of this work and its timeframe, has enough value of the right kinds been created to justify the investment?” That’s a more honest and generative question.

The evaluator as a trusted process-holder

All of this has significant implications for how we understand the evaluator’s role in systems-change VfM assessments – and it’s something I’ve spent a lot of time thinking about, both in practice and in my PhD.

Traditionally, evaluators have played the role of judge in VfM work. Maybe not overtly with a wig and a hammer, but by conducting highly technical analyses before presenting numbers that carry considerable weight – and can’t easily be interrogated by stakeholders – the evaluator is seen as an independent, neutral, objective arbiter.

I see a real opportunity – particularly in systems-change contexts – to rethink that role. When you work with stakeholders to surface what they value, and when you support a group of people to interrogate the evidence and reach a shared judgement, you’re shifting from judge to trusted process-holder. And I find that genuinely exciting! Our job becomes ensuring the right people can participate meaningfully, finding creative ways to present evidence so that diverse people can understand and interrogate it, and facilitating dialogue across different perspectives and power levels. Yes, it’s a different skillset, and yes, that might feel scary to some evaluators – but it’s a skillset I think we’re better placed to develop than we might expect.

This requires what colleagues have called “holding expertise in equal measure” (Bell and Anderson, 2023) – designing processes where technical knowledge, programmatic knowledge, and lived experience each have a legitimate place. It requires awareness of power, privilege, and positionality. And it calls for genuine humility and curiosity: stepping away from the role of “expert” in favour of facilitator, convenor, collaborator.

But this is also delicate work. We are often working within tight budgets and timelines, with groups that cannot possibly include everyone affected by the system. Some stakeholders may be unfamiliar with rubrics or sceptical about deliberative methods. Others may be accustomed to evaluations that deliver “the answer” without asking them to do much valuing work themselves. The evaluator is in the middle, shaping processes that are rigorous and trusted enough to support defensible judgements, but flexible enough to allow meaningful participation from diverse stakeholders.

A few practical notes for evaluators working in this space:

• Start with a modest but real deliberative space. You may not be able to convene the perfect group. Start by bringing together a small but diverse group who can credibly speak to different parts of the system – and design a process where their reasoning genuinely matters for the judgement.

• Make valuing schemes explicit. Use the rubric development process to surface whose values are shaping the criteria and standards. Ask openly: what kinds of value – and whose values – are we privileging?

• Treat the process itself as a learning intervention. The work of articulating criteria, agreeing standards, and working through evidence can itself shift how people think about the system and their role in it. As Gates and Schwandt argue, we’re not just determining value – we’re developing it. Make space for that.

Keeping the north star in view under real-world constraints

Every evaluation is a compromise. We make choices about who to involve, how much time we can reasonably ask for, what kinds of evidence we can collect, and how far we can push against institutional norms and expectations. There will always be limits on how democratic and inclusive our processes can be.

This can feel really hard for evaluators trying to conduct an evaluation with integrity. For me, this tension is exactly why having a north star matters. Democratic deliberation, anchored in transparent rubrics and a richer vocabulary of value, gives us an aspiration rather than a perfect recipe.

By treating value as something co-constructed, making evaluative reasoning visible, and situating our judgements within democratic deliberation rather than behind closed doors, we can make VfM assessments more trusted, more transparent, and more worthy of the complex work they’re asked to judge.

At the end of the day, we may still need to deliver a judgement: was this a good use of resources? But the way we answer it – who gets to be part of that process, how we reason together, and what we’re willing to count as valuable – can itself be part of the systems change we’re trying to create.

Final words from Julian

Heidi makes a point that I think many of us in evaluation feel in our bones: it is not enough to say an evaluation is participatory because we interviewed stakeholders or ran a workshop. We can run a “participatory” evaluation and still leave the valuing work untouched. If we accept that eVALUation is about determining the value of things, and that things do not have value, people place value on things, then meaningfully sharing power with stakeholders and rights-holders when systematically determining value is not just an intriguing methodological option - it becomes necessary for validity, credibility, ethics, and use.

That is why the shift she describes – from evaluator-as-judge to evaluator-as-trusted-process-holder – is so important. When you work with stakeholders to surface what they value, and when you support a group of people to interrogate the evidence and reach a shared judgement, you are changing both the answer and the way the answer is produced. Some contexts demand independent evaluation, while other evaluations are designed from the ground up to enable a group of stakeholders to form their own conclusions. Either way, I see this facilitative approach as an important part of working meaningfully with stakeholders to understand their values.

I also love Heidi’s line: “Rubrics create a shared language for disagreement”. You don’t have to agree with an evaluative judgement, and rubrics help us move from “who are you to have an opinion about my program?” to “there is a specific criterion, a specific standard, or a specific piece of evidence we need to debate here”. That is a massive win for clarity, and it reinforces her point about treating the process itself as a learning intervention. The rubric becomes the centrepiece for sense-making conversations, instead of being mistakenly treated as a set of “rules” for interpreting indicators.

Heidi’s Cycles of Value Creation model advances some long-term thinking I’ve been doing with colleagues across multiple projects to articulate value propositions (why might this be worth investing in?) and theories of value creation (through what magic does the investment create more value than it consumes?). Heidi’s schema gives evaluators and stakeholders a structured way to unpack those questions in systems-change work, inviting deliberative conversations to ask, not “are we there yet?” but, as Heidi put it, “given the nature of this work and its timeframe, has enough value of the right kinds been created to justify the investment?”

I’ll end on another line from Heidi that has stayed with me since I first heard it on Matt Healey and Tenille Moselen’s It Depends podcast: “I’d challenge people to not make their peace with what is keeping them up at night, and keep striving to do things better and differently.” Held with pragmatism, collegiality, and a healthy sense of humour, that kind of restlessness is exactly what our field needs. And it’s why I’m so pleased to be able to share her work with you here.

Further reading

Peterson, H. (2025). Democratic deliberation as a North Star: Showcasing a framework to assess the value for money of systems-change efforts. Evaluation, 0(0). https://doi.org/10.1177/13563890251386814

Peterson, H. (2022). Cost-Benefit Analysis (CBA) or the Highway? An alternative road to investigating the Value for Money of international development research. The European Journal of Development Research, 35, 260-280. https://doi.org/10.1057/s41287-022-00565-7

I’m sharing our conversation below, where we discuss my favourite topic - getting better answers to value-for-money questions. We talk about reframing how we think about value in the development sector, who should have a say in defining value, how AI might support evaluation, and more.

Impact & Beyond

The Julian King One

In this new episode of the Impact & Beyond Podcast I speak with Dr Julian King, New Zealand based public policy consultant, internationally recognised evaluation expert, and founder of King & Associates. Julian is widely known for having developed the…

Listen now

3 months ago · Michele Binci, PhD

While I’m here, two other things you might be interested in this week...

Revised Book Overview

Heartfelt thanks to everyone who commented or emailed me about the draft book overview. Your feedback was generous and incredibly helpful. I’ve now posted a revised overview that reflects many of your suggestions. If you’re interested, you can read the new version here:

Why I’m staying on Substack - for now

For those following recent debates about Substack’s moderation and its partnership with the Polymarket prediction market platform, I share many of the concerns that have led some writers to leave. For now, I’m choosing to stay, with eyes open, because Substack still does the one thing I most need: it lets me send you a simple email you’ve chosen to receive so I can keep doing careful, evidence‑and‑values‑focused work here.

Over the past couple of years, a small but visible group of Aotearoa New Zealand writers have left Substack. David Farrier moved Webworm off the platform after a long wrestle with its tolerance for Nazi and extremist newsletters, and with the way it has shifted from a simple email tool into a recommendation‑driven social pla…

Read more

• Several readers asked for a sharper, earlier distinction be…

Read more

When diverse traditions and knowledge are brought together with care, they can transform what practice looks like. For a Pacific Value for Investment Model, one of those moments was when the evaluation team took the familiar, eight‑step VfI process, contextualised each step through a distinctly Pacific lens, added extra stages at the beginning and the end, and re-imagined the entire sequence as a circle that begins and ends with relationships. Similarly, VfI’s four guiding principles were adapted and extended. The result is more than just “VfI for Pacific programmes”; it’s a Pacific‑led re‑expression of what good resource use can mean when it is grounded in Pacific values, worldviews and methodologies.​​

The evaluation design work was led by Dr Lana Perese and colleagues at TIRIA Ltd (formerly Malatest International). It was deeply collaborative from the outset. Colleagues from Auckland Council’s Economic Development Office Pacific team and Ministry of Business, Innovation and Employment (MBIE) Langa le Vā team commissioned and co-developed the framework, and an advisory group of senior cultural and technical leaders walked alongside through successive iterations of the model. Preston Davies from Sapere contributed advisory support on economics, and I supported the project with VfI advice.

Why a Pacific VfI Model?

The starting point was a practical commissioning problem. Auckland Council’s Economic Development team, with support from MBIE, wanted a framework that would ground economic evaluation in Pacific values and connect resources to value creation for Pacific peoples in Aotearoa New Zealand: skills, employment, wider social and cultural wellbeing. VfI was attractive because it already combines evaluative reasoning and economic analysis to answer value‑for‑money questions in nuanced ways, but it needed adaptation to fully capture socio‑cultural, relational and intergenerational impacts that matter for Pacific communities and the prosperity of Aotearoa New Zealand.​​

The resulting Pacific Value for Investment Model (P‑VIM) was developed between August and December 2025 through an iterative process: a literature scan across Pacific and Western knowledge bases, a fono (workshop) to explore potential synergies and tensions between VfI and Pacific methodologies, and engagements with Pacific experts, providers, academics and senior public servants. The intention was to produce the first unified framework for this commissioning context that “grounds economic evaluation in Pacific values and connects resources to value creation in ways that matter for Pacific communities, agencies, and funders – beyond what economic evaluation approaches alone can capture.”​ Having such a framework is increasingly important given the growing demographic presence and economic contribution of Pacific communities in Aotearoa New Zealand.2

Guiding principles: VfI re‑spoken in a Pacific voice

In standard VfI work, four principles are emphasised: interdisciplinary thinking, mixed methods, evaluative reasoning, and a participatory approach. The team kept this backbone but re-spoke it in a Pacific voice and added principles that are non‑negotiable in Pacific‑led evaluation. The resulting set sits at the heart of the P-VIM: cultural scholarship and interdisciplinary thinking, cultural credibility and mixed methods, evaluative reasoning with reflective and responsive practice, participatory approach and meaningful relationships, professional practice, and data stewardship and digital ethics.​​ I’ll unpack these principles below.

The framework is clear that these principles “are non‑negotiable and constitute the foundational core upon which the model is designed and implemented… a condition of engagement, application, and accountability within the P‑VIM framework.” Where VfI originally framed interdisciplinarity as integrating evaluation and economics, P‑VIM reframes it as prioritising Pacific scholarship as the lead discipline, with evaluation and economics in supporting roles. Where the general VfI approach emphasises mixed methods for validity and richness, P‑VIM anchors cultural credibility, Pacific‑led teams and data sovereignty as integral to what “mixed methods” means in practice.​​

This re‑expression of the principles is where the Pacific interpretation of VfI becomes visible. Each original VfI principle is both retained and transformed, as follows.

Interdisciplinary → Cultural scholarship and interdisciplinary thinking

The VfI approach integrates evaluation and economics to assess “the merit, worth and significance of resource use” in a way that is broader than pure economic evaluation or traditional programme evaluation. P‑VIM retains this integration, but combines three disciplines, “prioritising Pacific scholarship as the lead discipline governing the use of evaluation and economics.”​​

Pacific research paradigms such as fa’afaletui, talanoa and kakala, grounded in relational ethics and Pacific worldviews, are positioned as the primary intellectual home of the work.3 Theory and practice from evaluation and economics are intentionally woven in to fit with Pacific paradigms. This is an epistemological pluralism in practice: strong knowledge comes from the “convergence of diverse viewpoints and methodological rigour”, and in this framework, Pacific ways of knowing and being set the terms of that convergence.​

Mixed methods → Cultural credibility and mixed methods

VfI encourages mixed methods, recognising the validity of combining quantitative and qualitative evidence to understand how and why value is created, and to support transparent evaluative judgements. In P‑VIM, this becomes a broader principle of cultural credibility and mixed methods.​​

The framework document notes that in a Pacific context, credible evidence includes “collective and individual perspectives, objective measures and subjective experiences, and culturally grounded ways of knowing and being.” What counts as credible extends beyond Western standards of validity and reliability to incorporate cultural protocols, relationships, and the relevance of evidence to Pacific lives and contexts. Methodologically, this translates into Pacific‑led design and analysis, integrating quantitative and talanoa‑based qualitative work, and an explicit commitment to Pacific data sovereignty and ethics. It includes careful attention to Pacific data in administrative sources such as NZ’s Integrated Data Infrastructure, a large, de-identified database linking administrative data about people and households from across government for research purposes.​

Evaluative reasoning → Evaluative reasoning plus reflective and responsive practice

In general guidance, VfI emphasises explicit evaluative reasoning: clear criteria and standards, transparent synthesis of evidence, and collective deliberation to reach well‑reasoned judgements about resource use and value creation. P‑VIM keeps this and extends it by pairing it with reflective and responsive practice.​​

Reflective and responsive practice invites evaluators and agencies to pause on their conclusions and ask hard questions: Are processes extractive or misaligned with community aspirations? Have all reasonable efforts been made to support optimal development, delivery and achievement for both providers and participants? This principle legitimises recalibration – including the possibility of restarting programme or evaluation efforts – as a mark of integrity rather than failure. In the circular P‑VIM process, evaluative reasoning and reflective responsiveness recur across waves, rather than being a single “synthesis” moment at the end.​ This principle elevates evaluative thinking through a distinctively Pacific lens.

Participatory → Participatory approach and meaningful relationships

VfI stresses that the values guiding evaluation should be co‑constructed with a relevant and diverse group of stakeholders and expressed through shared criteria and standards (often as rubrics). VfI evaluation involves stakeholders in design and sense-making for ethical, credible, valid, useful evaluation. P‑VIM’s principle explains that “for programmes in Pacific communities, participatory approaches ensure the evaluation process aligns with community values, fosters deeper relationships, and generates findings that resonate and endure within the cultural contexts.”​​

This is framed through the concept of teu le vā – nurturing respectful and reciprocal relationships – which must be upheld throughout the initiative, not just during data collection or sense‑making workshops. The emphasis shifts to attending to the depth and quality of engagement, addressing power dynamics, and ensuring Pacific communities are centrally involved in design, analysis and dissemination.

8 steps in a row → 12 steps in a circle

Readers of my VfI work will be familiar with the eight‑step process: understand the investment and its value proposition, develop criteria and standards, identify and gather evidence, analyse and synthesise findings, and answer the evaluation questions. The team’s process work on P‑VIM began by asking what was missing if this was to be truly Pacific‑led. The answer resulted in the addition of extra steps at the beginning and end, and a different geometry.​

P‑VIM is organised into four waves: Wave 1 – Prepare (Teu le vā), Wave 2 – Design VfI, Wave 3 – Implement VfI evaluation, and Wave 4 – Inform (Tausi le vā). Each wave includes several components, bringing the total to 12. This process retains VfI’s 8 steps in the middle and expands it with Wave 1 “pre‑VfI” components at the start. These early components lay a strong foundation by clearly defining the purpose and scope of both the initiative and evaluation, recognising the partnership and two-way relationships between funders and evaluators through clear and shared roles, responsibilities and communication pathways; securing resources; and including diverse Pacific perspectives in the creating of the theory of change and evaluation logic.

At the other end, the Wave 4 “post‑VfI” components close the circle by turning findings into action through clear reporting and culturally respectful dissemination. This goes beyond reporting; it is about relational accountability, empowering communities, informing policy and practice, and strengthening relationships to drive meaningful, transformative change. Re-imagining the process as a circle makes this explicit: evaluation begins and ends with the state of the vā (relational space that binds people, communities and institutions), and the next cycle of investment and evaluation grows out of how well that vā is nurtured and maintained (teu le vā).​

Circles within oceans: P‑VIM and the Collective P‑VIM

Beyond individual initiatives, the team recognised that Pacific programmes rarely operate in isolation. They sit within ecosystems of disconnected activities, all intended to contribute to shared outcomes. So the framework extends P-VIM by zooming out and providing principles for a Collective Pacific Value‑for‑Investment Model (CP‑VIM). This extends P‑VIM to a system level, aligning with collective impact thinking while staying grounded in Pacific values.​

The CP‑VIM uses the metaphor of the moana nui, inspired by Epeli Hau’ofa’s vision of the Pacific as a sea of islands connected by ocean rather than divided by it. In the illustration, “the sand forms the foundation, representing Pacific values… the ocean currents flowing from each initiative symbolise the activities and outputs… the moana nui symbolises how collective outcomes and measurable steps toward change emerge from the combined contribution of all currents… the navigational star lines represent guiding principles and robust standards… [and] the surrounding environment symbolises evaluative criteria used to determine value, while the capitals represent sustainable and transformative societal impacts.” VfI’s emphasis on contextualised criteria and standards is still there, but embedded in the wider relational and systemic picture.​

Connecting to other contextual adaptations

The P‑VIM is one instance of a pattern that has been emerging in VfI work for some time: the general VfI approach is intended to be adapted to different contexts, while remaining true to its core guiding principles. Te Kounga o te Werawera, for example, is a kaupapa Māori-led VfI approach that uses Māori values and concepts to guide criteria, standards and questions about value, with VfI tools supporting rather than leading. In my Made‑in‑Africa VfI workshops and presentations, African indigenous contexts and aspirations define what counts as value, while VfI offers a process to articulate value propositions, connect investments to outcomes, and support decision‑making across value chains.​

Across these examples, the through‑line is that VfI is not intended to be a rigid formula; it’s a set of guiding principles and a process that help to make resource use and value creation questions explicit and answerable, which can be re‑spoken in different epistemologies and visual languages. P‑VIM is a powerful interpretation of those principles by Pacific scholars and practitioners who lead with their own concepts of value, relationships and accountability.​

One Pacific articulation among many

The P-VIM is not offered as a definitive or exhaustive “Pacific” model. It sits alongside many other Pacific research and practice frameworks, each grounded in its own cultural traditions, and it was developed through iterative, collective processes with Pacific advisors, providers, community leaders and officials, including both regional and ethnic perspectives, rather than by any one author or organisation.

Nor does the P-VIM pretend to speak for all Pacific peoples, nor to bridge all differences contained within the broad and often homogenising “Pacific” label. It draws on fa’afaletui‑informed processes that deliberately weave together multiple houses of knowledge – pan‑Pacific, ethnic‑specific, mixed ethnicity, cultural identity and orientation, community and government perspectives, Island‑born and New Zealand‑born experiences, and voices across generations, status and standing in families and communities – with a clear acknowledgement of diverse identities, contexts and worldviews.

What P-VIM teaches VfI practitioners

The Pacific-led process and principles in P-VIM extend and deepen ideas that are woven through VfI work. This makes the framework an important new resource for the VfI library. For example, in earlier posts and resources, you can find:

• Interdisciplinarity treated as evaluation, economics and whatever other disciplines the context demands - for example, bringing psychology alongside evaluation and economics in mental health work.

• Mixed methods presented as more than “quant plus qual”, inviting attention to how different ways of knowing and different kinds of subjective, objective, inter-subjective and inter-objective evidence work together.

• Evaluative reasoning described as a relational as well as technical practice, grounded in collective deliberation and explicit values.

• Participatory practice framed as power sharing and collaboration, not just consultation.

• VfI located inside decision-making cycles as a guiding, iterative logic rather than a linear recipe.

The P-VIM brings these threads to the foreground and makes them unmistakably Pacific, with Pacific leadership, knowledge and methodologies setting the terms. That offers the wider VfI community new language, imagery and practice to learn from, while standing firmly on Pacific leadership and authorship. This makes it more than a niche case study - it showcases important principles for all of us.

Thanks for reading!

And gratitude to Lana Perese for peer review. I take sole responsibility for all opinions and any errors or omissions in this piece.

Reference

Perese, L., King, J., Stephen, J., Pio-Bentley, F., Davies, P., & McLeod, D. (2025). The Pacific Value-for-Investment Model. TIRIA Ltd, Julian King & Associates and Sapere, for Auckland Council.

Infographic

Today I’m giving you a peek at one small but crucial piece: the draft Overview – the first pages readers will encounter, outlining what VfI is, how it works, and what the book is trying to do.

As fa…

Read more

I’m happy (and slightly nervous) to share that this strategy ha…

Read more

As a profession, I suspect we under-use economic methods of evaluati…

Read more

Read more

Read more