This is Evaluation and Value for Investment, a series about disrupting ‘value for money’ assessment to make it more valid and credible in a wider range of contexts, more inclusive, more value-focused, and more useful. I may also go off-piste occasionally and write about other interesting things.
It’s free to subscribe. Most articles will be free for the first two months after posting. To access the full archive, there’s a free 7-day trial period or you can become a paid subscriber.
Here’s a thematic summary of the main topics I’ve covered to date:
The Value for Investment approach
I developed the Value for Investment approach through doctoral research, to bring greater clarity to answering evaluative questions about how well resources are used, whether enough value is created, and how more value may be created from the resources invested in policies and programs.
As evaluators, development professionals, impact investors, and decision-makers, we know these ‘value for money’ questions are important questions but they can also be hard to answer. When I looked at the current state of VfM assessment I saw that it was often:
Too funder-centric, neglecting important perspectives like communities who were supposed to benefit from an intervention
Too accountability-focused, missing opportunities to inform learning and improvement
Too simplistic, ignoring the complexities of real-world programs
Too focused on cost reduction, quantitative evidence and monetary valuation, leaving stakeholders feeling short-changed because the assessment didn’t capture the full value of their investment.
We can address all of these problems, and we must, because VfM is critically important to making good decisions with limited resources. The Value for Investment approach is my contribution to better VfM assessment.
The following posts introduce the Value for Investment system and offer practical strategies for using the approach:
Value for Investment guide from Dovetail Consulting, with a case example from the NZ health sector
Better VfM assessment - four principles and a process to guide VfM evaluations
Getting to the real value in ‘value for money’ - a tale of two tribes
Rapid VfM assessment - four guiding principles and four examples
VfM reporting - features of a good report and strategies for structuring it
Ways we can work together - a VfI capability building pathway
Rubrics and evaluative reasoning
Rubrics and evaluative reasoning are the bedrock of the Value for Investment system. A rubric is a matrix of criteria (aspects of performance) and standards (levels of performance) that evaluators use as a scaffold to make sense of evidence and render a warranted value judgement. In other words, a rubric is a way of making evaluative reasoning explicit.
The following posts focus on the nature of evaluative reasoning, why I think rubrics are a good way to do it, and practical tips for working with rubrics.
Developing rubrics with stakeholders - tips and tricks from the coalface
Objectivity and subjectivity in evaluation - the science and art of combining values and other facts
Evaluative thinking - what is it, why is it complex, and can we boil it down a bit?
Program theory and value propositions
VfM assessments are too often unclear about the value and distracted by the money. We can address this by re-framing policies and programs as investments in value propositions. That’s why I prefer the term Value for Investment. Value propositions are useful constructs because we can define them and evaluate how well they’re met.
A value proposition extends a theory of change to make it more explicit about value. A theory of change typically describes how a policy, program or initiative is thought to contribute to a set of impacts through a set of actions. Implicitly, it may focus on activities and impacts that people value, but we can’t take this for granted.
A value proposition (or “theory of value creation”) augments a theory of change to consider questions about value, such as:
To whom is the investment valuable? How is it valuable to them?
What kinds of resources are invested, and by whom? What does good stewardship of those resources look like?
What ways of working will ensure we get the most value from the resources invested?
What changes should we pay attention to in the short- to medium-term to understand whether the investment is on track to create longer-term value?
What is an appropriate and fair distribution of resources, actions, impacts, and value?
How does the investment create value? What critical factors affect whether the investment creates a lot of value or a little?
These sorts of value questions can prompt us to think differently about a program. For example, in addition to what’s already in the theory of change, we may need to consider what it would mean for the program to manage resources well, deliver its services or products efficiently, create enough value and distribute the value fairly enough to be considered worth doing. Thinking about these issues can be helpful when it comes to developing context-specific VfI criteria - critical, observable aspects of performance that determine the extent to which a program provides good value for the resources invested in it.
The following posts explore theories of value creation with hypothetical and real examples:
Developing Value for Investment criteria from a value proposition
Developing VfI criteria from a value proposition - a case example
Value propositions - an example involving e-bikes and happiness
Value propositions - formative VfM assessment of the African Risk Capacity
Program Theory GPT - I asked ChatGPT to help unpack a program’s value proposition
Economic methods of evaluation
Value for Investment is an inter-disciplinary approach - because “good resource use” is a shared domain of evaluation (determining the goodness of things) and economics (studying how people choose to use resources). The VfI approach encourages the inclusion of economic methods of evaluation within a wider framework, where feasible and appropriate. Economic methods of evaluation contribute important pieces of evidence to help inform evaluative judgements.
I argue that evaluators should welcome these methods into the field of program evaluation and use them more often, but usually not on their own. I advocate for their use within mixed methods evaluations, alongside other methods.
Posts about cost-benefit analysis:
Social Return on Investment vs cost-benefit analysis - how different are they really?
With great power comes great responsibility - a presentation on CBA through an evaluative lens
Posts about other economic methods of evaluation:
Cubist Evaluation
Cubist Evaluation started life as a throwaway comment from a colleague. Like there are so many flavours of evaluation, why not one more? But somehow I just couldn’t let it go. I used it as the theme for a conference presentation, and have started developing the idea further in a couple of subsequent Substack posts.
Cubist Evaluation seeks to challenge and disrupt conceptions of rigorous evaluation in complexity by encouraging us to think beyond methods, beyond measurement, and beyond objectivity. It involves mixing methods and reasoning strategies, honouring multiple perspectives, challenging dominant narratives, contributing new meaning, and opening up possibilities. Overall it highlights a set of dispositions that I bring to evaluation and that can underpin the use of the Value for Investment approach.
Posts about Cubist Evaluation:
Mixed methods, bricolage, and eclectic methodological pluralism
My conference presentation where I first introduced the Cubist Evaluation idea
Other topics
Evaluation vs value for money - how VfM assessment differs (or not) from evaluation
Worthwhile travel time - valuing subjective experience of time in transport policy
Guest post: Functional review of a VfM strategy, by Danish Khan
The story so far - a retrospective on my first 6 months of Substack articles