Rapid value for money assessment
Four guiding principles, and four examples
I often get asked things like: "we're just a small program with a modest budget - how can we assess our value for money without breaking the piggy bank?" or "we don't have time for a full VfM assessment - we need to make a decision, and we need it yesterday!"
In any evaluation, the level of effort needs to be kept proportionate to the program and the purpose of VfM assessment. We want our VfM assessment to be good VfM!
So, what can we do when we're constrained by limited budget, time, or data? How can we provide the best answer to a VfM question with the resources we have? What's the minimum we should do to get a decent VfM assessment?
Here are four guiding principles for a good VfM assessment, and four examples of rapid, minimalist ways to implement them.
Four guiding principles
1. Use evaluative reasoning as the glue to combine evidence and values
Like any good evaluation, VfM assessment needs an explicit framework to help us get from evidence to evaluative judgements. There are lots of approaches to evaluative reasoning and you’re free to choose any. I often use rubrics. This involves defining criteria that represent features of good VfM, and standards that describe the difference between excellent, good, adequate, and poor VfM. This Guide explains how.
Compared to the way VfM assessments have been done in the past, this might feel like extra work - but rubrics are a transformative way to make an evaluation valid, credible, and useful. The time you invest up-front in developing rubrics pays back by supporting clear thinking and efficient work processes throughout the remainder of the VfM assessment.
Rubrics support clarity at every step of a VfM assessment. Developing rubrics brings stakeholders to the table to co-define what matters (criteria) and what good looks like (standards). Rubrics clarify the scope of the assessment and what evidence needs to be collected. They provide a framework for organising the evidence, an agreed basis for making sense of the evidence, and a logical structure for reporting findings.
When we use rubrics, we can tailor the criteria and standards to the scope and scale of the program, and to the available resources and time frames for the VfM assessment. The reasoning process remains the same, but we can vary the depth and breadth of the rubrics to make them fit for purpose.
2. Include evidence from multiple sources, in an intentional way
Traditionally, VfM assessments have tended to privilege quantitative methods. Keep using those, but don't stop there. We need a broader mix of evidence (qualitative and quantitative together) to build up a fuller picture of what's going on and to understand the story behind the numbers.
Jennifer Greene says mixed methods can strengthen evaluation in a number of different ways. For example, comparing different types of evidence can reveal to us where different sources of evidence agree and where they don't. We can get a broader and deeper view of the evidence by drawing on the relative strengths of different sources. We can hold the space for a greater diversity of values. We can use the results from one method to inform the design of another. These are all reasons to use mixed methods to strengthen evaluation.
What methods and tools you use, how you address causality, contribution, complexity… That's for you to decide. All I ask is: recognise the value of multiple forms of evidence, and don't rely on a single source of evidence if you can triangulate evidence from multiple sources.
3. Include stakeholders in co-designing the evaluation and making sense of the evidence
Thomas Schwandt has described evaluative thinking as a collaborative, social practice. I agree. At every step we have the opportunity (and often the obligation) to include stakeholders, end users, anybody with a right to a voice in the evaluation.
An inclusive, participatory approach supports people's understanding of evaluation, their ownership of the evaluation, their input into how it's designed and conducted, the credibility and validity of the evaluation findings, and makes it more likely you'll end up with a useful VfM assessment that people will actually use.
Who you include and how you include them is a contextual decision. All I ask is that you're not sitting alone in your evaluator cave, making judgements that affect others without engaging their expert knowledge and values.
4. Combine thinking from evaluation and economics
When we're posing evaluative questions about VfM, we're asking if something is good use of resources. Evaluation is the discipline responsible for determining how good something is. Economics is the discipline that studies how people use resources. Both disciplines are interested in value. Neither discipline has all the answers. So let's use both disciplines together.
From evaluation, you can draw on the core theory and practice that underpins making evaluative judgements (rubrics are one example). You’re also free to select an appropriate mix of evaluation strategies, theories, frameworks, methods and tools for your context. These sorts of contextual decisions are guided by evaluative thinking.
From economics, you might choose to include methods like cost-benefit analysis (CBA) or cost-effectiveness analysis (CEA). You could also bring in robust ways of thinking about costs (e.g., the concept of opportunity cost, sunk costs, fixed and variable costs). And useful ways of thinking about efficiency (e.g., allocative, technical and dynamic efficiency). These are just examples. They may be unfamiliar, but they're often important in VfM assessment and they're not hard to learn. What economic concepts you draw on will vary from case to case. Hot tip - get an economist on your team.
Four examples of small-budget approaches
These guiding principles I’ve just described may sound highfalutin, but you can vary the scale and complexity of the assessment to fit the circumstances. Here are some concrete examples of things you can do in a rapid VfM assessment - one for each principle.
1. Prioritise criteria
There are many potential criteria of good resource use, as I mentioned last week. Including all of them could get overwhelming and expensive. Good news - you don't have to include them all! Some will be critically important. Some will be less important. Some may be irrelevant. Designing an evaluation always includes drawing a boundary somewhere. Smaller budget, smaller boundary.
Different criteria are going to be important at different times in the life-cycle of a program. For example:
Informing a funding decision? You might want to focus on criteria like Relevance and ex-ante Return on Investment
Early days of a new program? You could prioritise Economy, and Equity of design
Mid-cycle? Top criteria could include Efficiency, and Equity of delivery
Endline? Maybe look at Effectiveness, Sustainability, and Equity of outcomes
Retrospective? Consider ex-post Return on Investment, and Equity of impacts.
Similarly, when you’re defining each criterion you can prioritise a minimum set of the most important sub-criteria that get to the heart of the program. And when you’re defining standards, think what are the key things that make a difference between ‘just good enough’ and ‘not good enough’, between ‘good’ and ‘great’, and so on.
In short: focus on the big stuff, and only bite off what you can chew.
2. Use the evidence you have to hand
In my early consulting years I had a colleague who talked about not "trying to drink the ocean". I like this metaphor. We'll never know everything we'd ideally want to know. And if we did, we'd drown in it. Instead, borrow an economic principle from Pareto: the first 20% of effort should get us about 80% of the evidence we need. Figure out what the best 20% looks like, and start with that.
Often, the low-hanging fruit includes evidence that already exists - e.g., in databases, surveys that have already been run, reports that have already been written, the advice of a few key experts (including but not limited to researchers, managers, frontline staff, and community representatives).
One day my colleague Judy Oakden called me, with an interesting challenge. While collating data in an evaluation that had not planned to look at VfM, she saw that there was possibly sufficient evidence to add a VfM component. This would considerably strengthen the evaluation and she realised that from the client’s point of view, this would be valued. She approached me to look at the existing data they had, to assess what warrantable claims we could make from a VfM perspective.
We decided there was enough data to add a section on the value generated by the program. So we worked together to define the value proposition of the program, and used the available evidence to make an assessment of the extent to which the program was meeting its value proposition - including transparency about what that evidence couldn’t tell us. You can see the report here. I think Judy Oakden and Kellie Spee did a great job within the constraints of the project!
3. Rapid sense-making workshops
What if all you needed to do was harness the expert knowledge of key stakeholders? Sometimes a workshop can provide the reasoning, the evidence and the engagement all rolled into one.
To illustrate, you could:
Organise a workshop of (say) 6-8 key stakeholders
Ply them with coffee
Facilitate a discussion to identify VfM criteria, prioritise the most important ones, and reach agreement on 'what good looks like'
Break for lunch
Come back and make a rapid assessment of how well the program is performing against the criteria - tapping into the formal and tacit knowledge of the group to judge performance, record the rationale, and brainstorm opportunities to improve VfM.
In one project where the organisation’s internal M&E team had already developed their own rubrics, the VfM assessment was completed in a 90-minute workshop! Usually though, it takes a little longer.
In another project, we held two 3-hour workshops, two weeks apart, for criteria development and judgement-making respectively. In between, we set participants homework of finding and reviewing available evidence.
These sorts of processes can work well when participants already have good evidence or experience of the program, and when the principal value of the assessment is the opportunity to reflect, learn and identify areas to strengthen VfM.
4. Include break-even analysis
Not enough time, budget, or data to do a full CBA? Consider doing a break-even analysis. While a CBA comprehensively identifies, values and compares the benefits and costs of a program, break-even analysis has the more modest aim of estimating what it would take (or what we'd have to assume) for benefits to just equal costs. This can give us a sense of the performance threshold a program would have to meet to give a positive return on investment. Even with incomplete information, break-even analysis can often help us make a defensible judgement about the prospects of a program creating more value than it consumes. For examples, see here and here.
Bottom line
VfM assessment needs to be proportionate. Four principles provide a minimum specification for designing and implementing a quality VfM assessment: evaluative reasoning, mixed methods, stakeholder participation, and inter-disciplinary thinking.
Like a good tailor, you can choose cloth that suits the budget and trim it to the right size. But just like a tailor, there's always a minimum level of skill and effort required to do a decent job. Too cheap or too quick will end in disappointment, perhaps even harm.
I’ve only shared four examples of minimalist strategies. They’re useful strategies but they’re not the last word, they aren’t suited to every occasion - and they’re no match for your creativity. Go forth and problem-solve!
***
Huge thanks to my first paid subscribers! This whole Substack thing started as an experiment. Thank you for seeing value in it and for pledging your support. I feel very encouraged knowing my work matters to you!