Evaluative reasoning and VfM
An intuitive process to get from questions to answers
If you’re looking for answers to value for money questions, here's a step-by-step process anybody can pick up and use.
Last week I outlined an inter-disciplinary system to help you combine evaluative and economic thinking to answer value for money questions. The previous post focused on the what. This one’s about the how. Here's an 8-step process that gives you a road map to get from questions to answers. You can use these steps to evaluate anything.
Start by understanding the program (Step 1), including its context, stakeholders, value proposition, and evaluation questions. Then, define criteria (Step 2) - aspects of VfM to focus on, aligned with the value proposition. The 5Es (economy, efficiency, effectiveness, cost-effectiveness, equity) are a generic example of VfM criteria; you need to develop specific criteria, tailored to context.
Next, define standards (Step 3) - levels of VfM, what 'good' VfM looks like, as well as excellent, adequate, and poor VfM. Criteria and standards (set out in a matrix called a rubric) provide an agreed set of lenses for making sense of the evidence. They guide every step of the evaluation process, from determining what evidence is needed all the way through to structuring the report.
Once you’ve developed rubrics, you can have an informed conversation about what evidence you need to collect (Step 4), where that evidence is going to come from, and what set of methods you should use. This could include qualitative, quantitative and/or economic methods, probably a mix.
After that, you can implement the evaluation. Gather the evidence (step 5) and analyse it (step 6), applying context-appropriate research and causal inference methods to understand what happened, what changed and what difference it made. Then use the rubrics to interpret the evidence and make transparent judgements on an explicit basis (step 7). Like, to what level of VfM does the evidence point? What are the strengths and opportunities to improve? What’s been learned?
And finally you can report the findings (step 8), in a way that gets straight down to business, answers the important questions and isn't torture to read!
All the way through, you're involving stakeholders (e.g. evaluation users and people with a right to a voice) in evaluation co-design, fact-finding and sense-making, building their values and expert knowledge into the fabric of the evaluation.
And all the way through, you’re combining economic and evaluative thinking. For example, economic concepts may be embedded in program theory, rubrics, evidence gathered, methods of analysis, judgement-making, and reporting. But you’re not limiting the evaluation to economic methods and metrics alone. You’re bringing your full evaluation tool kit and carefully selecting the right mix of methods for the context. The process is flexible and adaptable. This approach brings guiding principles, not rules.
You can do it! Free resources on my website; practical guide here; case example here. Training workshops coming up with ANZEA, AES and UKES. Experiential learning (learning by doing) is the best way to get the hang of rubrics. I can help you with that too.
Some readers will have seen this process before. Thanks for staying with me while I lay the foundations for my new connections. All-new content coming next week!
Kia ora rā