Interpreting evidence through the prism of explicit values
Short read
About ten years ago, while preparing some PowerPoint slides, I wanted a metaphor that would bring together the concepts of evaluative reasoning and mixed methods in one simple picture. The concept I landed on was a prism. Eventually, after many iterations, it became the VfI logo. One of the early versions looked like this:
My prism is backwards
If you studied high school physics, my prism might make you grimace. A prism splits white light into its component colours. Mine shows the process in reverse and, as a few people have told me, “that’s not how prisms work!”
Well, actually…
Sir Isaac Newton, who lived in a time before smartphones, amused himself by fooling about with prisms and lenses. He conducted a series of experiments to figure out how light and colours work. You can read his First Book of Opticks here, thanks to the Newton Project at the University of Oxford.
In one of his experiments, he ran a prism in reverse! In the following diagram (which is read from right to left), the light entered the experiment at point 0 and was dispersed into a spectrum by the first prism (labelled “ABC”). The spectrum was then re-focused by a convex lens (MN) and went into a second prism (DEG) which recombined the spectrum back into white light. After that, the white light passed through a third prism (HIK) and was dispersed into a spectrum again.
Disperse, refocus, recombine: a metaphor for an evaluation process
The process of designing and implementing an evaluation involves moving from holistic to analytic and back to holistic thinking - zooming in from context to program to aspects of the program, making values explicit, collecting facts to address the values, using the values to interrogate the facts, and putting it all back together into a cohesive whole. Disperse, refocus, recombine. This process mirrors Newton’s experiment in the diagram above.
Here’s how it aligns with the 8 steps of the Value for Investment process:
Step 1 involves gaining an understanding of the program, first thinking holistically and systemically about the context, value proposition, stakeholders and what is at stake, end users of the evaluation and their information needs, and so on. This conceptualising stage starts by exercising the right hemispheres of participants’ brains, tapping into their creativity, intuition, tacit knowledge, visual and nonverbal thinking before bringing in the left hemisphere and organising the concepts analytically, separating and organising evaluative questions, program theory, stakeholder groups, etc, into component parts.
Our journey from holistic to analytic thinking progresses as we move into steps 2-3, creating the prism (e.g., a rubric). In physics speak, splitting a beam of light into a spectrum is called dispersion. Similarly at this stage of the evaluation design we are identifying aspects of performance that matter (criteria) and what would constitute good performance on those aspects (standards). An agreed set of criteria and standards gives us a road map to further analyse the evaluation problem, identifying the pieces of evidence that are needed, and an appropriate mix of methods to gather and analyse the evidence (step 4).
Then we start to refocus - just like the convex lens in Newton’s experiment. We shift from designing the evaluation to implementing it. We collect data (step 5) and analyse it (step 6), producing multiple strands of evidence (like the spectrum of colours). We use the prism (rubric) to interrogate and synthesise (recombine) the evidence to get a lucid beam of evaluative clarity (step 7). Now we have nice clear findings to report (step 8).
A reflective, reflexive practice
Of course, none of this happens by magic (aka physics), as light does. The process of moving from holistic to analytic and back to holistic thinking is a messy, deliberative process at the interface of rigorous thinking and human interaction, part science, part art, part alchemy.
As a Māori colleague expressed it to me:
Atamaha (the prism) is a metaphor for the process of systematically combining evidence, tacit knowledge and values to help make sense of complexity and reach a place of transparency, honesty and clarity in order to make good decisions.
Hikina ake te kohu kia mārama te titiro (lift up the fog to see clearly).
The Dark Side of the Moon
Some 269 years after Sir Isaac’s experiment, in March 1973, Pink Floyd released their iconic album, The Dark Side of the Moon. The vinyl record came in a gatefold sleeve which, opened up, looks a little familiar…
The Dark Side of the Moon is preserved in the US National Recording Registry for its cultural, historical, and aesthetic significance. Celebrated for its sonic clarity and immersive beauty, the album weaves musical and electronic elements into a seamless, symphonic flow. The cover design reflects the band’s light shows, with the triangle symbolising thought and ambition as conveyed through the lyrics, which Roger Waters described as “an expression of political, philosophical, humanitarian empathy that was desperate to get out” and “an exhortation to join the flow of the river of natural history in a way that’s positive…”. The album was the band’s embodiment of truth, beauty, and justice. A worthy metaphor for evaluation.
Thanks for reading!
This is a rewrite of a post I shared two years ago. Since then, over 1,000 new subscribers have joined. From time to time, I’ll revisit, update and reshare earlier pieces.