Square peg, round hole
Integrating top-down criteria with what actually matters to stakeholders
Apollo 13 (April 1970) was meant to be NASA’s third landing on the moon. However, after an oxygen tank exploded 56 hours into the trip, 330,000km away from Earth, the mission was aborted and the crew used the lunar module as a lifeboat to try and get home. Amazingly, they made it. Engineers at Mission Control raced against the clock to solve a life-saving puzzle: How to fit the command module's square lithium hydroxide canisters to the lunar module's round holders. Without this, the lunar module's CO₂ scrubbers would be insufficient to maintain a breathable atmosphere for the astronauts. Using only materials available on board, like plastic bags, hoses from spacesuits, and duct tape, they successfully got a square peg to fit a round hole, ensuring the crew could keep breathing long enough to return. You might remember this scene from the 1995 movie:
Though all astronauts are evaluators, very few evaluators are astronauts. Nonetheless, we’re no strangers to square peg, round hole problems. For example, how about developing evaluation criteria that reflect what matters to stakeholders, while meeting top-down expectations of organisations to use a boxed set of criteria like OECD DAC (relevance, coherence, effectiveness, efficiency, impact, and sustainability) or the 5Es (economy, efficiency, effectiveness, cost-effectiveness, and equity)?
Marrying stakeholder values with top-down criteria doesn’t require plastic bags or duct tape - but it does involve similar levels of pragmatism and improvisation. First, we have to find out what stakeholders actually value. Then we need to classify their context-specific definitions of value within the standard criteria (if they fit). Finally, we might have to modify the list of criteria (by adding some or taking some away) so that the concepts are all correctly classified. Of course, all this is going to require decent skills in facilitation, negotiation and wordsmithing. See last week’s post for practical tips on engaging stakeholders in this process.
Criterial vs experiential evaluation - a false dichotomy?
Saville Kushner recently shared a short paper on LinkedIn which, as always, is well worth a read. In it he mentioned (after Stake) the contrast between “criterial approaches” in which criteria are handed down from on high, and “experiential approaches” in which criteria are determined and defined contextually. This is exactly what I’m talking about here. As far as what should happen for evaluation validity and use, I am firmly in the latter camp as readers of this Substack series will know. However, I’m also a pragmatist and I live and work in a world where evaluation criteria are often predetermined by commissioners. Fortunately, I don’t see criterial and experiential approaches as mutually exclusive. With a bit of mental gymnastics we can reach a satisfactory amalgam of both.
New guide
To help with developing contextually meaningful criteria in value for money (VfM) evaluations using the 5Es, I’ve put together a PDF guide based on my recent blogs on this topic. It’s free to download.
King, J. (2024). Value for Money and the 5Es: Designing a context-specific VfM framework. Julian King & Associates Ltd.
Examples
For examples of evaluations that defined the 5Es contextually to reflect stakeholder values, see:
Field, A., King, J., Moss, M., Parslow, G., Schiff, A., Chianca, T.K., Chianca, G., Mattiello, C. (2024). Urban 95/Ateliê Navio Value for Investment evaluation. Report for Van Leer Foundation. Dovetail Consulting Limited, Auckland.
Field, A., Crocket, A., Garden, E., King, J., Moss, M., Parslow, G., Schiff, A., Spee, K., Wehipeihana, N. (2023). Youth Primary Mental Health and Addictions Evaluation. Final Report for Te Whatu Ora – Health New Zealand. Dovetail Consulting Limited, Auckland.
Pipi, K., McKegg, K., Moselen, H. (2022). Ngā Wānanga o Hine Kōpū Evaluation Summary Report. Te Whatu Ora – Health New Zealand.