Discover more from Evaluation and Value for Investment
Evaluative reasoning in complexity
Blending democratic and technocratic approaches 🎈
Making value judgements on a transparent, defensible basis is a core function of evaluation, and there are many ways to approach it. Rubrics are one way of scaffolding evaluative judgements. There are multiple kinds of rubrics, and there are other options besides rubrics. This leaves us with many options when it comes to choosing an approach to evaluative reasoning.
To access earlier posts, check out the free 7-day trial. All new posts are free for the first two months. Paid subscribers have ongoing access to the full archive.
How to choose a suitable approach? I don’t think there can be a simple rule. Like so many things in evaluation, it’s a contextual decision based on multiple considerations and not always the same ones. However, I do think one important consideration is the nature and extent of complexity.
Complexity and the Stacey Matrix
In a recent LinkedIn post, Jeroen Kraaijenbrink gave an overview of the Stacey Matrix, a model for understanding complexity based on the level of certainty (or uncertainty) about future events and consequences of our actions, and the level of agreement (or disagreement) among stakeholders about the issue or problem and the potential consequences of our actions. Here’s Kraaijenbrink’s illustration of the Stacey Matrix. This is, of course, only one way of understanding complexity but it’s the one I’m going to focus on today.
The purpose of the Stacey Matrix, Kraaijenbrink says, is to help us choose decision-making strategies under different degrees of uncertainty. For example, if there is high certainty and high agreement, we are working in a simple space where decision-making is straightforward and we can rely on rationality and “best” practices to make decisions. On the other hand, as things become less certain and/or less agreed we may end up in situations that are complicated, complex, or even chaotic, in which judgement, politics and negotiation become increasingly important.
For example, complexity is characterised in this matrix by low certainty OR low agreement. Since we can’t fully anticipate or agree what will happen, an experimental or trial-and-error based approach to decision-making is likely to be more appropriate than a rational, deductive or optimising approach. If there’s low certainty AND low agreement, the situation is described as chaotic. Under these circumstances, it might not be possible to reach a sound decision. The strategy here is to try and increase certainty and/or agreement enough to move us into the realm of complexity.
Approaches to evaluative reasoning
I think there’s some wisdom here that we can use to help inform our selection of approaches to evaluative reasoning (combining criteria, standards and evidence to reach a warranted value judgement). There are many approaches to evaluative reasoning, as I’ve outlined in previous posts but to briefly recap:
Approaches to evaluative reasoning include tacit, all-things-considered, deliberative, and technocratic approaches (Schwandt, 2015).
Technocratic approaches to evaluative reasoning can be further broken down and include if-then statements, cost-benefit analysis, numerical weight and sum, qualitative weight and sum, and rubrics (although I am labelling all these as ‘technocratic’, they can be co-created and used in participatory or democratic ways).
Rubrics can take different forms including generic, analytic, holistic, or hybrids of these (see my earlier post for descriptions and illustrations of these different kinds of rubrics and trade-offs between them).
These approaches aren’t mutually exclusive. We can combine them. Different combinations of approaches may be preferable in different sets of circumstances.
Evaluative reasoning and the Stacey Matrix
The diagram below illustrates how I think the Stacey Matrix might be able to help inform the selection of contextually appropriate evaluative reasoning strategies.
Some general points:
Although the diagram visually suggests four categories of complexity, I prefer to think of it as a continuum.
Throughout the continuum, the General Logic of Evaluation applies, but the way we apply it can vary. As the situation tends toward greater complexity, the emphasis on politics and stakeholder participation increases, while technocratic structure and rational analysis can play a supporting role.
Throughout the continuum, stakeholder values are critically important and stakeholder engagement is (I would argue) almost always desirable. However, it may be that in simple circumstances, and even a few complicated circumstances, it is sometimes sufficient to rely on secondary or remote data about what stakeholders value (e.g., monetary valuations or policy documents). Conversely, the more complex our situation gets, the stronger the imperative to directly engage stakeholders in evaluation co-design and in making sense of evidence.
Strategies that are appropriate at higher levels of complexity (i.e. participatory approaches) are likely to also work at lower levels of complexity.
Strategies that may work at lower levels of complexity (i.e. technocratic approaches) are likely to be inappropriate or risky at higher levels of complexity unless they’re used in conjunction with democratic approaches.
I have included some examples of evaluative reasoning strategies at each level, but they are just examples.
To some degree, the ‘agreement’ axis may align with the ‘values’ component of evaluative reasoning and the ‘certainty’ axis may align with the ‘facts’ component - but I’ve resisted labelling them as such because the distinction isn’t absolute; values are facts, and facts are value-laden. There may be disagreement not only about values, but also about alternative interpretations of facts (that’s how science progresses), and there may be uncertainty not only about the issue or problem, but also about how people value things (e.g., because values vary between people and across time). This raises questions for me about whether we are ever really working in a ‘simple’ space.
As I’ve argued before, I have yet to find an evaluative reasoning strategy as practical and versatile as rubrics. My view (borne out by experience) is that rubrics can assist in the evaluative reasoning process at any level of complexity, though the ways they assist can vary. Used in participatory ways, a rubric is more than just a technocratic matrix of criteria and standards. It’s a stakeholder engagement tool:
The rubric development process can be an inclusive, power-sharing approach to collectively negotiate and articulate what matters (criteria) and what good looks like (standards)
Using rubric development as a focal point provides a doorway into important conversations about values that otherwise might not take place or might not reach such clarity
Negotiating criteria and standards helps stakeholders to recognise, accommodate and blend different viewpoints and ensure the evaluation doesn’t just reflect the values and assumptions of evaluators and evaluation commissioners
The processes of co-creating and using rubrics with stakeholders helps to demystify how evaluative judgements are made. This supports understanding, ownership, satisfaction with, and use of the evaluation.
You’ll see that the examples in the diagram include rubrics at every level from simple to complicated to complex - but the type of rubric and the way it is used varies:
In a simple context (high certainty and high agreement), analytic or holistic rubrics provide specific, detailed guidance for evaluating performance against a set of clear expectations - for example, assessing implementation fidelity against well-evidenced and well-accepted success factors. In some cases, desktop evaluation (without stakeholder engagement) may be feasible.
In a complicated context (medium certainty and/or medium agreement), any rubric structure could be made to work, and stakeholder engagement will be an important element of how the rubrics are developed and used.
In complex situations (low certainty OR low agreement), collective dialogic sense-making and judgement-making with stakeholders becomes increasingly important. These processes can still lean on technocratic approaches for a bit of structure. For example, generic rubrics can provide light-touch guidance to facilitate and focus the dialogue, with enough flexibility to make appropriate judgements. Analytic or holistic rubrics can work too if they contain suitably broad-brush descriptions, avoiding the temptation to over-specify expectations beyond what is knowable in advance. As we often find ourselves evaluating in complexity, I’ll share an example of what a light-touch rubric can look like:
In chaotic situations, according to the Stacey Matrix, certainty and agreement are both too low to enable clear conclusions to be reached. Therefore, evaluative judgements may not be warrantable: the criteria, standards and evidence upon which they are based may be too variable and unstable. Here, rubric development may still assist in facilitating agreement on a set of high level principles, representing just enough convergence of minds to nudge us toward complexity where we can make warrantable judgements. In this case, rubric development may assist in bringing us back through what the Stacey Matrix calls the “edge of chaos” into slightly more manageable territory.
Of course, the fundamental evaluation approach is also going to vary with complexity. For example, we’re more likely to use Developmental Evaluation in complex, emergent situations than simple ones. Conversely, tools like cost-benefit analysis are well-suited to simple and complicated situations where mathematical models can be structured and parameterised to give a reasonable representation of a system.
CBA can also be used to ‘cut through’ complexity by reducing a policy or program to two key aspects of performance (monetary valuations of resource use and outcomes). We just need to bear in mind that the complexity doesn’t actually go away. If we’re going to use rational, technocratic tools in high-complexity situations, we should use them with caution and in combination with complexity-informed approaches. A complexity lens can be valuable to inform evaluation designs, guide sensible interpretation of findings, and provide complementary insights.
🎈I’m flying a balloon here. Let me know your thoughts. Is the Stacey Matrix a helpful heuristic to guide us in tailoring our evaluative reasoning strategies to different contexts?
Update, 23 Nov 2023
Thanks to some great engagement on LinkedIn, I learned that the Stacey matrix “is deeply unfashionable in some complexity crowds since Stacey himself disowned it” (thanks Thomas Aston), and I’ve explored adapting the Cynefin framework as an alternative navigation aid for thinking about the different approaches we can take to evaluative reasoning in different circumstances.
Cynefin is, of course, another one of the many different ways of understanding complexity. As far as I’m concerned, there isn’t any one ‘right’ way to look at these things, nor should there be. Uncertainty and disagreement are part of the landscape we work in, and that includes diverse views about multiple options for understanding the things we evaluate. Also, I’m only scratching the surface of a deep and, well, complex topic (e.g. as Cathy Shutt rightly noted, there are epistemological dimensions to selecting evaluation approaches, and I would argue axiological ones too).
As Bob Williams commented in the LinkedIn convo:
“Many systems folks as well as complexity folks argue that all situations are complex. We can choose to consider situations as if they were simple or complicated, but that is more as a way of helping us address the impact of these complexities. Rubrics are excellent examples of seeking to ‘simplify’ the complex so that we can actually do something - in this case make evaluative judgements.”
If a free subscription feels like good value, sign up now!