Offering an updated training workshop in 2024
Discover more from Evaluation and Value for Investment
Offering an updated training workshop in 2024
A refresh of my current workshop, with renewed emphasis on rubrics and stakeholder-led valuing to answer Value for Investment questions
Originally posted 15 Nov 2023; Updated 22 Nov 2023
Value for Investment workshop
Here’s my new workshop offering for 2024. I’ll adapt its focus to offer tailor-made workshops addressing conference themes, such as Made in Africa VfM assessment for AfrEA, and VfM through collaborative action, thinking and practice for EES. Fingers crossed! I’ll also continue to offer my existing half-day, full-day and two-day workshops on evaluation and value for money.
Workshop overview
Effective resource allocation, learning and accountability are crucial for successful program impacts. This workshop, designed for evaluators at all levels, introduces the innovative Value for Investment (VfI) approach to assessing value for money (VfM). While building on familiar evaluation and economic principles, this workshop emphasises the role of VfM assessment led by the perspectives of local stakeholders and affected communities, while also meeting the needs of donors.
Workshop content
This interactive workshop places a spotlight on evaluative rubrics as a powerful tool for synthesising criteria, standards and evidence to make warranted judgements about VfM. Participants will delve into key topics including the collaborative development of rubrics with stakeholders, utilising rubrics for evaluative judgements, and using mixed methods evidence in VfM assessment. The workshop aims to empower evaluators to conduct VfM assessments that ensure representation of grassroots perspectives, bridging the 5Es (economy, efficiency, effectiveness, cost-effectiveness, and equity) with context-specific value from the viewpoints of those impacted.
Key topics include:
What is value for money?
What are rubrics and why use them?
A step-by-step process for developing and using rubrics in evaluation
Co-developing stakeholder-led VfM rubrics
Making sense of mixed methods evidence with stakeholders and rubrics
Reporting findings effectively, to reflect stakeholder values and meet donors’ needs.
Workshop objectives
Participants will learn how to engage stakeholders to co-create value by:
Articulating a program’s value proposition from the perspectives of those whose lives are affected
Co-developing rubrics that define VfM in program-specific terms
Using rubrics and mixed methods evidence to make nuanced evaluative judgements.
Who should attend?
This workshop is tailored for evaluators and commissioners aiming to enhance their VfM assessment skills, with a focus on aligning reporting with the local cultural context and donor expectations. While the content focuses on VfM assessment, the underlying principles are transferrable to any evaluation. The content is applicable to evaluators of all levels, from novices to experts.
Note
Value for money transcends financial metrics; it’s about good use of all invested resources to create value for people. This workshop advocates for the term “Value for Investment” to underscore that programs are more than costs - they are investments in value propositions. The approach encourages evaluators to adopt methods and tools they already possess, while borrowing valuable insights from economics, to provide meaningful answers to VfM questions. This workshop doesn’t include training in economic methods of evaluation as there are many other training courses already on offer that focus on these methods.
About the facilitator
Julian King, a specialist in evaluation and value for money, introduces the Value for Investment approach, a model combining insights from evaluation and economics. This workshop draws from his PhD research and extensive experience in assessing complex policies and programs globally. A regular presenter at evaluation and development conferences, Julian provides evaluation capability building through workshops, postgraduate education, mentoring, academic publications and blogs.
And here’s the original post and backstory from 15 Nov 2023:
It’s time for a refresh. I’ve been delivering training in the Value for Investment (VfI) approach in various forms since the Aotearoa New Zealand Evaluation Association (ANZEA) Conference in 2012. It has evolved significantly over the years, especially since completing my PhD on the topic in 2018.
In addition to ANZEA, I’ve delivered VfI training for members of the Australian, European, United Kingdom and American evaluation societies/associations, and regularly provide in-house training for consulting firms, government departments and non-governmental organisations.
I update the workshop each time, to reflect the interests and backgrounds of different groups and to incorporate new examples, exercises, and refinements to modules. No two workshops are the same, but the updates since 2018 have been incremental.
Next year’s a bit different. I know from feedback and questions on these Substack posts, as well as discussion in workshops, that there’s an appetite to dive in deeper on the practicalities of developing and using rubrics with stakeholders, and shifting the locus of power so that VfM assessments are led by the values and worldviews of people whose lives are affected.
Now, I’ve only had one coffee this morning so far and my thinking is a little fuzzy. So to help me get started, I gave ChatGPT my initial thoughts and asked it to have the first go. I didn’t even have time to make my second coffee before it came back with quite a good suggestion. Not perfect, but a solid foundation I can build on.
I’ve learned that it’s possible to get a reasonable quality first draft from ChatGPT, but there are some important provisos:
The lead author is the human operator, not the GPT. It’s on me to fact-check and sense-check everything it offers. It’s on me to edit the draft so that it says what I mean it to say.
I can influence the quality of the GPT’s work by giving it background information to work from, and setting clear expectations.
It’s imperative to treat the chat as a public domain conversation. Don’t share anything with ChatGPT that you wouldn’t share on social media. Accordingly, I’m happy to share the whole conversation here…
Here are the instructions and background I gave
I'm going to ask your help to write a proposal for a new training workshop. The workshop teaches program evaluators about a new approach to assessing value for money (VfM), called the Value for Investment (VfI) approach.
The VfI approach enables rigorous assessment of VfM in contexts where economic methods and metrics are challenging to apply or insufficient to comprehensively evaluate programme resources, processes, consequences, and value.
I already have a workshop that I have been delivering for several years. I'm going to give you some background text in two sections. In the first section, I'll show you the abstract for the existing workshop. In the second section, I'll explain to you how the new workshop is going to be different. Then, I will ask you to draft the abstract for the new workshop.
Certainly, I'd be happy to help you draft the abstract for the new workshop. Please go ahead and provide me with the background text for the existing workshop and the information on how the new workshop will differ.
First, here's the abstract for the existing workshop.
Workshop Overview
It is important for good resource allocation, accountability, learning and improvement that policies, programs, and other initiatives undergo rigorous evaluations of value for money (VfM). Many evaluators, however, lack training and experience in this area. This workshop offers a set of techniques to address that gap. Combining strengths of evaluation and economics, this approach, called Value for Investment, builds on theory and practice that will already be familiar to many evaluators, although there are also many evaluators and commissioners for whom these techniques may be new. The approach is intuitive to learn and practical to use.
Workshop Content
This participatory workshop provides practical guidance, underpinned by a sound theoretical framework, for evaluating VfM. It focuses on unpacking a process of explicit evaluative reasoning (using rubrics) and the use of mixed methods (qualitative, quantitative, and economic).
A sequence of steps will be shared, that when followed, help evaluators and commissioners develop and use context-specific definitions of good performance and VfM. These definitions provide a system for ensuring the evaluation: is aligned with the design and context of the policy, program, or initiative; collects and analyses the right kinds of evidence; draws sound conclusions; and answers VfM questions. Exercises and examples will be shared throughout the workshop.
Session One covers the practical steps involved in designing a VfM framework. Session Two covers the steps involved in conducting a VfM assessment, as well as some evaluation theory and conceptual principles underpinning the approach, and a few tips and tricks for applying the approach in the real world.
The workshop includes a brief overview of economic methods of evaluation (e.g., cost-benefit analysis) including considerations for determining when to use them in a VfM assessment, and when to combine them with other evidence and values. Note that it doesn’t provide detailed instruction in the design and implementation of economic evaluations.
Workshop Objectives
Participants will learn how to:
frame an evaluative question about VfM
develop rubrics, comprising criteria (aspects of performance) and standards (levels of performance), to provide a transparent basis for making sound judgements about VfM
combine multiple sources of evidence to support nuanced evaluative judgements
incorporate economic evaluation within a VfM framework, where feasible and appropriate
interpret the evidence, using criteria and standards, to make sound evaluative judgements
present a clear and robust performance story, guided by the criteria and standards.
Who should attend?
The workshop is designed for evaluators and commissioners of evaluation at all levels, from novice to expert. It is suitable for evaluators and commissioners who want to learn about rubrics generally and, in particular, for those who want to learn how to apply rubrics and mixed methods to VfM specifically.
About the facilitator
Julian King specialises in evaluation and value for money, helping decision makers, evaluation teams and stakeholders to assess the value of policies and programs. Julian is the Director of Julian King & Associates Ltd, a member of the Kinnect Group, an Associate of Oxford Policy Management, an Honorary Fellow at the University of Melbourne and University Fellow at Charles Darwin University. Julian’s PhD developed the Value for Investment approach, a model for combining evaluative reasoning with economic evaluation. The approach is used globally to evaluate complex and hard-to-measure programs and policy reforms.
Next, I'll explain how the new workshop will differ.
Thank you for providing the detailed background on the existing workshop. Please go ahead and share the information on how the new workshop will differ, and I'll help you draft the abstract for it.
Now, here are some notes on how the new workshop will be slightly different.
It will cover essentially the same content but with a different emphasis. The new workshop will focus on empowering evaluators to meet the VfM reporting needs of donors while understanding and communicating VfM (which I define as "good resource use") and value creation from the perspectives of local stakeholders and affected communities.
The workshop will focus on the use of evaluative rubrics as a way to support: 1) sound evaluative reasoning (i.e. synthesising criteria, standards, and evidence to make warranted judgements about VfM); and 2) stakeholder participation in rubric co-design and in using the rubrics to make judgements.
Key topics would include:
What are rubrics?
Why use rubrics?
How to develop rubrics with stakeholders
How to use rubrics with stakeholders to make evaluative judgements from the evidence
How to simultaneously meet the needs of donors and represent value from a grassroots perspective by building a bridge between the 5Es (economy, efficiency, effectiveness, cost-effectiveness, and equity) and actual value from the perspectives of people whose lives are affected
How to make sense of mixed methods evidence
How to report findings
Here are some additional ideas to incorporate:
Value for money is never just about the money. It's about good use of all the resources invested in a program, contributing to impacts that add value to people's lives. Programs are more than costs - they're investments in a set of value propositions. That's why I prefer the term "Value for Investment".
In this training workshop I'll walk participants through a practical, intuitive process for using methods and tools evaluators already have, and borrowing some useful ideas from economics, to ask clear evaluative questions and provide clear answers.
The first time I run this new workshop may be for African evaluators. I want to offer a vision of what made-in-Africa VfM assessment might look like. That means putting African values in the driver's seat to determine what good resource use and value creation look like. And it means creating space to discuss and address challenges in doing this.
One person gave me this advice: "I think like in many settings, challenges would vary depending on projects, programmes and sectors being evaluated and as well as type of stakeholders/client whether private sector, government, or I/NGOs, among others. For me, context is very crucial in Africa. It involves recognising the importance of indigenous knowledge and practices that may not align with typical western standards but are relevant for local development, so local ownership should be at the forefront of determining what good resource use and value creation means. Also, VfM should highly seek to assess how resources are allocated and whether value is created in a way that benefits the most disadvantaged groups. Inclusivity should be prioritised by considering the impact on marginalised and vulnerable populations. The two biggest challenges I think of are i) external pressures – where in most cases funding and programme design is done by external donors who influence the priorities and focus of VfM assessments, potentially diverting them from local values and priorities; and ii) strong political considerations and bureaucratic obstacles that in many times hinder the adoption of VfM assessments that challenged the status quo."
Please write me a new abstract, by adapting the old one. Use the same sequence of headings as the old abstract.
Here’s what ChatGPT drafted for me
Workshop Overview
Effective resource allocation and accountability are crucial for successful program outcomes. This workshop, designed for evaluators at all levels, introduces the innovative "Value for Investment" (VfI) approach to assessing value for money (VfM). While building on familiar evaluation and economic principles, this workshop distinguishes itself by emphasizing the alignment of VfM reporting with the needs of donors and the perspectives of local stakeholders and affected communities.
Workshop Content
This interactive workshop places a spotlight on evaluative rubrics as a powerful tool for synthesizing criteria, standards, and evidence to make warranted judgments about VfM. Participants will delve into key topics, including the collaborative development of rubrics with stakeholders, utilizing rubrics for evaluative judgments, and understanding mixed methods evidence. A unique emphasis will be on meeting the reporting needs of donors while ensuring representation of grassroots perspectives, bridging the 5Es (economy, efficiency, effectiveness, cost-effectiveness, and equity) with actual value from the viewpoint of those impacted.
Key Topics Include:
What are rubrics and why use them?
Stakeholder participation in rubric co-design and judgment-making
Balancing the needs of donors with representing value from a grassroots perspective
Making sense of mixed methods evidence
Reporting findings effectively
Workshop Objectives
Participants will learn how to:
Synthesize criteria, standards, and evidence for sound evaluative reasoning
Engage stakeholders in rubric co-design and judgment-making processes
Bridge the 5Es with grassroots perspectives to provide a comprehensive VfM assessment
Interpret mixed methods evidence to make nuanced evaluative judgments
Effectively report findings that meet the needs of donors and reflect local values
Who should attend?
This workshop is tailored for evaluators and commissioners aiming to enhance their VfM assessment skills, with a focus on aligning reporting with donor expectations and capturing local perspectives. The content is applicable to evaluators at all levels, from novices to experts.
About the facilitator
Julian King, a specialist in evaluation and value for money, introduces the Value for Investment approach, a model combining evaluative reasoning with economic evaluation. The workshop draws from his extensive experience in assessing complex programs globally. The inaugural session of this workshop will specifically cater to African evaluators, promoting a vision of VfM assessment rooted in African values and addressing challenges unique to the context, including external pressures and political considerations. Participants will explore how local ownership can shape assessments that prioritize inclusivity and benefit marginalized populations.
Note:
Value for money transcends financial metrics; it's about optimizing all invested resources to enhance the quality of life. This workshop advocates for the term "Value for Investment" to underscore that programs are more than costs – they are investments in a set of value propositions. The approach encourages evaluators to adopt methods and tools they already possess, while borrowing valuable insights from economics, to ask pertinent questions and provide meaningful answers in the VfI framework.
Feedback please!
Overall, ChatGPT’s suggestion gets me off to a good start, but the final submission will be my responsibility and I can see some adjustments I’d like to make. Some of the language feels a bit generic. I think it’s still over-emphasising accountability and donor perspectives, and there are some words and phrases that I need to switch out to put them in my voice.
Before I finalise and submit a workshop proposal, I would really like to hear from you. How well would this workshop meet your needs? Should I offer a version for your evaluation conference? Do you think it would attract the interest of relevant colleagues? Are there any improvements you would suggest?
