Program Theory GPT

I asked ChatGPT to help unpack a program's value proposition and it did an amazing job

When designing a process or outcome evaluation, a program’s theory of change can be an important tool for identifying organisational actions, impacts, causal mechanisms, and aspects of performance to focus on.

However, when we’re designing an evaluation that looks at value for money (VfM), it can take a conceptual leap to get from a theory of change to a set of VfM criteria - for example, defining what it would look like if the program was creating value equitably and efficiently.

This challenge arises because a theory of change focuses on how the program is expected to bring about change, whereas a VfM assessment is interested in how the program uses resources to create value. I argue that we should make program theories more explicit about:

• what kinds of value the program is expected to create;

• value to whom; and

• how it creates value.

I call this a theory of value creation, or value proposition.

In my view, it’s important to co-create program theories with stakeholders and end-users of the evaluation, because: i) they hold expert knowledge about the program and context; ii) it builds a shared understanding of the program and what success means; and iii) it enhances their understanding and ownership of the evaluation, increasing the likelihood that they will use it.

Nonetheless, before heading out to facilitate this kind of process, I like to do a bit of background prep so I’m schooled up on the situation, ready to ask stakeholders informed questions and engage with their answers. Typically this would include scanning program documents and literature.

While preparing course materials for a new workshop, I came up with a fictitious program (inspired by a real one) and sat down to ask ChatGPT a few questions, to see how well it could identify potential elements of a theory of value creation. The results, as you’ll see, were astonishing.

A good strategy for engaging with ChatGPT is to start a conversation, in much the same way you would with a person. Here’s how the conversation went…

The chat

Me:

Imagine you are a consulting economist, working for an international development agency. I will describe a governance and public financial management program to you, and I will ask you to summarize key economic theories that underpin it.

ChatGPT:

Sure, I'd be happy to help! Please go ahead and describe the governance and public financial management program that you have in mind, and I'll do my best to summarize the key economic theories that underpin it.

The program has three components: 1) technical assistance projects, aimed at enhancing public sector capacity to do things like needs-based budgeting; 2) civic projects, aimed at strengthening the capacity of civil society organisations to engage with government agencies, and 3) research projects, aimed at contributing knowledge to inform decisions and debate. Together, these three components are expected to result in the government collecting more taxes, and more effectively allocating them to meet needs. Consequently, this is expected to lead to improved access to needed services, which will in turn lead to improved population outcomes. Ultimately we hope this will contribute to more stable democracy and increased trust in government. The whole endeavour will be supported by a program of monitoring, evaluation, and learning, through regular feedback loops and adaptive management.

Being a language model, Chat GPT can’t read this picture. But for you, here is a diagrammatic representation of the program I described in the previous paragraph.

Thank you for describing the program. The economic theories that underpin this program include:

1. Public choice theory: This theory suggests that individuals act in their own self-interest, and that this behavior extends to those in public office. As such, public officials may not necessarily act in the best interests of the public, but may instead act in a way that benefits themselves or their constituencies. The technical assistance projects are aimed at enhancing public sector capacity to engage in needs-based budgeting, which may help to reduce the influence of special interest groups in public spending decisions.

2. Principal-agent theory: This theory suggests that there is often a divergence between the goals of principals (such as citizens) and the actions of agents (such as government officials). In the context of this program, the civic projects are aimed at strengthening civil society organizations' capacity to engage with government agencies, which may help to align the goals of citizens with the actions of government officials.

3. Taxation and public goods theory: This theory suggests that governments have a role to play in providing public goods (such as healthcare, education, and infrastructure) that the private sector may not provide efficiently or at all. The research projects in this program aim to contribute knowledge that can inform decisions about how to allocate public resources in a way that maximizes the provision of public goods and improves access to services.

4. Social contract theory: This theory suggests that individuals give up some of their individual rights in exchange for the protection and provision of public goods by the state. The program aims to improve access to needed services and contribute to more stable democracy and increased trust in government, which may help to strengthen the social contract between citizens and the state.

Overall, the program is based on the idea that effective public financial management and governance can contribute to improved population outcomes and social stability. The program's components are designed to address some of the key challenges and issues that arise in public sector decision-making, including the potential for self-interested behavior by public officials, the principal-agent problem, and the need to provide public goods efficiently and fairly.

I’m impressed. This summary of economic theories would be useful narrative to accompany the theory of change. It helps with understanding the program’s rationale, assumptions, and intended causal mechanisms.

Now we turn to a series of specific questions about value creation. When I talk about ’value’, I mean the merit, worth or significance that people and groups place on something. The ‘something’ is often an impact, but value is distinct from impact. An impact can have value (or not) in different ways from different perspectives. Since we are setting out to define what good value would look like for the resources invested in this program, we need to unpack its value to key groups.

So I asked ChatGPT…

What kinds of value does this program create, and for whom?

This program can create a range of values for various stakeholders, including:

1. Value for governments: The technical assistance projects can help governments improve their capacity to manage public finances and allocate resources more effectively. This can result in more efficient and transparent governance, better service delivery to citizens, and increased tax revenue.

2. Value for civil society organizations: The civic projects can help civil society organizations strengthen their capacity to engage with government agencies and hold them accountable for delivering services to citizens. This can result in more meaningful citizen participation in decision-making processes and increased transparency and responsiveness from government.

3. Value for citizens: Improved access to services, such as healthcare, education, and public infrastructure, can lead to better outcomes for citizens, such as improved health, education, and economic opportunities. Increased transparency and accountability from government can also lead to greater trust and confidence in the government.

4. Value for researchers and academics: The research projects can contribute to the generation of new knowledge and insights that can inform policy and practice in the field of public finance management and governance.

Overall, the program can create value by improving the capacity of public sector institutions, enhancing citizen participation and engagement, generating new knowledge and evidence, and promoting better outcomes for citizens. This can contribute to more stable democracy, increased trust in government, and improved quality of life for citizens.

Great stuff. ChatGPT has identified value from four different perspectives. As always, this is a starting point and we should consider whether there are additional perspectives to canvass, and maybe follow up questions we could ask the Generative Pre-trained Transformer. For example, how about the perspectives of key vulnerable groups? And donors?

We could also dig deeper into the nature of the value being created. For example, I would describe the value created by the technical assistance as growing a form of capital, something that lasts across multiple accounting periods and provides an ongoing return. In this case, the capital asset is governmental capability, and the immediate ongoing return is fiscal value: increased tax revenues and better allocation of funds. The fiscal value creates social value if and when it results in services more effectively and equitably meeting needs.

Similarly, we can conceptualise the long-term impact of more stable democracy and increased trust in government as a form of social capital. It’s important to understand social capital-building as an ongoing social process, not a destination. The ambition of our program is to make a meaningful contribution to this process with the limited resources and time it has. This might turn out to be out of our reach to evaluate. We may decide to focus on more immediate and direct outcomes. But there’s still value in specifying the long-term ambition and explaining how it may be logically linked to the program’s proposed actions.

Next we look at the processes by which the program transforms resources into the kinds of value listed above. So I asked…

How does it create value? What are the mechanisms of value creation?

The program creates value through various mechanisms, which can be grouped into three categories: capacity building, knowledge generation, and feedback loops.

1. Capacity building: The technical assistance and civic projects aim to enhance the capacity of government agencies and civil society organizations to perform their respective roles effectively. This can be achieved through activities such as training, coaching, mentoring, and the development of toolkits or guidelines. By building the capacity of these organizations, they can operate more efficiently, be more transparent, and provide better services to citizens.

2. Knowledge generation: The research projects aim to contribute knowledge and evidence to inform decisions and debate in the field of public finance management and governance. This can be achieved through the use of various research methods, such as surveys, case studies, and data analysis. By generating new knowledge and evidence, policymakers and practitioners can make informed decisions and design effective policies and programs.

3. Feedback loops: The program includes a monitoring, evaluation, and learning plan, which involves regular feedback loops and adaptive management. This allows for the program to be continuously evaluated and improved based on feedback from stakeholders. By incorporating feedback loops, the program can identify what is working well and what needs to be improved, and adjust its activities accordingly.

These mechanisms of value creation can result in a range of outcomes and impacts, such as improved public sector capacity, enhanced citizen engagement, better service delivery, and increased trust in government. These outcomes and impacts, in turn, contribute to more stable democracy and improved quality of life for citizens.

Amazing. In my simplified theory of change diagram (above), mechanisms aren’t specified. They’re implicit in the arrows between the boxes. ChatGPT has started to unpack some assumptions about how the program might create value, making proposed mechanisms explicit. When we specify VfM criteria, these mechanisms will help us consider what aspects of performance to look at.

The mechanisms identified above are underpinned by a theme I would describe as leverage; A small rudder steers a large ship by applying a little influence in the right place at the right time. Similarly, our program needs to identify the right points of influence where small actions can achieve significant results. In this program, it might take a bit of trial and error, and the ability to respond to changes and emergent opportunities, to be effective at leveraging value. This highlights the importance of feedback loops.

Now let’s take a look at success factors for value creation…

What factors will influence whether it creates enough value for the resources invested?

Several factors can influence whether the program creates enough value for the resources invested, including:

1. Program design and implementation: The design and implementation of the program can significantly influence its effectiveness and the value it creates. Key factors to consider include the alignment of the program's goals and objectives with stakeholder needs and priorities, the relevance and quality of the technical assistance, research and civic projects, and the extent to which the program is adaptable to changing circumstances.

2. Resource allocation: The allocation of resources, including funding, personnel, and technology, can impact the effectiveness of the program. Sufficient and appropriate resources must be allocated to ensure that the program can deliver on its objectives and expected outcomes.

3. Stakeholder engagement: The program's success depends on the engagement and participation of stakeholders, including government agencies, civil society organizations, and citizens. Therefore, stakeholder engagement should be an ongoing and iterative process throughout the program's implementation.

4. Political and social context: The political and social context in which the program operates can significantly influence its effectiveness. Political instability, corruption, and lack of social trust can pose significant challenges to program implementation and the achievement of its objectives.

5. Monitoring, evaluation, and learning: Regular monitoring and evaluation of program activities and outcomes, as well as the use of feedback loops and adaptive management, are essential to ensure that the program is achieving its intended results and creating sufficient value for the resources invested.

By considering these factors and actively managing them throughout the program's implementation, it is more likely that the program will create sufficient value for the resources invested.

What factors will influence how fairly value is distributed?

Several factors can influence how fairly value is distributed, including:

1. Targeting: The design and implementation of the program should consider targeting the most vulnerable and marginalized populations to ensure that they benefit from the program's activities. This can include measures such as geographic targeting, income targeting, and social targeting.

2. Inclusion: The program should be designed and implemented in a way that promotes inclusion and avoids discrimination against any particular group. This can involve ensuring that all stakeholders have equal opportunities to participate in the program's activities and that the benefits of the program are distributed fairly.

3. Transparency and accountability: Transparency and accountability are critical factors in ensuring that the benefits of the program are distributed fairly. This involves providing clear information about the program's goals, objectives, and activities, as well as ensuring that there are mechanisms in place for stakeholders to provide feedback and hold program implementers accountable.

4. Participation and consultation: Stakeholder participation and consultation can help ensure that the program's activities and outcomes are aligned with stakeholder needs and priorities. This can involve consulting with stakeholders during the program's design and implementation, as well as providing opportunities for stakeholder input and feedback throughout the program's lifespan.

5. Monitoring and evaluation: Regular monitoring and evaluation of program activities and outcomes can help identify any inequities or biases in the distribution of program benefits. This can involve disaggregating program data by demographic characteristics, such as gender, age, and socioeconomic status, to identify any disparities in program outcomes.

By considering these factors and actively managing them throughout the program's implementation, it is more likely that the benefits of the program will be distributed fairly among all stakeholders, including vulnerable and marginalized populations.

What do you think?

I think ChatGPT has done a remarkable job of priming us to think systematically about the program’s value proposition - what kinds of value it creates, for whom, and how - in a fraction of the time it would have taken to do some other way. ChatGPT only knows what little I told it about the program, but it can surmise an extraordinary amount from first principles.

Of course, this isn’t enough. We need to be aware of the potential for the GPT to perpetuate biases from the body of text it used as data. Context is also important, and ChatGPT knows nothing of the context. We still need to engage with stakeholders to refine this initial thinking. By the time we’ve done so, we’ll be well prepared to develop a set of VfM criteria targeting critical, observable propositions that underpin the program’s capacity to provide value for the resources invested in it.

The benefit of ChatGPT, in this instance, is in canvassing general concepts and issues for us to consider. Evaluative thinking remains “a collaborative, social practice” (Schwandt, 2018) and autoregressive language models aren’t (yet?) ready to take the lead on that.