Theory of Change: What is it good for?

Wednesday, October 6th 2021

Change

In this blog Kerry McCarthy, Sam Mackay and Tracey Gyateng reflect on how developing a theory of change can help organisations embark on a journey to improve their use of data.

We have recently completed work with the Law Centre Network (LCN) and hope these
reflections can help Law Centres, and others in the access to justice and non-profit sectors who are grappling with the question.

What data should my organisation collect?

This is a common question that many non-profits (and for-profits!) struggle with. Generally, we can agree that data is useful, and organisations are likely to collect data for many reasons, including funder requirements, operational purposes, to influence policy agendas, and monitoring and evaluating their work. So a useful starting point when faced with this question is to ask what questions are we trying to answer with our data?

For many organisations, including LCN, an important question is how do we use data to
support and evidence the difference we intend to make through our work? Having data
(both qualitative and quantitative) can help organisations go beyond statements of aspiration and belief, to having data that shows them what is and isn’t happening.

But to answer this question you need to be clear about the work your organisation is doing and the difference you think it will make. An important part of this process is being clear and unambiguous when describing your work. General terms like ‘support’, ‘more effective’, ‘transformational’ are not very tangible and can be difficult to translate into something that can inform data collection/collation requirements.

One process which can help with clarifying meaning in this way is theory of change. The
theory of change methodology was developed in the context of evaluating projects and
programmes, to understand how they are expected to work, and whether change happened as expected, in a different way or not at all. In this context, a theory of change would make explicit the short- and medium-term steps on the way to achieving the desired longer term change. This is often visualised in a linear way with arrows between the different parts, sometimes called inputs, outputs, outcomes, impact, although there are several different terms. These visualisations often fit conveniently on one piece of A4 paper!

However, in our work for LCN, we used the theory of change as an aid for LCN to reflect and articulate what they as an organisation do, and what changes result from this work. This will help LCN plan what data/information they should collect or collate to understand and evidence the difference they make. In embarking on a data journey, with LCN we have used the theory of change process to:

  1. Provide a framework to help LCN critically reflect on the work they are doing, how it will lead to the change they want to make and the assumptions this is based on. This is about the process of honest reflection, with the outputs of this work being for the organisation–staff and trustees– to act on, rather than for marketing purposes.
  2. Make implicit meaning explicit – so that LCN can identify specific data/ information
    that can inform and evidence their work.
  3. Identify where LCN expects to see and influence change, with a focus on what they
    are doing now, not on what they aspire to be doing in the future in order to focus the data collection and analysis process.

It isn’t easy to take a step back and critically assess the work that you are engaged with.
However, LCN were committed to the process and welcomed our questions and
discussions. This level of openness and curiosity created a safe space for us all to explore
how LCN conducts, and speaks about its work.

How did you do this with LCN?

Our work with LCN started with their question ‘what data should LCN and Law Centres
collect?’. And with our question ‘what are LCN & Law Centres trying to understand through data?’

For LCN this was about understanding outcomes for which they already have data;
outcomes where there is no data currently available; and identifying the data they already
have which does not relate to any outcomes, but may have another purpose (or no purpose at all!). Thereby allowing them to identify and prioritise areas for new data collection, and potentially stop collecting data that serves no purpose.

By going through this process themselves, LCN are then intending to support Law Centres to do the same, to identify the data they should be collecting.

To identify the outcomes LCN was working towards we reviewed all strategic documents,
finding a large number of very broad outcomes that were not always clearly defined. These were shared and discussed with all LCN staff in a workshop - to check what was relevant, to try and clarify meaning and to identify the resources and actions aimed at achieving each outcome.

From this discussion our intention was to produce a revised theory of change, perhaps a
visual or table to complement LCN’s existing visual on how they and Law Centres are
‘Agents for Change’, by adding a layer of detail and specifying the meaning of actions and outcomes in a way that it would be possible to identify them in existing data, or identify new indicators to collect.

But there was a disconnect between what staff were telling us they focused their time on,
and the descriptions of outcomes in the documents. One set of activities and outcomes were the day to day bread and butter work of LCN, and others were aspirational – relating to the change LCN would like to see but was not necessarily working on directly or could contribute to significantly. There was also still a lot of ambiguity in meaning around the phrases used to describe outcomes, with a need for a shared common understanding to be developed before it was possible to identify relevant data.

The large number of outcomes, some actively being worked on, and some aspirational, is
perhaps illustrative of the balancing act many charities face, between sticking to their core mission and feeling the need to reflect certain outcomes or ambitions in how they present themselves, in order to maximise opportunities for funding. A particular challenge for the access to justice sector, with the severe cuts to funding experienced. LCN refocused how they talked about their work, grouping it into three plain English
statements reflecting what they actually do (help Law Centres; amplify the voice and protect the reputation of Law Centres and LCN; strengthen the network).

Across three online workshops LCN staff answered the following questions for each area in turn, completing a table for each. There was a strong emphasis on including only what they, as LCN, are actively working towards, not what is aspirational, and not what is being done by others, for example, Law Centres.

  1. Clearly define your outcomes (what are you trying to change / achieve, avoid jargon,explain what is meant when using an ambiguous term)
  2. What LCN actions help achieve this outcome (e.g. what is it that you are doing that contributes to what you are trying to achieve)?
  3. Why do you think this is the right approach? (this is not about why you think a particular area of work is important, it is about why you think the actions in 2. contribute to the outcomes in 1, it could be based on knowledge, data, belief, hypothesis, assumption, experience etc.)

Through this process you can see how aspects of a theory of change approach play out -
supporting critical reflection on the work, making meaning explicit, the difference between shorter term outcomes, more within LCN’s sphere of influence, and those that are longer term where LCN plays only a contributory role. And through this being able to more clearly identify data that is relevant to what they are doing, and which is within the realms of possibility to collect or access.

The final tables produced through these workshops, do not look like a ‘traditional’ theory of change visual of inputs a, b and c leading to outcomes x, y and z. This is unsurprising as LCN is not trying to undertake a timebound project or programme impact evaluation that explores casual mechanisms. LCN are on an ongoing data journey, trying to identify, collect and use data that will help them achieve their organisation wide goals, and demonstrate their value.

And these tables are not, on their own, sufficient to tell the whole story of how different
elements of LCN’s work interact to support each other, or how they connect to the wider
networks and systems LCN is part of and contributes to. Their primary purpose is to
articulate what LCN is doing and the difference they think that will make, in a way that makes it possible to look at their data and see if it is represented there, and if not, to think about future data collection needs. LCN, like other organisations, will need a variety of approaches for telling their story depending on the audience. But the tables produced during these workshops should help them tell parts of their story with data.

What was the experience like for LCN?

LCN, please could you describe how it has been to go through the process of
developing the theory of change and what actions you intend to take going forward.

The process of thinking through our theory of change has been an intellectually challenging exercise that made the whole team at LCN, as well as key Trustees, think about our work at its most basic level, about what we do and why, not solely the ambition of our work. When we approached Tracey, Sam and Kerry for help to work out what data we needed, it became obvious that there was a bit of work to do on our theory of change, which at the time focused on how LCN and Law Centres work together to create change, rather than LCN’s part of this work.

It was interesting, and in some ways unexpected, just how much we needed to unpack the terminology in our strategic and planning documents until it expressed clearly what we do.

Before starting, we had not thought about this project as concerning language and communication. Our strategic documents were full of jargon, topical buzz words like
‘resilience’, and aspirational outcomes. As we progressed, we realised that it was
impossible or very difficult to evidence those outcomes with data or even attribute them to our work. We also recognised that, despite collecting a lot of data, we were not always clear about the reason for collecting it and what we do with it afterwards. We needed to embed our data within a clear strategic framework that would help us learn, evidence and use it to best effect. Working with Kerry, Sam and Tracey has helped us to take time to listen to one another, consider the language we use, be clearer about which activities we actually do to deliver our desired outcomes, and alignment about the steps we take (including what data we need or not) to get there. The external perspective really helped us to gain clarity and objectivity, it guided our conversation and challenged us when this was needed.

As a result , we now have:

  1. Clarity over outcomes, activities and data points needed to evidence how we meet our objectives, as well as a better way to communicate what LCN does
  2. An improved overview of the data available across the whole organisation and how this can be used to evidence our Theory of Change
  3. Data Principles to guide the ongoing collection and use of data, not only at LCN but across our Network.

The Data Principles have themselves raised some fundamentally important questions for
how we improve our data maturity as an organisation and a network. In particular,
we learned what skills, culture, and leadership we need to keep improving and to bring data to the forefront of everything we do. The next steps for LCN will involve embedding these data principles through our work, gaining further clarity on data points needed to evidence our Theory of Change, and alignment of data architectures across LCN tools and
systems. It is encouraging to recognise that we already have a lot of useful and meaningful data that we can draw on. This makes us more confident about how we use this to evidence our own work, and to encourage more Law Centres to collaborate on data with LCN and the wider Access to Justice Sector.

We are fortunate to have been supported by the Legal Education Foundation to go
through this process.

The next step for us will be to better understand how we can best work with Law Centres to progress their own data journeys. This will help us assess how LCN as the network body can better use data to strengthen our network and amplify the voice and reputation of Law Centres with funders and policy makers. We sincerely thank Tracey, Sam and Kerry for helping us on our journey. We’re just getting started!

Where can we find more information and support to guide us on a data journey?

There are a number of organisations that can support you with your data needs, some of
which provide free support such as ProBono OR, or DataKind UK; some are region specific, but provide free resources to all such as DataWiseLondon. Small charities can also access low cost support from Coalition For Efficiency and some funders such as Lloyds Bank Foundation run grant plus programs which can provide data support. You can also contact Kerry or Sam who both provide consultancy work, or if you work in the access to justice sector Tracey is always happy to have a conversation about data!

Doing this work will take time and given the squeezed state of access to justice, with
reduced funding, issues in recruitment and increased demand– you may (rightly) ask how
can organisations take time out to do this work? But hopefully this blog has shown the need to take time out to critically reflect on your work and consider how you know (and show) what difference you are making. We continue to work in an area where public spending is under huge strain, knowing that you are providing the best service for the people you work with, and being able to provide evidence of this work will be an essential part of your story. Starting with a theory of change, or any framework [1] for critical reflection and clarification of your work and the role of your organisation is crucial.

A theory of change shouldn’t be a one-time only tick box exercise to create a pretty, but
quickly out of date visual, ready to be admired in the vault of a funder. When embarking on a journey to improve how you collect and use your data, it will act as an essential map, which you update as you develop greater and greater understanding of the terrain of your work.

A theory of change for your data journey, what is it good for? (It’s not absolutely
nothin’)![2]-

[1] e.g. OKRs (Objectives and Key Results)
[2] to borrow from the singer Edwin Starr ‘War’

If you would like more information about LCN’s work on Data, please get in touch with Alex Charles, IT and Digital Officer, Law Centres Network at alex@lawcentres.org.uk.

Data in the Advice Sector - a Case StudyDownload[271 KB]