Well the snacks are the most important element, they keep your energy levels up! When I mention that to groups of stakeholders that have gathered to work on developing a hypothesis for a programme or intervention, they realise that they are not getting off lightly.
There is lots of debate about the value or otherwise of Theory of Change and Log Frames. I’ve previously written about the times when you don’t need a Theory of Change and personally, I’d never design one for an organisation I was running (I prefer strategic frameworks with ‘old fashioned’ KPIs). At the moment however they are the preferred fad and we learn how to work with them. However Theories of Change can hamper an evaluation by prescribing to the organisation what they think has actually happened, not what was intended to happen. There is a difference. Evaluations need to understand what actually happened and whether these events contributed towards the stated objective or outcome. Basing an evaluation on a pre-existing Theory of Change can cloud that process.
So instead I always start with a session where I facilitate the stakeholders involved in the intervention / project / programme to develop a hypothesis of what they think happened from their perspectives and understanding what their stakes are in the intervention. This session also considers barriers and the world as it was when the intervention was first conceived so that we can understand the extent of the intervention, what problem it was designed to address and the different interactions and perceptions of the stakeholders involved.
The resulting hypothesis drives everything and principally the evaluation or learning framework. This might look like a Log Frame but is in reality a framework for assessing evidence gathered. While I do try to incorporate the evaluation questions that the client has come up with for the Terms of Reference (I mean really why do they do that before they’ve hired an evaluator?!?), this is also a chance to review those questions with the client and make amendments based on the workshop and the hypothesis that has been developed.
And then we’re off, down into the detail of the evaluation process before coming back up the other side to consider whether what actually happened, after the available evidence has been assessed, reflects the hypothesis that we started off with. You’d be surprised at some of the results that this process throws up. More importantly this process does two key things that sets it apart:
- It amplifies the voice of stakeholders and beneficiaries in particular in the evaluation process, ensuring that their experiences and perceptions of the changes that have happened are included, and
- It results in tangible recommendations that organisations can implement immediately to improve project design, organisational strategy, programme performance, value for money or quality of results.
Increasingly clients appreciate that I include increasing internal evaluation capability into my work too, which this process, because its about facilitating the project team through regular review sessions, enables beautifully!
How does evaluation feature in your work? Does your consultant get you up on your feet on day one?