So I’m in the middle of setting up a new evaluation for a client, well a group of clients really and it is quite a complicated funding programme that they have asked me to look at.
As usual I have set up my systems and processes for this project and started data collection already. It occurred to me that there are certain things that I always do, certain expectations that I have of clients and a level of efficiency that I rely on in order to be as effective as possible. Even more so when I’m running a complex evaluation process.
I’ve been thinking about how to distill this into a system or set of guidelines.
So here is my set of (golden) rules for running an evaluation process:
Communicate, communicate, communicate
This is the most important thing to do. Sounds obvious I know, but an evaluator has to come into an organisation and quickly establish a rapport with a (usually) close-knit team who invariably work in a different way to that of the evaluator. Always keep the team updated with what is going well and what is not going well. I build a two-weekly formal reporting process into my evaluation management and supplement that with informal emails and telephone/skype calls as needed. This way everyone knows what is going on and what stage the evaluation process is at.
Always use the OECD DAC criteria
Relevance, Effectiveness, Efficiency, Results, Sustainability. I always use these as my evaluation framework. Martin Ravallion at the World Bank has also distilled a set of generic evaluation questions that are very useful for unpacking how to apply the DAC criteria in impact evaluations. He didn’t develop these questions with the DAC criteria in mind of course, but I’ve used his work in this way and it has been very useful. Even if the evaluation is not about humanitarian, development or international development issues, these criteria can be applied as they are appropriate for any intervention that aims to make a sustained difference in people’s lives (or indeed animals and the environment).
Always use more than two forms of qualitative and quantitative data collection
You always hear about how important it is to triangulate data collected in the field. We can have the qualitative versus quantitative debate another time, but I think we all agree that it is important to have both types of data collection included in an evaluation. I always try to include at least two approaches for both qualitative and quantitative data collection, so that I can test the data received using one methodology against the data received using another methodology (for example: focus group discussions against key informant interviews or participatory rapid appraisals, etc.). The more ways that you can collect data, the more robust your findings will be.
Always include value for money
I know this will be contentious and you won’t all agree with me on this one. No matter who your audience is and what the aim of the evaluation is, value for money assessments are essential these days. Not only does it help the NGO to understand how it is using the funds allocated, but it gives the organisation evidence to communicate how it is performing on value for money to donors, supporters, partners, beneficiaries, etc. And who hasn’t been asked where the money goes?
Always deliver value for money
Again this might seem obvious but evaluators, like other suppliers, need to be able to demonstrate that it was a good idea hiring them to do an evaluation. But be careful about providing value for money and not working for free or working yourself to death!
Be realistic about how long it will take you to deliver an evaluation
This is crucial. Often have NGOs asked that an evaluation should be delivered within a month or so. Some NGOs might be notoriously bad at planning, but it is important that evaluators are clear about how long it is going to take to delver a piece of work. Not only does it manage expectations, but it also underlines that an evaluator will deliver a valuable and comprehensive piece of work within a particular timeframe.
Understand the value and cost of your services
You might think that evaluators cannot prescribe payment to a client with a fixed budget. You’re right you cannot always get the budget changed, but like anyone else, evaluators need to know what their time and talents are worth and how much to sell them for. For example, I recently had a client who had a very limited budget, but wanted a particular type of evaluation delivered. We talked about what was possible and arrived at an agreement of how much of my time they could afford at the lowest possible rate I could afford. We agreed. I did the work in the time and budget available and the client called me again to help with another evaluation. So while I could have turned them down initially because they didn’t have the usual evaluation budget, I understood what my costs and profit margins were and worked to maintain those while finding a solution for the client. Basic business really.
So what are the golden rules that you use to commission evaluations or deliver evaluations? What are some of the things that haven’t worked and why?