I recently took on a project that included identifying the key elements of a successful intervention. This gave me the opportunity to use a rarely used tool in the evaluator tool box: qualitative comparative analysis (QCA). Qualitative Comparative Analysis (Ragin, 1987) in a case-based method that allows evaluators to systematically compare projects, identifying key factors that are responsible for the success of an intervention.
What I found was that applying QCA made the whole evaluation process more rigorous and required an additional layer of analysis, which isn’t a bad thing really. Using QCA gave me an additional certainty in my conclusions and a robustness to the narrative of the final report.
It is important to note that QCA doesn’t determine whether or not a project is performing well against the outcome indicators, theory of change or any other measure, but whether there are sufficient or necessary components in place in each location for a particular result to be seen. You could have set up a project identically in multiple locations and some might still perform better than others for contextual reasons. The QCA analysis will identify where differences lie, but you will still have to go and investigate those differences. For example, in the evaluation that I was doing, one location appeared to be more focused on advocacy and policy than other locations, however when it came to the QCA, this wasn’t supported and further investigation determined that another location, while less vocal about its policy work, had better evidence for actual contribution to policy change over a longer period and across more legislation and regulation. The first location may yet make a huge difference in time and you can’t take away from the successes that they achieved. But not everything is at it appears and this is where the value of QCA was for me.
Recently I’ve also seen a paper by the ODI on adding QCA to their toolbox, and they had similar experiences of increased robustness of evidence as I did. I don’t know that QCA is appropriate in every instance, but it is always worth considering.
Have you used QCA before? Would you consider using it? What are some of the challenges and benefits that you can think of?