Thoughtful comments

Our inception report has received a thoughtful review by Rick Davies on his blog Rick on the Road. Titled "The Challenges of Using QCA", the posting summarises some of the methodological difficulties we have encountered so far.

Inception report ready for sharing

The inception report, which summarises our review methodology, is finally ready for wider dissemination. You can download it by clicking on THIS LINK. It is a large file of some 1.9MB. 

Due to some technical complication that we do not want to explore in detail, some text magically disappears when you try to read the report in your browser. (For instance, in Firefox our response to the points raised by SEQUAS disappear.) So: if you wish to read the report, do not try to read it in your browser - rather click on "download" and use Adobe Reader to make sure you get the full text.

The annexes to the report include the tools we have used so far (coding instructions, survey questions and interview guides), as well as documentation on our initial dialogue with the External Reference Group, and with the Specialised Evaluation and Quality Assurance Service (SEQUAS). We will post more updates in the coming weeks.

Final report planned for June 2014!

We have received a few queries as to when our final report will be available. The plan is to complete it by/ in June 2014. It will include our findings from Qualitative Comparative Analysis and Process Tracing, as well as short descriptions of 15-20 evaluation approaches and designs that we have found effective or promising. 
Furthermore, we will produce a couple of papers:

• A paper summarising our findings for development and evaluation practitioners.
• A more academic, peer-reviewed article that will explain our review methodology, in particular the use of QCA in this study. Peer review means that it'll probably take until 2015 or even 2016 until the article is published.
• We will also post summaries of our findings and of key steps in our research on this blog. It has been a bit quiet in recent weeks because we wanted to complete our dialogue with the external reference group that accompanies the review before sharing the details we explain in our Inception Report (see post above).

We are still looking for suitable international events in late May or in June to present our findings. If any interesting events come to your mind, please let us know!

Helpful comments

A few comments have been posted on our blog. Apologies we are a bit slow with our reaction: We have been so absorbed by our hunt for contact details (see earlier post) and the launch of our survey with evaluation stakeholders - that we sort of lost sight of the useful comments which have appeared.

Rick Davies has shared a reference – thank you! – and directed us to an interesting blog posting on the question whether evaluations must fulfil certain quality standards to produce positive effects. The comments appear when you click on the titles “Inception” and “Evaluations identified for the first coding round” below.

Carol Miller (under “Soon to come: QCA conditions”) hopes we will look at evaluation processes in terms of how they contribute to empowerment of key stakeholders. Empowerment of stakeholders is indeed among the effects we intend to measure. By the way, our model for QCA will be published here - on this blog - with our final inception report, by March 2014. A couple of updates will be posted before that date.

One caveat: this project is not a huge piece of original research – it is a review of evaluation reports that is enriched by some primary data collection on evaluation effects, chiefly through a short web-based survey with four types of evaluation stakeholders – evaluators, people who commission evaluations, people who have implemented the interventions evaluated, and representatives of organisations that have funded the interventions. (We realise these categories sometimes overlap.) So, we won’t go into the fine detail of every possible evaluation effect. But we are confident we can find some interesting contours.

Message from DfID to the survey participants

This week we have sent out our web-based survey. Our DFID counterparts, Zoe Stephenson and Clare McCrum, are sharing this message, to encourage prospective respondents to respond to our request to engage with the survey:

This piece of work is a review of VAWG evaluations, commissioned by DFID’s Evaluation Department.  It’s a really interesting piece of work exploring what makes evaluations effective/influential. The consultants have identified about 70 VAWG evaluations to explore and they will be using QCA (Qualitative Comparative Analysis) and process tracing to explore the factors that influence whether the evaluations were used/useful.  Doing this depends on seeking the views of some key stakeholders who were involved in the evaluations – ideally the commissioner, the evaluator and someone involved with the programme’s implementation.

The work is funded by DFID and the engagement of those who have received the survey would be greatly appreciated by us. It is important for a large number of persons to participate in the survey - otherwise we may have to remove some evaluations from the set that the researchers work on and that would be a shame. The larger the set the better!

Your help is needed: names and contacts

As explained in the previous post, we plan to contact (i) evaluators, (ii) people who have commissioned evaluations ("evaluation commissioners") and (iii) representatives of the organisations whose work has been evaluated - for the full set of evaluations. It has proven difficult to identify contact persons for all evaluation reports. 

Hence this crowdsourcing action: We would be immensely grateful if you could have a look at the list available under THIS LINK (click on "THIS LINK" to get there). Would you happen to know anyone who is knowlegeable about any of these evaluations? If so, please write to review-team@gmx.de, possibly with the name and e-mail address of the person who can share information about the evaluation.

Inception

The review process encompasses three phases: Scoping, Inception and the actual Review, i.e. the analysis of evaluations. We have completed the scoping phase, and we are deep into the inception phase now.

In parallel with our search for evaluations (Scoping Phase), we have reviewed literature and initiated a virtual discussion with the Review Reference Group on the dimensions of evaluation practice. We are interested in the characteristics of evaluations and the positive or negative results they produce. To obtain a first understanding as to the characteristics and effects we need to look for, we have studied relevant literature. (See your Scoping Report for the full literature list.)

We have identified a wide range of elements that are considered as influencing the effect of evaluations. That is, they are likely conditions for positive evaluation effects, in QCA terminology. These conditions have been provisionally clustered into five dimensions: 

1. Conducive circumstances, which are present when the intervention is evaluable and the political environment (among and beyond evaluation stakeholders) favourable.
2. Powerful mandate, something evaluators have if resources are appropriate, the evaluation is timely and the evaluation team commands high esteem.
3. Convincing methodology that leads to compelling evidence, is well documented  participatory and ethically sound (‘do no harm’).
4. Effective communication, which rests on presentation and dissemination of findings, conclusions, recommendations and lessons learned.
5. High context sensitivity, in particular regarding gender, cultural and professional issues.

This is tentative and fairly abstract – our inception report will come with more precise definitions and calibrations to make fuller sense of these concepts. There is no hierarchy in these conditions. For the time being, the purpose of this initial inventory is to find out what could possibly influence evaluation effects. The provisional model we have built is a ‘maximum model’ in that it attempts to integrate a wide range of possible conditions.

We have also looked more closely at the effects of evaluations, clustering them into three groups:

• Effects on development practice  – i.e. changes in the further implementation of the intervention evaluated, or in the implementation of subsequent interventions.
• Effects on accountability and advocacy.
• Effects on the wider knowledge base– in terms of learning beyond the actual intervention, for example the contribution an evaluation makes to the global knowledge base on “what works” in efforts to end violence against women. 

The Review Reference Group (RRG) examined the tentative model in October and provided rich comments. The dialogue with the RRG and our DfID counterparts has helped us to clarify the terminology used and to appreciate the many facets of these dimensions.

Following from that, we have developed detailed reporting sheets for the coders. The coders have started their first coding round, examining all 74 reports we identified in our search (see earlier post). At this point, their job is to map the data on conditions they find in the reports

As to the effects generated from the evaluations, we cannot rely on the reports for data. Therefore we are building a survey. For every evaluation in our set, we are planning to question at least two out of three types of stakeholders: (1) the evaluator, (2) a person who has commissioned the evaluation, (3) a representative of the organisation that has implemented the intervention evaluated and who can report on the effects of the evaluation. We have interviewed 2-3 representatives of each category to further enrich our picture of the effects evaluations can generate. At the moment, we are building a web-based survey that will be sent out in early January.

By the end of January, we expect to have:

• An accurate picture of the data available from each of the 74 evaluation reports.
• Rich data on many of the conditions in our model, from 74 evaluation reports.
• Information from our survey respondents on evaluation characteristics which the reports have not provided sufficient data on.
• Data on the effects the evaluations have produced.

Qualitative comparative analysis (QCA) is at the heart of our review methodology. If we obtain meaningful data on conditions and effects, we can go ahead with QCA. That is why we have put in extra shifts to make sure we can contact a large number of evaluation stakeholders – a task that has proven more difficult than expected! (See post below, “Review + detective work”.)

Review and detective work

Most of the 74 evaluation reports in our first coding round do not display the evaluator’s or the commissioner’s contact details. In some cases, the evaluators remain anonymous; in other cases, the only e-mail address available in the report is a generic info@xyz.org. This has surprised us – in our own evaluation practice, we always include our e-mail addresses so that our counterparts can get in touch with us in case, say, they wish to work with us again.

Even where we could find an e-mail address, it was not easy to elicit a response. One could blame the busy season - early December, when in many countries the festive season starts and/ or the fiscal year is about to end. But I was puzzled to see that even in organisations with dedicated monitoring and evaluation staff, knowledge about evaluations – including fairly recent ones (2011-2012) in the public domain – appeared uneven.

Our hunt for addresses continues; we are seeing light at the end of the tunnel. Many thanks to everyone who has helped finding evaluation stakeholders around the world! We are particularly indebted to people in organisations with several evaluations of VAWG-related work and who have been specially generous in sharing information at this busy time: extra thanks to CARE, FOKUS (fokuskvinner), the International Rescue Committee, Oxfam, the Population Council, UNICEF and WOMANKIND. We have also benefited from the support of the Review Reference Group members (special thanks to Amanda Sim, Helen Lindley and Krishna Belbase). Some UNDP country registries have also proven effective in identifying evaluation stakeholders when we had no other contacts.

For a handful of evaluations found via the web, we have not yet managed to obtain any addresses that work. We will post the list shortly to ask for 'crowdsourcing' support in identifying stakeholders. If the authors and users remain shrouded in mystery, we will have to remove these evaluations from our QCA set. Which is OK – QCA also works with small sets of cases. But it would be hard to draw conclusions for the overall evaluation landscape if we ended up with, say, just a dozen of evaluations.

Evaluations identified for the first coding round

One thing that is special about our approach is that we do not only apply established quality standards to the evaluations we review. Instead, we will look into evaluation effects as well. Whether or not an evaluation has to fulfil established quality standards to produce positive effects is an open research question. To answer it, we have to include in our review evaluations that vary in the degree to which they fulfil certain methodological standards. We hope that our research will shed light on the factors that contribute to negative and positive evaluation effects.

We initially cast a large net, searching for any evaluations on work related to violence against women and girls.  A first, cursory exam of the reports we netted showed that summaries tended to contain too little information on evaluation approaches and methods.  Therefore we decided to work with full evaluation reports only.

We found 140 such reports. In many reports that included VAWG as a secondary component (e.g. evaluations of multi-sector country programmes, reproductive health initiatives and humanitarian aid), VAWG-related work tended to occupy a marginal position. Analysing those reports could yield useful information on the quality and effects of evaluations in general – but our focus are evaluations that are specifically designed for interventions on violence against women and girls.

In a further step, we narrowed down our set to reports completed in 2008-2012, excluding evaluations produced in 2013. This is because we will question (through interviews and a web-based survey) evaluation stakeholders about the effects the evaluation has produced. To make sure we can take into account effects that occur after an evaluation, we must allow for some time. One year seems a reasonable time-frame, even though we realise that some effects often occur at a later stage (for instance, the use of ‘lessons learned’ published in an article).

Of the 140 full evaluation reports, we have excluded 16 because they fell outside the 2008-2012 period, 43 because they evaluated interventions which included VAWG as a minor component, and 6 because both exclusion criteria applied. (One report did not show the year of publication.)

The remaining set includes 74 evaluations of VAWG-related interventions in low- to middle-income countries. This is the full set of evaluations we have found to meet all our criteria – i.e. we do not draw any sample. The evaluations cover three different contexts – development, humanitarian and conflict / post-conflict, and the four strategic priorities that inform DFID’s work on violence against women and girls (see figure below). Figures refer to the number of evaluations that match the criteria; the total exceeds 74 because some evaluations match several criteria.

DFID priorities	Development	Humanitarian		Post-/Conflict
Building political will and institutional capacity	22	7	12
Changing social norms	37	2	6
Empowering women and girls	12	3	3
Providing comprehensive services	16	7	7

The evaluations deal with a broad spectrum of interventions of varying complexity carried out by public and not-for profit actors (including women’s rights organisations), ranging from a single training project to multi-country programmes that bring together different types of interventions.  Most evaluations found have occurred near or after the end of an intervention, a smaller number are mid-term reviews.

The reports vary in size (8-258 pages); their median length is 52 pages (average length: 62). The degree to which they fulfil established quality standards (with regard to the methodology employed, protection of VAWG survivors and other aspects) is assessed in the first coding round. What can be said at this point is that quality, understood in this way, appears to vary significantly. This is also true for the appearance of the reports.

All published reports we have identified will be shared with DFID. 19 out of the 74 reports are unpublished or of uncertain publication status. We cannot share these reports with others, but we have obtained permission to extract data from these reports. It is important to keep them in the set of evaluations to be reviewed, as this is an opportunity to work on material that is not easily accessible to a wider public. 

For those who would like to take a peek at the published evaluation reports: the reports can be retrieved via this link. The link takes you to a folder which includes our full scoping report and a brief guide to the folder. 

Meet the coders

We are delighted to announce that we have found highly qualified coders: Miruna Bucurescu, Scout Burghardt, Sanja Kruse, Astrid Matten and Paula Pustulka. 

Astrid Matten holds a  Master's degree in Political Sciences, Sociology  and Social Politics from the University of Göttingen. Her research  focuses on migration, state borders and gender issues.

With a background in development studies, Miruna Bucurescu is a polyglot with expertise in women, peace and security. Since 2011 she has worked as an independent researcher focusing on social science statistics and evaluation.

Paula Pustulka is a sociologist, involved mostly in qualitative research projects in the fields of gender and migration studies. 

Sanja Kruse is a sociology graduate with a wide range of practical experiences in research and international cooperation, particularly related to employment, education and gender issues.

Scout Burghardt has a degree in Gender Studies and European Ethnology. She participated in the 2-year research project “Samenbanken – Samenspender” on reproductive technologies at Humboldt University Berlin.

For some reason, our provider refuses to upload the photographs that come with the text. You'll find them here as soon as it works again!

Our search for evaluations

As explained in our October 2013 post “What evaluations do we review?” we have looked for evaluation reports in English that meet certain criteria. Our search started on 23 September 2013. We used a three-pronged strategy: (1) web-based search (“web search”), (2) communication with contacts in the fields of VAWG and evaluation, and snowballing via these contacts (“snowballing”) and (3) specialised web-based networks (“DFID and helpdesk”). In addition to evaluation reports, we identified meta-evaluations and specialised publications on “best practices” to end VAWG, as well as literature on evaluation quality and effectiveness.

For the web search, we used combinations of the terms “evaluation”, “review”, “assessment”, “best practice” and 38 terms closely related to violence against women and girls and work to end VAWG, such as “violence against women”, “gender-based violence”, “forced sexual initiation”, “forced marriage”, “human trafficking”, “masculinities”. The web search yielded many duplicates and triplicates, i.e. it reached a high degree of saturation which could be expected in view of the overlaps between these terms.

However, focussing on the web search would have yielded incomplete results. As shown in the figure below, 84 per cent of the evaluation reports that we identified came from a single source, i.e. either from web-search, direct communication with our networks or snowballing.

This confirms that it was a good idea to combine three search methods. We have been particularly impressed by the effects of our snowballing action. It was initiated through two channels – (i) direct contact to evaluation and VAWG specialists known by DFID and the Review team, and (ii) publication of our call for evaluation reports on the following list servers and social web sites: Platform for Evidence-Based Learning list server (PELICAN), professional groups on LinkedIn (AEA, Evaluations and Development), Michaela’s Facebook and Twitter accounts, and our blogs www.developblog.org and, after registration of the new domain, www.evawreview.de. 

Sending out e-mails yielded an excellent response. Examining the messages received within three weeks from our call, we counted some 175 persons who had received the call by e-mail (as direct addressees or in copy). The actual number is probably higher, as we cannot assume that we have been copied into all e-mail correspondence. The interest raised by social web posts was considerable: www.developblog.org (Michaela’s blog) registered a peak of 270 page views on the day our call for evaluation proposals was posted on the above-mentioned platforms (as compared to about 40-120/day in “normal” times).

Soon we will describe what kinds of evaluations we have found. Our draft Scoping Report is with the Review Reference Group; when we'll have the final version we'll provide a link to it on this blog.

Soon to come: QCA conditions

We are on the last straight line of two -now that I think of it, three- processes:

• Finalising the scoping - i.e. the gathering and sorting of evaluation reports in a systematic manner and preparing a scoping report explaining how we have done it. The good news is, there are more evaluations - especially published ones - than we had expected to find! The bad news is, we cannot conduct qualitative comparative analysis (QCA - see the posts below for a quick introduction) on all of them. Not because QCA would not allow for it - in the opposite, maximum openness is a key feature and the beauty of QCA - but because going through all reports would take much more time and resources than what we can afford. We must sample! Watch this space. 
• Defining the conditions that -we (and the literature we have reviewed) suspect- contribute to making an evaluation of a VAWG-related intervention useful (or not), i.e. that increase or diminish positive evaluation effects. Our tentative conditions have been reviewed by the Review Reference Group, whose members have come up with useful questions and comments. We are refining the conditions now; soon they will be tested 'in earnest' in a first round of coding.
• Recruiting and instructing coders - a highly qualified team of five (we'll ask them whether they would like to be presented on this blog; if they do, you'll read more about them) has been brought together and awaits our detailed definitions and directions for coding.

We will share more on these points (including our set of conditions) on this blog near the end of this month, when this busy phase of scoping, sorting, sampling, refining our model and instructing the coders will be over. An intensive and exhilarating process!

Hopes and Misgivings

Rick Davies has shared his hopes and reserves about our Review on his Monitoring and Evaluation NEWS blog. We share Rick's hopes, and we realise our approach is ambitious. That is one reason why we are excited about it! We'll respond in more detail near the end of our scoping phase.
Right now we are super-busy coping with an  avalanche of evaluation reports. Our combination of web-search and snowballing has yielded an overwhelming response. We were worried that we might unearth too few evaluation reports. It turns out that there are many, especially from recent years and even many published ones.

Huge thanks to DFID, the Reference Group and everyone who have contributed to the avalanche by responding to or forwarding our request for evaluations!

Why qualitative comparative analysis (QCA)?

To generate realistic, practice-oriented findings and recommendations, the Review needs to differentiate between a wide range of evaluation approaches, methods and contexts. The number of existing evaluations in the field of VAWG is too small for statistical analysis to yield accurate conclusions. Yet, it would not do justice to the variety of evaluation settings if we selected only few evaluations for detailed analysis, as a conventional comparative case study would do. Qualitative comparative analysis (QCA) enables us to make full use of evidence from a wide spectrum of evaluations - without jeopardizing the applicability and generalisability of our findings. QCA has been designed for “medium-N” situations, i.e. situations where there are more than a handful of cases, but too few for meaningful statistical analysis.

QCA rests on the assumption that several cause-to-effect chains coexist. It matches sets of characteristics (in our case, the characteristics of evaluations) with specific outcomes (for instance, improved results of advocacy efforts). This method helps reveal which interactions between different kinds of methodology, resources and other conditions are necessary to achieve high quality evaluations under specific sets of circumstantial factors. 

QCA is transparent and replicable: It makes it possible and necessary to explain the iterative process of categorizing and coding evaluation reports that will be included in the analysis. We will go back and forth between conceptual work (categorisations of evaluation practice) and the evidence (evaluation reports and users’ narratives on evaluation processes and outcomes). We will thereby refine the definitions of dimensions of evaluation practice, and indicators that can be used to categorise evaluations. New factors will be taken into account when they prove necessary; old differentiations between evaluation settings will be given up if they prove superfluous.

Statistical methods or “conventional” comparative case studies may include similarly iterative processes, but their movement between theoretical levels and the evidence tends to be unsystematic and implicit. This “black box” situation may lead to the omission of important explanatory factors, and makes it difficult to replicate the findings.

What is the review about?

The UK Department for International Development (DFID; LINK) has commissioned a review of evaluation approaches and methods for violence against women and girls (VAWG)-related interventions (‘the Review’). Its purpose is to generate a robust understanding of the strengths, weaknesses and appropriateness of evaluation approaches and methods in interventions addressing violence against women and girls (VAWG), particularly in international development and humanitarian contexts. Review findings and recommendations are expected to support practitioners’ efforts to (i) commission or implement evaluations that yield robust findings, and (ii) assess the relevance and generalizability of the evidence gathered in evaluations. Furthermore, the Review will contribute to the growing body of literature on applied research on VAWG in development and humanitarian contexts, and to the current debate on broadening impact of evaluation designs.

Interventions tackling violence against women and girls (VAWG) have characteristics that make them difficult to evaluate. VAWG takes many forms and potentially affects all stages and spaces of women’s and girls’ lives. Programmes tackling VAWG tend to combine different types of activities – such as a mix of services for VAWG survivors, public sensitisation campaigns and policy advocacy – to address multiple causes. Some of the changes pursued, for example, reduced social acceptance of VAWG, take many years and are complicated to measure. Social stigma and the risk of re-traumatising survivors make it problematic to gather data from beneficiaries. Seemingly simple indicators – for example, the numbers of clients at counselling centres, or of court cases on VAWG – lend themselves to contradictory interpretations: An increase in reported VAWG cases suggests a welcome attitude change in places where under-reporting has been a problem, while in a different context it could indicate an undesired increase in VAWG incidence. Existing reviews on ‘what works’ (e.g. Bott et al 2005, Heise 2011) have noted issues with evaluation quality, but there is no clear consensus as to what a good evaluation should look like in this field. A number of VAWG-related evaluations exhibit gaps in the validity, reliability and generalisability of their findings. 

Our review team is convinced that there is no single evaluation method likely to produce the best possible results for all VAWG-related interventions. That is why we strive to examine a broad spectrum of evaluation approaches and designs. We will use qualitative comparative analysis (QCA) to identify which combination of factors is needed to produce a good evaluation and in what context. We trust that this approach will reveal a variety of effective ways to assess interventions tackling VAWG in development and humanitarian work – as well as the pitfalls that come with different evaluation designs. Our findings will be distilled into concrete recommendations, illustrated with exemplary evaluations. Process tracing, the second pillar of our methodology, will allow us to precisely identify best practices for successful evaluations.