Our search for evaluations

As explained in our October 2013 post “What evaluations do we review?” we have looked for evaluation reports in English that meet certain criteria. Our search started on 23 September 2013. We used a three-pronged strategy: (1) web-based search (“web search”), (2) communication with contacts in the fields of VAWG and evaluation, and snowballing via these contacts (“snowballing”) and (3) specialised web-based networks (“DFID and helpdesk”). In addition to evaluation reports, we identified meta-evaluations and specialised publications on “best practices” to end VAWG, as well as literature on evaluation quality and effectiveness.

For the web search, we used combinations of the terms “evaluation”, “review”, “assessment”, “best practice” and 38 terms closely related to violence against women and girls and work to end VAWG, such as “violence against women”, “gender-based violence”, “forced sexual initiation”, “forced marriage”, “human trafficking”, “masculinities”. The web search yielded many duplicates and triplicates, i.e. it reached a high degree of saturation which could be expected in view of the overlaps between these terms.

However, focussing on the web search would have yielded incomplete results. As shown in the figure below, 84 per cent of the evaluation reports that we identified came from a single source, i.e. either from web-search, direct communication with our networks or snowballing.

This confirms that it was a good idea to combine three search methods. We have been particularly impressed by the effects of our snowballing action. It was initiated through two channels – (i) direct contact to evaluation and VAWG specialists known by DFID and the Review team, and (ii) publication of our call for evaluation reports on the following list servers and social web sites: Platform for Evidence-Based Learning list server (PELICAN), professional groups on LinkedIn (AEA, Evaluations and Development), Michaela’s Facebook and Twitter accounts, and our blogs www.developblog.org and, after registration of the new domain, www.evawreview.de. 

Sending out e-mails yielded an excellent response. Examining the messages received within three weeks from our call, we counted some 175 persons who had received the call by e-mail (as direct addressees or in copy). The actual number is probably higher, as we cannot assume that we have been copied into all e-mail correspondence. The interest raised by social web posts was considerable: www.developblog.org (Michaela’s blog) registered a peak of 270 page views on the day our call for evaluation proposals was posted on the above-mentioned platforms (as compared to about 40-120/day in “normal” times).

Soon we will describe what kinds of evaluations we have found. Our draft Scoping Report is with the Review Reference Group; when we'll have the final version we'll provide a link to it on this blog.

Soon to come: QCA conditions

We are on the last straight line of two -now that I think of it, three- processes:

• Finalising the scoping - i.e. the gathering and sorting of evaluation reports in a systematic manner and preparing a scoping report explaining how we have done it. The good news is, there are more evaluations - especially published ones - than we had expected to find! The bad news is, we cannot conduct qualitative comparative analysis (QCA - see the posts below for a quick introduction) on all of them. Not because QCA would not allow for it - in the opposite, maximum openness is a key feature and the beauty of QCA - but because going through all reports would take much more time and resources than what we can afford. We must sample! Watch this space. 
• Defining the conditions that -we (and the literature we have reviewed) suspect- contribute to making an evaluation of a VAWG-related intervention useful (or not), i.e. that increase or diminish positive evaluation effects. Our tentative conditions have been reviewed by the Review Reference Group, whose members have come up with useful questions and comments. We are refining the conditions now; soon they will be tested 'in earnest' in a first round of coding.
• Recruiting and instructing coders - a highly qualified team of five (we'll ask them whether they would like to be presented on this blog; if they do, you'll read more about them) has been brought together and awaits our detailed definitions and directions for coding.

We will share more on these points (including our set of conditions) on this blog near the end of this month, when this busy phase of scoping, sorting, sampling, refining our model and instructing the coders will be over. An intensive and exhilarating process!