tag:blogger.com,1999:blog-72299831599997663072024-03-05T02:10:04.408-08:00EVAW ReviewThis blog documents the review of evaluation approaches and methods in interventions related to violence against women and girls (VAWG) in development and humanitarian setting. The review runs from October 2013 to July 2014; it uses qualitative comparative analysis (QCA) and process tracing. EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-7229983159999766307.post-31129437186078235172015-01-29T03:23:00.001-08:002015-01-29T03:23:31.569-08:00PublicationsThis blog has been quiet because we have completed our review and our presentations!<br />
But there are two journal articles in the pipeline -, and the November 2014 issue of the newsletter of the European Evaluation Society (EES) includes a short article that accompanies our award-winning poster. You download the newsletter via <a href="http://europeanevaluation.org/resources/connections">this link</a>. It'll take you to the EES website.EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-61360724482572492452014-10-07T07:37:00.002-07:002014-10-07T07:37:31.853-07:00More reading on QCA and decision tree analysis<span style="font-family: Georgia, "Times New Roman", serif;">For those interested in our conversation with Rick </span>Davies comparing the virtues of Qualitative Comparative Analysis (QCA) and decision tree analysis respectively, there is a new instalment on Rick's blog <a href="http://mandenews.blogspot.co.uk/2014/10/comparing-qca-and-decision-tree-models.html" target="_blank">Rick On the Road</a>. Renewed thanks to Rick and to everyone who have shown their interest in Qualitative Comparative Analysis!EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-19965905588464528002014-10-06T10:52:00.000-07:002015-01-29T03:24:11.915-08:00Qualifying Rick Davies's findings from 'triangulating QCA'<span style="font-family: Georgia, "Times New Roman", serif;">We are delighted to see that there is interest in the resources we have posted on this blog (more than 5,000 page views as of today). Most recently, Rick Davies has used our dataset for a presentation at the 11th biennial EES conference (see earlier posts below) which compares <b>decision tree analysis</b> with Qualitative Comparative Analysis (QCA). Rick comes to the conclusions that (1) a decision tree analysis of our data would have yielded fewer paths to evaluation effectiveness, and that (2) those paths would have operated a more precise differentiation between cases of effective and ineffective evaluation respectively. Rick's presentation is available on <a href="http://www.youtube.com/watch?v=dHhSQHX01L0" target="_blank">YouTube</a>. We have found it stimulating to examine the merits of decision tree analysis (and other methods) as compared to QCA. </span><br />
<br />
<span style="font-family: Georgia, "Times New Roman", serif;">Meanwhile, we believe that Rick’s conclusions are flawed, for the following reasons</span><br />
<a name='more'></a><span style="font-family: Georgia, "Times New Roman", serif;"> <i>(this is going to be a bit technical)</i>:</span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"></span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"> <b>1. The decision tree analysis is <u>not</u> based on the same data set as our QCA.</b> Rick has converted our data set into a binary format, and based his decision tree analysis on this changed data set. </span><br />
<span style="font-family: Georgia, "Times New Roman", serif;">Our QCA rests on a data set which specifies four different values for each condition – not just two, as in a binary data set. Our data is therefore twice as nuanced as the data used in Rick’s decision tree. While the datasets display some similarities, they are certainly not equivalent. An equivalent decision tree would have to take into account four different values for each condition in the tree, not just two. It is likely that in that case, a decision tree of similar precision would need to be much more complex to explain the outcome.</span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><br /><b>2. Decision tree analysis is compared with a type of QCA solution that is <u>not</u> meant to maximise parsimony</b> – although the comparison focuses on parsimony. </span><br />
<span style="font-family: Georgia, "Times New Roman", serif;">QCA produces three types of solutions, called ‘conservative’, ‘intermediate’ and ‘parsimonious’ respectively. Rick has compared the results of the decision tree analysis with our intermediate QCA solution. An <b>intermediate QCA solution</b> is generally recommended for communication to audiences interested in the practical implications of QCA analyses; that is why we have used and argued for an intermediate QCA solution in our review report. However, an intermediate solution is not the most parsimonious solution QCA provides. If the purpose was to compare the parsimony of QCA results with those of decision trees, then the 'parsimonious' QCA solution should be used.</span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><br /><b>3. The decision tree analysis performs less well than stated in the presentation. </b><br />Similar criteria must be applied when comparing the number of paths (and the consistency value) in the QCA and in decision tree analysis respectively.</span><br />
<span style="font-family: Georgia, "Times New Roman", serif;">If the question is how many paths the respective method uses to cover cases of effective evaluations (i.e. to explain positive outcomes), then the answer for the decision tree is four, not three. There are four paths in the decision tree that cover cases with positive outcomes, and the consistency value for these four paths is not 82%, but 76%. </span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><i><br />A side note: Contrary to the contention in Rick's presentation, <b>none of the paths in our QCA is redundant</b>. An aggregated total coverage of more than 100% is <u>no</u> indication of the redundancy of an individual path. While there may be cases that are covered by two or three paths, there can still be cases for each path that are covered only by that single path.<b></b> </i></span><span style="font-family: Georgia, "Times New Roman", serif;"><i><b>In our solution, all paths have a unique coverage</b></i></span><span style="font-family: Georgia, "Times New Roman", serif;"><i><b> of more than 0</b>. Otherwise, they would not feature in t</i></span><span style="font-family: Georgia, "Times New Roman", serif;"><i>he solution, as they would be eliminated by the QCA algorithm.</i></span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><br /><b>4. The decision tree consistency measure is less rigorous than in QCA.</b> Consistency values in fsQCA take into account not only the fact whether a case shows the outcome or not, but also to which degree the case is in line with the posited set relationship. This is possible due to the “gradual” nature of the data in fsQCA. For the consistency value, it means that even if all cases covered by a path show the outcome, the consistency value can be smaller than 1. As the decision tree analysis is based on binary data, the extent to which cases contradict the posited explanation cannot be taken into account – a higher consistency value is therefore more easily achieved.</span><br />
<span style="font-family: Georgia, "Times New Roman", serif;">*** </span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><br /></span>
<span style="font-family: Georgia, "Times New Roman", serif;">Intrigued by the idea of 'triangulating' QCA results with decision tree analysis, we have converted our QCA dataset into a binary format (as Rick did, see point 1 above) and conducted a csQCA with that data. This allows for a more precise comparison of QCA with the decision tree analysis as conducted by Rick.<i> (Btw Rick
Davies may have made a slight error when converting the data, as his
decision tree features 29, not 28 effective evaluations.)</i></span><br />
<span style="font-family: Georgia, "Times New Roman", serif;"><br /></span><span style="font-family: Georgia, "Times New Roman", serif;">The resulting parsimonious solution consists of 7 paths and has a consistency value of 96%. The comparable values for the decision tree analysis are 4 paths and a consistency value of 76%. For the non-occurrence of outcomes, the parsimonious solution consists of 3 paths and has a consistency value of 100%. The comparable values for the decision tree analysis are 4 paths and a consistency value of 29%. </span><br />
<br />
<span style="font-family: Georgia, "Times New Roman", serif;">This means that, if we run the identical dataset through QCA and the decision tree algorithm respectively, then the decision tree analysis yields slightly more parsimonious findings (i.e. fewer paths) than QCA for the occurrence of effective evaluations. For the non-occurrence of effective evaluations, the QCA solution is more parsimonious. At the same time, applying the decision tree algorithm to our data set comes with a massive loss in accuracy for the findings regarding both outcomes.</span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-19403193899368183542014-10-06T10:16:00.000-07:002014-10-06T10:16:18.552-07:00The poster - againApparently some organisations' servers make it difficult to download documents from dropbox, which is where many of the links on this site will take you. Please send in a comment if you experience any such problems and we'll send the file by e-mail. Meanwhile, here is our poster in its .jpg incarnation - a bit difficult to read on this page, I am afraid.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFBgjqQTI3RnqvbLex_hlfrDKwAHVzUq9itVucWDx5WgU7TAXPyBptLvJyFn2q6M3oNfCvHzv828tbSvAYrgBxNvQyZYbGANPO4XfjPxK2eG6qmVpzk7lg6NAtBiggPpl8EMUWFuuRgeM/s1600/Raab_Stuppert+review+EES+poster2014.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFBgjqQTI3RnqvbLex_hlfrDKwAHVzUq9itVucWDx5WgU7TAXPyBptLvJyFn2q6M3oNfCvHzv828tbSvAYrgBxNvQyZYbGANPO4XfjPxK2eG6qmVpzk7lg6NAtBiggPpl8EMUWFuuRgeM/s1600/Raab_Stuppert+review+EES+poster2014.jpg" height="640" width="451" /></a></div>
<br />EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-41725733989649180882014-10-06T01:35:00.001-07:002014-10-06T01:39:52.795-07:00The link to our award-winning poster<span style="font-family: Georgia,"Times New Roman",serif;">As promised in the previous post, this is the <a href="https://www.dropbox.com/s/ngsu6wnzwkrg5oc/Raab_Stuppert%20EvalEVAWReview%20poster%20EES.pdf?dl=0" target="_blank">LINK</a> to our poster presenting the findings of our Qualitative Comparative Analysis (QCA). For those who are interested in innovative ways of visualising QCA findings, the diagram in the middle of the poster (diagram reproduced below) may be particularly interesting. A 2-pager presenting the findings can be downloaded <a href="https://www.dropbox.com/s/x4fb90dhrf4fj9f/Raab_Stuppert%20EvalEVAWReview%202-page%20summary%20of%20findings%20-%20final%20version.pdf?dl=0" target="_blank">here</a>; links the full review report are available in the 'welcome' section on the right side of this blog.</span><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqLSzhaMKUyvj_KND63S6iL59eggwsdJ4_jZwUMMj6QYyK35VVbFPhEiNQ-wV5plC1VmnTp7eFyJ3IryR-PHnIMLe8xCqmZPUTGpKM-pyBAERVd4qZGUYbre3klylaQMB41PELyuJioPw/s1600/paths+as+of+Oct+2014.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqLSzhaMKUyvj_KND63S6iL59eggwsdJ4_jZwUMMj6QYyK35VVbFPhEiNQ-wV5plC1VmnTp7eFyJ3IryR-PHnIMLe8xCqmZPUTGpKM-pyBAERVd4qZGUYbre3klylaQMB41PELyuJioPw/s1600/paths+as+of+Oct+2014.jpg" height="300" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Paths to effective evaluation in the field of VAWG - Diagram from the Raab/Stuppert poster</td></tr>
</tbody></table>
<br />
<br />EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-7787110138232899322014-10-03T01:22:00.003-07:002015-01-29T03:24:55.003-08:00Our poster at the EES Conference<span style="font-family: Georgia, Times New Roman, serif;">The poster presenting the results of our review has won the Best Poster Award at the 11th biennial conference of the European Evaluation Society (EES) in Dublin. If you happen to be around, come and have a look at it - find it in poster area 06 in Liffey A on the 1st floor (where the coffee and lunch breaks happen). </span><br />
<br />
<span style="font-family: Georgia, Times New Roman, serif;">We are delighted about this enthusiastic acknowledgement of our efforts to present the results of a QCA truth table in an accessible form.</span><br />
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-66061570151846785222014-10-01T07:07:00.001-07:002014-10-01T07:07:33.349-07:00EES Conference - pick up a leaflet<span style="font-family: Georgia, "Times New Roman", serif;">If you are at the 11th EES Conference these days, please have a look at our poster (poster area Liffey A, 1st floor) summarising our review findings, and pick up one of the leaflets we have placed near the poster. In case no leaflets are left - you can also download a copy using the link in the previous blog post. </span><br />
<br />
<span style="font-family: Georgia;">And you are warmly invited to join us in Zoe Stephenson's panel session on Thurday 2 October at 9h45 in Wicklow (2nd floor), meeting room 2b.</span><br />
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-68055390814321935092014-09-22T02:36:00.001-07:002015-01-29T03:25:18.225-08:00Presentations at the EES Conference<span style="font-family: Georgia, Times New Roman, serif;">For your diaries in early October:</span><br />
<span style="font-family: Georgia, Times New Roman, serif;">The findings of our review will be presented at the <a href="http://www.ees2014.eu/" target="_blank">11th Biennial Conference of the European Evaluation Society</a> (EES) at two occasions:</span><br />
<a name='more'></a><br />
<br />
<ul>
<li><span style="font-family: Georgia, Times New Roman, serif;">The session facilitated by Zoe Stephenson (DFID) "Effective evaluation of violence against women and children" on Thursday <b>2 October at 9h45</b> at the venue called <b>Wicklow, Meeting Room 2b</b>. Panelists will include Sally Neville (UK), Alexandra Chambel and Krishna Belbase (both USA), as well as Michaela Raab.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">On Wednesday<b> 1 October at 2pm</b> (14h), Michaela will present a poster summarising our review findings. All poster sessions will happen from 14h to 14h45; you will find Michaela in <b>Poster area Liffey A</b>. For those with no or little time to spend near our poster, 2-page leaflets to take away will be available in the poster area. A few small tweaks in the decision tree visualising our findings have made it more user-friendly than ever before - so even if you have read our report, <b>come and take a look</b>! If you can't make it to Dublin, feel free to download the leaflet <a href="https://www.dropbox.com/s/x4fb90dhrf4fj9f/Raab_Stuppert%20EvalEVAWReview%202-page%20summary%20of%20findings%20-%20final%20version.pdf?dl=0" target="_blank">here</a>.</span></li>
</ul>
<div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">Michaela has also volunteered to chair a totally different session. She has been assigned the session S097 - "Evaluation at local level", which will include Dries Verlet (Belgium), Fation Luli (Albania), Melanie Kitchener (UK) and Ludger Niemann (Netherlands). You can find us on Friday 3 October at 2pm in Wicklow.</span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;">If you wish to find out more about our review, please have a look at the documentation on this blog (in particular "Welcome to our blog" on the right-hand column). Michaela would be delighted to meet you at one of the sessions above. You can also contact us via the addresses you find on our reports.</span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;">Unfortunately the QCA experts on our review team, Wolfgang Stuppert and Julian Brückner, cannot make it to the EES conference this year. But you will find several papers and sessions on QCA in the conference programme. For instance, Barbara Befani will speak about "Applying QCA to real life evaluations: potential and pitfalls" as part of the session S082 "Enhancing Evaluation Practice" on 3 October at 11h45.</span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;">The full conference programme is available in the form of a </span><a href="http://www.czech-in.org/cm/ees2014/download/EES_2014-Scientific_programme.pdf" style="font-family: Georgia, 'Times New Roman', serif;" target="_blank">huge table</a>.<span style="font-family: Georgia, 'Times New Roman', serif;"> The list of poster sessions is separate from the conference programme; find it via </span><a href="http://www.czech-in.org/cm/ees2014/CM.NET.WebUI/CM.NET.WEBUI.scpr/SCPRsessions.aspx?conferenceid=05000000-0000-0000-0000-000000000051&sessionId=05000000-0000-0000-0000-000000002261" style="font-family: Georgia, 'Times New Roman', serif;" target="_blank">this link</a><span style="font-family: Georgia, 'Times New Roman', serif;">.</span></div>
<br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-45785528071487557662014-07-29T02:59:00.000-07:002015-01-29T03:25:37.342-08:00Presenting and discussing our findings<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">On 1 July 2014, we – Michaela Raab and
Wolfgang Stuppert – presented the findings of our <a href="http://r4d.dfid.gov.uk/Output/196633/Default.aspx" target="_blank">review</a> at the DFID office in
Whitehall (London). Some 30 participants from DFID and external organisations –
representing chiefly NGOs and consulting firms working on violence against women and girls - attended the 2-hour event, including DFID staff outside Whitehall (who joined us
via videoconference). After our presentation (slides available<a href="https://www.dropbox.com/s/w2lon0nwtw4czfz/Raab%20Stuppert%20Review%20of%20VAWG%20evaluations%20July%202014.pptx" target="_blank"><span style="color: blue;"> via this link</span></a>), Sam
Coope from DFID in Zimbabwe and Asmita Naik, an independent consultant,
reported about their experience in VAWG-related evaluation.</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">The discussion that followed drew from the
participants’ rich and diverse programming and research experience. We would
like to flag a few points</span></span><br />
<a name='more'></a><span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB"> that we have found particularly interesting. Those
who have been in the meeting will notice that we have added a couple of extra thoughts.</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span lang="EN-GB">Do we want to have evaluation without high
quality data</span></b><span lang="EN-GB"><b>?</b> If you put it that
way, no. And if, in our review, you don’t see “compelling evidence” in most paths to evaluation
effectiveness, that does <u>not</u> mean that good data is
superfluous. If your evaluation is going to be influential, you better make
sure you base it on accurate data.</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">However, there are evaluations that do not
need to collect data according to social science standards. For instance, one
evaluation we examined traced the history of a long-term programme on the basis
of interviews and group discussions with key staff only. The commissioner found
it effective because it made a previously implicit theory of change more
explicit, and helped the organisation to further develop its strategies.
Arguably, this type of participatory sense-making exercise does not require any
particularly rigorous data collection.</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span lang="EN-GB">Choice of qualitative or quantitative
approach:</span></b><span lang="EN-GB"> We did not discover any pattern that
would link a certain evaluation approach to a certain type of intervention. It
is commonly assumed that qualitative approaches work particularly well with formative
evaluation, while in impact evaluation, quantitative research or a combination of qualitative and quantitative approaches can yield robust
evidence. </span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">Every so often we run into people who
believe that the only rigorous evidence you can get is quantitative. That is
wrong. Poor qualitative research yields poor data, and so does poor
quantitative research. Good research of any kind tries to prevent bias through appropriate sampling and questions design; it gathers data from different
perspectives, and is transparent about the tools it uses and their limitations.</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span lang="EN-GB">Many impact evaluations happen too early</span></b><span lang="EN-GB"> to make sense, for instance if the theory of
change of an intervention is still emerging, or if it is applied inconsistently
or incompletely. There are programmes that start without a clear idea of the
outcomes they want to achieve and how exactly they intend to get there. Where a
programme is still searching its form, it makes no sense to spend money on rigorous
impact evaluation – it won’t yield any evidence that can be used elsewhere.
Better go for some different form of research and reflection, possibly
something that involves substantive participation and sensitivity to
VAWG-related issues, to improve the programme.</span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">There seems to be a divide between
evidence generated by academic research and evidence generated through
evaluations. Lori Heise (London School of Hygiene and Tropical Medicine) noted
that she could not see any overlap between the research-based publications on
VAWG her team worked on, and the evaluation reports our review was based on.
She advocated for tearing down the “Chinese Wall” between research and
programming. <o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">One could argue that the primary purpose
of academic research is to generate knowledge (generalisable answers), while
the primary purpose of evaluation is to improve programmes and to assess their
effectiveness (specific answers). VAWG programming is mainly about
improving women’s and girls’ lives – not necessarily about increasing
knowledge. </span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">Where a programme strives to produce scientific
evidence, researchers should be involved in the programme starting from its
design phase, to ensure a good match between programme implementation and the
conditions for research. Our review report describes an excellent example for
such work, Julia Kim’s evaluation of the Refentse Model of Post-Rape Care (2009;
summarised on pages 30-33 of our <a href="http://r4d.dfid.gov.uk/Output/196633/Default.aspx" target="_blank">review</a>).</span><span lang="EN-US"><o:p></o:p></span></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<br /></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-60111143874470812952014-07-21T05:10:00.000-07:002015-01-29T03:25:55.875-08:00Dataset available now! <div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">At the
early stages of our review we promised we would share the QCA dataset. We –
Michaela Raab and Wolfgang Stuppert – hold the intellectual property rights to
the information we have generated. Our DFID counterparts would like the data to
be freely accessible for people wanting to explore it, replicate our analysis
or do further analysis – and so do we. </span></span><br />
<a name='more'></a><span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">We have
anonymised the dataset so that users cannot track any data to any specific
evaluations. (This is to protect the confidentiality of the survey and
interview responses we have received, and because some of the evaluations we
have analysed are unpublished.) The full dataset is available via<a href="https://www.dropbox.com/s/zwog5aspzihw4rt/QCA%20dataset%20for%20blog.dat" target="_blank"> this link</a>. It contains values for the seven conditions we have used
in QCA, for the different types of evaluation effects and the aggregated
effects, as well as a case indicator variable which only contains numbers.</span></span></div>
<br />
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">If you wish
to use the data and publish anything that uses the data, please acknowledge
DFID’s funding of the original data collection, and Michaela Raab and Wolf
Stuppert as the researchers who have generated it.</span><o:p></o:p></span></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-66003011191968675252014-07-09T05:40:00.002-07:002014-07-09T05:40:41.054-07:00Full Review report on-line!<span style="font-family: Georgia, Times New Roman, serif;">Our full Review report is now available on-line, on the DFID Research for Development (R4D) site. This <a href="http://r4d.dfid.gov.uk/Output/196633/Default.aspx" target="_blank">link</a> will take you there; you can download the report for free. The actual report encompasses 30 pages with plenty of tables in-between, but the annexes are huge as we have appended all our data collection instruments and plenty of extra information.</span><div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">For very rushed readers, a 2-page document summarising practical recommendations will be added at a later point. (Meanwhile feel free to study the short executive summary, which is part of the report.)</span><div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">We would be delighted to have your comments, here or via the e-mail address you'll find in the report.</span><div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
</div>
</div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-66438490286839305432014-07-03T07:56:00.000-07:002015-01-29T03:26:14.800-08:00Presentation of our findings and recommendations<span style="font-family: Georgia, Times New Roman, serif;">On Tuesday 1 July, the two of us - Michaela Raab and Wolf Stuppert - presented our review findings and recommendations at the DFID office in Whitehall, London. Many thanks to everyone who attended and who contributed useful questions and insights. Within the next couple of weeks</span><span style="font-family: Georgia, 'Times New Roman', serif;">, we will summarise a few points that we have found particularly interesting in the discussion, and share them here.</span><br />
<a name='more'></a><br />
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">We will present our findings again at the European Evaluation Society Conference in Dublin, 1-3 October 2014, in one dedicated session and a separate poster presentation. Furthermore, we'll produce:</span><br />
<br />
<ul>
<li><span style="font-family: Georgia, 'Times New Roman', serif;">a 2-pager summarising what our findings mean to commissioners of evaluations</span></li>
<li><span style="font-family: Georgia, 'Times New Roman', serif;">a short article on approaches and methods in evaluating VAWG</span></li>
<li><span style="font-family: Georgia, 'Times New Roman', serif;">and a more scholarly article as to how we have used QCA and presented our findings to a lay audience. </span></li>
</ul>
<span style="font-family: Georgia, 'Times New Roman', serif;"><span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span>
The articles will be submitted to peer-reviewed journals, so it'll take some time before they are published. But the 2-pager will be made accessible on this blog as soon as it is ready.</span><br />
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">The full review report is about to be published on the DFID R4D website; we'll provide the link when we have it. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Please feel free to share your comments on this blog.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Renewed thanks for everyone who has contributed ideas, information, questions and precious time to this Review!</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-22510622017470581892014-07-03T07:44:00.000-07:002015-01-29T03:26:36.530-08:00Q&A on definitions<span style="font-family: Georgia, Times New Roman, serif;">Some snag in our 'comments' function makes that comments are only shown if readers click on the titles of our posts. A couple of weeks ago, Rick Davies asked a few questions about our definitions:</span><br />
<a name='more'></a><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<br />
<div class="MsoNormal">
<span lang="EN-US"><i>Thanks again for another useful update on
your work. </i></span><i>Three questions, to help those of us
following in your footsteps:</i></div>
<div class="MsoNormal">
<span lang="EN-US"><i>1. You have on aggregate Outcome (strong
evaluation effect) made up of 4 types of effects. Why bother with the aggregate
level, why not do 4 separate QCA analysis, one per specific effect?<o:p></o:p></i></span></div>
<div class="MsoNormal">
<span lang="EN-US"><i>2. Re the tree diagrams for each condition,
how will you aggregate judgements up from the particulars (leaves) to the more
general (root)?</i></span></div>
<div class="MsoNormal">
<i>3. Re the figures 2, 4, 5, 6, where are
these explained. They look like cutoff points on the scales used to judge each
condition, a fuzzy set value, but I may be way off.</i></div>
<div class="MsoNormal">
<span lang="EN-US"><i><br /></i></span></div>
<span style="font-family: Georgia, Times New Roman, serif;">And here are our answers. A warning: this is going to be quite technical!</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">1. We believe that to call an evaluation<b> effective </b>or not implies more than the evaluation having caused (or not) one particular type of effect. It is an overall judgment of the consequences an evaluation has had. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;">We think that the three types of effects on active stakeholders are substitutive, i.e. having caused one type of effect is enough for the evaluation to be effective for active stakeholders. This is because we believe that different kinds of evaluations pursue different kinds of goals. If a formative evaluation achieves a high degree of action effects, to us, it is as effective as an impact evaluation achieving a high degree of persuasion or learning effects. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;">Furthermore, we believe that one cannot call an evaluation effective if it has produced a negative effect on beneficiaries, no matter how strong its effects on active stakeholders. This is why we believe an evaluation should have caused at least one type of effects on active stakeholders and have had a positive effect on beneficiaries to merit the designation of being overall “effective”. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;">Conducting QCAs for each type of effect and reporting the respective results would have sent the wrong message to our audience.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">2. For <b>favourable context</b>, we assumed partial compensation of the three components. This is why we took the mean of the three values for the components as the value for the condition itself. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;">For the three types of <b>effects on active stakeholders</b>, we assumed full substitutability. That’s why in this case we have taken the maximum value for the three components as the value for the outcome component “effect on active stakeholders”. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;">All other components were aggregated by taking the minimum of the values for the components, including the aggregation of “effect on active stakeholders” and “empowerment effect”.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"></span><br />
<span style="font-family: Georgia, 'Times New Roman', serif;">3. As we decided against explaining the aggregation rules to our readers, we wanted to indicate what the presence of a given condition means empirically with regard to the presence or absence of its components. Hence, the</span><b style="font-family: Georgia, 'Times New Roman', serif;"> diagrams</b><span style="font-family: Georgia, 'Times New Roman', serif;"> Rick refers to show the distribution of a given sub-dimension among the 39 evaluations, given the presence or absence of the condition they are part of.</span><br />
<div>
<br /></div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-70317793738744797952014-05-06T06:21:00.003-07:002015-01-29T03:28:28.552-08:00Definitions for QCA<span style="font-family: Georgia, "Times New Roman", serif;">As announced earlier, we are now in a position to share the definitions we have used in our Qualitative Comparative Analysis (QCA). You can download the definitions as a PDF document by clicking <a href="https://www.dropbox.com/s/g4d3o0s9civrut0/Raab_Stuppert%20Definitions%20for%20QCA%20VAWG%20related%20evaluation.pdf?dl=0" target="_blank">here</a>. </span><br />
<a name='more'></a><br />
<br />
<span style="font-family: Georgia, "Times New Roman", serif;">The document presents the top-level, aggregated conditions we have employed in QCA. Behind every individual condition there is a host of detailed definitions. We have written up some 60 pages of such definitions, with instructions for the precise measurement of the factors that have entered our analysis. If you have any questions regarding a specific element, please do not hesitate to get in touch via the comments function on this blog.</span><br />
<br />
<span style="font-family: Georgia, "Times New Roman", serif;">Our draft Review report is with DFID and the Reference Group now; by mid-June you can expect to find the final product here. </span><br />
<br />
<span style="font-family: Georgia, "Times New Roman", serif;">Furthermore, we are working on two articles - one on the findings, the other one on QCA, for peer reviewed Journals. We'll be in London to present our findings at an event hosted by DFID (date & venue to be confirmed). Michaela will present a poster on our Review findings and how we used QCA in it at the Conference of the European Evaluation Society in Dublin (October 2015). We'll keep you posted!</span><br />
<br />EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-12583055083468553702014-03-31T02:00:00.003-07:002015-01-29T03:28:49.362-08:00Thoughtful comments<span style="font-family: Georgia, Times New Roman, serif;">Our inception report has received a thoughtful review by Rick Davies on his blog <i><a href="http://mandenews.blogspot.de/" target="_blank">Rick on the Road</a>. </i>Titled <i>"The Challenges of Using QCA", </i>the posting summarises some of the methodological difficulties we have encountered so far.</span><br />
<a name='more'></a><span style="font-family: Georgia, Times New Roman, serif;"> </span><span style="font-family: Georgia, 'Times New Roman', serif;"><span style="font-size: x-small;">(There have been a few non-methodological ones as well. For example, the detective work we carried out to identify and contact hundreds of evaluation stakeholders so as to ask them about the effects the evaluations had produced.)</span></span><br />
<div>
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">A couple of clarifications, to read with Rick's blog post: : The "model" reproduced in Rick's posting has a very minor place in our inception report (that is why it is part of the annexes). We we have used it only as a communication device to solicit ideas and comments from the Reference Group. The purpose of our QCA is _not_ to test the model, but to identify combinations of conditions that lead to effective evaluations. In a way, the model is just a list of potential ingredients for effective evaluation.</span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">Since our dialogue with the Reference Group in late 2013 we have adjusted and defined the five central conditions and their components. They look different now. For instance "convincing methodology" has been replaced by "compelling evidence", which is about quality standards in research (such as triangulation of data sources, transparent documentation...). Methodological choices enter the analysis as separate conditions.</span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div>
<span style="font-family: Georgia, Times New Roman, serif;">We will post the precise definitions of the conditions (or "ingredients") in April. Watch this space. </span></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-40545937880355599392014-03-26T12:02:00.000-07:002014-05-12T02:27:24.576-07:00Inception report ready for sharing<span style="font-family: Georgia,"Times New Roman",serif;">The inception report, which summarises our review methodology, is finally ready for wider dissemination. You can download it by clicking on <a href="https://www.dropbox.com/s/k77026t127aypil/Raab_Stuppert%20Inception%20Report%20VAWG%20Evaluations%20Review%2020140324%20min.pdf" target="_blank">THIS LINK</a>. It is a large file of some 1.9MB. </span><br />
<br />
<span style="font-family: Georgia,"Times New Roman",serif;"><i>Due to some technical complication that we do not want to explore in detail, some text magically disappears when you try to read the report in your browser. (For instance, in Firefox our response to the points raised by SEQUAS disappear.) So: if you wish to read the report, do <u>not</u> try to read it in your browser - <b>rather click on "download" and use Adobe Reader </b>to make sure you get the full text. </i></span><br />
<span style="font-family: Georgia,"Times New Roman",serif;"><br /></span>
<span style="font-family: Georgia,"Times New Roman",serif;">The annexes to the report include the tools we have used so far (coding instructions, survey questions and interview guides), as well as documentation on our initial dialogue with the External Reference Group, and with the Specialised Evaluation and Quality Assurance Service (SEQUAS). We will post more updates in the coming weeks.</span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-38501428760640579252014-03-26T11:53:00.000-07:002014-03-26T12:15:23.965-07:00Final report planned for June 2014!<span style="font-family: Georgia,"Times New Roman",serif;">We have received a few queries as to when our final report will be available. The plan is to complete it by/ in June 2014. It will include our findings from Qualitative Comparative Analysis and Process Tracing, as well as short descriptions of 15-20 evaluation approaches and designs that we have found effective or promising. </span><br />
<span style="font-family: Georgia,"Times New Roman",serif;">Furthermore, we will produce a couple of papers:</span><br />
<ul>
<li><span style="font-family: Georgia,"Times New Roman",serif;">A paper summarising our findings for development and evaluation practitioners.</span></li>
<li><span style="font-family: Georgia,"Times New Roman",serif;">A more academic, peer-reviewed article that will explain our review methodology, in particular the use of QCA in this study. Peer review means that it'll probably take until 2015 or even 2016 until the article is published.</span></li>
<li><span style="font-family: Georgia,"Times New Roman",serif;">We will also post summaries of our findings and of key steps in our research on this blog. It has been a bit quiet in recent weeks because we wanted to complete our dialogue with the external reference group that accompanies the review before sharing the details we explain in our Inception Report (see post above).</span></li>
</ul>
<span style="font-family: Georgia,"Times New Roman",serif;">We are still looking for suitable international events in late May or in June to present our findings. If any interesting events come to your mind, please let us know!</span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-16192829028773363802014-01-30T13:44:00.000-08:002014-01-30T13:44:12.676-08:00Helpful comments<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;">A few comments have been posted on our blog. Apologies we are a bit slow with our
reaction: We have been so absorbed by our hunt for contact details (see
earlier post) and the launch of our survey with evaluation stakeholders - that we
sort of lost sight of the useful comments which have appeared.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;">Rick Davies
has shared a reference – thank you! – and directed us to an interesting
blog posting on the question whether evaluations must fulfil certain quality
standards to produce positive effects. The comments appear when you click on
the titles “Inception” and “Evaluations identified for the first coding round”
below. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;">Carol
Miller (under “Soon to come: QCA conditions”) hopes we will look at evaluation
processes in terms of how they contribute to empowerment of key stakeholders.
Empowerment of stakeholders is indeed among the effects we intend to measure.
By the way, our model for QCA will be published here - on this blog - with our final inception report,
by March 2014. A couple of updates will be posted before that date.</span></span></div>
<div class="MsoNormal">
<span lang="EN-US"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-US" style="font-family: Georgia, 'Times New Roman', serif;">One caveat:
this project is not a huge piece of original research – it is a review of
evaluation reports that is enriched by some primary data collection on
evaluation effects, chiefly through a short web-based survey with four types
of evaluation stakeholders – evaluators, people who commission evaluations, people who have implemented the interventions evaluated, and representatives of organisations that have funded the interventions. (We realise these categories sometimes overlap.) </span><span style="font-family: Georgia, 'Times New Roman', serif;">So, we won’t go into
the fine detail of every possible evaluation effect. But we are confident we
can find some interesting contours.</span></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-11185782917028196002014-01-23T07:31:00.000-08:002014-01-23T07:56:52.761-08:00Message from DfID to the survey participants<span style="font-family: Georgia,"Times New Roman",serif;"><i>This week we have sent out our web-based survey. Our DFID counterparts, Zoe Stephenson and Clare McCrum, are sharing this message, to encourage prospective respondents to respond to our request to engage with the survey:</i></span><br />
<span style="font-family: Georgia,"Times New Roman",serif;"><br />This piece of work is a review of VAWG evaluations, commissioned by DFID’s Evaluation Department. It’s a really interesting piece of work exploring what makes evaluations effective/influential. The consultants have identified about 70 VAWG evaluations to explore and they will be using QCA (Qualitative Comparative Analysis) and process tracing to explore the factors that influence whether the evaluations were used/useful. Doing this depends on seeking the views of some key stakeholders who were involved in the evaluations – ideally the commissioner, the evaluator and someone involved with the programme’s implementation.</span><br />
<span style="font-family: Georgia,"Times New Roman",serif;"><br />The work is funded by DFID and the engagement of those who have received the survey would be greatly appreciated by us. It is important for a large number of persons to participate in the survey - otherwise we may have to remove some evaluations from the set that the researchers work on and that would be a shame. The larger the set the better!</span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-60481865352247844552014-01-07T06:01:00.000-08:002014-01-07T06:01:02.809-08:00Your help is needed: names and contacts<span style="font-family: Georgia, Times New Roman, serif;">As explained in the previous post, we plan to contact (i) evaluators, (ii) people
who have commissioned evaluations ("evaluation commissioners") and (iii)
representatives of the organisations whose work has been evaluated - for
the full set of evaluations. It has proven difficult to identify contact
persons for all evaluation reports. </span><br />
<div>
<span style="font-family: Georgia, Times New Roman, serif;"><br />Hence this crowdsourcing action: We would be immensely grateful if you
could have a look at the list available under <a href="https://www.dropbox.com/s/h8zjbud81hmy1n8/Wanted%20-%20contacts%20for%20these%20evaluations.pdf" target="_blank">THIS LINK</a> (click on "THIS LINK" to get there). Would you happen to know anyone who
is knowlegeable about any of these evaluations? If so, please write to
<a class="moz-txt-link-abbreviated" href="mailto:review-team@gmx.de">review-team@gmx.de</a>, possibly with the name and e-mail address of the
person who can share information about the evaluation.
</span></div>
<div>
<br /></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-22944609848944544072014-01-06T09:19:00.000-08:002014-05-12T02:33:28.655-07:00Inception<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">The review
process encompasses three phases: Scoping, Inception and the actual Review,
i.e. the analysis of evaluations. We have completed the scoping phase, and we
are deep into the inception phase now.</span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;">In parallel
with our search for evaluations (Scoping Phase), we have reviewed literature
and initiated a virtual discussion with the Review Reference Group on the <b>dimensions of evaluation practice</b>. We
are interested in the characteristics of evaluations and the positive or
negative results they produce. To obtain a first understanding as to the
characteristics and effects we need to look for, we have studied relevant
literature. (See your <a href="https://www.dropbox.com/s/e7hla5qp49mvgfm/Raab_Stuppert%20Scoping%20Report%20VAWG%20Evaluations%20Review%20DFID%202013%20blog%20Jan%202014.pdf" target="_blank">Scoping Report</a> for the full literature list.)</span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">We have
identified a wide range of elements that are considered as influencing the
effect of evaluations. That is, they are likely <b>conditions for positive
evaluation effects</b>, in QCA terminology. These conditions have been
provisionally clustered into five dimensions: </span></div>
<div class="MsoNormal">
</div>
<ol>
<li><span style="font-family: Georgia, Times New Roman, serif;">Conducive circumstances, which are present when the intervention is evaluable and the political environment (among and beyond evaluation stakeholders) favourable.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">Powerful mandate, something evaluators have if resources are appropriate, the evaluation is timely and the evaluation team commands high esteem.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">Convincing methodology that leads to compelling evidence, is well documented participatory and ethically sound (‘do no harm’).</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">Effective communication, which rests on presentation and dissemination of findings, conclusions, recommendations and lessons learned.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">High context sensitivity, in particular regarding gender, cultural and professional issues.</span></li>
</ol>
<span style="font-family: Georgia, Times New Roman, serif;">This is tentative and fairly abstract – our inception report will come with more precise definitions and calibrations to make fuller sense of these concepts. There is no hierarchy in these conditions. For the time being, the purpose of this initial inventory is to find out what could possibly influence evaluation effects. The provisional model we have built is a ‘maximum model’ in that it attempts to integrate a wide range of possible conditions.</span><br />
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">We have
also looked more closely at the <b>effects
of evaluations</b>, clustering them into three groups:</span></div>
<ul>
<li><span style="font-family: Georgia, Times New Roman, serif;">Effects on development practice – i.e. changes in the further implementation of the intervention evaluated, or in the implementation of subsequent interventions.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">Effects on accountability and advocacy.</span></li>
<li><span style="font-family: Georgia, Times New Roman, serif;">Effects on the wider knowledge base– in terms of learning beyond the actual intervention, for example the contribution an evaluation makes to the global knowledge base on “what works” in efforts to end violence against women. </span></li>
</ul>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">The Review
Reference Group (RRG) examined the tentative model in October and provided rich
comments. The dialogue with the RRG and our DfID counterparts has helped us to
clarify the terminology used and to appreciate the many facets of these
dimensions. <o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">Following
from that, we have developed detailed reporting sheets for the coders. The
coders have started their first coding round, examining all 74 reports we
identified in our search (see earlier post). At this point, their job is to map
the data on conditions they find in the reports<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">As to the
effects generated from the evaluations, we cannot rely on the reports for data.
Therefore we are building a survey. For every evaluation in our set, we are
planning to question at least two out of three types of stakeholders: (1) the
evaluator, (2) a person who has commissioned the evaluation, (3) a
representative of the organisation that has implemented the intervention
evaluated and who can report on the effects of the evaluation. We have
interviewed 2-3 representatives of each category to further enrich our picture
of the effects evaluations can generate. At the moment, we are building <b>a web-based survey</b> that will be sent
out in early January.<o:p></o:p></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif;">By the end
of January, we expect to have:</span></div>
<div class="MsoNormal">
</div>
<ul>
<li><span style="font-family: Georgia, Times New Roman, serif; text-indent: -18pt;">An
accurate picture of the data available from each of the 74 evaluation reports.</span></li>
<li><span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif; text-indent: -18pt;">Rich
data on many of the conditions in our model, from 74 evaluation reports.</span></li>
<li><span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif; text-indent: -18pt;">Information
from our survey respondents on evaluation characteristics which the reports have
not provided sufficient data on.</span></li>
<li><span lang="EN-GB" style="font-family: Georgia, Times New Roman, serif; text-indent: -18pt;">Data
on the effects the evaluations have produced.</span></li>
</ul>
<span style="font-family: Georgia, Times New Roman, serif;">Qualitative
comparative analysis (QCA) is at the heart of our review methodology. If we
obtain meaningful data on conditions and effects, we can go ahead with QCA.
That is why we have put in extra shifts to make sure we can contact a large
number of evaluation stakeholders – a task that has proven more difficult than
expected! (See post below, “Review + detective work”.)</span>EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-82318929069155667612014-01-06T09:10:00.000-08:002014-01-06T09:10:15.404-08:00Review and detective work<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB">Most of the
74 evaluation reports in our first coding round do not display the evaluator’s
or the commissioner’s contact details. In some cases, the evaluators remain
anonymous; in other cases, the only e-mail address available in the report is a
generic info@xyz.org</span><span lang="EN-GB">.
This has surprised us – in our own evaluation practice, we always include our
e-mail addresses so that our counterparts can get in touch with us in case, say,
they wish to work with us again. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-GB"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">Even where
we could find an e-mail address, it was not easy to elicit a response. One
could blame the busy season - early December, when in many countries the
festive season starts and/ or the fiscal year is about to end. But I was puzzled
to see that even in organisations with dedicated monitoring and evaluation
staff, knowledge about evaluations – including fairly recent ones (2011-2012) in the public domain – appeared uneven.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">Our hunt
for addresses continues; we are seeing light at the end of the tunnel. Many thanks to everyone who has helped finding evaluation stakeholders around the world! We are particularly indebted to people in organisations with several evaluations of VAWG-related
work and who have been specially generous in sharing information at this busy time: extra thanks to CARE, FOKUS (fokuskvinner), the International
Rescue Committee, Oxfam, the Population Council, UNICEF and WOMANKIND. We have also benefited from the support of the
Review Reference Group members (special thanks to Amanda Sim, Helen
Lindley and Krishna Belbase). Some UNDP country registries have also proven effective in identifying evaluation stakeholders when we had no other contacts.</span></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, 'Times New Roman', serif;">For a handful
of evaluations found via the web, we have not yet managed to obtain any
addresses that work. We will post the list shortly to ask for 'crowdsourcing' support in identifying stakeholders. If the authors and users remain shrouded in mystery, we
will have to remove these evaluations from our QCA set. Which is OK – QCA also works with
small sets of cases. But it would be hard to draw conclusions for the overall
evaluation landscape if we ended up with, say, just a dozen of evaluations.</span></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-9925977019814226042014-01-04T07:17:00.002-08:002014-10-20T02:11:34.204-07:00Evaluations identified for the first coding round<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">One thing
that is special about our approach is that we do not only apply established
quality standards to the evaluations we review. Instead, we will look into <b>evaluation effects</b> as well. Whether or
not an evaluation has to fulfil established quality standards to produce
positive effects is an open research question. To answer it, we have to include
in our review evaluations that vary in the degree to which they fulfil certain
methodological standards. We hope that our research will shed light on the
factors that contribute to negative and positive evaluation effects. <o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">We
initially <b>cast a large net</b>, searching
for any evaluations on work related to violence against women and girls. A first, cursory exam of the reports we netted
showed that summaries tended to contain too little information on evaluation
approaches and methods. Therefore we
decided to work with full evaluation reports only.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">We found
140 such reports. In many reports that included VAWG as a secondary component (e.g.
evaluations of multi-sector country programmes, reproductive health initiatives
and humanitarian aid), VAWG-related work tended to occupy a marginal position.
Analysing those reports could yield useful information on the quality and
effects of evaluations in general – but our focus are <b>evaluations that are
specifically designed for interventions on violence against women and girls</b>.</span></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, 'Times New Roman', serif;">In a further step, we narrowed down our set to reports </span><b style="font-family: Georgia, 'Times New Roman', serif;">completed
in 2008-2012</b><span style="font-family: Georgia, 'Times New Roman', serif;">, excluding evaluations produced in 2013. This is because we
will question (through interviews and a web-based survey) evaluation stakeholders
about the effects the evaluation has produced. To make sure we can take into
account effects that occur after an evaluation, we must allow for some time.
One year seems a reasonable time-frame, even though we realise that some
effects often occur at a later stage (for instance, the use of ‘lessons
learned’ published in an article).</span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, 'Times New Roman', serif;"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">Of the 140
full evaluation reports, we have excluded 16 because they fell outside the
2008-2012 period, 43 because they evaluated interventions which included VAWG
as a minor component, and 6 because both exclusion criteria applied. (One report did not show the year of publication.)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-top: 10.0pt;">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">The remaining set includes <b>74
evaluations of VAWG-related interventions in low- to middle-income</b>
countries. This is the <b>full set</b> of
evaluations we have found to meet all our criteria – i.e. we do not draw any
sample. The evaluations cover three different contexts – development,
humanitarian and conflict / post-conflict, and the four strategic priorities
that inform DFID’s work on violence against women and girls (see figure below).
Figures refer to the number of evaluations that match the criteria; the total
exceeds 74 because some evaluations match several criteria.</span></span></div>
<div class="MsoNormal" style="margin-top: 10.0pt;">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<table border="1" cellpadding="0" cellspacing="0" class="MsoTableMediumGrid3Accent1" style="border-collapse: collapse; border: none; mso-border-alt: solid white 1.0pt; mso-border-themecolor: background1; mso-padding-alt: 0cm 5.4pt 0cm 5.4pt; mso-table-layout-alt: fixed; mso-yfti-tbllook: 1184; width: 510px;">
<tbody>
<tr style="height: 1.0cm; mso-yfti-firstrow: yes; mso-yfti-irow: -1;">
<td style="background: #4F81BD; border-bottom: solid white 3.0pt; border: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-bottom-themecolor: background1; mso-border-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 168.45pt;" width="225"><div class="MsoNormal" style="margin: 4pt 0cm 0.0001pt; page-break-after: avoid;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">DFID
priorities</span></b><span lang="EN-GB"><o:p></o:p></span></span></div>
</td>
<td style="background: #4F81BD; border-bottom: solid white 3.0pt; border-left: none; border-right: solid white 1.0pt; border-top: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 78.45pt;" width="105"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; font-size: 10.5pt; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Development</span></b><span lang="EN-GB" style="font-size: 10.5pt; mso-ansi-language: EN-GB; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></span></div>
</td>
<td colspan="2" style="background: #4F81BD; border-bottom: solid white 3.0pt; border-left: none; border-right: solid white 1.0pt; border-top: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 77.45pt;" width="103"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; font-size: 10.5pt; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Humanitarian</span></b><span lang="EN-GB" style="font-size: 10.5pt; mso-ansi-language: EN-GB; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></span></div>
</td>
<td style="background: #4F81BD; border-bottom: solid white 3.0pt; border-left: none; border-right: solid white 1.0pt; border-top: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 58.15pt;" width="78"><div align="center" class="MsoNormal" style="margin: 2pt 0cm 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; font-size: 10.5pt; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Post-/Conflict</span></b><span lang="EN-GB" style="font-size: 10.5pt; mso-ansi-language: EN-GB; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman";"><o:p></o:p></span></span></div>
</td>
</tr>
<tr style="height: 1.0cm; mso-yfti-irow: 0;">
<td style="background: #4F81BD; border-bottom: none; border-left: solid white 1.0pt; border-right: solid white 3.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 168.45pt;" width="225"><div class="MsoNormal" style="margin: 2pt 0cm 0.0001pt; page-break-after: avoid;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span lang="EN-US" style="color: white; mso-ansi-language: EN-US; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Building political will and institutional capacity</span></b><span lang="EN-GB"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 78.45pt;" width="105"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">22</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 70.35pt;" width="94"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">7</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td colspan="2" nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.25pt;" width="87"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">12</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
</tr>
<tr style="height: 1.0cm; mso-yfti-irow: 1;">
<td style="background: #4F81BD; border-bottom: none; border-left: solid white 1.0pt; border-right: solid white 3.0pt; border-top: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-left-themecolor: background1; mso-border-right-alt: solid white 3.0pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 168.45pt;" width="225"><div class="MsoNormal" style="margin: 2pt 0cm 0.0001pt; page-break-after: avoid;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Changing
social norms</span></b><span lang="EN-GB"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-alt: solid white .75pt; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 78.45pt;" width="105"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">37</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-alt: solid white .75pt; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 70.35pt;" width="94"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">2</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td colspan="2" nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-alt: solid white .75pt; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-alt: solid white 1.0pt; mso-border-right-themecolor: background1; mso-border-right-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.25pt;" width="87"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">6</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
</tr>
<tr style="height: 1.0cm; mso-yfti-irow: 2;">
<td style="background: #4F81BD; border-bottom: none; border-left: solid white 1.0pt; border-right: solid white 3.0pt; border-top: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 168.45pt;" width="225"><div class="MsoNormal" style="margin: 2pt 0cm 0.0001pt; page-break-after: avoid;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Empowering
women and girls</span></b><span lang="EN-GB"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 78.45pt;" width="105"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">12</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 70.35pt;" width="94"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">3</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td colspan="2" nowrap="" style="background: #A7BFDE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 127; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white 1.0pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white 1.0pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.25pt;" width="87"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">3</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
</tr>
<tr style="height: 1.0cm; mso-yfti-irow: 3; mso-yfti-lastrow: yes;">
<td style="background: #4F81BD; border-right: solid white 3.0pt; border: solid white 1.0pt; height: 1.0cm; mso-background-themecolor: accent1; mso-border-bottom-alt: 1.0pt; mso-border-color-alt: white; mso-border-left-alt: 1.0pt; mso-border-right-alt: 3.0pt; mso-border-right-themecolor: background1; mso-border-style-alt: solid; mso-border-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: .75pt; padding: 0cm 5.4pt 0cm 5.4pt; width: 168.45pt;" width="225"><div class="MsoNormal" style="margin: 2pt 0cm 0.0001pt; page-break-after: avoid;">
<span style="font-family: Georgia, Times New Roman, serif;"><b><span style="color: white; mso-bidi-font-family: Arial; mso-fareast-font-family: "Times New Roman"; mso-themecolor: background1;">Providing
comprehensive services</span></b><span lang="EN-GB"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-alt: solid white .75pt; mso-border-bottom-alt: solid white 1.0pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 78.45pt;" width="105"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">16</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-alt: solid white .75pt; mso-border-bottom-alt: solid white 1.0pt; mso-border-bottom-themecolor: background1; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 70.35pt;" width="94"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">7</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
<td colspan="2" nowrap="" style="background: #D3DFEE; border-bottom: solid white 1.0pt; border-left: none; border-right: solid white 1.0pt; border-top: none; height: 1.0cm; mso-background-themecolor: accent1; mso-background-themetint: 63; mso-border-bottom-themecolor: background1; mso-border-left-alt: solid white .75pt; mso-border-left-themecolor: background1; mso-border-right-themecolor: background1; mso-border-top-alt: solid white .75pt; mso-border-top-themecolor: background1; padding: 0cm 5.4pt 0cm 5.4pt; width: 65.25pt;" width="87"><div align="center" class="MsoNormal" style="margin-bottom: 0.0001pt; page-break-after: avoid; text-align: center;">
<span style="font-family: Georgia, Times New Roman, serif;"><span style="font-size: 9pt;">7</span><span lang="EN-GB" style="font-size: 9pt;"><o:p></o:p></span></span></div>
</td>
</tr>
<!--[if !supportMisalignedColumns]-->
<tr height="0">
<td style="border: none;" width="225"></td>
<td style="border: none;" width="105"></td>
<td style="border: none;" width="94"></td>
<td style="border: none;" width="9"></td>
<td style="border: none;" width="78"></td>
</tr>
<!--[endif]-->
</tbody></table>
<div class="MsoNormal" style="margin-top: 10.0pt; page-break-after: avoid;">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">The evaluations deal with a broad
spectrum of interventions of varying complexity carried out by public and
not-for profit actors (including women’s rights organisations), ranging from a
single training project to multi-country programmes that bring together
different types of interventions. Most
evaluations found have occurred near or after the end of an intervention, a
smaller number are mid-term reviews.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-top: 10.0pt;">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">The reports vary in size (8-258 pages); their median length is 52 pages
(average length: 62). The degree to which they fulfil established quality
standards (with regard to the methodology employed, protection of VAWG
survivors and other aspects) is assessed in the first coding round. What can be
said at this point is that quality, understood in this way, appears to vary
significantly. This is also true for the appearance of the reports. <o:p></o:p></span></span></div>
<div class="MsoNormal" style="margin-top: 10.0pt;">
<span style="font-family: Georgia, 'Times New Roman', serif;">All published reports we have identified will be shared with DFID. 19
out of the 74 reports are unpublished or of uncertain publication status. We
cannot share these reports with others, but we have obtained permission to
extract data from these reports. It is important to keep them in the set of
evaluations to be reviewed, as this is an opportunity to work on material that
is not easily accessible to a wider public.</span><span style="font-family: Georgia, 'Times New Roman', serif;"> </span></div>
<div class="MsoNormal" style="margin-top: 10.0pt;">
<span style="font-family: Georgia, 'Times New Roman', serif;">F</span><span style="font-family: Georgia, 'Times New Roman', serif;">or those who would like to take a peek at the published evaluation
reports: the reports can be retrieved via <a href="https://www.dropbox.com/sh/bxlcsp4hqb84yl4/AAB684TJYkAbioFcTxOJKCkba?dl=0" target="_blank">this link</a>. The link
takes you to a folder which includes our full scoping report and a brief guide
to the folder. </span></div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-75189155278685024542014-01-04T06:54:00.002-08:002014-01-04T06:54:53.972-08:00Meet the coders<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;">We are delighted to announce that we have found highly qualified coders: Miruna Bucurescu, Scout Burghardt, Sanja Kruse, Astrid Matten and Paula Pustulka. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Astrid Matten holds a Master's degree in Political Sciences, Sociology and Social Politics from the University of Göttingen. Her research focuses on migration, state borders and gender issues.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">With a background in development studies, Miruna Bucurescu is a polyglot with expertise in women, peace and security. Since 2011 she has worked as an independent researcher focusing on social science statistics and evaluation.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Paula Pustulka is a sociologist, involved mostly in qualitative research projects in the fields of gender and migration studies. </span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Sanja Kruse is a sociology graduate with a wide range of practical experiences in research and international cooperation, particularly related to employment, education and gender issues.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">Scout Burghardt has a degree in Gender Studies and European Ethnology. She participated in the 2-year research project “Samenbanken – Samenspender” on reproductive technologies at Humboldt University Berlin.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<span style="font-family: Georgia, Times New Roman, serif;">For some reason, our provider refuses to upload the photographs that come with the text. You'll find them here as soon as it works again!</span><br />
<span lang="EN-GB"><span lang="EN-GB" style="font-family: "Georgia","serif"; font-size: 11.0pt; line-height: 115%; mso-ansi-language: EN-GB; mso-bidi-font-family: Arial; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin;"><br /></span></span>
<span lang="EN-GB"><span lang="EN-GB" style="font-family: "Georgia","serif"; font-size: 11.0pt; line-height: 115%; mso-ansi-language: EN-GB; mso-bidi-font-family: Arial; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin;"><br /></span></span>
<span lang="EN-GB"><br /></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><br /></span>
<br />
<div align="center">
<br /></div>
</div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0tag:blogger.com,1999:blog-7229983159999766307.post-4892363228042566562013-11-30T03:37:00.000-08:002013-11-30T03:46:41.631-08:00Our search for evaluations<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;"><span lang="EN-US">As
explained in our October 2013 post “<i>What
evaluations do we review?”</i> we have looked for evaluation reports in English
that meet certain criteria. </span><span lang="EN-GB">Our search started on 23
September 2013. We used a three-pronged strategy: (1) web-based search (“web
search”), (2) communication with contacts in the fields of VAWG and evaluation,
and snowballing via these contacts (“snowballing”) and (3) specialised
web-based networks (“DFID and helpdesk”). In addition to evaluation reports, we
identified meta-evaluations and specialised publications on “best practices” to
end VAWG, as well as literature on evaluation quality and effectiveness.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">For the <b>web search</b>, we used combinations of the terms “evaluation”,
“review”, “assessment”, “best practice” and 38 terms closely related to
violence against women and girls and work to end VAWG, such as “violence
against women”, “gender-based violence”, “forced sexual initiation”, “forced
marriage”, “human trafficking”, “masculinities”. The web search yielded many
duplicates and triplicates, i.e. it reached a high degree of saturation which
could be expected in view of the overlaps between these terms. <o:p></o:p></span></span><br />
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">However, focussing on the web search would
have yielded incomplete results. As shown in the figure below, 84 per cent of
the evaluation reports that we identified came from a single source, i.e.
either from web-search, direct communication with our networks or snowballing.</span></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlyevwlyShqU2xctyV8lsB_RHnhCrkMHbwn4wb68RLmaBqSXk9laaLOXhKpi4qxeC5e09DcxtM1spYYvGyS0NIY7abpKNNXKqlW6m4ZI3fK-_PCoBbeLEC9nAeVQi1v7SYaFs1G8YUe9A/s1600/saturation+of+scoping.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlyevwlyShqU2xctyV8lsB_RHnhCrkMHbwn4wb68RLmaBqSXk9laaLOXhKpi4qxeC5e09DcxtM1spYYvGyS0NIY7abpKNNXKqlW6m4ZI3fK-_PCoBbeLEC9nAeVQi1v7SYaFs1G8YUe9A/s1600/saturation+of+scoping.jpg" height="220" width="320" /></a><span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">This confirms that it was a good idea to
combine three search methods. We have been particularly impressed by the
effects of our <b>snowballing </b>action.
It was initiated through two channels – (i) direct contact to evaluation and
VAWG specialists known by DFID and the Review team, and (ii) publication of our
call for evaluation reports on the following list servers and social web sites:
Platform for Evidence-Based Learning list server (PELICAN), professional groups
on LinkedIn (AEA, Evaluations and Development), Michaela’s Facebook and Twitter
accounts, and our blogs <a href="http://www.developblog.org/">www.developblog.org</a>
and, after registration of the new domain, <a href="http://www.evawreview.de/">www.evawreview.de</a>. </span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span>
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;">Sending out e-mails yielded an excellent response. Examining the messages
received within three weeks from our call, we counted some 175 persons who had received
the call by e-mail (as direct addressees or in copy). The actual number is
probably higher, as we cannot assume that we have been copied into all e-mail
correspondence. The interest raised by social web posts was considerable: <a href="http://www.developblog.org/">www.developblog.org</a> (Michaela’s blog) registered
a peak of 270 page views on the day our call for evaluation proposals was
posted on the above-mentioned platforms (as compared to about 40-120/day in
“normal” times).<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span lang="EN-GB"><span style="font-family: Georgia, Times New Roman, serif;"><br /></span></span></div>
<div class="MsoNormal">
<span style="font-family: Georgia, Times New Roman, serif;">Soon we will describe what kinds of
evaluations we have found. Our draft Scoping Report is with the Review Reference Group; when we'll have the final version we'll provide a link to it on this blog</span>.</div>
EVAW Review Teamhttp://www.blogger.com/profile/10478062519964590910noreply@blogger.com0