Q&A on definitions

Some snag in our 'comments' function makes that comments are only shown if readers click on the titles of our posts. A couple of weeks ago, Rick Davies asked a few questions about our definitions:

Thanks again for another useful update on your work. Three questions, to help those of us following in your footsteps:

1. You have on aggregate Outcome (strong evaluation effect) made up of 4 types of effects. Why bother with the aggregate level, why not do 4 separate QCA analysis, one per specific effect?

2. Re the tree diagrams for each condition, how will you aggregate judgements up from the particulars (leaves) to the more general (root)?

3. Re the figures 2, 4, 5, 6, where are these explained. They look like cutoff points on the scales used to judge each condition, a fuzzy set value, but I may be way off.

And here are our answers. A warning: this is going to be quite technical!

1. We believe that to call an evaluation effective or not implies more than the evaluation having caused (or not) one particular type of effect. It is an overall judgment of the consequences an evaluation has had. 
We think that the three types of effects on active stakeholders are substitutive, i.e. having caused one type of effect is enough for the evaluation to be effective for active stakeholders. This is because we believe that different kinds of evaluations pursue different kinds of goals. If a formative evaluation achieves a high degree of action effects, to us, it is as effective as an impact evaluation achieving a high degree of persuasion or learning effects. 
Furthermore, we believe that one cannot call an evaluation effective if it has produced a negative effect on beneficiaries, no matter how strong its effects on active stakeholders. This is why we believe an evaluation should have caused at least one type of effects on active stakeholders and have had a positive effect on beneficiaries to merit the designation of being overall “effective”. 
Conducting QCAs for each type of effect and reporting the respective results would have sent the wrong message to our audience.

2. For favourable context, we assumed partial compensation of the three components. This is why we took the mean of the three values for the components as the value for the condition itself. 
For the three types of effects on active stakeholders, we assumed full substitutability. That’s why in this case we have taken the maximum value for the three components as the value for the outcome component “effect on active stakeholders”. 
All other components were aggregated by taking the minimum of the values for the components, including the aggregation of “effect on active stakeholders” and “empowerment effect”.

3. As we decided against explaining the aggregation rules to our readers, we wanted to indicate what the presence of a given condition means empirically with regard to the presence or absence of its components. Hence, the diagrams Rick refers to show the distribution of a given sub-dimension among the 39 evaluations, given the presence or absence of the condition they are part of.