This blog post is based on the Evidence & Policy article, ‘Does evaluation quality enhance evaluation use?‘
Pirmin Bundi, Kathrin Frey and Thomas Widmer
Evaluations provide important information to improve public services, but only if they yield valid and reliable findings – so we believed for a long time. Evaluation communities have therefore established certain criteria that should define evaluation quality. Yet against prior studies on evaluation utilisation, we show that evaluation quality measured by the criteria is not necessarily associated with evaluation use, but rather linked to the perception of quality and impact of the evaluation. Evaluators should adjust their communications strategies accordingly.
You wouldn’t eat partially baked bread, would you? Probably not, but what if the bread looks perfect from the outside? Baked golden brown to the perfection with a crispy crust? It happens to the best bakers that a bread already taken out of the oven is not yet completely cooked. Even though it looks tasty, it is still indigestible. Consequently, the quality of the bread’s production is central to its consumption.
Much like baking bread, the process of doing evaluation impacts on the quality of the evaluation. Many studies have made a connection between evaluation’s quality and use. Literature argues that high-quality evaluations not only generate more useful findings, but also better engage stakeholders so that they make superior use of the evaluation results. Yet hardly any studies have investigated whether this connection upholds empirically. Are better evaluations really used more often? Albeit a higher frequency does not mean that they have more impact, it is nevertheless an indicator of decision-makers’ engagements in evaluation.
The quality of an evaluation can be assessed in two distinct ways. On the one hand, we can simply ask stakeholders whether they perceive the evaluation to be of high quality. This ‘subjective’ evaluation quality is often grasped by the perception of the involved actors with the help of a survey or other interview techniques. On the other hand, evaluation communities have defined sets of criteria for assessing the quality of an evaluation – so-called evaluation standards. These standards define the ‘objective’ evaluation quality that includes the desired qualities an evaluation should possess, which is often identified with meta-evaluations.
Using the example of external school evaluations, we assessed both the ‘subjective’ and ‘objective’ evaluation quality in our recent Evidence & Policy article, ‘Does evaluation quality enhance evaluation use?’. While we asked the school representatives how they perceive the quality of external school evaluation, we also meta-evaluated the evaluation reports using a set of evaluation standards. Our findings show that the ‘subjective’ quality of an evaluation is not always congruent with its ‘objective’ quality. Moreover, the latter does not particularly influence the use of evaluation, while the perception of the evaluation quality correlates strongly with the perceived impact of an evaluation.
In short, evaluation quality does indeed enhance evaluation use, but only if the stakeholders perceive the evaluation to be of high quality. Moreover, we have identified a gap between subjective and objective evaluation quality, which is both unsatisfactory and worrisome. Evaluations might be used which contain inaccurate or even false information. In contrast to the unfinished bread, stakeholders do not realise whether an evaluation is of high quality and will use the results of an evaluation anyway. This might lead to misuse or even abuse. But what can evaluators do about it?
Evaluators of good quality evaluation should be more active in advising stakeholders when it comes to use. In doing so, they should not only focus on direct utilisation of evaluation. But above all, evaluators have it in their own hands to produce good evaluations. If this is not possible, they have at least the duty to inform their clients about the limitations of the evaluation. This is crucial, since there are significant risks of negative impacts and, unlike bread, these impacts are likely to affect many more people than the original consumers.
Pirmin Bundi is an Assistant Professor in the Institut de hautes études en administration publique (IDHEAP) at the University of Lausanne, Switzerland. Pirmin’s research focus is on policy evaluation, political elites and public administration.
Kathrin Frey is a Partner and Senior Consultant at KEK-CDC Consultants, Zurich, Switzerland.
Thomas Widmer is a Professor in the Department of Political Science at the University of Zurich, Switzerland. His research focus includes evaluation, policy analysis, Swiss politics, institutional change and methodology.
You can read the original research in Evidence & Policy:
Bundi, P. Frey, K. and Widmer, T. (2021). Does evaluation quality enhance evaluation use? Evidence & Policy, DOI: 10.1332/174426421X16141794148067.
Image credit: Photo by Jay Wennington on Unsplash
If you enjoyed this blog post, you may also be interested to read:
Evaluations that fail: nasty emails, small samples and tenuous futures
An experimental evaluation tool for the Public Innovation Lab of the Uruguayan government
‘I see a totally different picture now’: an evaluation of knowledge exchange in childcare practice
Evidence & Policy Blog 