Learning from failure: improving behavioural health treatments through understanding mis-implementation

Grace Hindmarch, Alex R. Dopp, Karen Chan Osilla, Lisa S. Meredith, Jennifer K. Manuel, Kirsten Becker, Lina Tarhuni, Michael Schoenbaum, Miriam Komaromy, Andrea Cassells and Katherine E. Watkins

This blog post is based on the Evidence & Policy article, Mis-implementation of evidence-based behavioural health practices in primary care: lessons from randomised trials in Federally Qualified Health Centers, part of the Special Issue: ‘Learning from Failures in Knowledge Exchange’.

“This is disappointing, but I agree we’ve done the best we can.” 

– CEO of a rural health care system

In October 2021, a rural healthcare system in the US discontinued implementation of a new program to improve access to quality care for patients with co-occurring opioid use disorder and mental health disorders. The program’s mission, fueled by passion for patients, was to help complex patients not fall through the cracks. After two years of immense effort, the system experienced ‘mis-implementation.’  Mis-implementation refers to unsuccessful efforts to implement treatments in real-world settings. Although it is a disappointing outcome, studying mis-implementation can provide insights to improve processes and make changes more successfully in the future.

Continue reading

Productive interactions without impact?

Magnus Gulbrandsen and Silje Maria Tellmann

This blog post is based on the Evidence & Policy article, Productive interactions without impact? An empirical investigation of researchers’ struggle to improve elderly’s oral health, part of the Special Issue: ‘Learning from Failures in Knowledge Exchange’.

Even if researchers do everything that is expected of them – collaborate with stakeholders, target important societal problems, engage in intensive science communication – societal impact may still not happen. What are the possible explanations?

A recurring observation in studies of the societal impacts of research is that substantial change typically involves a great deal of ‘productive interaction’ between stakeholders and researchers. However, not all interactions provide the desired societal impacts, as our empirical study of a cross-disciplinary research group focused on improving oral health in the elderly shows. In our Evidence & Policy article, we examined the subtleties of productive interactions and the intricate web of stakeholders, to shed light on the gaps that keep research efforts from having the desired societal impact.

We followed a group of researchers for six years, and even though they carried out many of the recommended activities to make impact happen, they were unable to achieve the expected outcomes. Even if many events took place that may – in an optimistic perspective – prepare the groundwork for future impact, no decisions in policy or practice targeting elderly’s oral health emerged. To analyse this process, we began by considering the oral health of the elderly as a problem area in which a wide range of stakeholders have a stake, but with varying interest, sense of urgency or capacity to make changes happen.

Continue reading

What can we learn from co-production approaches in voluntary sector evaluation work?


Louise Warwick-Booth, Ruth Cross and James Woodall

This blog post is based on the Evidence & Policy article, Obstacles to co-producing evaluation knowledge: power, control and voluntary sector dynamics’, part of the Special Issue: ‘Learning from Failures in Knowledge Exchange.

Co-production has been increasingly discussed as a positive and useful approach in health and social care research, based on principles such as partnership working, reciprocation, power sharing and the appreciation of all expertise. We have used co-production values to inform our evaluation work for many years, but in our Evidence & Policy article we reflect upon the challenges that such approaches bring, specifically in relation to sharing findings, known as knowledge exchange. Our article discusses evaluation work across three interventions that constitute perhaps the most challenging of our experiences in over a decade of such work. Conflict in evaluation work remains largely underreported, but we feel our experiences provide a useful contribution for readers.

Continue reading

Why failure isn’t the f-word in knowledge brokering


Stephen MacGregor

This blog post is based on the Evidence & Policy article, Theorising a spectrum of reasons for failure in knowledge brokering: a developmental evaluation’, part of the Special Issue: ‘Learning from Failures in Knowledge Exchange.

Failure often gets a bad rap, especially in professional settings. It’s usually seen as a waste of time and resources, something to steer clear of. But failure is not just an unfortunate outcome; it can be a crucial learning opportunity.

Particularly in higher education, the pressure is on for academics and universities to show the real-world impact of research. Here, knowledge brokers play a critical role: they are the human force behind efforts to connect research production and use contexts. Yet, the challenges and failures that these professionals face are not often discussed.

My recent Evidence & Policy article aimed to shed light on the spectrum of reasons for failure in the professional practice of knowledge brokering, drawing on a set of semi-structured interviews with a network of knowledge brokers. To understand knowledge brokers’ experiences, two frameworks were integrated: (a) the integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework, and (b) Dr. Amy Edmondson’s Spectrum of Reasons for Failure framework.

Continue reading

Learning from failures in knowledge exchange: how hard can it be?


Peter van der Graaf, Ien van de Goor and Amanda Drake Purington

This blog post is based on the Evidence & Policy article, ‘Learning from failures in knowledge exchange and turning them into successes, which introduces the Special Issue: ‘Learning from Failures in Knowledge Exchange.

We don’t like talking about failures, as it signals loss of time, resources and reputation, but failures present opportunities for learning in knowledge exchange. However, this requires a ‘failure culture’ in academia and policy, in which failures are no longer avoided but actively encouraged. To learn how to turn failures into successes, we need to share and publish our failures, have early engagement with all stakeholders in the knowledge exchange process, and make more use of boundary spanners.

There are plenty of papers celebrating successes in knowledge exchange, but not many researchers and policy makers talk openly about their failures. However, learning from failures is just as important, if not more crucial, than celebrating successes. Allowing partners to reflect in a safe space on knowledge exchange practices and research projects gone wrong, in which communication broke down, partners did not engage or dropped out, and evidence was not taken up or ignored, will provide important lessons on how knowledge exchange practices and research can be improved.

At the 5th Fuse conference on knowledge exchange in public health, held in Newcastle, UK on 15-16 June 2022, we created such a space by bringing together over 100 academic researchers, policy makers, practitioners, and community members to share and reflect on thier failures and how to turn them into success. Our special issue brings together selected papers from the conference and papers that were submitted in response to an open call afterwards. From 23 original submissions from 14 different countries (including the UK, USA, Cananda, Norway, Switzerland, Kenya, Chile, South Korea, Canada and Portugal) and from a range of disciplines and areas of focus (Public Health, Primary Care, Oral Health, Sociology, Anthropology, Public Management, Policy-Making, and Community and Voluntary Sector), we invited four research papers and three practice papers for full submissions.

Continue reading

Embracing creativity in co-production using the arts

Stephen MacGregor, Amanda Cooper, Michelle Searle and Tiina Kukkonen

This blog post is based on the Evidence & Policy article, ‘Co-production and arts-informed inquiry as creative power for knowledge mobilisation’, part of the Special Issue on Creativity and Co-production.

The days of research reports going unread by all but their authors and articles being hidden behind publisher paywalls are giving way to more collaborative research approaches. One that has provoked great attention in recent years is co-production, an approach that acknowledges the unique knowledge and expertise different individuals can bring to the research process. However, the evidence base for co-production has not kept pace with the excitement surrounding it.

In our recent Evidence & Policy article, we asked, ‘How can seeing co-production as a creative endeavour create opportunities to move knowledge into action?’ To answer this question, we examined three cases focused on promoting shared understanding and action in the Canadian education sector. Each case used artful practices to promote meaningful reflection, understanding and representation of individual and communal experiences.

Unique to our study was the use of a realist perspective. Realist explanations look to develop reasoned pathways from specific mechanisms and contexts to observed outcomes (see Pawson & Tilley, 1997). Researchers typically refer to these as CMO configurations and represent the expression as: context + mechanism = outcome. These explanations are helpful because we can learn about the possibility of transferring lessons learned from one instance of co-production to another. What’s more, by comparing these CMO configurations across our three cases, we can identify common propositions about arts-informed approaches to co-production.

Continue reading

Engaging refugee women and girls as safeguarding experts, using creative and participatory methods

Alina Potts

This blog post is based on the Evidence & Policy article, Engaging refugee women and girls as experts: co-creating evidence on sexual exploitation and abuse in humanitarian crises using creative, participatory methods’, part of the Special Issue on Creativity and Co-production.

Efforts to build evidence on how best to deliver humanitarian assistance have grown over the past decade, at a time when the number of forcibly displaced people in the world has risen to over 84 million. Yet crisis-affected people are often left out of shaping the questions asked, and participating in answering them. Creative, participatory research methods can break down these silos and enable the co-production of evidence with displaced populations, and its uptake for practice and policy. The ‘Empowered Aid‘ study engages in participatory action research with refugee women and girls in Uganda and Lebanon to examine how to better prevent sexual exploitation and abuse (SEA) in aid delivery. Co-producing knowledge about violence with those most affected by it creates actionable evidence to reduce risks.

In humanitarian settings, pre-existing power imbalances due to gender, age, and other factors can be exacerbated. While women and children account for a large share of the displaced, they are often left out of decision making, despite the impact aid delivery has on their lives and their heightened risk of gender-based violence, including sexual exploitation and abuse (SEA). While a system of reporting and response has been put in place over the last two decades, many survivors are discouraged from using it due to a lack of access, information, and trust in the process or the organisations leading it. Accountability mechanisms have also focused on responding to abuses already perpetrated, rather than working to prevent them.

Continue reading

I’m not in the ‘too hard basket’ – I wove my kete to create my success

Katey Thom, Stella Black, David Burnside and Jessica Hastings

This blog post is based on the Evidence & Policy article, ‘He Ture Kia Tika/Let the Law Be Right: informing evidence-based policy through kaupapa Māori and co-production of lived experience’, part of the Special Issue on Creativity and Co-production.

A kaupapa Māori co-production project that honours the voices of those with lived experience of incarceration

Research often tells us most prisoners have experienced mental distress or addiction within their lifetime but often end up in the ‘too hard basket’. The criminal justice system of Aotearoa, with its strong Westminster roots, significantly contributes to intergenerational traumatic experiences and struggles to get help that is needed. Our project aimed to reject this basket, replacing it with a diverse array of kete (baskets) filled with localised mātauranga (ancient knowledge derived from a te ao Māori worldview), strategies and solutions to improve wellbeing and reduce reoffending.

Continue reading

Harnessing creativity in participatory research – the tension between process and product

Louise Phillips, Maria Bee Christensen-Strynø and Lisbeth Frølunde

This blog post is based on the Evidence & Policy article, ‘Arts-based co-production in participatory research: harnessing creativity in the tension between process and product’, part of the Special Issue on Creativity and Co-production.

In participatory research, researchers share the ideal of democratising knowledge production, on the basis of an expanded understanding of what counts as knowledge and whose knowledge counts. People with knowledge based on their own lived experience take part as co-researchers in processes of co-producing knowledge together with academic researchers. This process of harnessing the knowledge of people with lived experience can make a valuable contribution to the transformation of health and social care practice, as well as to the research field.

Arts-based research methods are often used to draw out the personal knowledge of co-researchers, including the emotional and aesthetic dimensions. But the use of arts-based co-production in participatory research does not easily get rid of the difficulties of putting the principles into practice – due to the tensions that arise between cultivating the collaborative, creative process and generating specific research results.

Continue reading

A new way for system design: building a relational infrastructure

Mandy D. Owens, Sally Ngo, Sue Grinnell, Dana Pearlman, Betty Bekemeier and Sarah Cusworth

This blog post is based on the Evidence & Policy article, ‘Co-producing evidence-informed criminal legal re-entry policy with the community: an application of policy codesign’, part of the Special Issue on Creativity and Co-production.

Health service researchers are plagued by the fear that policy and system-level improvement efforts will ignore or under-utilize research. Consequently, efforts at system improvement that come out of research centers tend to use “research-first” approaches that include protocols, trainings, and coaching sessions around evidence-based programs. But oftentimes the issue is not that a system is unaware of the research, it is uncertainty about how to get something going that fits the local context. This has as much or more to do with local values, personalities, and working relationships as it does with the specifics of a protocol.

Our study finds that engaging a community in a policy codesign process that prioritizes mutual learning, rather than a protocol, not only yielded a high-quality plan but built the relational infrastructure for local collaboration long after the external design facilitators left.

Continue reading