How do policymakers perceive qualitative research?

Rebecca S. Natow

Qualitative research has the potential to be of great value in policymaking. By examining stakeholders’ lived experiences, providing rich detail about policy contexts, and offering nuanced insights about the processes through which programmes are implemented, qualitative research can supply useful information that is not easily, if at all, obtainable through surveys and other quantitative methods. However, policymakers consistently express a preference for quantitative research. This is particularly true for randomised controlled trials (RCTs), which have been called the ‘gold standard’ of evaluation methods.

Why do policymakers tend to favour quantitative research when qualitative research has so much to offer in policy debates? My study of how research has been used in the US government’s higher education rulemaking process, published in Evidence & Policy, provided the opportunity to examine this issue. Higher education rulemaking is the process through which the United States Department of Education creates regulations govern programmes under the Higher Education Act. Regulations created through the rulemaking process include rules on administration of the US government’s enormous and impactful student financial aid programme. The higher education rulemaking process provides an apt case study for examining research utilisation in policymaking for several reasons.

First, federal law requires agency regulations to have a ‘reasoned basis’, and research evidence can provide such a basis for decision-making in the rulemaking process. Second, regulators are required to conduct impact assessments and cost-benefit analysis when creating many higher education rules. This provides natural opportunities to rely upon research. Finally, higher education rulemaking often includes stakeholders in the process of developing proposed rules. Because research-oriented institutions such as universities are stakeholders of higher education regulations, the rulemaking processes has the potential to bring researchers directly into conversation with policymakers.

My study found there were concerns around both the credibility and the generalisability of qualitative data. One interviewee said that whether such research is considered in rulemaking depends on factors such as ‘who brought the data’ and whether that source is ‘trusted’. Quantitative research, on the other hand, was viewed as only ‘numbers’, and therefore considered ‘just data’.

Although studies have shown quantitative methods and even RCTs having their own biases, some policy actors viewed quantitative research as more credible than qualitative research. Participants also indicated the lack of generalisability of qualitative findings may render them less utilised by policymakers. Other participants in the rulemaking process expressed misunderstandings about what qualitative research is. For example, some conflated research with anecdotal evidence or individual testimony.

Despite the infrequent use of qualitative research in higher education rulemaking, some participants expressed respect for qualitative research and indicated it can be useful in policymaking. This was particularly true for qualitative research that produces detailed stories, such as narrative research or case studies that identify stories as illustrative examples of a phenomenon. As one interviewee said:

‘The stories, the compelling stories, are the ones that can sway people. Because numbers are hard and cold and they say one thing, but when you have a strong message from particularly students saying, “This is how it impacted me,” that really does speak to the issue for a lot of folks, and you have to balance that somehow with the numbers’.

In sum, my study found the reasons why qualitative research is used infrequently in rulemaking relate to policy actors’ concerns about the credibility and non-generalisability of qualitative findings and misconceptions of qualitative research being anecdotal. However, a number of participants in the rulemaking process expressed respect for qualitative methods and indicated such research may be particularly useful in policymaking when it produces compelling stories that can supplement the numeric findings of quantitative research.

By educating policymakers regarding the validity and value of qualitative research, and by identifying compelling stories in data and communicating those stories to policymakers, qualitative researchers can take steps toward enhancing the role of qualitative research in policymaking.

Rebecca S. Natow is Assistant Professor of Educational Leadership and Policy at Hofstra University. Her research focuses on higher education policy, federal policy processes in the United States, and higher education leadership.

You can read the original research in Evidence & Policy:

Natow, R.S. Policy actors’ perceptions of qualitative research in policymaking: the case of higher education rulemaking in the United States. Evidence & Policy, DOI: 10.1332/174426420X16047229138089.

Image credit: Photo by Agence Olloweb on Unsplash

If you enjoyed this blog post, you may also be interested to read:

Explaining variation in evidence-based policy making in the American states

How much evidence is in evidence-based policymaking: a case study of an expert group of the European Commission

Analysing use of evidence in public policymaking processes: a theory-grounded content analysis methodology

One thought on “How do policymakers perceive qualitative research?

  1. Ian Greener April 7, 2021 / 11:03 am

    I absolutely accept your main findings (the value of qual research, and its potential contributions), but also share the concerns of at least some policymakers about being sometimes sceptical about qual findings.
    I think there are (at least) three main issues here. The first is that a great deal of qual research is really poor in reporting method. The number of times research reports ‘We read the data and coded it thematically’ (without reporting how themes were derived) or ‘Data was coded in qualitative methods software’ (without any sense of exactly what that involved) is really concerning. Both qual and quant methods carry biases (as you rightly say), but it is crucial we get those out in the open and report (using both methods) exactly what was done, why and how. That simply doesn’t happen at the moment and I think qual research is generally poorer at this than quant.
    The second issue is around ‘reproduceability’. There is a big push in quant research around ‘reproducible research’. This doesn’t translate at neatly into qual work, but in qual work it is now routine to have to make data and, where it exists, computer coding, available in order for others to be able to see what was done with the data. Qual research is still a long way away from this. We need to talk about transparency in method, but also in practice to make clearer the rigour in qual research. How do we achieve greater transparency while also accepting that qual researchers are often dealing with more sensitive data?
    The third issue is around cherry-picking. Qual research is great – it allows us to tell stories. But often it isn’t clear if those stories are representative in any sense of the data, or the social groups we are reporting on. We need to think about how we can assure policymakers that our stories are important – not only for the individual involved – but in a wider sense. You raise this in terms of generalisability, but I do know qual researchers who simply look for ‘cool’ bits in their data and write those up. That leads to the kinds of suspicions I’ve already raised above.
    I absolutely accept your main points. But we also might want to reflect on why some policymakers are suspicious, and what more we can do to alleviate those fears.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s