Rebecca S. Natow
Qualitative research has the potential to be of great value in policymaking. By examining stakeholders’ lived experiences, providing rich detail about policy contexts, and offering nuanced insights about the processes through which programmes are implemented, qualitative research can supply useful information that is not easily, if at all, obtainable through surveys and other quantitative methods. However, policymakers consistently express a preference for quantitative research. This is particularly true for randomised controlled trials (RCTs), which have been called the ‘gold standard’ of evaluation methods.
Why do policymakers tend to favour quantitative research when qualitative research has so much to offer in policy debates? My study of how research has been used in the US government’s higher education rulemaking process, published in Evidence & Policy, provided the opportunity to examine this issue. Higher education rulemaking is the process through which the United States Department of Education creates regulations govern programmes under the Higher Education Act. Regulations created through the rulemaking process include rules on administration of the US government’s enormous and impactful student financial aid programme. The higher education rulemaking process provides an apt case study for examining research utilisation in policymaking for several reasons.
First, federal law requires agency regulations to have a ‘reasoned basis’, and research evidence can provide such a basis for decision-making in the rulemaking process. Second, regulators are required to conduct impact assessments and cost-benefit analysis when creating many higher education rules. This provides natural opportunities to rely upon research. Finally, higher education rulemaking often includes stakeholders in the process of developing proposed rules. Because research-oriented institutions such as universities are stakeholders of higher education regulations, the rulemaking processes has the potential to bring researchers directly into conversation with policymakers.
My study found there were concerns around both the credibility and the generalisability of qualitative data. One interviewee said that whether such research is considered in rulemaking depends on factors such as ‘who brought the data’ and whether that source is ‘trusted’. Quantitative research, on the other hand, was viewed as only ‘numbers’, and therefore considered ‘just data’.
Although studies have shown quantitative methods and even RCTs having their own biases, some policy actors viewed quantitative research as more credible than qualitative research. Participants also indicated the lack of generalisability of qualitative findings may render them less utilised by policymakers. Other participants in the rulemaking process expressed misunderstandings about what qualitative research is. For example, some conflated research with anecdotal evidence or individual testimony.
Despite the infrequent use of qualitative research in higher education rulemaking, some participants expressed respect for qualitative research and indicated it can be useful in policymaking. This was particularly true for qualitative research that produces detailed stories, such as narrative research or case studies that identify stories as illustrative examples of a phenomenon. As one interviewee said:
‘The stories, the compelling stories, are the ones that can sway people. Because numbers are hard and cold and they say one thing, but when you have a strong message from particularly students saying, “This is how it impacted me,” that really does speak to the issue for a lot of folks, and you have to balance that somehow with the numbers’.
In sum, my study found the reasons why qualitative research is used infrequently in rulemaking relate to policy actors’ concerns about the credibility and non-generalisability of qualitative findings and misconceptions of qualitative research being anecdotal. However, a number of participants in the rulemaking process expressed respect for qualitative methods and indicated such research may be particularly useful in policymaking when it produces compelling stories that can supplement the numeric findings of quantitative research.
By educating policymakers regarding the validity and value of qualitative research, and by identifying compelling stories in data and communicating those stories to policymakers, qualitative researchers can take steps toward enhancing the role of qualitative research in policymaking.
Rebecca S. Natow is Assistant Professor of Educational Leadership and Policy at Hofstra University. Her research focuses on higher education policy, federal policy processes in the United States, and higher education leadership.
You can read the original research in Evidence & Policy:
If you enjoyed this blog post, you may also be interested to read: