Elizabeth Day
This blog post is based on the Evidence & Policy article, ‘How legislators define research evidence’.
When people ask about my research area, I answer that I study how policymakers use research evidence. Their response always follows a similar thread: ‘That sounds hard’ and ‘Ha! Do they even know what research is?’ These reactions align with a broader opinion in the United States that elected officials are clueless when it comes to using research evidence in the decision-making process.
Yet there are plenty of examples in research, legislation, and regulations where policymakers do use research in their work. My colleague Karen Bogenschneider and I wondered if this mismatch – assuming policymakers don’t use research when there are examples that they do – might have to do with a jingle-jangle problem: Do researchers and legislators actually mean the same thing when they say ‘research evidence’?
We wanted to dig into this question by getting right to the source, so in this Evidence and Policy article we asked legislators directly, ‘How do you define research evidence?’ We interviewed legislators from Indiana and Wisconsin who had been nominated by their colleagues as exemplary users of research in policy.
Overall, legislator definitions largely aligned with what researchers themselves might describe as ‘research evidence’. Legislators mentioned peer-reviewed studies, which they described as, ‘kind of like the gold standard for whether or not something is considered good research’. Legislators also mentioned the scientific method and qualities that researchers highly value: lack of bias, credibility, sound statistical analysis. As an Indiana legislator described, ‘I’d get back into my old basic science: you have a theory or hypothesis and then try to prove it or not prove it or disprove it’.
Legislators also differentiated research from ‘data’. They used words such as data, statistics, and ‘fact finding’ to describe the process of compiling information. This was different from research, which they described as a more in-depth exploration of why things are happening or how things may be related. As one Wisconsin legislator summarized, ‘Research is what provides meat to the bones of the data’.
An Indiana legislator explained that a lack of access to research might be why his colleagues rely more on data than research:
‘People use the word research to mean a lot of different things. And they say, “Well, the research says”, and it’s like that’s information, that’s not research… For what I call our high-stakes policies, it’s very difficult to get your hands on research, quite honestly’.
Legislators also mentioned different methods and study designs in their definitions. They included quantitative or qualitative studies, as well as experiments, quasi-experimental designs, and anecdotes. Legislators described how ‘stories make a big difference’ in the policy world. As an Indiana legislator described, ‘I think anecdotes can be useful in illustrating research or explaining a problem or an issue…They have a role to play and should be included in some broad sense in research, but they’re not primary’.
So, after talking with legislators, do we think researchers and legislators mean the same thing when they say ‘research evidence’? We think yes, most of the time they do. Legislators may differ from researchers by including anecdotes in their definition, but legislators also clarified that stories are just one piece of a larger puzzle of information. Their broader definition included peer-reviewed studies, the scientific method, sound statistical analysis, experiments, quantitative and qualitative studies, and a lack of bias. Researchers can better engage policy audiences by referring to these qualities when discussing their research. Policymakers prefer evidence from rigorous studies to those that are poorly executed or politically motivated, which legislators called ‘bogus’ research, ‘party research’ or ‘pseudoscience’.
Image credit: Photo by Chad Morehead on Unsplash
Elizabeth Day is a Research Assistant Professor with the HEDCO Institute for Evidence-Based Educational Practice in the College of Education at the University of Oregon. Her research focuses on bridging research, practice and policy, with a particular focus on child and adolescent well-being and social policy at the state level.
Read the original research in Evidence & Policy:
Bogenschneider, K. and Day, E. (2023). How legislators define research evidence. Evidence & Policy, DOI: 10.1332/17442648Y2023D000000012.
If you enjoyed this blog post, you may also be interested in reading:
The extent of use of surveys in policymaking: the case of Hong Kong
Use of research evidence in legislatures: a systematic review OPEN ACCESS
Examining research systems and models for local government: a systematic review
Disclaimer: The views and opinions expressed on this blog site are solely those of the original blog post authors and other contributors. These views and opinions do not necessarily represent those of the Policy Press and/or any/all contributors to this site.
Evidence & Policy Blog 