Assessing Individual & Organizational Use of Climate Adaptation Science

An article, Development of a survey instrument to assess individual and organizational use of climate adaptation science, authored by Assistant Tribal Climate Science Liaison, Steph Courtney, was published in the journal Environmental Science & Policy in November 2022. SE CASC Consortium PI Paul Armsworth (U. Tennessee-Knoxville) and Consortium Co-PI Karen McNeal (Auburn University) are co-authors to the article along with Amanda Hyman (U. Tennessee-Knoxville) and Lindsay Maudlin (Iowa State University). The publication is the product of the SE CASC project, Best Practices for Project Design: Effectively Addressing Natural Resource Management Needs. The following summary was prepared by lead author, Steph Courtney.

Employees of public and private organizations, including various government bodies, are tasked with managing natural and cultural resources in ways that will minimize the negative impacts of climate change (i.e., climate change adaptation). Those management decisions will be more effective if they are based on the best available science, which is why the CASCs (among other agencies) seek to provide actionable science to these partners. However, understanding whether this science is truly actionable — that is, whether it has been used by partners as intended — is difficult to assess.

In this study, we wanted to create a quantitative tool, specifically a survey, to measure the use of climate adaptation science by identified partners. Developing a valid survey requires a deliberate and thorough process with frequent input from the intended audience, and we followed suggestions from previous research to do so. Our primary motivation for creating the survey was to distribute it to partners and use those results as part of a larger project. In the larger project, we collected data about many aspects of climate adaptation research projects and analyzed them to understand which aspects are most associated with different outcomes.

The survey was also informed by existing literature about evaluation and the use of knowledge, including other studies of the impacts of climate adaptation science. Previous research suggests that there are at least three distinct ways to use climate adaptation science (conceptual, instrumental, and justification), but this topic has not been researched in depth, and understanding different types of use will help us to measure it better. To address this question, we also analyzed the results of the survey using factor analysis, to see if the data showed patterns that would suggest there were distinct types of use present.

We defined conceptual use as “information [that] was reported to enhance knowledge base or to inform process or planning,” instrumental use as “information [that] was reported to influence decision-making directly with respect to action, process, or plan,” and justification use as “information [that] was reported to justify an action, process, or plan.”

Our development process resulted in a survey that contained 22 questions that we thought would describe the three types of use described by previous researchers. Instead, we only found evidence of two distinct types of use, i.e. two distinct response patterns among the questions in the factor analysis. We used the factor analysis to remove questions that were redundant or did not clearly fit into one type of use. We also used information gathered in interviews with partners to understand what the types of use might represent or describe. From our study, we concluded that the survey revealed individual use (which may be related to conceptual and justification use) and organizational use (may be related to instrumental use).

We hope that others will use the survey to measure use of climate adaptation science by their partners and collaborators. This is important for the continued assessment of the survey too, since our study did not have a very large sample size, and our results may not be the same as future applications of the survey with different contexts or participants. If you are interested in using this survey, you can read the full text of our manuscript, and you may reach out to Steph Courtney (stephcourtneyphd@gmail.com) with any thoughts or questions or for guidance in implementing the tool.

Note to author: An abstract of the paper follows.

Steph L. Courtney, Amanda A. Hyman, Karen S. McNeal, Lindsay C. Maudlin, Paul R. Armsworth,
Development of a survey instrument to assess individual and organizational use of climate adaptation science, Environmental Science & Policy, Volume 137, 2022, Pages 271-279, ISSN 1462-9011, https://doi.org/10.1016/j.envsci.2022.08.023. (https://www.sciencedirect.com/science/article/pii/S1462901122002738)

Abstract: Research that can improve the resilience of social and natural systems to climate change has become more common. Many climate adaptation science organizations and agencies now focus on actionable science, a model that aims to have greater impacts on policy and practice than traditionally produced and distributed science. However, evaluations of research projects are needed to examine and verify the impact of climate science on adaptation and society. Better understanding the types and mechanisms of impact will allow organizations to design, fund, and facilitate more useful climate adaptation science. Many existing actionable science evaluation approaches are qualitative in nature and take considerable time and effort for funders and administrators to implement. Quantitative methods could provide a valuable option for evaluation, specifically for making comparisons across many projects. Thus, we have designed a quantitative survey instrument for measuring the use of climate adaptation science. We designed the survey using best practices and iterative input from social scientists as well as climate adaptation scientists and practitioners. We then distributed the survey to a sample of users of climate adaptation science and analyzed those responses to further refine the survey. Quantitative and qualitative results show that use of climate adaptation science may be described as either individual use or organizational use, which contrasts with popular models of use in existing evaluation literature. The survey is made available for future efforts to evaluate and improve climate adaptation science and to advance efforts to measure different kinds of use.
Keywords: Climate change; Climate adaptation science; Science evaluation; Survey development