ScholarWorks ScholarWorks

Background: Efforts to achieve improved student outcomes in STEM are critically reliant on the success of reform efforts associated with teaching and learning. Reform efforts include the transformation of course-based practices, community values, and the institutional policies and structures associated with teaching and learning in higher education. Enacting change is a complex process that can be guided by change theories that describe how and why a desired change takes place. We analyzed the utility of a theory-based change model applied in a higher education setting. Our results provide guidance for change efforts at other institutions. Results: Use of the CACAO model to guide the transformation of STEM instruction at a large public university resulted in changes to faculty teaching practices and department culture consistent with the vision defined for the project. Such changes varied across STEM departments in accordance with the emergent nature of project activities at the department level. Our application of the CACAO model demonstrates the importance of (1) creating a vision statement (statement of desired change or end-state); (2) attending to different levels of the organization (e.g., individuals, departments, and colleges); (3) working with change agents who are situated to be effective at different organizational levels; and (4) employing strategies to meet the needs and interests of faculty at different stages of adoption with respect to the desired change. Conclusion: Our work, which demonstrates the utility of the CACAO model for change and captures its key elements in a matrix, provides a potential foundation for others considering how to frame and study change efforts. It reinforces the value of using change theories to inform change efforts and creates a structure that others can build on and modify, either by applying our CACAO matrix in their own setting or by using the matrix to identify elements that connect to other change theories. We contribute to the growing body of literature which seeks to understand how change theories can be useful and generalizable beyond a single project.


Introduction
Education reform in the STEM disciplines is necessary both to improve student learning and to achieve equitable outcomes for students (Committee on STEM Education, 2018; Harris et al., 2020;Kober, 2015;Seymour & Hunter, 2019). A significant focus of work in this area is on shifting faculty pedagogical practice, supporting the move from lecture-based delivery to evidence-based instructional practices (EBIPs) (Freeman et al., 2014;Stains et al., 2018;Theobald et al., 2020). While supporting individual faculty growth is essential, focusing solely on individual teaching practice may actually contribute to the lack of widespread use of EBIPs (Walter et al., 2021). Attention to organizational context and the systems in which faculty work is necessary for change to be successful and sustained (Elrod & Kezar, 2015;Lund & Stains, 2015;Stupnisky et al., 2018;Walter et al., 2021). For example, Walter et al. (2021) suggest that leadership at the department level is a key variable in catalyzing Page 2 of 20 Viskupic et al. International Journal of STEM Education (2022) 9:6 instructional change, as well as institutional support and guidance in executing a change.
The process of change in higher education is as complex as the pedagogies and learning processes that are being promoted (Borrego & Henderson, 2014). With few examples of wide-spread departmental or institutional adoption of EBIPs to draw upon, practical, operational models of change are needed to guide large-scale reform efforts (Austin, 2011;Owens et al., 2018;Reinholz & Apkarian, 2018). Models of change, based on scholarly knowledge of how change happens, are needed to allow efforts to focus on factors most likely to yield success (Reinholz et al., 2021). Further, "linking change efforts to existing theory ensures that new initiatives are informed by and built on prior efforts" (Borrego & Henderson, 2014, p. 222). It is important to identify effective strategies for promoting educational reform that can be generalized and applied in a variety of environments. Therefore, we share our experience with a theory-based model to support institutional changes to the teaching and learning environment on our campus. This manuscript addresses the need to demonstrate the use of "a theory as a lens or guide that directly informs specific components" of STEM education reform efforts (Reinholz et al., 2021;p. 17). We probe the extent to which the model was effective in both shaping the actions of a change project and in bringing about the desired change. In doing so, we help to frame a specific theory-based organizational change model for use in higher-education settings that can be used by campus change leaders, faculty developers, and higher education and discipline-based education researchers. Dormant's (2011) CACAO model for supporting and increasing the effectiveness of organizational change efforts was developed for and applied to business contexts. The model incorporates theoretical underpinnings about how and why change does or does not occur, and helps practitioners "understand and plan for the key dimensions of a change process" (Dormant, 2011, p. 10). The model identifies four elements important to any change process: the Change, the Adopters, the Change Agents, and the Organization (CACAO). Here we describe the four elements of the model and their connections to other ideas used to inform STEM Education change. We then describe the elements of the CACAO model in the context of our specific higher education reform project.

The change
The first element of the CACAO model calls for "specifying the change. " It requires that change leaders thoroughly understand the change they seek, including its strengths and weaknesses, and explicitly consider the change from the adopters' perspective. When adopters perceive negative attributes associated with change characteristics, they will be less likely to adopt that particular change. Therefore, change agents can build in support or actions to mitigate perceived negativities in order to increase the chances of adoption.
Characteristics of the change described by the CACAO model include its relative advantage, simplicity, compatibility, adaptability, and social impact, all from the adopters' point of view. For example, is the change better than alternative options (relative advantage) and easy to understand (simplicity)? Is the change similar to current practices (compatibility) and also easy to modify (adaptability)? Does the change promote positive responses and social interactions (social impact)? These characteristics have been modified from those outlined originally in Roger's Diffusion of Innovation (Rogers, 2003). The CACAO model's focus on "the change" is similar to the emphasis in other models on a clear vision. For example, an important element of the Keck/PKAL change model "encourage[s] a vision that is clear, shared, and aligned with institutional priorities" (Elrod & Kezar, 2017).

The adopters and adoption stages
The CACAO model emphasizes that those seeking change must know their adopters well and understand that adopters move toward the change through "evolving stages. " Dormant's (2011) stages of adoption-awareness, curiosity, mental try-out, hands-on try-out, and adoption-are based on the research of Lewin (1951), Rogers (2003), and the Concerns-Based Adoption Model (Hall & Hord, 2010). Adopters move from a stage of passivity to one in which the adopter has implemented the change but may still need assistance sustaining the change in their context. This focus on the adoption process has been widely considered in the higher education setting. Individual faculty making changes to their teaching are likely to move through phases of adoption (Andrews & Lemons, 2015;Henderson et al., 2012;Lund & Stains, 2015;Marbach-Ad & Rietschel, 2016). This process may start with simple awareness that a variety of evidencebased strategies exist and, ideally, progress to a rich practice in which pedagogy is built on scholarly findings and informed by assessment of one's own teaching practice. Such progression is, however, not simple, nor inevitable. Some faculty begin to explore EBIPs and then revert to their previous teaching methods (Henderson et al., 2012). Changing from teacher-centered to student-centered instruction is challenging and the factors that influence these faculty choices are complex (Andrews & Lemons, Page 3 of 20 Viskupic et al. International Journal of STEM Education (2022) 9:6 2015; Bathgate et al., 2019;Lane et al., 2019;Lund & Stains, 2015;Stupnisky et al., 2018). Importantly, the CACAO model makes clear that those who seek to create change must intentionally attend to the needs of adopters at each stage of the process. Adopters' needs and the strategies designed to address them will differ depending on the stage of the individual adopter. Dormant (2011) draws from Rogers' (2003 innovation-decision process to identify the following needs of adopters at different stages: Adopters in the awareness stage benefit from advertisement of the desired change that promotes a positive vision; Adopters in the curiosity stage benefit from information and reassurance that address their questions and concerns; Adopters in the mental tryout stage benefit from demonstrations of the change in action that shows successful use among their peers; Adopters in the hands-on tryout stage benefit from training; and Adopters in the adoption stage benefit from technical support and recognition of their efforts. Dormant (2011) advocates for the formation of a change implementation team, as well as the engagement of organizational leadership and other critical actors who can influence the change process. Change agents are considered those who define, stimulate, facilitate, and coordinate change activities (Lunenburg, 2010). The team of change agents responsible for facilitating the change should represent a range of organizational expertise and include relevant stakeholders (Dormant, 2011). Drawing from Kotter and Posner (1990), Dormant indicates change agents must both manage and lead change which includes setting the direction and communicating the change to adopters. The change agent team needs to possess a broad range of skills, expertise, and personal attributes, such as empathy and credibility, as they must influence the beliefs, intentions and practices of the target adopters (Armenakis et al., 1993;Dormant, 2011). In higher education, engaging faculty embedded in departments as change agents has shown potential in affecting change due to their accessibility to their colleagues, their knowledge of the department context, and pre-existing relationships and trust (Andrews et al., 2017).

Organizational culture and structure
The CACAO model emphasizes the importance of considering the structure of the organization in choosing strategies and in planning for sustainability. The organizational context can influence the extent to which a change is successfully adopted among its members, and some features of the organization might need modification to support and sustain the change over time. The organizational context becomes even more important when the change sought is a cultural change. Dormant (2011) draws from Schein's (1999 work focusing on indicators of the existing culture: artifacts, espoused values, and shared assumptions. Cultural change requires broadbased organizational support and takes years to achieve (Dormant, 2011). Whether or not an adopter engages in the change process is influenced by factors such as the adopter's personal beliefs, organizational culture and external factors (Austin, 2011;Reinholz & Andrews, 2020). In a university setting, the institutional, college, and departmental cultures and structures influence individual and collective behaviors through the transmission of information from members of the organization (Grunspan et al., 2018;Kezar, 2014;Reinholz & Apkarian, 2018); the transmission of information influences practices and beliefs of others in the organization leading to social norms (Grunspan et al., 2018).

The CACAO model as applied at Boise State University
At Boise State University we used the CACAO model (Dormant, 2011) and its accompanying processes to guide a STEM education reform project. In this project, the Change is the state of transformed teaching and learning that we aspired to achieve, the Adopters are the faculty who must alter their practices to achieve the Change, the Change Agents are players engaged with helping to promote or facilitate the Change, and the Organization includes the institution and organizational subunits, such as colleges and departments. To our knowledge, our project was the first large-scale project to adapt the CACAO model for use in higher education. Here we present the key components of the model as they were applied in our context.
The CACAO model guided the project leadership team (PLT) to articulate a project vision statement (the Change) and to spend considerable time and effort communicating with adopters about the vision for the project (Earl et al., 2020;Landrum et al., 2017;Shadle et al., 2017). Project efforts were oriented toward an aspirational vision in which the culture of teaching and learning at Boise State University would be characterized by: On-going exploration and adoption of evidence-based instructional practices Faculty engaged in continuous improvement of teaching and learning Dialogue around teaching supported through a community of practice Teaching evidenced and informed by meaningful assessment The adopters in this project were all STEM faculty (hereafter, faculty). We focused attention on faculty Page 4 of 20 Viskupic et al. International Journal of STEM Education (2022) 9:6 knowledge and use of evidence-based instructional practices (EBIPs), and we sought to provide resources and strategies for faculty at all stages of adoption, promoting the movement of all faculty toward achievement of the vision statement. Two groups of change agents were active. The first group of change agents, the project leadership team (PLT), included the director of the institution's Center for Teaching and Learning, the deans of the College of Engineering and the College of Arts and Sciences, a project manager, and four STEM faculty with expertise and leadership in STEM education, faculty development, and teaching reform. The PLT members acted individually and collectively as project change agents, and had the resources and positional power to affect change at the institutional level. Additionally, the PLT identified potential department change agents through discussions with deans and department chairs. At least one of these faculty members from each STEM department was invited by the PLT to serve as part of a Faculty Advocates for STEM Transformation team (FAST team). Individuals on the FAST team were envisioned as department-level change agents, and were important to the change process because, compared to the change agents serving on the PLT, they had a better understanding of their department culture and had established relationships with their colleagues. FAST team members were department insiders and could facilitate changes "with" their colleagues rather than "to" (Dormant, 2011).

Research questions
To explore how the CACAO change model was applied to support institutional change in higher education reform, we conducted an analysis of the activities of our STEM education reform project relative to the structures of the CACAO model and measured the impact of those activities. We address the following research questions: • How can the CACAO model be adapted for a project focused on institutional STEM education reform? • What evidence is there that the CACAO framework successfully supported STEM education reform?
The second question is important because understanding how the model functions is only valuable if the model can successfully support the desired change. After the presentation of our results, we discuss our use of the CACAO model with respect to change theories in STEM educational reform.

Mapping project activities
Project activities were identified by document content analysis of annual reports submitted to the National Science Foundation (NSF). Two researchers simultaneously reviewed each of the annual reports and the final project report to identify specific activities that were implemented as part of the change project. Annual reports were used to systematically identify key project activities reported during the project period in order to reduce bias as the researchers played an active role in the project; all researchers were members of the PLT. Activities reported to the NSF that were related to the administration of the grant and dissemination efforts were excluded from the analysis. Because these reports were often a broad representation of project activities, more detailed information about salient strategies was identified through the analysis of proposals and status reports from Partner Projects (described below) as well as regular reports and action plans from FAST Team members.
Once all of the reported activities were identified, the researchers used the following questions to identify which components of the CACAO model were being enacted: 1) What stages of faculty adoption were supported by the activity? 2) Who were the project change agents carrying out the activity? 3) What level of the organization was impacted by the activity? We used the intersections of these three questions to develop a representation of the strategies framed by the CACAO model.

Understanding the CACAO model's impact
We assessed changes to individual and departmental practices during the project period by analyzing faculty responses to a questionnaire. Boise State University's Institutional Review Board reviewed and approved all research protocols and consent was obtained from all participants.

Questionnaire
At the start of the fifth and final year of the project (Fall 2017), members of the PLT attended a regular department meeting in all (12) STEM departments at Boise State University. The goal of these meetings was to engage faculty in a department dialogue about teaching and student success, and to collect data from faculty related to the impact of the project. The participants were STEM faculty from each STEM department. Before each meeting a message was sent to the faculty in the department explaining the purpose of the meeting.
During the meetings, faculty participants were asked to respond individually to a paper questionnaire, one question at a time. For questions specific to a course, faculty were instructed to think about a "target course" defined as the lowest level, largest enrollment undergraduate course that they teach. After faculty had individual time to reflect and record their response, small group and whole group discussions occurred before moving to the next question. The group discussions were not recorded and were not considered in our data analysis. At the end of the questionnaire, faculty were asked to list their department name, rank, and years teaching at Boise State University. In order to maintain a minimum level of confidentiality, no other identifying information was collected from respondents. Written questionnaires were collected and transcribed. Faculty responded to the following items: 1 EBIP Adoption Scale: Faculty responded to the six yes/no items of the EBIP Adoption Scale (Landrum et al., 2017). Scores on this scale can range from 0 (pre-awareness/awareness) to 6 (adoption with evidence of improvement in student learning or success) and reflect the degree to which an instructor is aware of or uses EBIPs. Faculty were asked to respond to the six questions twice, the first time considering what they currently know about EBIPs and the second time thinking retrospectively about what they knew about EBIPs four years earlier (when the project started). A pre-post analysis was not possible in this case because the EBIP Adoption Scale was developed as part of the project and thus was not available when the project began. The use of the current/retrospective approach provides insight into what faculty perceive had changed. 2 Description of changes to teaching practice: Faculty were asked to "Please list 2-3 changes you have made to improve teaching and learning in your courses over the last 4 years. " 3 Description of changes to departmental teaching culture and practice: Faculty were asked "What has changed in your department's teaching culture or practices over the last four years?" Codes for all open-ended faculty reflection questions were developed through an inductive approach, meaning the codes were derived from the data themselves rather than using pre-existing codes (Braun & Clarke, 2006). Responses to each question were coded separately, by two researchers (first two authors) with a separate set of emergent codes, although there was some overlap in codes among questions. Coding each question separately ensured that the results were aligned with the context of each question and reflected the respondents' ideas and interpretations of the terms "improve, " "culture, " and "practices. " The two researchers collaboratively coded faculty responses from one department in order to clarify the meaning of each code, and through that process identified nuances to the codes and refined or identified new codes as needed (Saldaña, 2015). Coding then proceeded independently for the remaining STEM departments with regular meetings during which the researchers engaged in ongoing, reflexive dialog to ensure that codes were being used consistently (Braun & Clarke, 2006;Saldaña, 2015). The researchers discussed any instances of disagreement until consensus was reached (Saldaña, 2015). Codes, definitions, examples, and the aggregated results across the 12 STEM departments are provided in Additional file 1: Table S1 and Additional file 2: Table S2. Not all faculty responded to every question. In addition, some responses were not coded because the relationship of the response to the question was unclear.
Of the 224 total full-time STEM faculty, 69% (155) attended the department meeting at which data were collected and completed the questionnaire, with participation in single departments ranging from 47 to 100%. For comparison, 58% of STEM faculty participated in project activities, with participation rates in departments ranging from 14 to 85%. Therefore, in each departmental sample, we have captured the perspectives of a broad spectrum of faculty, and not only those who participated directly in project activities. Questionnaire respondents reported their rank as professors (20%), associate professors (24%), assistant professors (29%), research faculty (1%), clinical faculty (6%), lecturers (15%), and no response provided (5%). Adjunct faculty were not part of the data collection because they are not regular attendees at department meetings. Participants reported their number of years teaching as less than 1 year (6%), 1-4 years (35%), 5-9 years (16%), 10-14 years (16%), 15-19 years (12%), 20-24 years (8%), and 25 years or more (3%).
With respect to the department-level results, our analysis focuses on coded changes reported by at least one-third of respondents within a department. Because many of these changes were similar across departments, we have chosen to focus on the results of a subset of six of the twelve STEM departments for brevity and clarity. The results for these six departments include the range of most frequently reported changes observed in the full data set of twelve STEM departments. The number of faculty participating in the department meeting and completing the questionnaire are as follows: Dept 1, n = 9; Dept 2, n = 25; Dept 3, n = 9; Dept 4, n = 12; Dept 5, n = 12; Dept 6, n = 8. Changes for all twelve departments and changes reported with frequencies lower than 33% are reported in Additional file 1: Table S1 and Additional file 2: Table S2.

Project activities and the CACAO model
The analysis of project activities led us to recognize and represent the key components of the CACAO model in relation to each other within the higher education context. These key components are the change agents, the organizational levels within the institution, and the change adopters who are at varying stages of adoption. All change strategies employed during the project are mapped onto these key components in Table 1. The project vision statement (the Change sought) is included at the top of the table as a reminder that the vision statement guided all project activities. Below the vision statement, the change agents and components of the organization are represented as columns, and adopters are represented as rows according to the stages of adoption.
PLT change agents focused on guiding overall project activities and on affecting change at the institutional level such as changes to infrastructure, policies, and the allocation of resources. Change agents on the FAST Team were tasked with supporting departmentally-focused dialogue around teaching and student success, developing a departmental action plan for change, and informing the project leadership team of nuances in the department culture, barriers to EBIP adoption, and needed resources.
Change strategies were enacted within three organizational levels. At the broadest level (university/college), strategies focused on structural supports, such as classroom infrastructure, policy revisions, and changes to the tenure and promotion policies and hiring practices at both the institutional and college levels. Department and individual level strategies are described in more detail below. While space does not allow for a thorough description of every strategy listed in Table 1, we describe below three strategies drawn from different positions in the table to illustrate how the strategies map onto components of the model. These strategies are noted with bold text in Table 1.

Example strategy 1: the vision and department conversations
Creating and promoting a vision statement is an example of a project strategy carried out at the institutional/ college level of the organization. This strategy, led by the PLT, was primarily focused on moving individuals into the first stages of adoption: awareness and curiosity. The CACAO model suggests that change agents, as leaders of a change, set the direction for change and align people to engage in the change process. The PLT set the direction for change by articulating a vision, the creation of which provided common institutional language for our aspirational view of teaching at Boise State University, and was critical for the project's communication strategies. In order to align people to the change, PLT members visited each STEM department at the beginning of the project to introduce the project's vision statement, and used the opportunity to define EBIPs and the related components of the vision. The PLT members then facilitated a conversation with faculty during the meeting about how faculty perceived the proposed change with respect to characteristics identified by the CACAO model (e.g., relative advantage, simplicity, etc.). This process and the results have been reported by Shadle et al. (2017). While the information gained from the department discussions provided insights into challenges identified by individual faculty, the aggregate results provided critical information about department and college level differences that would need to be addressed during the change process.

Example strategy 2: partner projects
Partner Projects were subawards based on proposals from departments or teams of faculty to increase the use of EBIPs in a single course or a course sequence, or to create supportive teaching structures within a department. All proposals were expected to align with the project's vision statement (the Change). This strategy appears several times in Table 1 because both change agent groups were involved in implementing this strategy, it spans multiple stages of adoption, and supports changes at multiple levels within the organization.
Both the PLT and the FAST Team were actively engaged in supporting Partner Projects. The specific EBIPs to be adopted were not prescribed by the PLT; instead, the PLT encouraged faculty to explore and adopt EBIPs best suited to their teaching context. Through the Partner Projects, the PLT provided resources, including time (e.g., course buyouts or summer salary), access to expertise and training (e.g., funding to attend disciplinebased workshops), and/or incentives for participation. FAST Team members encouraged their colleagues to engage in Partner Projects and in many cases led Partner Projects themselves.
Partner Projects targeted both the individual and department levels of the organization. Partner Projects were designed to target the department level by explicitly encouraging projects to involve teams of faculty and by requiring each Partner Project to include mechanisms for sharing their activities and results with the rest of the department or institution. While Partner Projects had a department focus, they often supported individual faculty to change their teaching practices. Therefore, Partner Projects also targeted the individual level of the organization. Partner Projects targeted faculty at different stages of adoption. Generally speaking, Partner Projects can be viewed as a strategy that engaged faculty who were ready to move to hands-on tryout or adoption, with the goal of giving them the support they needed to solidify their practice. These projects generally focused on course redesigns, course coordination, or the development of new lab activities. In a few cases, Partner Projects focused on creating departmental conditions that would support future explorations and adoption of EBIPs, therefore engaging faculty who were in earlier stages of adoption. For example, a project in Dept. 5 focused on program level assessment. This project was designed to serve as an on-ramp for faculty to think about where improvements in student learning outcomes were needed, and through this lens to motivate faculty to think about new ways of teaching. Faculty involved in this project were generally at early adoption stages such as curiosity and mental tryout. As another example, a project in Dept. 2 focused on creating a formal teaching discussion forum. The forum was designed to help faculty learn about what pedagogical choices were possible and also targeted faculty in early stages of adoption.

Example strategy 3: FAST team action plans
The primary role of FAST team members was to encourage faculty exploration of EBIPs and to promote dialogue around teaching, learning, and assessment within their department. Dormant (2011) suggests that those leading the change create an action plan after thoroughly understanding their adopters. In our implementation of the CACAO model, we engaged each FAST team member in developing an action plan for promoting dialogue and activities within their department based on their knowledge of department culture, interests, and norms. FAST team members were aware of Dormant's stages of adoption and the barriers to change and drivers of change most salient in their department . However, FAST team action plans were not explicitly focused on targeting adopters at specific stages. For this reason, FAST team action plans represent a variety of strategies and are presented in the "all stages'' row of Table 1. The PLT provided ideas and resources, but did not prescribe activities to the FAST team members; rather, FAST team members enacted plans they thought best suited their department's needs and their own abilities and capacities. Additional details about strategies that FAST team members planned and implemented are described in Table 2, where variations among department plans can be seen.
As the analysis of our project makes clear, the strategies we adopted span the different levels of the organization, the needs of faculty at different adoption stages, and were carried out by different types of change agents.

Department level activities
The strategies highlighted above demonstrate the emergent nature of activities and that department efforts fit their local context within the broad framework set by the project. In order to illustrate how local context led to different project activities, Table 2 documents the strategies of FAST Team members and the Partner Projects in six representative departments.
The data in Table 2 show both similarities and important differences among department activities. For example, not surprisingly, every FAST team member reported informal work to stimulate discussion about teaching/ EBIPs, usually in department meetings, and nearly all FAST team members were involved in encouraging engagement with the project (e.g., to submit Partner Project Proposals, to visit others' classrooms, etc.). There are contrasts, as well. For example, in some departments FAST team efforts were narrowly focused on activities such as encouraging colleagues to participate in project activities and increasing dialogue (Dept. 4 and Dept. 6). In other departments, FAST team efforts included a broader range of activities such as providing professional development opportunities, curating and sharing resources, and recognizing successes (Dept. 2 and Dept. 5). The types of Partner Projects also varied among departments. For example, Partner Projects in departments 2 and 3 focused on coordination of content and activities across sections of the same course or across a sequence of courses, while Dept. 4 focused on implementing EBIPs in specific courses or redesigning labs, and Dept. 5 focused on assessment.
Reviewing and categorizing project activities allowed us to demonstrate how the CACAO model was applied in a higher education setting and to identify the key project elements and change strategies. Next, we examine the model's utility to higher education change projects by evaluating the degree to which changes toward the project's vision statement took place.

Impact of using the CACAO Model to Enact Change.
We used the elements of the CACAO model to guide our approach to understanding and assessing the project's impact on different levels of the institution and on adopters at different stages. If the CACAO model was effective in guiding the change process, we would expect to see changes at different levels of the institution, changes in individuals across all adoption stages, and changes that vary among departments and are consistent with department activities. Here we share results related to changes during the course of the project with respect to these key components of the CACAO model as applied in higher education. Changes are presented first based on institutional level (college/ institution, department, individuals), followed by changes across adopters at different adoption stages described in the section about individual-level changes.

Changes to college and institution teaching culture and practices
Changes at the institutional and college level were the result of actions that could impact all individuals and departments. These changes were largely focused on policy issues and included active engagement with the Faculty Senate to revise the institution's tenure and promotion policy as well as secondary changes made to the tenure and promotion policies at the college level. The policy revisions prioritized the evaluation of teaching effectiveness based on the assessment of student learning outcomes and faculty efforts to improve their teaching, rather than a singular reliance on student course evaluations to demonstrate teaching effectiveness. Midway through the project, the institution implemented a new incentive-based budget model, and while this was not a result of the project, the PLT used it as an opportunity to model different undergraduate student retention scenarios to illustrate to stakeholders how improved student retention would impact department budgets. With this heightened and focused sense of the practical consequences of improved student success, departments increased their focus on implementing strategies to improve student retention, such as EBIP adoption.
Additional changes occurred in the institutional approach to classroom design and remodels. Many faculty identified the lack of classroom spaces that would make active learning easy to implement as a barrier to the adoption of EBIPs . Similarly, faculty who already adopted a variety of EBIPs reported challenging classroom spaces. As a result, members of the PLT worked to assess classroom spaces across campus and make recommendations for classroom remodels and the design of classrooms in new buildings. For example, a large-enrollment lecture hall with stadium seating was remodeled to include tiers of tables with flexible seating to facilitate student-student and student-instructor interactions.

Changes to departmental teaching culture and practice
The most frequently reported changes in departments' teaching culture or practices across all 12 STEM departments were increased dialogue about teaching and learning (33%), and increased exploration or adoption of EBIPs (18%; Fig. 1). Perceived changes varied among the departments; for example, 50% of faculty respondents in Dept. 4 described increased emphasis or value placed on teaching effectiveness and EBIPs within their department, while 42% of faculty respondents in Dept. 5 reported an increased focus on outcomes and assessment Fig. 1). Neither of these changes were commonly reported across all STEM departments, highlighting the variations in the types of changes that happened across this organizational level. Code descriptions, example quotations, and aggregate data are provided in Additional file 1: Table S1.

Changes in individual faculty teaching practice
Changes reported in individual faculty behaviors show that people at all stages of the adoption scale made progress toward the vision of EBIP exploration and adoption. Faculty reported their teaching practices using the EBIP Adoption Scale (Landrum et al., 2017) at the end of the project and (retrospectively) at the beginning of the project. At the end of the project, faculty respondents span the full range of adoption stages, with only a few (5%) being unaware of EBIPs and many being in at least one of the adoption stages (78%). Nearly half of faculty respondents (44%) indicated they were not aware of EBIPs when the project started, and another 29% of faculty respondents indicated they were in pre-adoption stages when the project started. The remaining 21% indicated they were in one of the adoption stages (at least using EBIPs) when the project started. Only 6% of faculty respondents reported having evidence of improved student success with EBIP use (e.g., they are engaging with assessment data) at the beginning of the project, whereas 34% of faculty respondents reported being in this final stage of EBIP adoption at the end of the project. Of the faculty who completed both the retrospective and end-of-project EBIP adoption stage questions, 73% reported progressing at least one stage, with 40% progressing through four or more stages (Fig. 2) of respondents reported regression to an earlier stage of adoption, and 24% reported no movement on the EBIP adoption scale. Faculty in all departments showed a shift toward later stages of adoption (Fig. 3). Further, several departments in which the majority of faculty respondents reported being in early adoption stages at the beginning of the project ended the project with all faculty respondents at later stages of adoption (e.g., Departments 1, 3, 4) (  When asked to describe changes they had made to improve teaching and learning during the project period, faculty respondents most frequently described increased use of, or modification to, in-class activities, as well as the use of low-threshold EBIPs. Low-threshold EBIPs are those that take minimal amounts of time and effort to incorporate into courses, such as think-pair-share, clickers, and/or opportunities for reflection. The most frequent types of changes reported by faculty across the STEM departments are summarized in Fig. 4 (code descriptions and example quotations provided in Additional file 2: Table S2). Many faculty reported making more than one change. For example, 42% of those who reported adopting low-threshold EBIPs also reported making other changes to in-class activities.
Aggregate changes made to teaching practice varied among faculty from different departments, with some departments exhibiting a majority of faculty respondents making a specific type of change, while other departments reported a greater variety of changes (Fig. 4). While participation in professional development does not guarantee changes to teaching practice, faculty reporting increased participation in professional development as a change in their teaching practice is noteworthy, especially since it connects to the project's vision for teaching to be characterized by "on-going exploration and adoption of evidence-based instructional practices" and by faculty engagement "in continuous improvement of teaching and learning. " Faculty at all stages on the EBIP adoption scale reported making changes to their teaching. However, changes made to teaching practice varied among faculty at different stages of EBIP adoption (Fig. 5). Faculty respondents at the earlier stages of EBIP adoption more frequently reported adding or revising homework assignments and resources and/or revising assessment methods. Faculty respondents at the later stages of EBIP adoption more frequently reported integrating highthreshold EBIPs into their courses and/or adding or modifying in-class activities. High-threshold EBIPs are those likely to require more time and effort to incorporate into a course compared to low-threshold EBIPs because they tend to impact course structure rather than a single class period. Examples of high-threshold EBIPs shared by faculty include Process Oriented Guided Inquiry Learning, Team Based Learning, Case Based Learning, and Service Learning among others.

Factors perceived to support change
Although not specifically asked for on the questionnaire, slightly less than half (43%, n = 68) of faculty respondents reported a factor that influenced the individual or departmental changes they described. The most frequently cited factors were Partner Projects, participation in CTL programming, other project activities, and discussions with colleagues and FAST team members. Example quotes citing each of these factors are provided in Table 4.
Comparing reported department and individual changes with FAST Team and Partner Project activities shows variations among departments, as well as a general alignment of activities and changes within departments (Table 5). For example, in Dept. 5, the FAST team member supported efforts to review program learning outcomes, curriculum mapping, and assessment. Partner Table 3 Percent of faculty in EBIP adoption stages by department at beginning (retrospective) and end of the project *Two respondents from this department did not complete the EBIP scale so the sample size for this item is two less than what is reported in all other tables Project efforts in Dept. 5 focused on developing skillsbased activities designed to support program learning outcomes, participation in professional development, and coordinated efforts to embed assessments throughout the curriculum. Faculty respondents in this department most frequently reported integrating low-threshold EBIPs, modifying course design, and revising assessments at the individual level. Similarly, departmental changes frequently reported by faculty respondents in Dept. 5 included an increase in dialogue, an increased focus on outcomes and assessments, and exploration of EBIPs. In contrast, FAST team efforts in Dept. 1 focused on encouraging exploration of new teaching practices and holding informal meetings to discuss teaching and learning within the department. Partner Project activities in Dept. 1included incorporating EBIPs into upper division courses and holding a summer teaching retreat. Faculty respondents in this department frequently reported individual changes of increased adoption of EBIPs, revised assessments, and increased classroom interactions. Some departmental changes in Dept. 1, such as increased dialogue and exploration of EBIPs, are similar to changes reported in other departments, but unlike other departments, a third of the faculty respondents in Dept. 1 indicated a strong sense of community within the department. Finally, we examine Dept. 6, where faculty engaged in making changes only mid-way through the project period. FAST team efforts in Dept. 6 focused on trying to increase awareness of EBIPs and establishing time in department meetings dedicated to discussing teaching-related topics. Partner Project activities in Dept. 6 were focused on individual adoption of EBIPs within a subset of courses. Faculty respondents reported an increase in adoption of EBIPs at the individual and department level, but unlike many other departments, faculty did not note an increase in dialogue.

A matrix for using the CACAO model in higher education
Our analysis of project activities allowed us to identify key components needed to apply the CACAO model in higher education settings and resulted in a matrix ( Table 1) that can be generalized for use by others seeking to affect change. In a general version of this matrix (Fig. 6), the change agents and components of the organization are represented as columns, and adopters are represented as rows according to the stages of adoption. The planned change implicitly surrounds all components, as it is the driver of the efforts overall. The matrix allows one to identify strategies that connect the needs of adopters with the actions of change agents operating at different    Table 4 Factors cited that prompted change to individual teaching practices or department changes related to teaching

Influencing Factor Example Faculty Quotes
Partner Projects "Also through the partner project we implemented carousel and concept maps. " "Participating in two partner projects gave us the incentive to think actively & make changes to our courses. " "In the last year, there has been robust discussion…regarding consistency across sections to help with success and vertical integration. This has been possible because of the creation of a [discipline] curriculum committee and a funded… partner project. " CTL Workshops and Programming "Introduced and fully implemented POGIL, prompted by CTL course design workshop. " "…joined Boise State University Teaching Scholars and started to educate myself about EBIPs (CTL!) via workshops and faculty communities. " Other project activities or general reference to project "… more freedom to approach others and get information on their teaching techniques/materials-it is more okay to do this; prompted significantly by participating in [the project]" "Teaching is something that is consistently discussed within the department….. Prior to 4 years ago this was not the case. I think this shift has happened partially because of [the project] but also because of other organizations [discipline specific] have put a lot of importance on teaching methods" Discussion with colleagues and FAST Team members "Team-based Learning. This was prompted by discussions with colleagues and support from [the project]. " "Integrated EBIPs in every class. 200 level to 500 level motivated by improving evals, discussions with colleagues…" "We have more discussions about teaching and the curriculum. This change is related to several things, FAST Team activities in dept: spending time in faculty meetings talking about EBIPs; [short workshops] in faculty meeting…" Page 15 of 20 Viskupic et al. International Journal of STEM Education (2022) 9:6 levels within a higher education institution. Representing the CACAO model as a matrix makes clear that there are different strategies that can be employed by different change agents at each organizational level. Viewing project activities in the matrix provides an overview of a complex change process, and can be used to help identify potential targets for project activities and for collecting project evaluation data.

The CACAO model as a guide for STEM education reform
Any project with ambitious, institutional level goals for change is necessarily going to be complex. In the case of our STEM education change project, having a theorydriven framework helped to guide planning and connect disparate parts of the large project. The CACAO model informed our efforts as the project unfolded, influencing our choice of activities and interventions. The postproject analysis of activities, presented in this study, shows how our interventions map onto the framework of the CACAO model, and helps to illustrate the complexity of a full project and how such a project can span the full space of the model. Further, it stands in contrast to much of the literature about STEM education reform efforts, which frequently focus on the description and assessment of a more narrowly defined intervention. Our study serves as an example of a change effort that demonstrates a "deep connection" between project activities and the change theory informing the project (Reinholz et al., 2021). In addition to guiding project activities, the CACAO model helped guide the assessment of the project's impact. Based on the organizational components needed to affect change, we specifically assessed impacts on individual faculty practice, on department level practice, and on institutional policies and practice.
Individual change was assessed both with respect to movement along the adoption curve and based on concrete changes to practice. Changes in individual faculty behaviors show that people at all stages of the adoption scale made progress toward the vision of EBIP exploration and adoption; faculty moved toward greater adoption of EBIPs (movement along the adoption curve) and documented changes to their practice consistent with this movement. Project activities were intentionally designed to help faculty at all stages of adoption (with perhaps the exception of firm resisters), and this aspect of the CACAO framework led to success in impacting changes at the individual level.
The CACAO model acknowledges that individuals work in context and so it was also important to measure the changes faculty experienced in their departments. Faculty reported a variety of changes at the department level with respect to practice and culture, including increased dialogue, use of assessment data, coordination across different sections of the same course, and perceived changes to the value of teaching in the department. Many of these changes were ones explicitly sought by our project's vision, which called for changes in faculty practice in the context of changed culture and shared expectations around effective teaching. Our results suggest that the activities guided by the CACAO framework were successful in impacting changes at the department level. The alignment between faculty perceptions of change and the project vision statement, as well as the direct acknowledgement of project efforts by faculty describing changes, suggests that intentional project activities developed with the guidance of the CACAO framework were successful. Interestingly, faculty at different stages of EBIP adoption reported different changes to their teaching practices, with faculty at later adoption stages generally reporting the use of more varied and sophisticated pedagogical strategies. These findings reinforce the need for targeted interventions for faculty at different adoption stages, which are based on the kinds of changes faculty may be ready for at each stage.
The variations in changes across departments are a reflection of the importance of context when promoting change and are similar to the results of the Science Education Initiative, where department-focused reform efforts led to changes to 10-93% of courses across 12 departments at two institutions (Wieman, 2017). The variation in changes reported among departments in our project reflects the emphasis of understanding the organizational context described in the CACAO model. We focused on the department as an important locus for change and allowed project activities to be emergent and meaningful to the departments in which they took place. The PLT did not dictate what departments should do, but instead provided resources and support in response to emergent needs and interests. We intentionally worked with FAST Team members as change agents within each department who were familiar with department norms and could suggest and carry out activities that were likely to be well received by faculty members. Thus, activities within each department were different, as were outcomes described in each department. The CACAO model helped us to support changes in the different department contexts, allowing the activities and efforts to align with the interests and needs of the department, while still achieving meaningful change. The results reinforce the value of focusing STEM education change efforts at the departmental level (Austin, 2011;Bager-Elsborg, 2019;Lund & Stains, 2015;Ngai et al., 2020;Reinholz & Apkarian, 2018;Wieman & Gilbert, 2015).
Our analysis suggests that the CACAO model was successfully adapted for use in the higher education context and supported substantive movement towards our aspirational vision. Changes that occurred at the institutional level (e.g., changed policy, classroom space) should help to sustain changes at other levels. While the CACAO model does not explicitly address sustainability, Dormant (2011) describes that cultural changes are challenging to enact and take five years or longer. We are hopeful that the shifts in departmental practice and culture, individual pedagogical practice, and structural supports will serve to sustain continued improvements in teaching and learning at the institution. That the transformation is not yet fully realized underscores the challenge of enacting changes of this type across an institution in a relatively short time frame, and highlights the need for ongoing investments of time, intentional activity, and financial resources beyond a five-year period. Choosing a change theory Different change theories offer different insights into how to think about and approach change . This can make selecting a change theory challenging, especially for those new to thinking about grounding one's change efforts in a change theory or framework. At the start of our project, most members of our PLT had no experience with grounding a project like this in a theoretical framework and we had modest knowledge of the variety of change theories we might have leveraged. We selected the CACAO model because we had a local expert in organizational, cultural, and systems change who was familiar with the model and its theoretical underpinnings; that individual served on the PLT throughout the project. As we engaged with the CACAO model, we found the ideas straightforward to use and we particularly appreciated the insights gained from the activities that Dormant (2011) outlines as part of the model. Our engagement with the model and its activities helped us to apply the underlying theoretical ideas to support change at our institution. Further, as we gained facility with the model, we found it especially helpful to focus on the faculty stages of adoption and engagement at multiple levels of the institution. While we cannot know how our project might have unfolded if it had been framed with a different theory, we believe the CACAO model helped us to manage a complex change process by allowing us to think intentionally about processes and activities across the institution, how they are connected, and how they could be leveraged to move toward a common vision. In this way, the CACAO model supports a change process which contrasts with other models which, for example, more narrowly prescribe activities (e.g., Communities of Practice; Wenger, 2011) or which focus change efforts at a particular organizational level (e.g., Teacher-Centric Systemic Reform; Gess-Newsome, et al., 2003). Importantly, multiple scholars (Kezar, 2014;Reinholz et al., 2021) have asserted that change efforts may benefit from the use of multiple change theories. Throughout our project, we continued to learn about additional change theories that could help us understand certain phenomena or challenges. For example, while our project vision articulated the goal of faculty in regular dialogue about teaching, we came to appreciate how increased dialogue was critical to the change process itself. As our project unfolded, we recognized this as a form of sensemaking (Kezar, 2013). We believe discussion was critical to helping faculty move along the stages of the adoption curve to make sense of new ideas and new expectations. In the academic context, which lacks the hierarchical nature of organizations in business contexts, for which the CACAO model was originally designed, the inclusion of sensemaking as an underlying change theory informed the ongoing use of dialogue as a change strategy. As another example, we reviewed the Theory of Planned Behavior (Ajzen, 1991), which provides a framework for considering the reasons faculty move (or not) along the adoption curve by illuminating the ways in which an individual's intent to engage in a behavior is informed by their attitudes, in interaction with local norms and perceived behavioral controls. While these supplemental change theories did not guide our overall work, they did provide deeper insights into particular components of the model that helped us to implement change. Combining CACAO with other models (e.g., those which focus on prescribed activities or focus on a particular level of an organization) may be fruitful for future change efforts.

Building on the CACAO model
Our depiction of the CACAO model is an adaptation based on our use in the higher education context. Presenting our project activities in the context of this model helped us to think about how change occurred and has potential value for others interested in driving or studying change. The matrix presented in Fig. 6 explicitly reflects important features of higher education organizations, including the possibility of activity by different kinds of change agents who are well-positioned to influence the organization at a particular level. In our own project, FAST team members were explicitly focused on their departments and were not in positions to enact change at the institutional or college level. Administrators and faculty leaders on the PLT were in positions that enabled them to make changes to policy and allocate resources to support the work of the project overall.
Similar to ideas that come from change theories based on systems thinking (e.g., Meadows, 2008;Offerdahl et al., 2020;Wasserman, 2010), our adaptation of the CACAO model can help change agents think explicitly about the important interconnected components of the higher education change environment. Further, the matrix can help to identify both where efforts are needed to promote change and where to look for and assess impact. It is important to acknowledge that the full complexity of a change effort cannot be collapsed into a simple matrix. Table 1 is an overview of activities that took place in our project over a five-year period. The strategies were not all implemented at one time, nor were they always viewed in this interconnected way as we worked to implement the CACAO model in our project. As we gained facility with the CACAO model we found we built its ideas more naturally into the cycles of continuous improvement in which we engaged. For example, as the project unfolded, we added or modified strategies with regard to faculty needs at different stages of adoption and the desire to target change at multiple levels of the organization. In the context of the model, we tried some approaches and assessed results, they worked in some places and not others, and those outcomes informed what we chose to do next. The Keck/PKAL change model (Elrod & Kezar, 2017) captures the temporal nature of change and acknowledges that projects do not move forward linearly and at a constant rate toward effective adoption.
The CACAO matrix introduced here reinforces the value of using change theory to inform change efforts (Borrego & Henderson, 2014;Reinholz & Andrews, 2020). Rather than seeing our efforts simply as a series of activities, however intentionally chosen, mapping them onto the matrix makes clear how they are connected to important levers for change. It also creates a structure that others can build on and modify, either by applying our CACAO matrix in their own setting or by using the matrix to identify elements that connect to other change theories. Capturing the key elements of the CACAO model in the matrix provides a valuable tool for others considering how to frame change efforts. Our work contributes to the growing body of literature which seeks to understand how change theories can be useful and generalizable beyond a single project. Another project's use of the framework might be populated with different activities, but using the framework as a lens provides the potential for important future comparisons that will provide insight into how to be more effective at influencing change.

Conclusion
Achieving pedagogical reform in higher education is complex and challenging. The theory-based CACAO model has now been adapted for use in higher education and applied to a STEM education reform effort. The representation of the interconnected elements of the CACAO framework in a matrix allows others to build on this work as change agents in higher education. In our application of the model, we were able to leverage multiple types of change agents, operating at the individual, departmental, and organization level to impact the way that STEM faculty on our campus engage pedagogically. The changes to faculty practice, departmental discourse, and institutional structures demonstrate the utility of the CACAO model for making change.