Possibilities and pitfalls of practitioners in trying to apply change theory as viewed through the lens of Reinholz, White, and Andrews “Change theory in STEM higher education: a systematic review”

Introduction

In their 2021 review paper, Reinholz, White, and Andrews noted that change efforts in STEM higher education do not engage deeply in change theory to guide interventions and research efforts, leading to disjointed attempts to influence change and a proliferation of theories. This superficial use of theory and tendency toward disjointed approaches creates barriers to generalized knowledge about change that can be applied by practitioners desiring to bring about change in their particular context. Reinholz and Andrews (2020) point out that the key elements of a theory of change include context, interventions, and indicators while considering rationale and assumptions in a specific project. Any project’s theory of change should be informed by change theory, and Reinholz et al. (2021) provide a systematic review of the change theories upon which a particular project might draw.

For those STEM education change agents who are not change researchers, the literature suggests mixed expectations about the level of familiarity with using change theory in the development of a project’s theory of change. For example, Borrego and Henderson (2014) distill the goals, assumptions, and underlying logic of change strategies (also known as interventions) for STEM education change agents. They note it is important that practitioners are aware of the array of change strategies available, but at the same time, they suggest the complexity and variability associated with developing a theory of change makes it challenging for those without social science backgrounds to fully engage with change theories. Kezar et al. (2015) note that practitioners often have difficulty identifying all the elements of a theory of change when they are planning a project, and they warn that one of the implicit theories of change that practitioners adhere to is that meaningful change can start with interventions. Kezar et al. (2015) observe that starting with interventions comes at the expense of understanding the context, including considerations of rationale and assumptions. With little acknowledgement of how challenging it is for those who do not come from a place of familiarity with theories of change and change theories, Reinholz et al. (2021) conclude their paper with a call to action asking change agents to state clearly the change theories that guided their work and then to explicitly analyze how those theories played out in practice to help guide others, including practitioners, in their change projects.

In this commentary, we describe our experience working with a project aimed at developing approaches to support student quantitative skills (QS), including working with data, reasoning with numbers, and using mathematical models within disciplines, in the context of a consortium of highly selective small liberal arts colleges—the Liberal Arts Collaborative for Digital Innovation (LACOL). We highlight some of the tensions with regard to how much understanding of change theory STEM educators can be expected to have and illustrate how some of the themes identified by Reinholz et al. (2021) were relevant to our specific project. We provide an overview of the context in which we were trying to initiate change, the frameworks and approaches that we intended to use to promote and measure change, and what we have learned about our efforts—in terms of successes and failures, as well as the more nuanced understanding of employing change strategies in our consortial context. Our project focused on engaging individuals in addressing a common challenge, but sought to do so within a consortium of similarly situated institutions. In developing our project's theory of change, we planned to use faculty learning communities, an intervention informed by the Community of Practice change theory, but deeper understanding of the Community of Practice change theory, including its extension to Communities of Transformation (Gehrke & Kezar, 2016; Reinholz et al., 2021), would have been valuable in our efforts to work across institutions.

Before engaging in further discussion of our project and how we planned to employ change theories in our work, it is important to highlight our position within the context of STEM higher education. We are not change researchers nor are we discipline-based education researchers. We had engaged previously in the scholarship of teaching and learning as higher education faculty members and academic technologists. At the start of the project, the four project leaders (three of whom are authors on this paper) were in roles where we were able to see institutional challenges around inclusive support of STEM students and were interested in working toward institutional change. Melissa Eblen-Zayas was Director of the Perlman Center for Learning and Teaching at Carleton College, Laura J. Muller was Director of Quantitative Skills Programs and Peer Support at Williams College, and Janet S. Russell was Director of Academic Technology at Carleton College. The fourth project leader was Director of Instructional Technology at Williams College.

The project we describe here emerged from a consortial shared desire to improve the persistence in quantitative disciplines of students who enter college lacking fluency and confidence with foundational QS. The peer-reviewed literature demonstrated that online modules for embedded skills building within a particular disciplinary context had an impact on student success in STEM courses (Burn et al., 2013; Forrest et al., 2017; Jackson & Johnson, 2013; Thompson et al., 2010; Wenner et al., 2011). Faculty at our institutions expressed interest in using online resources, and we wanted to expand this approach of embedded online modules and develop multidisciplinary modules relevant to a wide variety of introductory courses. Starting from the strategies in the change framework presented by Borrego and Henderson (2014), we chose relevant interventions in designing the consortial project. We planned to encourage faculty to contribute to the development of, and then adopt, these multidisciplinary modules to add one more tool to their toolbox of approaches to support student QS.

Ultimately, the COVID-19 pandemic significantly interrupted the implementation of the project, but in reflecting on the original project plan and how it played out, we find that we fell short in some of the ways identified by Reinholz et al. (2021), and our project would have benefitted if, as practitioners, we had begun the project design with a broader consideration of change theories. Our retrospective analysis has helped us see that, like many practitioners, we focused more effort on choosing specific interventions and indicators and focused less on clearly articulating the context, rationale, and assumptions that are key elements of a project’s theory of change (Kezar et al., 2015). At the same time, emergent efforts like ours do not always lend themselves to explicit articulation of a complete theory of change at the beginning of the work. Considering when STEM educators might benefit most from learning about relevant elements of change theory is not necessarily straightforward, and the example of our project can inform both practitioners and researchers about considerations in translating change theory into practice.

Project origin and development

In the past decade, there has been increased attention in higher education to what changes might ensure the persistence of all students in STEM learning environments. Racial and socioeconomic gaps resulting in uneven early math preparation of students affect how well-positioned they are for 2-year or 4-year college experiences (Lee, 2012). In addition, gaps in math preparation can lead students to opt out of quantitatively focused majors (Brown et al., 2017). QS are increasingly relevant in disciplines that previously did not have a significant quantitative focus, and students often need additional support in strengthening QS to thrive (Feser et al., 2013; National Research Council, 2003). Thus, faculty are faced with the dual problem of linking math concepts to their courses to support the burgeoning role of QS in their disciplinary context while supporting students with less fluency or confidence in their quantitative abilities. For many types of institutions, developmental math courses and related topics of QS readiness in disciplines have been a long-standing issue, but these issues are relatively new to the LACOL institutions, all of which are highly selective and historically have not provided developmental coursework. As these colleges are now admitting students from more diverse high school backgrounds, faculty are challenged to support a student population that has a broader range of math preparation than in the past, and at the same time, institutional initiatives have focused on encouraging the incorporation and application of QS and quantitative reasoning across the curriculum (Elrod, 2014; Grawe & Rutz, 2009). Supporting faculty to meet this dual challenge is key to managing faculty workload and ensuring student success.

Two primary approaches have been developed to address this dual challenge (Aikens & Dolan, 2014): (1) construction of interdisciplinary courses or curricula and (2) embedding skills development modules within existing subjects. Constructing new curricula requires major curricular overhaul that creates a barrier to adoption at more than one institution (Lent et al., 2021). Embedded QS development can take a variety of forms, but online modules are of particular interest because of the possibility for self-paced engagement and portability. Students benefit from embedded QS development, whether using commercial online materials (Forrest et al., 2017; Jackson & Johnson, 2013) or faculty-developed online modules (Burn et al., 2013; Thompson et al., 2010; Wenner et al., 2011). One of the limitations is that most online modules are designed for exploring the application of QS within a single discipline. While this approach helps students succeed in particular courses, it does not help students transfer skills to other contexts and recognize that they might be using the same skills in different disciplines.

Our cross-institutional project, known as QLAB, which is not an acronym but a name representing the collaborative exploration of approaches to building QS, focused on developing online modules to review QS taught in high school math courses and demonstrate their relevance in a variety of science and social science contexts. Institutions involved in the project were part of LACOL, a consortium of eleven highly selective liberal arts colleges with individual enrollments of about 2000 students. Originally formed in 2014, in response to an era when MOOCs (Massively Open Online Courses) were seen as a potential disruptive force in higher education, this consortium was an opportunity for similar institutions that valued personal relationships between students, faculty, and staff and student-centered residential learning experiences to explore possible approaches to and benefits of digital and technology-enhanced teaching and learning. All 11 institutions have a strong tradition of shared governance in which faculty are responsible for decisions about curriculum and instructional techniques, so projects driven by faculty interests were at the center of the work of this consortium. Neither administrators nor consortial leaders had much power to shape decisions about teaching in the new, technologically driven, landscape.

LACOL was formed around rethinking digital and online education in the specific setting of selective, residential liberal arts colleges and supporting individuals who participated in these efforts at their home institution. Although not informed by the Community of Transformation change theory, early LACOL efforts contained many of the key elements of a Community of Transformation, namely, fostering peer-to-peer learning through gatherings and workshops, online forums and colloquia in a manner that was infused by a shared culture that valued the liberal arts and high-touch, residential learning and teaching communities (Kezar et al., 2015). The consortium aimed to encourage projects that connected individuals on different campuses who were interested in experimenting with new models of technology-enhanced, student-centered teaching, learning, and research. At the inaugural consortial workshop in 2014, the shared challenge of addressing the range of entering students’ QS presented an opportunity for cross-institutional collaboration that would capitalize on the use of technology. LACOL faculty benefited from hearing about each other’s approaches and continuing conversations led to the emergent development of the QLAB project (Fig. 1). QLAB was designed to build on recently formed LACOL connections.

Fig. 1

Timeline of emergent process leading to a formal project that employed change theory

The QLAB project relied on the emerging consensus of LACOL participants that consortial efforts among peer institutions in the development of approaches to QS support employed across the curriculum would have advantages beyond what might be achieved on a single campus. We planned to work together to develop and test shared online resources that could be used in a variety of different instructional contexts including introductory STEM and social science courses, peer tutoring programs, or quantitative resource centers. In particular, we wanted to demonstrate to students that QS would be transferable across many different settings and show the relevance of these skills in the context of a liberal arts curriculum. Creating materials is time intensive, and we envisioned that having a tangible product that we were working to develop would catalyze action and that sharing that effort consortially rather than developing resources individually could sustain energy.

Having a tangible product that we were working to develop did help to catalyze action, and several targeted gatherings of interested individuals, including an in-person design workshop and a proof-of-concept project, were critical in cementing interest and identifying action steps. The design workshop included 14 faculty and staff participants from seven different institutions. Eight of those individuals continued developing the proof-of-concept project, which involved building multidisciplinary modules designed to help students review and practice selected QS. Nineteen additional faculty and staff from seven different institutions joined the project—contributing materials for the modules, providing suggestions for module structure, or testing the modules. The response from both faculty and students who used the modules was positive and showed us clear areas for improvement. For this early work, faculty received no compensation for their contributions; it was a volunteer effort.

Those who participated in these activities collectively agreed that pursuing the shared effort to build QS resources would be beneficial and four project leaders stepped forward to launch the QLAB project (at the time, two of us were at one institution and two of us were at another institution). For MEZ and LJM, our institutional roles working with faculty and students across our colleges gave us a broad view of student QS support, and we were drawn to the idea of moving the conversation beyond individual classrooms. The other two project leaders brought institutional views on using technology to support instruction. We believed we had a “shared enough” vision for what change we wanted to see happen. Furthermore, we believed that this vision aligned with both the consortial vision for collaborations around technology-enhanced teaching, as well as movement on the individual campuses toward both greater diversity and greater persistence of students in quantitatively-rich majors.

In the early activities described above, we found strong support from individuals who were interested in specific QS topics or student support in particular disciplines, but these individuals were not at any one institution or in any one field. Seeing as quantitative proficiency has “across the curriculum” relevance in the same way as writing skills (Grawe & Rutz, 2009), we designed this project to connect these enthusiastic individuals across the consortium to develop one tool that could be used by many. Our goal was then to empower and support these individuals through a consortial project that would engage in collective, iterative development of modules, and then encourage these individuals to leverage disciplinary networks and institutional networks to foster the adoption and adaptation of the modules. One of the limitations of this approach was that the positionality of some of these enthusiastic individuals within their institutions, as untenured or contingent faculty or staff, had significant implications for their ability to leverage networks across their campus and across the consortium.

Through the proof-of-concept project that involved seven different institutions, we realized that organizing activities at all LACOL institutions simultaneously was difficult. To be more nimble, we identified three anchor institutions, where the active and enthusiastic individuals resided, to begin the QLAB work with the aim of moving conversations out to more LACOL institutions in successive years. By engaging individual faculty across the consortium in a collaborative building process, we were trying to give resources to grassroots efforts that emerged as the project proceeded. Collective development of modules could then lead to the addition of existing materials into the modules, and we could track patterns of adoption and adaptation of modules across various departments and institutions of the consortium. Our project was initially focused on individual choice and change, but also considered how disciplinary and institutional contexts might impact choices and how individuals might influence others. In retrospect, we took this constructivist approach based on our personal experiences with change in our local contexts (Kezar et al., 2015).

Change theories that guided the design of our project

From the beginning, our work was grounded in the literature demonstrating that online modules could improve student quantitative skills in disciplinary contexts (Burn et al., 2013; Forrest et al., 2017; Jackson & Johnson, 2013; Thompson et al., 2010; Wenner et al., 2011), but we needed to determine how to get faculty to use the modules and reflect on how they complemented their existing approaches to strengthening and supporting students’ QS development. Research shows that dissemination of curricular resources rarely leads to adoption of those resources (Froyd et al., 2017; Hennessy et al., 2021), and although the existing online modules demonstrably improved student outcomes (Burn et al., 2013; Thompson et al., 2010; Wenner et al., 2011), they were not resources that were developed in institutional contexts similar to LACOL institutions. Therefore, we knew it would be a challenge to convince faculty to use the existing modules. In addition, modules had only been developed for a few specific disciplines, such as biology, geology, and economics. Recognizing the shortcomings of dissemination alone in leading to change, we wanted to leverage the findings that interactive development and dissemination (Khatri et al., 2016), influenced by informal social networks (Andrews & Lemons, 2015; Dancy et al., 2016; Lane et al., 2022; Quardokus & Henderson, 2015) leads to successful adoption of the materials that are developed. Our goal was to have intra-institutional cross-departmental and cross-institutional intra-disciplinary faculty involvement in the development of multidisciplinary modules relevant for a variety of introductory STEM and social science courses. Rather than creating online modules and then disseminating them to faculty, we wanted to create modules with faculty, and in particular, with faculty from a variety of different disciplines.

To be able to support this work adequately, we originally applied for, and received, external funding from the NSF Improving Undergraduate STEM Education (IUSE) program for 3 years of work, though the timeframe was extended due the pandemic impact on project progress. We chose to apply to the IUSE Institutional and Community Transformation track because our goal was the expansion of the college-wide toolbox for supporting student QS; we planned to achieve that goal through faculty development in discussions around approaches to supporting student QS as part of module creation and testing. Our NSF proposal required us as project leaders to clearly identify how we planned to promote institutional change, although it has been noted that funding agencies often use conflicting terminology in their calls for the use of change theories and theories of change (Reinholz & Andrews, 2020). We did not have a deep knowledge of change theories as STEM practitioners, and in retrospect, the theory of change that we developed for our project included a mix of implicit and explicit elements. In addition, we found that many change theories assumed some element of top-down vision or positional leadership by key participants, which seemed at odds with the emergent development of the project that we hoped to pull through from the proof-of-concept work.

During the proposal process, the work of Borrego and Henderson (2014) gave us the vocabulary and broader perspective with which to articulate the design of our project (Figs. 2 and 3). Although Borrego and Henderson (2014) discuss change strategies, they explicitly state that they are aiming their work at change agents, and they focus on explaining the goals, assumptions, and underlying logic of those change strategies, so that STEM higher education change agents can make an informed decision about which strategies to use. They direct those without social science backgrounds to focus on choosing strategies that fit their situation to overcome the barriers of articulating an entire change theory framework before starting their project. To help STEM education practitioners with selection of strategies, Borrego and Henderson (2014) organize change strategies into quadrants, each quadrant characterized by the change environment (personal vs. institutional), and whether the change is emergent or prescribed, and identify change strategies that fall within each quadrant. In developing a theory of change for our project, we explicitly articulated interventions (or “change strategies”) and indicators (as shown in Table 1), but the rationale/assumptions and context remained implicit. Kezar et al. (2015) identified that one of the widely held, but unstated, theories of change that exists among STEM education change agents is that change can be meaningfully created by starting with interventions. Indeed, we focused our project design on interventions and indicators, leaving other key elements of the theory of change for our project less clearly articulated or developed. A greater understanding of the relevant context for various change theories as laid out in Reinholz et al. (2021), in particular the Community of Practice and Community of Transformation theories would have challenged us to consider the ways in which emergent institutional change might be possible through a consortial project.

Fig. 2

QLAB project activities as they fit within Borrego and Henderson’s (2014) quadrant change strategy framework

Fig. 3

Cyclical nature of QLAB project design including change strategies

Table 1 QLAB project change strategies (interventions) with relevant activities, context, and indicators

When making choices about change strategies (interventions), we were most attracted to the two quadrants of emergent change in the Borrego and Henderson (2014) framework (Fig. 2 and Table 1)—developing reflective teachers (at the individual level) and developing shared vision (at the institutional level). However, we also recognized that there needed to be some individual focused prescriptive elements in years 1 and 2 of the project (Fig. 2). Emergent strategies fell into the category of reflective teaching, specifically treating teaching as a scholarly activity and employing faculty learning communities to support development of new pedagogical practice around QS development. In year 1, our efforts primarily would take place at each of the three anchor institutions. We envisioned faculty on a single campus engaging in discussion about the QS relevant for their courses and the types of materials they provided to help students learn/review and practice these skills within their disciplines. The year 1 discussions at the anchor institutions would seed discussions at other LACOL institutions in subsequent years (Fig. 3).

In the proof-of-concept project, we had developed a shared structural framework for the modules, and, in year 1, we planned for a round-robin discussion of QS development and usage in different disciplinary contexts to inform choices about the content of the modules. After a first round of these conversations and development of prototype modules with materials and examples provided by faculty from across the consortium, we planned for faculty members to use the modules and provide feedback that would then be used to refine the modules. This iterative process would allow us to refine modules until they met the needs of consortium faculty members and institutions. In addition, this approach would provide the opportunity to refine and solidify a shared vision for the goals and framing of the online modules. Cyclical collaborative design of curricular materials promotes faculty agency and, therefore, can be an effective tool in promoting professional development (Burrell et al., 2015; Vallis et al., 2022; Voogt et al., 2015), but we did not explicitly choose a theory-driven process for the round-robin discussions. The structure of these discussions was informed by conversations around interdisciplinary mathematics curricular reform work as part of the NSF-supported Synergistic Undergraduate Mathematics via Multi-Institutional Interdisciplinary Teaching Partnerships (SUMMIT-P) (NSF Award #18222451). This once again highlights a shortcoming as STEM practitioners—we grounded all of our decisions in research-based literature, but we did not ground every single element of our process in change theory literature.

In the first year after building modules, we anticipated that faculty participating in campus-based faculty learning communities (FLCs) would discuss how they were using the online modules as well as other support mechanisms to help students strengthen their quantitative skills in their courses. Because conversations at the first LACOL consortium-wide workshop and in the pilot program were so rich, we believed that the FLCs would lead to the formation of a Community of Practice dedicated to adding to each individual faculty member’s QS support toolbox.

In the second year, we had planned for cross-institutional FLCs involving faculty from multiple LACOL institutions. We believed that dialog with faculty members across campuses would catalyze changes in the ways faculty thought about supporting QS development. In addition, all participants—faculty in FLCs and project leaders—would gain insight into how different institutional structures and cultures impact faculty thinking about and approaches to students’ QS development. Although the first year of grant activities focused on interventions, a not uncommon scenario for STEM change agents according to Kezar et al. (2015), we did realize that we would need to accommodate the institutional culture on each campus to gain traction.

We had planned to probe environmental factors and their impact on change in the third year of the grant, through offering block grants to individuals to organize activities that they thought might encourage the adoption of the modules by new users or the adaptation of modules to better suit a particular context. By cataloging the types of activities that existing users proposed, including both intra- and inter-institutional projects, we hoped to learn about the departmental and institutional cultures that were impacting faculty choices around QS support. We were able to articulate the understanding that each campus had a very specific culture, including different approaches to supporting students and faculty development, varying levels of departmental autonomy, and differing degrees of openness to innovation. However, we made understanding each campus culture and its impact on faculty choices a goal of the project rather than trying to determine all of the campus contexts in advance of developing the project’s theory of change.

Although online modules were the product around which most of the interactions were designed, the focus of our work was not about the efficacy of online modules in supporting student learning or measuring student engagement with the modules. In fact, we took as a baseline previous research that online modules did help students’ QS in disciplinary contexts and we planned to incorporate some of those existing disciplinary materials into our modules. Our focus was on faculty, and we had two objectives: (1) to develop an interactive model for collaborative adaptation of existing, research-based but disparate online resources into coherent online modules, a model that results in a shared vision for the use and value of these modules and that meets the curricular needs of consortium faculty members and institutions; and (2) To understand what motivates or inhibits a range of faculty to adopt and adapt these online modules to support student QS development. Developing online modules as one additional tool in the toolbox to support student QS development was the means through which we hoped to understand what motivated faculty to collaborate on the development, adaptation, or adoption of curricular resources. Making this rationale for our work more explicit might have helped us advance our project more successfully.

Project implementation and pitfalls identified

Almost 2 years elapsed from the completion of the pilot project to when we obtained funding, so once funded, we started by considering the needs of various stakeholders. Although we were not as attuned to context in the development of our project’s theory of change as we should have been, we did launch our project with an effort to understand existing faculty practices and perceptions around QS support. In a September 2019 survey sent to 540 LACOL faculty members who taught introductory STEM or social science courses that used QS, 44% of the 220 faculty respondents indicated that they were using online resources (Eblen-Zayas et al., 2020). Among faculty members who were not currently using online resources to teach QS (56%), an overwhelming majority (80%) expressed interest in using online resources for this purpose in their introductory courses. This seemed like a promising level of interest from which to begin work.

Other responses on the 2019 needs assessment survey about existing modes of QS support should have caused us to think more concretely about the assumptions we were making in moving to our next steps. The most popular approach faculty respondents used to support students’ QS development was one-on-one meetings outside of class time (82%). Faculty also supported students through referrals to peer tutors (69%) or to textbooks and their associated resources (61%). Faculty were much more reluctant to refer students to online resources (44%) or to staff colleagues (37%). Consistent with the strong emphasis on high-quality teaching and individualized support at LACOL institutions, these results suggested that faculty were most comfortable with encouraging personalized support (individual meetings with the faculty member or peer tutors) or course-specific support (textbooks). Although there was interest in online resources, we should have seen the dedication to personalized support and course-specific resources as an indication that we needed to invest more time building an understanding of the benefits of using shared online resources (just-in-time, accessible 24 h a day, available to be revisited multiple times, facilitating transfer) in tandem with faculty members’ personalized approaches.

Using the information from the needs assessment, we identified three priority topics for module development (Eblen-Zayas et al., 2020), and then we carried out three rounds of round-robin dialogs, built modules, and then had faculty review them at workshops at the anchor institutions. The entire process included about 75 faculty members. The round-robin dialogs involved faculty on a single campus engaging in discussions around one specific QS, how it was relevant for their courses, and the types of materials they provided to help students learn, review and practice these skills within their disciplines. That information was then passed to facilitators at a subsequent anchor institution (who included the local project leader), and faculty on that campus could both provide their own thoughts and materials as well as react to the information that had been collected at the previous campus. As part of the dialog process, we were able to gather individual faculty reflections about how each QS topic was relevant to their discipline and, once the preliminary modules were built, about the modules themselves. In addition, we heard cross-departmental conversations at each institution about what pieces of a module would be useful to students in particular courses, and we could identify common themes that emerged across institutions. All of these conversations and perspectives helped project leaders further refine the modules.

While the round-robin dialogs and the workshops helped faculty understand how others might teach or ask students to use a specific skill, we did not get to a point where faculty participants understood that they were being asked to work together to develop approaches to teaching foundational QS in their own disciplinary contexts as well as more broadly. Perhaps, because both the round-robin dialog conversations and the workshops were not broad enough or because the project next steps were disrupted by the COVID-19 pandemic, faculty did not get beyond thinking about the mechanics of the materials and approaches they used to consider how multidisciplinary online modules might fit with existing approaches or how staff might encourage module use in other support contexts. Faculty conversations revolved around what each did in their classroom, but did not acknowledge that the approach of another could work in their context. In contrast, participants in the SUMMIT-P project gained meaningful movement toward supporting mathematics fluency across the curriculum using a similar construct (Bishop et al., 2020).

We planned to encourage testing and adoption of the modules and discussion about how they fit within other modes of QS support in two ways. In the first year, one group would use these modules in FLCs comprised of faculty from their own institutions and only the anchor institutions would participate. These FLCs would give faculty a chance to discuss both how they were framing and using the modules in their courses as well as engaging in a broader discussion of additional approaches to supporting student QS in their courses. Another group of faculty (from the non-anchor institutions) would use the modules and would develop their own framing but without engagement in an FLC. We envisioned that the FLCs would lead to emergent change by connecting module users across departments within the institution, and would be more likely to lead to continued adoption and adaptation as compared to users who did not participate in an FLC. Others have noted that a collaborative culture such as is often seen in FLCs can lead to more sustained and successful curricular reform (Chasteen et al., 2015).

The pandemic hit before we had fully refined the modules, and project work that involved organizing FLCs was put on hold. Two students continued module development in the summer of 2020, providing valuable student perspectives, and in the fall of 2020, we were able to have four faculty members from the anchor institutions test the modules developed from the dialog process but without support of the originally planned FLC. Of these faculty testers, all had participated in the dialog process and some had contributed material, except for one new assistant professor. In focus groups of the module testers, faculty reported that the modules supplemented and reinforced their in-class treatment of the focus QS. Faculty were pleased with the video curation of materials for the disciplinary-agnostic QS review section. In addition, faculty appreciated the contextualization of the practice problems. As one tester noted, “I really did appreciate the range of applications that I could point [students] to. And I appreciated the quality of the context. Sometimes you’ll see contextualized problems that are—they seem to be written by somebody who doesn’t understand the context.”

In trying to resume work on module development after the onset of the pandemic, we heard that faculty were tired and did not have the energy for engagement that they may have had in the past. Although pandemic-related disruptions did not allow us to carry through our work as we had originally intended, in reflecting on how far we got in implementing our project, we identified two major pitfalls—one related to how we had imagined collaborative development unfolding as compared to how faculty colleagues experienced collaborative development and one related to implementing the project at a consortial level—that might have been mitigated with a deeper knowledge of change theory.

Revisiting rationale and assumptions associated with collaborative development

Our approach to the QLAB project relied on the rationale that collaborative development of modules among faculty from different departments and institutions would give rise to emergent change in the QS support ecosystem. Although the rationale behind collaborative development was based on the literature (Burrell et al., 2015; Vallis et al., 2022; Voogt et al., 2015), we made the mistake of assuming that collaborative development of one particular kind of intervention to assist student QS development would challenge faculty to address their overall approach to this issue. Reinholz et al. (2021) emphasize the importance of bringing forward underlying assumptions when engaging in change efforts, and we did not articulate the assumption we were making about the connection between collaborative development of modules and broader consideration of existing approaches to supporting QS development.

As we began the round-robin discussions with faculty members in the first year, our focus was primarily on the development of online modules. We assumed the positive response to the online modules developed in the proof-of-concept project indicated that faculty would be interested in continuing to build on that earlier work. We wanted to build on the earlier efforts of the network of enthusiastic individuals who were engaged in the proof-of-concept project. We also assumed this network of individuals would grow to become a larger, consortium-wide, Community of Transformation through ongoing module development and cross-institutional FLCs. Because of these assumptions, we may have inadvertently caused those participants who were not part of earlier module development to see the project as prescribing change rather than engaging participants in the development of a shared vision of how a variety of QS supports, including online modules, could work in their classrooms. We had intended for discussions about multiple approaches to QS support to occur as part of the FLCs after the modules were built. Perhaps if the COVID-19 pandemic had not foreclosed this opportunity, the FLCs would have been successful in enhancing collaborative development and encouraging adoption of online resources, but perhaps more pointed discussions of how the modules might fit within existing approaches to QS support should have been foregrounded.

To some extent, this issue reflects the long timeline for project development (Fig. 1). Since the founding of the consortium, faculty and staff had been considering what online approaches could broaden the toolbox for supporting students, but only a handful of faculty who participated in the NSF-funded QLAB project had been a part of the earlier consortial discussions. In focusing on the module development as the first part of the QLAB project, we missed opportunities to engage in broader conversations about the many existing strategies for QS support for students and how online approaches might be a valuable addition. Appreciative Inquiry change theory (Cooperrider et al., 2008), which promotes change that builds on existing strengths, might have been valuable as we considered where to (re)start the on-going conversations and efforts once we received funding. Rather than kicking-off the round robin conversations with a focus on online modules specifically, we could have encouraged faculty to collectively take an inventory of all the approaches that they were successfully employing, as well as the success of the proof-of-concept project, and used that as the starting point for discussions.

Revisiting context and assumptions associated with working across departments and across institutions

The QLAB project developed in the context of a newly formed consortium of similar institutions to tackle a challenge that impacted multiple departments at each institution. We had planned a layered model to reach our goal of creating and then propagating the modules and expanding the related conversations. Initial module development was done by our round-robin discussions involving stakeholders at each anchor institution. We hoped this model would foster investment in the QLAB project generally, and module development specifically, by leveraging intra-institutional relationships to encourage conversations across disciplinary boundaries and, in the process, illuminating themes that might be common across different institutions.

Although the topics for the modules were chosen from responses to the 2019 needs assessment survey, the disciplinary audiences for each module topic did not necessarily overlap significantly. At each institution, no more than one or two faculty members participated in all module development conversations. After the round-robin conversations and the building of prototype modules, we held workshops on each anchor campus to provide faculty an opportunity to see modules that were being developed and provide additional input or contribute additional disciplinary examples. The workshops included between nine and 13 faculty participants, but at each institution, only two workshop participants also had participated in at least one round of the round-robin conversations. Thus, the conversations around module development included 75 faculty members from the three anchor institutions, but few of those faculty members engaged in the process in a sustained manner, and, as a result, a community of practice did not form around module development. When we decided to engage as broad a range of disciplines as possible by specifically choosing module topics that were relevant across many disciplines, we did not realize one outcome would be that faculty often only engaged in one discussion (for the module topic that was most relevant to their discipline) in the round-robin. In retrospect, the project would have benefitted from having a smaller range of disciplines engage in more of the round-robin discussions, because more of the topics were relevant to their discipline. As Reinholz et al. (2021) note, change efforts that employ FLCs often superficially draw on the Community of Practice literature. Our own lack of deep understanding of the Community of Practice change theory (Wenger et al., 2002) limited our ability to effectively design faculty engagement for module development and likely would have limited the success of the FLCs.

In the change theory literature, the Departmental Action Team (DAT) project has identified the department as an effective level to work at when it comes to making change. DATs work across all four quadrants of Borrego and Henderson’s (2014) typology of change. In examples of DATs (Reinholz et al., 2017), the department is usually large (maybe 30+ faculty) and the DAT might consist of 4–8 individuals who participate, focusing on an educational issue in the department. In LACOL institutions, departments tend to be small (sometimes there may only be four or five faculty members, and a large department might be 12 faculty members), and for a topic like QS in introductory courses, the issue is relevant across the curriculum. Students are often exploring the curriculum of the institution broadly and taking relevant courses before they have a departmental affiliation. Siloing the efforts within departments seemed counter-productive when the skill building was happening across the curriculum. We wanted to foster interdisciplinary conversations within institutions and disciplinary conversations across institutions, but we did not have a nuanced understanding of the variations in existing departmental and institutional cultures around change (Reinholz et al., 2019). Choosing to work with, and then connect, faculty members from a handful of departments might have been more productive than trying to appeal to the whole range of STEM and social science disciplines. A project that focused on developing quantitative biology modules for community college biology and math courses that occurred during a similar timeframe in the midst of the pandemic reported much more success in engaging faculty (Esquibel et al., 2023), suggesting a tighter disciplinary focus might have fostered more commitment.

Building on both Appreciative Inquiry, on which DATs are based, and the Community of Practice, on which FLCs are based, could have been valuable in developing or revising the theory of change for our project. In particular, the work of Reinholz et al. (2017) comparing how DATs and FLCs contribute to change might have been instructive in considering how we might have integrated elements of both. For example, our planned second year included supporting enthusiastic module developers and testers to facilitate discussions with others across the consortium about approaches to QS development and needed revisions to the modules with the goal of discovering and implementing those that would work in specific contexts. Perhaps cultivating this group of individuals so we had strategic representation from within a handful of departments would have been beneficial for promoting change. Recent research on team-based instructional change is exploring what makes teams successful, but interestingly, the definition of a team assumes that the individuals are working on change within a department or an institution (Sachmpazidi et al, 2021).

Returning to the idea of LACOL as a Community of Transformation, QLAB aimed to provide a peer-to-peer network for those faculty from across LACOL that shared the belief that students in introductory STEM and social science courses are more equipped to succeed when they have greater QS facility. We assumed the fact that the work was sponsored by the consortium would give individuals credibility within their home departments and reduce isolation. However, the COVID-19 pandemic limited our ability to grow the network. Although module testers were enthusiastic about the modules in their classrooms—both the quality of the material and the authenticity of the disciplinary practice problems—that positive response did not lead to testers recommending module use to others or engaging with the project in other ways. This could have been because testing took place during the pandemic when faculty were often not interacting much with colleagues in their departments or their institutions, or it could have represented a lack of strong affinity for online modules as a source of student support. The interruptions of the COVID-19 pandemic mean we will never know how the original project design would have played out if uninterrupted. More recently, we have found new ways to leverage individual campus leaders in engaging in cross-institutional conversations about institutional approaches to QS support, although the details of that are beyond the scope of this paper.

Conclusions

The QLAB project arose from a shared need within a consortium with a structure similar to, but not articulated as, a Community of Transformation. Consortial activities were designed to empower peer-to-peer learning and collaboration. The emergent nature of the QLAB project made it unclear who should lead the collective effort. Because the authors’ roles at our respective institutions allowed us to see a perspective beyond individual courses, we stepped in to lead the project, but we had not intentionally considered how we had influenced change in our institutions in the past. We read about change theories and strategies and designed our project by explicitly articulating the interventions and indicators that seemed to be a good match based on what we had learned from the published literature, workshops, and conferences. Although we were implicitly aware of context, we had planned to gain a better understanding of the contextual factors that impacted the interventions we chose as the project unfolded. The project would have benefited if we, as practitioners, had more explicitly articulated all the elements of the theory of change we were using, but it is unclear when we would have benefited most from a more explicit and strategic introduction to the change theory literature, and what would have been the best format for learning about existing change theories. A clear roadmap, such as Reinholz et al. (2021) prompts for framing the use for change theory according to context, indicators, and intervention while considering rationale and assumptions, would have been helpful.

We encourage change researchers and organizations promoting institutional change to more clearly consider the process by which practitioners can learn about and apply change theories. For example, although many grant solicitations ask for a project’s theory of change, prompts asking for specific discussion of rationale, context, interventions, and indicators could help practitioners articulate some of the implicit elements of a project’s theory of change. We also encourage additional consideration of when and how emergent grassroots efforts transition to a more structured undertaking. Not every effort begins as a formal project that articulates a theory of change. The switch from “let’s talk about changing the way we support X and try a few things” to “let’s think formally about how to change things and the associated strategies to use” can be fraught. Additional research on how these transitions occur would be helpful for practitioners, who often find themselves working on grassroots efforts that are initially catalyzed by their experiences inside and outside STEM higher education classrooms. Such efforts have the potential to transform departments or institutions if they can transition into more structured and intentional efforts. We strongly support Reinholz et al. (2021) call for additional research on how practitioners learn about change theory and how change theories apply at different stages in the lifecycle of a project, with a particular consideration of when in an emergent change effort one needs to have an explicit articulation of context, rationale and assumptions, interventions, and indicators.