Documenting professional learning focused on implementing high-quality instructional materials in mathematics: the AIM–TRU learning cycle

Abstract

Background

To increase teachers’ capacity to implement high-quality instructional materials with fidelity in their classrooms through a video-based professional learning cycle, the Analyzing Instruction in Mathematics Using the Teaching for Robust Understanding framework (AIM–TRU) research–practice partnership was formed. Drawing upon the design-based research paradigm, AIM–TRU created the initial design for the professional learning cycle and wanted to engage in continued iterative redesign as the year progressed. This necessitated a method, common among those who adjust their designs when applying them in context, by which to document and justify changes made over time to our model. The research contained in this article used qualitative methods to articulate and test the design underlying our professional learning cycle by advancing conjecture mapping, a device by which the embodiments of the design are made transparent to be analyzed in practice.

Results

The initial design conjectures and activity structures teachers engaged in through our model of professional learning were refined to address three themes that emerged. Firstly, it was found that the ways participants engaged with the mathematics of the lesson were underwhelming, in large part, because our own definition of what rich talk around mathematics should entail was lacking in details such as the mathematical objects in the lesson, the presence of multiple solution pathways, or the various representations that students could use. Second, talk structures did not always allow for equitable exchanges among all teachers. Finally, activity structures did not encourage teachers to delve deeply into the mathematics so they could perceive the lesson as a coherent piece of their own classroom curriculum. Our design conjectures and activity structures were revised over the course of the year.

Conclusions

Our use of conjecture mapping allowed us to address the concern with research–practice partnerships that they should develop and utilize tools that make the systemic inquiry they engage in transparent, allowing for other researchers, practitioners, and stakeholders to see the complete design process and make use of the findings for their local context. Implications for this process as a tool for those who pilot and scale professional development are raised and addressed.

Introduction

When the National Council of Teachers of Mathematics (NCTM) published the Principles and Standards for School Mathematics (PSSM) in 2000, it drew upon research to articulate a broad vision of what ambitious mathematical instruction should look like in classrooms across the United States. In particular, the PSSM outlined basic precepts fundamental to high-quality mathematics education, understanding that components such as curriculum, assessment, and equity are parts of a coherent whole (NCTM, 2000). These articulations focused on how students should be doers of mathematics in the classroom, deeply influencing the later development of the Common Core State Standards (CCSS) in 2009. Many believe that, based upon a more complete vision of instruction, the CCSS had the potential for meaningful improvements in the United States (e.g., Cobb & Jackson, 2011; Schmidt & Houang, 2012), especially if, as the PSSM explains, they are used in combination with well-designed instructional materials that keep the goals of these standards in mind (Coburn et al., 2016).

The use of instructional materials is of particular interest in effecting change; in the most recent National Survey of Science and Mathematics Education, it was found that around 76%, 65%, and 61% of elementary, middle, and high school mathematics classes, respectively, based their instruction on commercially published instructional materials at least once a week and that districts mandated those materials in approximately 90% of those classrooms (Banilower et al, 2018). These results show that, no matter the grade level, the reach of developed instructional materials is widespread.

Despite this widespread availability, instructional materials are rife with issues, especially in how they are being implemented in the classroom. Some textbooks are still misaligned with CCSS, stressing memorization and procedural knowledge over higher cognitive skills (Polikoff, 2015). However, what may be more alarming is that, even when the materials themselves use the recommendations of PSSM and embed activities that emphasize the construction of knowledge by students, their enactment by teachers in the classroom often undercuts the cognitive demand of the tasks, typically because the teacher is unaware why the task was built the way that it was (Heck et al, 2012; McDuffie et al., 2018). Improper materials and insufficient support for using instructional materials make it no surprise that, in the majority of mathematics classrooms at all grade levels across the United States, teachers put aside textbooks and other instructional materials to use units or lessons they have developed on their own as a basis for instruction at least once a week (Banilower et al., 2018). The basis by which teachers create or adapt instructional materials in ways that undercut the goals of instruction is not a problem in the United States alone; recent studies outside of the United States have also focused on the improper support systems for curricula that result in teachers developing and using their own instructional materials (Leong et al., 2019; Prediger et al., 2019).

A critical lens for professional learning should be in the ways it can support teachers in understanding high-quality instructional materials well enough to improve fidelity, both by increasing teachers’ likelihood to use the materials and to implement the materials as intended. This is the primary argument for developing educative instructional materials that target both student and teacher learning simultaneously (Davis & Krajcik, 2005; Davis et al., 2006). However, in practice, the results of teachers using instructional materials without ongoing supportive professional development are mixed at best. Even teachers who take the learning of instructional materials to heart during professional development may not implement those same materials with fidelity in their classrooms (Marco-Bujosa et al., 2017). To address this problem, recent work has suggested focusing less on supports within the materials and rather on developing professional learning opportunities that support teachers throughout implementation, primarily by providing spaces for teachers to engage in active reflection and adaptation of the materials themselves (Marco-Bujosa et al., 2017; Remillard, 2016; Shirrell et al., 2019).

As we look to address the issue of supporting teachers in implementing high-quality instructional materials, it is paramount to bring the researcher and practitioner perspectives together to gain a more complete picture of the problem of practice (Bevan & Penuel, 2018). Research–practice partnerships (RPPs) allow researchers and educators to collaborate while taking the realities of communities, schools, and classrooms into account (Campbell et al., 2019). In addition, a strength of RPPs is that they engage in design work that “develop[s] theories yet must place these theories in harm’s way” (Cobb et al., 2003, p. 10), holding researchers accountable for ensuring that an intervention works in actual classrooms and engaging in continued iterative redesign if it does not (Donovan & Snow, 2018). This strength is also a struggle for RPPs, as it requires a method by which those involved in the RPP can document and justify the changes made during the iterative design process (Coburn & Penuel, 2016).

To address the need to support teachers in better understanding and fully implementing instructional materials in the classroom, the Analyzing Instruction in Mathematics Using the Teaching for Robust Understanding framework (AIM–TRU) RPP was established in 2017. The research we have undergone has been situated in the design-based research (DBR) paradigm (Collins, 1992; Design-Based Research Collective, 2003), with the aim of increasing the capacity for use of high-quality instructional materials aligned with a framework that builds equitable, mathematically rich classrooms through professional learning housed in a community of practice. While recent work has demonstrated the impact the AIM–TRU learning cycle has had on the implementation of instructional materials in teachers’ classrooms (Russell, 2020; Russell & Wilson, 2021), the issue of understanding our iterative design process is the focus of this paper. In particular, we have worked to advance conjecture maps as a possible method for documenting and justifying the iterative changes that took place to our professional learning design over time.

The principal goal of this paper is to make transparent the joint work the AIM–TRU RPP engaged in to design and adapt to the central problem of practice, connecting the revision process of conjecture mapping to a qualitative methodology. We will illustrate in the sections that follow how RPPs can document learning while undergoing iterative design processes. In doing so, we keep within AIM–TRU’s general goals of advancing understanding around how to better design professional learning experiences that enable teachers to implement high-quality instructional materials with fidelity.

Background literature and conceptual framing

We draw from multiple bodies of literature frame our work: the role of RPPs in understanding problems of practice from different perspectives, what is understood about effective professional development, and how to tie effective professional development to high-quality instructional materials. We begin by introducing the RPP as our structure for understanding and approaching problems of practice using DBR and then move to examine what can be gleaned from the literature on professional learning centered around tenets of effective professional development and focused on high-quality instructional materials. Conjecture maps and their connection to DBR are introduced as a method for making our design choices clear before we summarize the literature review and expand upon the purpose for this paper.

Research–practice partnerships to understand problems of practice

RPPs are long-term collaborations between researchers and practitioners that focus on investigating and addressing problems of practice to improve educational systems (Penuel & Gallagher, 2017). In addressing problems of practice, the RPP centers on mutual goals, overcoming the traditional position of the researcher as expert directing others through the enactment of externally developed interventions. Instead, sustained interactions between researchers and practitioners occur throughout the design, implementation, and evaluation of the intervention (Campbell et al., 2017; Penuel, 2017). These sustained interactions result in a collective understanding of a problem, producing original analysis that benefits from the perspectives of all stakeholders (Penuel et al., 2013).

RPPs involve relationships that are productive to both practitioners as well as researchers, overcoming the issue of differing goals known as the research–practice dichotomy by engaging in collaborative work around jointly negotiated goals (Coburn et al., 2013; Penuel & Gallagher, 2017). For instance, Jefferson County Public Schools and researchers at Vanderbilt University engaged in an 8-year RPP in which the researchers supported partner districts in their work to improve middle school mathematics instruction, with a focus on developing students’ conceptual understanding and procedural fluency. Middle-School Mathematics and the Institutional Setting of Teaching (MIST) program engaged in a variety of activities, from classroom use of instructional materials to school-level professional learning communities and even principal development (Cobb & Jackson, 2011). This partnership was mutually beneficial, as district leaders gained feedback and guidance on how efforts were working, while researchers gained opportunities to study how to improve mathematics instruction at scale (Henrick et al., 2016).

In engaging in RPPs, DBR is often employed as an approach, because it centers design and research on producing outcomes that are both theoretical and practical, typically by seeking to learn how interventions work in real-life contexts (McKenney & Reeves, 2013). In this sense, DBR goes beyond merely designing and testing interventions to attempting to understand the relationships between theory and design and how they are enacted in practice (Design-Based Research Collective, 2003). The ability to gain the perspective of practitioners in constructing practical knowledge is an important reason that DBR benefits from the work that RPPs engage in (Bereiter, 2014).

In doing this work, the flexibility of RPPs as they adapt to better solve problems of practice is seen as both a strength and a focus for continued work. While a core part of the work of RPPs engaging in DBR is to shape the innovation through iterative design as more is learned about a problem of practice in and surrounding classrooms (Design-Based Research Collective, 2003), the tools that record these changes over time are underdeveloped and the focus of current research (Campbell et al., 2019). In particular, RPPs should find, develop, and utilize tools that make transparent the systemic inquiry in which they engage. This allows for other partnerships to see the initial design process, the method by which re-design occurred, and the information necessary to make use of these findings for the local context in which they are located (Coburn & Penuel, 2016).

Framing the design for professional learning

To create our initial design, we reviewed the literature on creating opportunities for powerful professional learning. For many years, the characteristics of effective professional development have been espoused to the point of cliché; effective professional development should focus on the practice of teaching through active learning, be of appropriate duration, form a coherent whole, and be aligned with reform documents (e.g., Ball & Cohen, 1999; Desimone, 2011; Garet et al., 2001; Putnam & Borko, 2000). More recent work has questioned or extended these features, specifically calling attention to the lack of classroom context as a place for substantive learning in standard professional development (Lampert, 2010). In particular, researchers have theorized the need for effective professional development to have an overarching goal of powerful disciplinary instruction that guides teachers in the analysis of practice with a content focus, to have a grounding in artifacts, such as student work or video, and to use instructional materials that serve the dual purpose of helping to teach the target content to students while serving as a source of investigation in the classroom (Roth et al., 2017, 2019).

Content focused analysis of practice

In their most recent review of emergent literature on professional development, the Learning Policy Institute’s first defining feature for effective professional development was that it be content focused. However, this focus on content is not in the way traditionally espoused as a tenet of effective professional development (Desimone, 2011; Garet et al., 2001). Specifically, effective professional development involves content in the way it “links content learning to pedagogies supporting teachers’ students and practice” (Darling-Hammond et al., 2017, p. 7). This view of content as connected to the context of teachers’ classrooms is important, as reviews done showing the effects of professional development focused purely on increasing teachers’ content knowledge without connections to pedagogical practice have shown mixed results, especially when evaluated at the student level (Garet et al., 2016). In informing our design for professional learning, we wanted to make sure that it gave teachers opportunities to see mathematics as more than a set of arbitrary rules and to identify productive patterns of mathematical thinking. To address this, we wanted to make sure that a core part of our professional development design provides opportunities for rich talk focused on the mathematics of the lesson, and specifically, that this talk allowed teachers “to prioritize and organize content so that students are introduced to big ideas rather than getting lost in a welter of details, and allows them to respond flexibly to questions raised by their students” (Schoenfeld & Kilpatrick, 2008, p. 322).

The use of video

Video has the ability to powerfully shape professional learning by situating teachers in the classroom without necessitating in-the-moment decision-making that can become a barrier for reflective practice (Sherin & Han, 2004). When done well, professional development utilizing video creates a shared experience and highlights aspects of classroom life that a teacher might not (and perhaps cannot) notice in the midst of carrying out a lesson (Erickson, 2007). In particular, using video clips to highlight interesting student and teacher conversations allows the focus of the professional development to become clearer, especially when the videotaped conversations address specific goals, are embedded within activities, and activate mathematical knowledge that can be used to orchestrate productive conversations (Carroll & Mumme, 2001). In addition, while the selection of video clips is a nontrivial matter, researchers and teachers can be positioned to do joint work bringing their respective expertise to co-create video cases that become the backbone of reflective professional development models (Borko et al., 2011; Roth et al., 2019).

Video cases are most powerful when combined with specific guidelines, questions, or protocols (Teuscher et al., 2016). In particular, Martinez et al. (2015) state that “the use of video should be accompanied with a specific framework targeting desired learning goals” (p. 76). This same research indicates that the use of video case studies, when analyzed with a specific framework for powerful instruction, can enhance teachers’ anticipation of students’ mathematical thinking. Our work builds on these ideas by coupling video case study analysis with a particular framework for effective mathematics teaching: the TRU framework (Schoenfeld, 2014). The TRU framework characterizes five dimensions of powerful mathematics teaching: (a) Mathematical Content; (b) Cognitive Demand; (c) Equitable Access to Content; (d) Agency, Authority, and Identity; and (e) Formative Assessment. The five dimensions delineated by TRU have been deemed necessary and sufficient to support high-quality instruction, especially as they relate to the vision of PSSM and CCSS (Schoenfeld, 2013, 2017).

Use of instructional materials

Content-focused and video-embedded professional development is most effective when the classroom materials it draws from are of high quality and able to elicit productive student conversations to investigate (Borko et al., 2011; Prediger et al., 2019). Inherent to our design for professional learning are embedded video cases centered upon the use of proven high-quality instructional materials. We chose the Mathematics Assessment Project’s 100 “Formative Assessment Lessons,” (FALS) which were developed to support the kinds of instruction proposed by the five dimensions of TRU (Schoenfeld, 2019). When these materials were evaluated by the National Center for Research on Evaluation, Standards and Student Testing (CRESST) in algebra classrooms in Kentucky, they found a positive impact on both teacher and student learning (Herman et al., 2015). CRESST also suggested areas for future improvement when using the materials; over half of the teachers saw the preparation required to teach the lessons as a major impediment to use, requiring more professional learning. In particular, teachers wished that there was professional development that focused explicitly on implementing the lessons in their classroom, especially around differentiation, facilitating discussion, and addressing student misconceptions (Research for Action, 2013). We saw the finds from CRESST as further illustrating the desire for our professional learning model.

Summarizing our system of professional learning and instructional materials

In creating our pilot, we looked to design a model of professional development that utilized similar work in video-based professional learning to disentangle important facets of knowledge about mathematical content as they present themselves in the classroom (Ball & Bass, 2000; Ball et al., 2008). In particular, we were interested in looking at models of adaptive professional development, such as the Problem-Solving Cycle, in which the goals and resources of the professional development are derived from the local context (Borko et al., 2011). While this style of professional development has been found to have potential for supporting learning and promoting changes in instructional practices (Seidel et al., 2013), a shortcoming that Borko et al. (2011) found is that the “selection of video clips is limited by the quality of both the lessons taught by participants and the video recordings” (p. 185).

By grounding our video cases in the FALs built using the TRU framework, we directly deal with the shortcomings that Borko and colleagues illustrate. The work of the lessons not only raises the floor for the quality of the lessons taught, but because the FALs focus on student engagement with a rich, challenging mathematical task, it is possible for the video cases to be composed entirely of episodes of mathematical talk by teachers and students as they are engaged in the lesson. In addition, as with many iterations of the Problem-Solving Cycle but unlike some other models, videotaping was not handled by the teachers, allowing them to focus entirely on enacting the lesson and reacting to student ideas.

Conjecture maps within DBR

For the purposes of this work, our design for professional learning and the research on it are intertwined, and we embraced the paradigm of DBR to frame our collective work. DBR recognizes that the design of learning environments is a theoretical activity that embodies hypotheses about how learning occurs within the local context and how that learning might best be supported (Cobb et al., 2003; Sandoval, 2014). As embodied hypotheses are put to the test within the local environment, the DBR paradigm considers how best to learn fast to improve (Bryk et al., 2015; Fishman et al., 2013), which led us to consider how to tie data collection and analysis explicitly to conjecture mapping.

Drawing upon the DBR paradigm requires the development and use of tools that keep five central characteristics in mind: researchers engaged in the DBR paradigm accept the intertwined nature of developing theories and designing the learning environment; continue through continuous cycles of design, enactment, analysis, and redesign; posit sharable theories that help communicate relevant implications to practitioners; account for how designs function in authentic settings; and rely on methods that can document and connect enactment to outcomes (Design-Based Research Collective, 2003).

In working to expand upon the methods used to make explicit the connection between enactment and outcomes and to communicate continuous cycles of design, Sandoval (2004, 2014) proposed the use of conjecture maps. Conjecture maps serve to specify the theoretically salient features of a learning environment design and illustrate how they are predicted to produce desired outcomes. This tool allows the connections between design and theory embodied in a designed learning environment to be reified before the beginning of a study (Boelens et al., 2020) and are an instrument that can be examined when going from one context to another (Wozniak, 2015).

Conjecture maps typically have the following components (Sandoval, 2014):

• a high-level conjecture, which serves as a hypothesis about how to support learning and implementation in the context of interest;

• embodiments, which are reifications of the high-level conjecture, that may be broken down into elements, such as the materials, task structures (the structures of the task learners are expected to do), and participant structures (the structures for how learners are expected to participate in the learning environment);

• mediating processes, which are the salient features of the design that lead to desired outcomes; and

• outcomes, which reflect a commitment that the designers should have to articulating the ways in which the high-level conjecture can be observed or measured.

The interactions between the embodiments, mediating processes, and outcomes are referred to as design and theoretical conjectures, which individually serve as the basis for testing and resultant redesign. The design conjectures propose how the embodiments lead to mediating processes, and the theoretical conjectures hypothesize how the mediating processes lead to certain outcomes (Sandoval, 2004, 2014).

Conjecture maps can also be revisited and revised over the course of the study as more is learned, addressing the iterative need of DBR (Campbell et al., 2019). Because a conjecture map breaks down a design and makes its embodiments and how these lead to theoretical processes and outcomes transparent within the specific setting, it is an ideal tool for those working within the DBR paradigm.

Summary and purpose

As an RPP, we understand that learning occurs as researchers and practitioners engage with a central problem of practice and that an effective RPP adapts their work as more is learned. The AIM–TRU RPP aspires to increase the capacity for use of high-quality instructional materials in the classroom by creating, testing, and revising a coherent system of professional learning centered on video cases that supports teachers in creating mathematically powerful learning environments. As we deepen our understanding about how teachers experienced this model of professional learning, we required a tool that we could revisit to make informed choices about where to redesign and why.

The design for our model of professional learning was based on characteristics of effective professional development as well as work within video-based mathematics professional development. Our design is based on a cycle of learning focusing on teachers’ analysis of practice through carefully selected video cases showing the implementation of high-quality instructional materials in the classroom. Using materials based on the TRU framework and designing activities for teachers focused on the five dimensions, this overarching conceptual framework for both the instructional materials and teachers’ analysis of practice helped to form a coherent vision for our design.

One way to improve the implementation of instructional materials is through a system of professional learning focused on the analysis of practice of collaboratively built video cases to understand how the teachers present can create mathematically powerful environments for all learners (Borko et al., 2011). To test this in our work using the DBR paradigm, we generated a conjecture map to codify the design and allow for qualitative data analysis. The goal of this paper is to advance the knowledge base around how RPPs can make transparent their revision process using conjecture maps to plainly communicate how they addressed the central problem of practice. Specifically, this paper describes our work within the DBR paradigm using a conjecture map to articulate and test the design conjectures underlying our professional development model to address the pressing problem of supporting teachers in understanding high-quality instructional materials to increase their implementation in the classroom with fidelity.

Methods

Local context

The following sections take care to explain the localized contexts and approaches that characterized the RPP’s work. We believe it is essential to paint a complete picture to ensure that others can take the full design into account when considering how it would apply in their own context (Tseng et al., 2017).

Research context

This research focused on work that took place over a 12-month period between June 2018 and June 2019, following earlier iterations of the professional learning design dating back to the beginning of 2017 at both Montclair State University and Math for America. This 12-month period included the initiation of the project through a summer of preliminary design when we looked at previous work done with teachers in New York City and Buffalo and redesigned for the local context of Math for America, where the majority of professional learning opportunities are facilitated by teacher leaders. Designing professional learning to be facilitated by teacher leaders gave us the chance to be explicit with our work, welcome more teacher voices into the partnership, and draw from and integrate recent research into the supports teacher leaders could call on when leading other teachers (Borko et al., 2017; Lesseig et al., 2017; Roth et al., 2017). Putting teachers in the position of leading the professional development also gave us the opportunity to engage in the work as participants rather than facilitators. The first author took the role of participant during the professional learning that occurred and debriefed at the end of each session with the two teacher leaders who facilitated.

During the 2018–2019 academic year, teachers met for 2 h monthly over the course of each semester. As the year progressed, the research team analyzed data and occasionally iterated upon the professional learning model explicated later in this paper. All sessions took place in a classroom in a central location in New York City.

Participants

The 19 teachers who were a part of the study, either as facilitators or participants, were all a part of the Math for America fellowship program. This fellowship program provides stipends and opportunities for professional learning to over one 1,000 math and science public school teachers in New York City to reward their work in the classroom. Being a part of the professional learning that occurred as a part of this study was one of many opportunities they were given. Of the 19 teachers who participated, 3 teachers worked with the research team to better understand the model for professional learning and lead the group over the course of the 2 semesters.

The teachers who were a part of this study had taught a minimum of 5 and a maximum of 27 years. Ten teachers taught high school, eight taught middle school, and one taught fifth grade. Together, they represented four of the five boroughs across the city.

Study design, data collection, and data analysis

Study design

We drew upon DBR and its tools, with the use of a conjecture map as a central feature of the professional development design process. The initial conjecture map is shown in Fig. 1, which gave the research team the base for designing and refining the professional learning environment. The conjecture map also informed the analytical framework we used to code interactions with teachers to improve the AIM–TRU professional learning model over the course of the year.

Fig. 1

Conjecture map that guided our early RPP work

As can be seen in Fig. 1, the high-level conjecture underpinning the work of the RPP was “to improve the implementation of instructional materials, teachers need professional learning that uses the classroom as a resource to understand how math learning occurs within the materials, as well as how they can use the materials to create mathematically powerful classrooms for all learners.” This high-level conjecture serves as a hypothesis about how to support learning and implementation in our context as well as how to overcome the research–practice dichotomy with regards to the adaptation of instructional materials in the classroom.

For the purposes of this paper, we pay attention to our design conjectures, which will become the base for our data analysis. As an example, a design conjecture we propose is that, within our professional learning model, places exist for rich talk focused on the mathematics of the lesson that is predicated on using the FALs based on the TRU framework (this conjecture is represented by a dashed line in Fig. 1). The use of individual design conjectures as units for analysis is not novel (Campbell et al., 2019), and we look to build upon this past work by codifying the conjecture map to test and revise design conjectures to drive change to the resultant professional learning model.

Based on our conjecture map as well as past work on professional development and adaptive video-based professional development in particular, the activity structures that formed our initial professional development model are found in Fig. 2.

Fig. 2

Initial AIM–TRU learning cycle

Participants engaged in the AIM–TRU learning cycle by focusing on one lesson and its associated video case. Each session started by grappling with the same mathematical task in the formative assessment lesson that videotaped students tackled (Box 1 of Fig. 2). The practice of doing the math before watching the video was similar to other video-based professional development models (Borko et al., 2011). After participants worked through the task and discussed the various solution pathways that they took and that students could take, they gained context for the video case, which was often taken from a participant’s classroom (Box 2). At this point, participants watched a short video clip (typically around 4 min) of students engaged in mathematical talk with partners or in small groups as they grappled with the task. The video case was discussed using one of the dimensions of TRU (Box 3) to give a perspective on the classroom that was not purely focused on the mathematics of the task. Finally, the participants and facilitators made a plan for the next session, which may have involved a teacher volunteering their classroom for study as they implemented a formative assessment lesson (Box 4).

We utilized the design conjectures in our initial model. Each activity structures involves the collective investigation of content and practice (participant structure) associated with formative assessment lessons (materials) through a professional development model involving video of students and teachers engaged in those lessons (task structure). These structures allow teachers to grow through conversations around the mathematics of the task and the ways in which they can implement the lesson using teaching practices aligned with a framework for powerful instruction (the mediating processes).

Data collection

This research relied on a combination of digital recordings of the professional development sessions, artifacts generated by the participants, and follow-up semi structured interviews as necessary. As the sessions met on a monthly basis with the lead author as a participant, there was adequate time for the research team to look at notes, review the videotapes, and start analysis that led to changes to the conjecture map and to the initial design for professional learning over the year. This work was done based on the data analysis outlined below.

The central problem of practice for the AIM–TRU partnership is how to increase teacher capacity to use high-quality instructional materials while aligning classrooms with the TRU framework. The primary goal of our research in this paper was to illustrate how we leveraged conjecture mapping to aid in the design and refinement of the professional learning environment to meet our problem of practice. Therefore, we limited the data collection for this element of our research to the ways that the learning environment itself allowed us to test and revise design conjectures to drive changes to the professional learning model.

Data analysis

To analyze the data, the first and second authors examined the digital recordings and artifacts of group meetings and then engaged in a close read of the transcripts, allowing for confirmation of the accuracy of the transcripts and as a first pass on the content of the sessions. Furthermore, the conjecture map came to be the basis for first-level coding using a provisional coding schema (Miles et al., 2014; Saldaña, 2015). The following is a brief summary of how coding was used in conjunction with the conjecture map:

• 1. The conjecture map served as an a priori coding scheme to help identify units of meaning. More specifically, any point when the participants were revealing connections between the embodiments (e.g., the adaptive professional development model) and mediating processes (e.g., rich talk focused on the mathematical meaning of the lesson) was considered a unit of meaning. Within a community of practice, these units of meaning typically involved the reification of ideas gained during participation by the teachers (Kirschner & Lai, 2007). Coding was first done independently by two of the authors, then discrepant data were analyzed collaboratively through a second analysis.

• 2. Units of meaning were clustered to form themes that could be used to confirm or revise the initial provisional codes, which in turn could be used to revise parts of the conjecture map.

• 3. As themes developed, they were used to either explicate and validate the design conjectures or reexamine, revise, and further iterate the conjecture map.

• 4. As the conjecture map changed, these changes were communicated to teacher facilitators, the professional development design was revisited, and revisions were made as necessary.

The advantage of this technique was that it allowed research and design to continually drive each other. This solved many of the pressing concerns associated with design-based research: it allows for continuous cycles of design, enactment, analysis, and redesign (Cobb, 2001); accounts for how the design functions in authentic settings; and perhaps most importantly, reveals the best way to “respond to emergent features of the setting” (Design-Based Research Collective, 2003, p. 6).

Findings

We organize our findings by the common activity structures of the professional development model seen across meetings (Fig. 2) as well as by the iterative revisions to these structures. We based the iterative revisions made to the model upon these findings and in consultation with the teacher facilitators. Participation within the activity structures was coded using the methods outlined earlier to reveal the findings related to the design conjectures (Fig. 1). Within each activity structure below, we present an example of talk typified during the structure, the design conjectures that primarily emerged through coding, and the revisions we decided upon based on the evidence.

One of our first noticings was that the design conjectures were not represented equally across the activity structures, as teachers participated in different parts of the professional development in different ways. As such, this gave the researchers an opportunity to use the activity structures to think carefully about individual conjectures, instead of making general statements about the conjectures writ large. As we looked at the participation of teachers within each activity structure, we were able to use the coding as a way to make reasoned changes to the AIM–TRU learning cycle and document change over time.

Structure 1: discussions focused on solving the lesson task

We looked at this activity structure understanding that merely doing the mathematical task together is not enough. It is important that the conversations around the mathematical task consider the mathematical skills, procedures, and concepts that students may apply; the mathematical reasoning that students may employ (correctly or incorrectly); and the affordances and constraints of different mathematical representations (Schoenfeld, 2017; Tekkumru-kisa, 2020). To support the facilitation of these discussions by teacher leaders at scale, the conversations were guided by a set of discussion questions the research team adapted from TRU materials focused specifically on thinking about mathematical content (Schoenfeld & the Teaching for Robust Understanding Project, 2016).

The discussion focused on solving the lesson task took place after the participants had grappled with the task on their own or in small groups (whatever form the lesson asked students to take) and included making their thinking visible on chart paper. As an example, in one of the sessions, teachers were investigating a video case centered on the Applying Properties of Exponents FAL. This lesson involves a card sorting activity in which students form groups of equivalent statements. In this sort, cards with a single term expression with one exponent (e.g., 2⁶) are to be matched with other expressions including multiple terms (e.g., 2³ × 2³) or single term expressions with more than one exponent [e.g., (2³)²]. The following exchanges show participants grappling with approaches students could take:

Danielle: We talked about different groupings or different ways to write a number. So, for the one that was 2⁶ and then (2³)² and 4³, we talked about how 2³ twice would end up giving you 2 groups of 2³. Which can be 2⁶. And how 4 can be broken down into 2². So, then you end up having 2² three times, which also gives you 2⁶.

Lisa: I’m going to share my mistake. And I've done this task. I have very tired eyes tonight I guess [laughter]. And I was so confident I'm going to take the easy one. We were actually following protocol. And we were going one at a time. And I matched 6⁸ divided by 6⁴ to 6². And so [laughter] I've done this before. And it was just so interesting. I was so confident. And then they were like, “Wait. Wait. Wait. I’m going to challenge that [laughter].” So, and then we had a discussion about why that was wrong. So, I could see students doing that.

Andy: To build on that, we noticed that 6⁴ and its partner quantity (3² × 2²)². We noticed that those were large numbers. And we used that strategy to eliminate others.

In analyzing conversations like these that took place around the task, the research team found the design conjectures connected to the mediating process of rich mathematical talk were apparent in some form throughout each session (shown in Fig. 3). For instance, in the exchange above, participants were engaged in mathematical talk that was grounded in the choices of cards offered by the lesson and were collectively investigating the mathematics to understand different solution pathways students could take while grouping the cards. However, the quality of the talk throughout each cycle of professional learning varied greatly in the degree to which teachers saw big ideas in a way that would allow them to respond flexibly to students when implementing the lesson in their own classroom.

Fig. 3

Elements of the conjecture map that show up in the first activity structure. The dashed lines indicate an element coding targeted for revision

In some sessions, the diversity of mathematical representations elicited would spur incredibly rich conversations around the solution pathways that students could take and the various ways to support them in the pathway they chose (for instance, a conditional probability problem based on whether someone has a false positive test inspired teachers to create tree diagrams, two-way tables, and sample spaces to supplement typical algebraic representations). At other times, and especially when the learning cycle employed fewer open-ended tasks (for instance, a card sort or domino matching activity), teachers were less likely to press each other to make the representations behind their thinking public, which did not allow for what we defined as rich talk within the group to occur.

In analyzing the video across the professional development, the research team noticed that this variance was due, in large part, to the ways in which our discussion questions were not fully eliciting conversations around ideas that would generate rich talk among our teachers. By looking closely at the diversity of conversations in which teachers were engaging, and the ways in which those conversations did or did not lead to rich talk, we revised our definition of rich talk within our conjecture map to focus more on what rich talk actually is: the diversity of mathematical objects, representations, and solution pathways that students can take to engage with big ideas (building on Schoenfeld, 2017). Once we expanded our definition of rich talk in the conjecture map, we worked with facilitators to adapt the discussion questions appropriately in our model as well.

Structure 2: Analyzing the episode of student mathematical talk

The design team was aware that the amount of information that can be gained from watching a video in the classroom has been compared to drinking water from a firehose (Erickson, 2007). To ensure that the discussions in this activity structure were focused, the facilitators used one of the dimensions of the TRU framework (Cognitive Demand; Equitable Access to Content; Agency, Ownership, and Identity; or Formative Assessment) to serve as a lens for the video using a set of discussion questions adapted from available tools (Schoenfeld & the Teaching for Robust Understanding Project, 2016).

For instance, the following exchange is an example of a conversation that took place during this activity structure. In the following exchange, two teachers are discussing why a breakdown occurred in the TRU dimension of Agency, Ownership, and Identity using a video case centered on students sharing their mathematical thinking while engaging in a lesson targeting representations of conditional probability. In this lesson, students try to figure out whether a game involving consecutive draws of white and black balls from a bag is fair.

Jaxn: That’s something that came up later on that they had to clarify to some people. They’re able to make a sample space of just different combinations of these balls but didn’t really link it back to well, what ball means what? Like maybe if they had a key from the beginning, then they would have been able to link that back.

Luis: Yeah. It’s like I could feel what he was saying was making sense to me. Although the diagram … like I feel like his thinking wasn’t represented on the diagram, but he had like a counterargument and he says, “Yeah, I know what you’re talking about with the first one being a 50/50 chance.” Then, he was going into that second point, proving that that’s the point where it’s unequal.

In analyzing conversations like this one, the research team identified the design conjectures found in Fig. 4 were apparent throughout this activity structure. In the exchange above, teachers were constructing an explanation for why a student’s thinking was not recognized by others through engagement in the video case as well an analysis of practice through the TRU framework.

Fig. 4

Elements of the conjecture map that show up in the second activity structure. The dashed lines indicate an element coding recommended to add

In early forms of the professional development, however, there were no processes for all participants to have the opportunity to contribute equitably to the conversation throughout any of the activity structures, as our coding demonstrated when we started not only looking at what was said, but who was saying it. The lack of procedure had a direct impact on whose voices were heard, which we noticed when the same people kept appearing in our coding. To build better conversations and opportunities for participants to grapple with the video that they saw, we used a common discussion protocol called a discussion diamond in this activity structure in which participants wrote answers to a prompt in separate corners of a piece of chart paper before coming to a consensus answer in the center (see Fig. 5).

Fig. 5

Discussion diamond

Adding these protocols helped ensure that the AIM–TRU learning cycle was giving everyone a voice during the sessions. In reflecting upon these decisions to revise our model, we realized there was a need to communicate this in our conjecture map, which we did by adding the following to the participant structure: Teachers participate through protocols that ensure rigor and equity. Specifically, we judged the protocols of the learning cycle as equitable and rigorous by making sure that they provided opportunities for all teachers to participate in the intellectual activities of the session, to be valued as important contributors to the group’s understanding, and to see themselves as full members of the learning community.

Structure 3: Adding the big mathematical picture

In the work of the previous activity structures, the design team, upon reflecting on early coding, tweaked the design of the professional development to strengthen the mediating processes of the conjecture map. A more substantive change to the design was realized when it was noticed that rich mathematical talk and TRU-supported explanations of practice were typically seen in the analysis of the task and the student discussions held in the video case and not in the ways that the lesson itself, when implemented well, fostered a productive, equitable, and mathematically rich environment for students (see the design conjectures represented in Fig. 6). In addition, even when the teachers were engaged in rich conversations, these were often of a fine-grained nature of what to do in individual scenarios and to overcome specific problems, and there was no space for “linking in meaningful ways to the big ideas behind them” (Schoenfeld, 2019, p. 7).

Fig. 6

Elements of the conjecture map that were the focus of the third activity structure

To more explicitly connect the circumstances the teachers were analyzing to the lessons themselves and situate the content of the lesson in larger mathematical ideas, the research team worked with teacher facilitators to add a precursor to solving the lesson task that was about forming the big mathematical picture for the lesson and placing it within the mathematical landscape that students would see over the course of the year. Now, at the start of each session, participants would pick a lens to generate perspectives on possible big mathematical ideas that were explored within the lesson. This lens could have been about the mathematical objects that were being explored, patterns that formed through the lesson, representations that students may be building, or the placement of the lesson in relationship to past and future knowledge. Once participants had thought clearly on their own about the lesson through that lens, they would talk to others to build a consensus answer, which would get charted and shared before a whole group discussion would look for larger mathematical ideas that connected these lenses.

As an example, the following episode occurred within the new activity structure in which teachers were placing the big mathematical ideas found in a high school lesson on “Representing Probabilities” within their own grade level.

Luis: Does anyone want to maybe weigh in on number two about the mathematical relationships or ideas that we really want students to understand given the goals that we wrote?

Nancy: I mean, we kind of discussed it in ours as like relationship between part to part to part and part to whole. Because I know that’s a huge understanding in seventh-grade math for proportional relationships. I don’t know. It’s something that came up.

Andy: I mean, I had sort of started an answer to number three and I said sort of a similar thing. But proportionality is actually huge like that’s what probability is. You’re being asked to think proportionally in order to understand—it’s sort of foundational, being able to have foundational— in percentages or another example of proportionality, proportional thinking.

Kevin: And I think something also based off of that was that you don’t always break up the whole all at once, which is, I think, something that is tricky to get.

To supplement the idea of a complete mathematical landscape, the other two activity structures had an extra discussion question attached that would return to the big mathematical picture and explore how engagement in the task and observations of the video gave them an additional understanding of the way that they could engage students in big mathematical ideas using this lesson.

Discussion

In this section, we first give the reader an overview of the context with which we thought about the codesign of the professional development to form critical experiences for teachers that increase their capacity for implementing high-quality instructional materials. This is followed by details about the ways in which our findings provide new insight related to the literature, our conjecture map, and other RPPs looking for methods to document the changes their design takes over time.

As alluded to in the introduction, we formed the AIM–TRU RPP to address the pressing problem of supporting teachers in their implementation of high-quality instructional materials while aligning classrooms with the TRU framework. Our model of professional learning uses analysis of practice through video cases to better understand how to implement instructional materials with fidelity. We took care to use principles of what constitutes good professional development (Garet et al., 2001) and specifically good professional development grounded in a framework that uses video (Roth et al., 2017). RPPs are charged with providing a way to bridge the research–practice dichotomy (Lampert, 2010) by affording teachers opportunities to investigate strategies for addressing educational challenges connected to their classrooms while also ensuring that the research done is timely, tuned to teachers’ local contexts, and leads to original analysis (Campbell et al., 2019). In addition, RPPs should develop and utilize tools that make the systemic inquiry they engage in transparent, allowing others to see the complete design process and make use of the findings for the local context in which they find themselves (Coburn & Penuel, 2016). Engaging with the DBR paradigm is one step RPPs can take in generating knowledge that is both theoretical and practical, but it is not enough without tools that reflect the perspectives of researchers and practitioners.

Using design principles for effective professional development, high-quality instructional materials, video cases based on the TRU framework, and a conjecture map, our current research identified and put in harm’s way the design and theoretical conjectures embodied in the AIM–TRU learning cycle. Throughout the first year engaging with the AIM–TRU learning cycle, we were able to use coding based on the conjecture map to better understand and revise activity structures that positioned and supported teachers in powerful learning to build their capacity to use high-quality instructional materials with fidelity. Our final revised conjecture map can be found in Fig. 7.

Fig. 7

Revised conjecture map

In thinking about revising the conjecture map based on a close analysis of these activity structures, we looked at one of the embodiments and one of the mediating processes that coding deemed problematic. The sole participant structure in our original conjecture map was a “collective investigation of content and practice utilizing the TRU framework,” which did the work of saying what participants were doing but not how they were engaging in activity equitably. For our work as an RPP, making sure that everyone was able to contribute became a core tenet. In particular, equity gave new meaning for us as coders, as the only way we could revise our model to best meet the needs of all of our teachers was if all of their voices were heard during each session of the professional development, echoing similar work connecting ideas of rigor, equity, and the importance of making thinking public in the design of learning experiences (Windschitl & Calabrese Barton, 2016).

We also found that our ideas of what composed “rich talk” as a mediating process were defined based on what they enabled instead of what they were and, in particular, the need for teachers to make their mental representations public for others. Revising our discussion questions to refer explicitly to seeing the diversity of mathematical objects, representations, and solution pathways pushed all of the participants to think deeper. As mathematical objects and representations were made public, this encouraged risk-taking as teachers would try alternate methods to see where they would go. As an example, when two teachers were presenting a representation that they used to explain conditional probability, the act of explaining the representation caused reflective discourse, as in the following situation:

Robert: Yeah, sure. So, it was interesting that we actually did two different methods. I liked using the table just so I could see the numbers very clearly, right, and you liked using the Venn diagram—

Laila: Yeah. So, I’m the person thinking in Venn diagrams [laughter].

Robert: So, it was funny because I’m listening to the explanation, and as you were explaining it to me, you were realizing that you had set it up incorrectly or you read it incorrectly. But the nice thing is−and I find this is always the case and I tell my students this−is that when you explain it to somebody else, you start realizing things just by just saying it. And I think that’s what−you started to fix your assumptions in fixing the actual Venn diagram.

As we worked to make teacher thinking apparent throughout the activity structures, it immediately caused the rigor of conversations to increase, as teachers at times struggled to voice thinking that they thought of as automatic when they had to slow down and explain their ideas to others. In one case, a card matching task in which operations with fractions were shown using various representations, one of the participants explained:

Kevin: We noticed ourselves, whether we were saying it or not, algebraically solving it and then sort of looking for a calculate−we weren’t even sure. I’m thinking of the [card for] what fraction of one-third is one-sixth? And we kind of knew intuitively that there’s one [sixth] shaded in and there’s one not, so it’s one half. However, it was pointed out that I had picked the wrong calculation even though I said the answer was one half. And so, I knew what the answer was, but I didn’t necessarily know what the calculation was that led to that answer.

The use of the protocol, with a better idea of what rich talk should look like, gave the design team a better envisioning of what the design conjecture could look like to result in rigorous talk among the participants.

Beyond what was found initially, what was equally informative about this research, especially as it relates to design-based research, is what was not found. More specifically, as early coding progressed, we realized that the design conjectures related to the instructional materials themselves were underrealized, especially in relation to mathematical talk placing the lesson within a landscape of learning for students. One of the strengths of the DBR paradigm is that it stresses methods that allow you to “respond to emergent features of the setting” (Design-Based Research Collective, 2003, p. 6). Instead of finding after the professional learning that teachers were defaulting to seeing the ideas students were bringing with a fine-grained lens, we were able to refine the design of professional learning during the course of the year. Our improvements to the model helped explain how we not only increased the odds that teachers would implement the materials, but that they would implement them well (Russell, 2020; Russell & Wilson, 2021). Codifying the conjecture map and resultant data analysis also allowed us to add an activity structure in the middle of the year and see, through continued analysis, how it helped teachers understand the ways they could support students in visualizing the mathematical landscape that the lesson is situated within. The revised AIM–TRU learning cycle with the additional activity structure can be found in Fig. 8. It should also be noted that, even within the activity structures that looked the same as in the original learning cycle, changes to protocols and discussion questions occurred in line with themes we noticed during coding.

Fig. 8

Revised AIM–TRU learning cycle

Conclusions and implications

RPPs must develop and utilize tools that make the systemic inquiry they engage in transparent, allowing for others to see the complete design process and make use of the findings for the local context in which they are situated (Coburn & Penuel, 2016). Using a conjecture map within the DBR paradigm, we have been able to tell the story of the AIM–TRU learning cycle and demonstrate how other RPPs can do the same with their own initiatives.

Conjecture maps are not novel as a method for designing learning environments (Boelens et al., 2020; Wozniak, 2015), which is unsurprising considering their stated purpose (Sandoval, 2004). Conjecture maps have also been a tool to demonstrate how the work of those engaging in the DBR paradigm have changed their approach to design over time (Campbell et al., 2019; Sandoval, 2014). However, because they do not always have an explicit tie to methods other than those used to evaluate outcome success, they have not been fully utilized to look at data collected in the moment to allow the RPP to jointly negotiate changes in design. By codifying the conjecture map, the AIM–TRU RPP was able to work more quickly to revise and implement changes than it could have had it waited until the end of the year.

An additional concern for all professional learning models is how to design them so that they are able to sustainably scale up (Fishman et al., 2013). In addressing this concern, being able to see the design conjectures in action is of paramount importance; otherwise, it would be difficult to realize the parts of the model that must be adapted because of the challenges and opportunities of the local context. In the year after this study, our RPP expanded and now encompasses three urban areas across the United States. We believe that the work we did under the DBR paradigm in this pilot year was critical to understanding what worked well, what needed to change, and how we could support the AIM–TRU learning cycle before it started to scale to other locations. In addition, it gave us a vocabulary to use across sites.

We hope that the conjecture map can continue to be explored as a tool that RPPs use to investigate the joint work that they are engaged in and to communicate what they have learned along the way. This paper focused primarily on codifying the conjecture map to study design conjectures and how they influenced the design of professional learning, but we also see the need for future work to consider how the theoretical conjectures can be similarly tested to understand the causal relationship between the mediating processes of a design and the ultimate outcomes that we use to evaluate success. In this manner, the conjecture map is able to tell the full story of the intervention.