Skip to main content

Developing student 21st Century skills in selected exemplary inclusive STEM high schools



There is a need to arm students with noncognitive, or 21st Century, skills to prepare them for a more STEM-based job market. As STEM schools are created in a response to this call to action, research is needed to better understand how exemplary STEM schools successfully accomplish this goal. This conversion mixed method study analyzed student work samples and teacher lesson plans from seven exemplary inclusive STEM high schools to better understand at what level teachers at these schools are engaging and developing student 21st Century skills.


We found of the 67 lesson plans collected at the inclusive STEM high schools, 50 included instruction on 21st Century skills. Most of these lesson plans designed instruction for 21st Century skills at an introductory level. Few lesson plans encouraged multiple 21st Century skills and addressed higher levels of those skills. Although there was not a significant difference between levels of 21st Century skills by grade level, there was an overall trend of higher levels of 21st Century skills demonstrated in lesson plans designed for grades 11 and 12. We also found that lesson plans that lasted three or more days had higher levels of 21st Century skills.


These findings suggest that inclusive STEM high schools provide environments that support the development of 21st Century skills. Yet, more can be done in the area of teacher professional development to improve instruction of high levels of 21st Century skills.


School-aged students in the USA are underperforming, particularly in science, technology, engineering, and mathematics (STEM) subjects. National Assessment of Educational Progress (U.S. Department of Education, 2015a) scores show that in science, only 34% of 8th graders are performing at or above proficiency and 12th grade students at or above proficient US students drop to 22%. Similarly, mathematics scores show 33% of 8th graders and 22% of 12th graders were at or above proficiency (U.S. Department of Education, 2015a). Additionally, the US mathematics scores for the Programme for International Student Assessment (PISA) for 2015 were lower than the scores for 2009 and 2012 (Organisation for Economic Co-operation and Development; OECD, 2018). US students not only underachieve in mathematics and science, but are also not engaging successfully in engineering and technology. At the secondary level, there are relatively few students in the USA that take engineering (2%) and computer science (5.7%) (National Science Board, 2016). The NAEP technology and engineering literacy (TEL) assessment found that for technology and engineering literacy, only 43% of 8th graders were at or above the proficiency level (U.S. Department of Education, 2015b). This consistent trend of underperformance has focused many national, state, and local efforts to improve student experiences in integrated STEM subjects (cf. President’s Council of Advisors on Science and Technology, 2010; Texas Education Association (n.d.) for school-aged students and beyond.

The efforts for improvement in STEM teaching in K-12 environments have yielded a slight increase in the enrollment of STEM majors recently (National Science Board, 2016). However, roughly half of students who declare a STEM major when entering college either switch majors or drop out of college (National Science Board, 2016). One approach to helping students persist in undergraduate education is a stronger foundation in content knowledge, academic skills, and noncognitive skills (Farrington et al., 2012). Academic skills, including analysis and problem solving skills, allow students to engage with content knowledge at higher levels of cognition. Noncognitive skills, including study skills, time management, and self-management, assist students in optimizing their ability to gain content knowledge and use their academic skills to solve problems. Students who possess these skills have high-quality academic behaviors, characterized by a pursuit of academic goals despite any setbacks (Farrington et al., 2012).

Because academic skills, noncognitive skills, and content knowledge have fluid definitions and may not be directly observable, for the purposes of this study we used 21st Century skills consisting of knowledge construction, real-world problem solving, skilled communication, collaboration, use of information and communication technology for learning, and self-regulation (Partnership for 21st Century Learning, 2016). Graduates who possess 21st Century skills are sought out by employers (National Research Council, 2013). In the environment of rapid advancements in technology and globalization, employees need to be flexible and perpetual learners in order to keep up with new developments (Bybee, 2013; Johnson, Peters-Burton, & Moore, 2016). There is a need to ensure that students who graduate the K-12 system are adept in 21st Century skills so that they can be successful in this new workforce landscape (Bybee, 2013).

Not only do 21st Century skills help students be successful in all areas of formal school, these skills are also necessary for a person to adapt and thrive in an ever changing world (Partnership for 21st Century Learning, 2016). One movement embracing the need for the development of student 21st Century skills is the proliferation of inclusive STEM high schools (ISHSs), schools that serve all students regardless of prior academic achievement (LaForce et al., 2016; Lynch et al., 2018). ISHSs promote student research experiences by using inquiry-based curricular models to scaffold independent learning and encourage personal responsibility (Tofel-Grehl & Callahan, 2014). The goal for ISHSs to facilitate this type of student-centered learning is to build students’ 21st Century skills such as adaptability, communication, problem solving, critical thinking, collaboration, and self-management (Bybee, 2013; Johnson et al., 2016; LaForce et al., 2016). Although there has been some evidence that not all ISHSs are advantageous in offering STEM opportunities (Eisenhart et al., 2015), there is an accumulation of evidence that ISHSs can increase college and career readiness for students from groups who are typically underrepresented in STEM careers (Erdogan & Stuessy, 2015; Means, Wang, Viki, Peters, & Lynch, 2016). As the number of inclusive STEM schools continue to increase across the USA, there is a need to understand the ways these schools successfully engage students in 21st Century skills. The purpose of this paper is to systematically analyze teacher-constructed lessons and student work from seven exemplar ISHSs in order to better understand how teachers are engaging and developing student 21st Century skills.

Specifically, this study looked at the extent to which teachers at these exemplar ISHSs ask students to practice the 21st Century skills and at the level of student performance of the following categories: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation (SRI International, n.d.-a; SRI International, n.d.-b). An examination of the lesson plans and student work products at exemplar ISHSs provides insight into effective development of student 21st Century skills in a variety of contexts.

Conceptual framework

In an attempt to clearly define the skills, content knowledge and literacies that students would need to be successful in their future endeavors, the Partnership for 21st Century Learning (P21; 2016) created a framework that includes (a) life and career skills; (b) learning and innovation skills; (c) information, media, and technology skills; and (d) key subjects (Partnership for 21st Century Learning, 2016). The first three parts of the framework, (a) life and career skills, (b) learning and innovation skills, and (c) information, media, and technology skills, describe proficiencies or literacies students should develop and can be integrated and developed in any academic lesson. The fourth piece, key subjects, suggests 21st Century interdisciplinary themes or content to engage students in authentic study (Partnership for 21st Century Learning, 2016).

Due to the need to build 21st Century skills, this study focused on the teaching and learning of (a) learning and innovation skills; (b) information, media, and technology skills; and (c) life and career skills at exemplar ISHSs. In order to operationalize and measure the three categories, we searched for instruments that measured the learning of 21st Century skills. Microsoft, in collaboration with SRI Education, developed two rubrics that are designed to assess the extent to which 21st Century skills are present in lessons and the extent to which students demonstrate the skills from these lessons (SRI International, n.d.-a; SRI International, n.d.-b). The 21st Century Learning Design Learning Activity Rubric examined the proficiency of teacher lesson plans for the development of 21st Century skills while the 21st Century Learning Design Student Work Rubric examined the level of competency for each 21st Century skill. Although the rubrics did not align exactly with the P21 Framework, we felt that there was enough alignment with the categories that the rubrics would be useful in measuring the extent to which lessons in ISHSs taught 21st Century skills and the extent to which students demonstrated these skills. The rubrics had the same categories for lesson assessment and student work assessment: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation in teacher lesson plans and student work samples (SRI International, n.d.-a; SRI International, n.d.-b). Table 1 shows how the categories assessed in the two rubrics align with the categories in the P21 Framework. Further, as we reviewed the literature on these categories, a model of their relationship emerged. Our literature review discusses the individual categories followed by the conceptual model of how these categories work together in 21st Century skill development.

Table 1 The alignment of 21st century learning components with the 21st century learning design rubrics

Knowledge construction

Knowledge construction occurs when students create new knowledge themselves rather than reproducing or consuming information (Prettyman, Ward, Jauk, & Awad, 2012; Shear, Novais, Means, Gallagher, & Langworthy, 2010). When students participate in knowledge construction rather than reproduction, they build a deeper understanding of the content. Learning environments that are designed for knowledge construction promote self-regulated and self-directed learners as well as building grit (Carpenter & Pease, 2013).

Although knowledge construction helps students to build deep understandings and skills to be self-directed and resilient learners, many students are unfamiliar with this approach to learning and frequently need scaffolding to take on joint responsibility of learning (Carpenter & Pease, 2013; Peters, 2010). When transitioning to a more student-centered learning environment that supports knowledge construction, the teacher becomes more of a facilitator rather than a lecturer (McCabe & O’Connor, 2014). A student-centered learning environment encourages students to shift from a paradigm of expecting one convergent answer and toward deeper meaning-making when learning (Peters, 2010). Knowledge construction anchors the development of 21st Century skills because students need to be able to have background knowledge in order to perform the skills in an authentic context.

Real-world problem solving

Sometimes called project-based learning (Warin, Talbi, Kolski, & Hoogstoel, 2016), real-world problem solving is characterized by students working to solve problems that have no current solution and where the students can implement their own approach (Shear et al., 2010). When solving a real-world problem, students work to identify the problem, propose a solution for a specific client, test the solution, and share their ideas (Prettyman et al., 2012; Warin et al., 2016). The design aspect of the process encourages students to be creative and learn from failures (Carroll, 2015). When using real-world problem solving, students develop knowledge in a meaningful way (White & Frederiksen, 1998), must regulate their cognition and behavior in a way to reach their goals (Brown, Bransford, Ferrara, & Campione, 1983; Flavell, 1987), and gain experience defending their choices through evidence and effective communication skills (Voss & Post, 1988).

Teachers can develop real-world problem solving skills in their students by modeling inquiry after research actual scientist are involved in, using databases with real-life data, and evaluating evidence from current events (Chinn & Malhortra, 2002). Designing real-world problem scenarios for the classroom provide a framework by which students can engage in 21st Century learning and can help to encourage a more positive attitude towards STEM careers (Williams & Mangan, 2016). Together, knowledge construction and real-world problem solving create the foundation from which students can engage in self-regulation, collaboration, and communication.


Self-regulation is a key 21st Century skill for independent learners. Students who are self-regulated plan their approach to problem solving, monitor their progress, and reflect on their work given feedback (Shear et al., 2010; Zimmerman, 2000). During the self-regulation process, a student motivates himself or herself to control impulses in order to efficiently solve problems (Carpenter & Pease, 2013; English & Kitsantas, 2013). Fortunately, these skills are teachable; however, students need time to accomplish regulatory tasks and guidance for the key processes of reflection and revision (Zimmerman, 2000). Therefore, long-term projects give a more appropriate time frame than short-term projects to hone these regulatory skills.

Students have different levels of self-regulation (English & Kitsantas, 2013) and teachers may need to integrate strategies and ways of monitoring students into lessons (Bell & Pape, 2014; English & Kitsantas, 2013). Incorporating self-regulated learning strategies helps students to stay engaged and deal with any adversity that may come up in the process (Boekaerts, 2016; Peters & Kitsantas, 2010). A tangible way teachers can support student self-regulation is by using Zimmerman’s (1998) four-stage model of self-regulated learning support: modeling, emulation, self-control, and self-regulation (Peters, 2010). First, teachers explicitly model the target learning strategy that the student should acquire, pointing out key processes (modeling). Second, teachers can provide students with verbal or written support for the key processes of the learning strategy while the student attempts to emulate the modeling from the teacher (emulation). Once students can roughly emulate the learning strategy, the teacher can fade support and have the student try to do the learning strategy on their own (self-control). After students attempt it on their own, the teacher provides feedback to the student to help them improve their attempted learning strategy (self-regulation). When a student can successfully perform the learning strategy on their own, they have become self-regulated in that aspect of their learning. Students who have mastered self-regulated learning have the ability to be proactive in knowledge building and in problem solving, which are characteristics that STEM industry employers value.


Collaboration occurs when students take on roles and interact with one another in groups while working to produce a product (Shear et al., 2010). Collaborative interactions include taking on leadership roles, making decisions, building trust, communicating, reflecting, and managing conflicts (Carpenter & Pease, 2013). Students who collaborate solve problems at higher levels than students who work individually because students respond to feedback and questions to create solutions that better fit the problem (Care, Scoular, & Griffin, 2016). Collaboration is an important skill to enhance knowledge building and problem solving. Conversations among peers can support student self-regulated learning through modeling of verbalized thinking.

Skilled communication

“Even the most brilliant scientific discovery, if not communicated widely and accurately, is of little value” (McNutt, 2013, p. 13). For the purpose of this paper, skilled communication is defined as types of communication used to present or explain information, not discourse communication. Skilled communicators present their ideas and demonstrate how they use relevant evidence (Shear et al., 2010). An important part of being able to communicate successfully is the ability to connect a product to the needs of a specific audience or client (Warin et al., 2016). In doing so, the students need to take into account both the media they are using and the ideas they are communicating so that it is appropriate for the audience (Claro et al., 2012; van Laar, van Deursen, van Dijk, & de Haan, 2017). Like collaboration, skilled communication is a necessary process to successfully employ knowledge construction and real-world problem solving.

Use of information and communication technology for learning

When students use information and communication technology (ICT) for learning, they are designing, creating, representing, evaluating, or improving a product, not merely demonstrating their knowledge (Koh, Chai, Benjamin, & Hong, 2015). In doing so, they need to choose how and when to use the ICT as well as know how to recognize credible online resources (Shear et al., 2010). The effective use of ICT requires self-regulation in order to use these tools independently and to keep up with technological advances. Given the continuous advancements in technology, it is essential that students know how to manage and communicate information in order to solve problems (Ainley, Fraillon, Schulz, & Gebhardt, 2016).

Conceptual Model of 21st Century Skills

The six 21st Century skills presented above are critical for students to develop to prepare for both college (National Science Board, 2016) and the future employment (Bybee, 2013; Johnson et al., 2016). Twenty-first century skills do not exists in isolation. By building one skill, others are reinforced. For example, knowledge construction and real-world problem solving can be enhanced by self-regulation. Likewise, collaboration requires skilled communication to build knowledge and solve problems. These skills coalesce to build the necessary toolkit for students who can learn on their own. Figure 1 shows a working hypothesis of how these six skills, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, interact to foster lifelong learning for student.

Fig. 1
figure 1

Working hypothesis of how 21st Century skills work together to build a 21st Century student

Knowledge construction and real-world problem solving are the keystones of the model and typically represent the two main goals of student-centered lessons. Knowledge construction is the conceptual formation while real-world problem solving represents the process skills that students are expected to develop. Knowledge construction and real-world problem solving feed each other in a circular fashion. Knowledge construction is built through the inquiry process of real-world problem solving. At the same time, real-world problem solving requires new knowledge to be constructed in order to solve the problem at hand. The connection between knowledge construction and real work problem solving is mediated by collaboration and communication.

While communication and collaboration allow a student to work with others to build their conceptual knowledge and work toward a solution to their real-world problem, self-regulation is an internal process that occurs simultaneously. The student’s self-regulation guides the student’s individual connections, reflections, and revisions between knowledge construction and real-world problem solving.

Information and communication technology provides tools for the students to facilitate communication and collaboration as well as other 21st Century skills. ICT helps to simplify and assist the communication and collaboration for groups of students. ICT can help streamline the process of analysis and record keeping as well as facilitating the sharing ideas with others. It allows students to more easily document their progress and express their ideas for later reflection. Although ICT is not directly connected with other elements in the model, the use of ICT allows for the learning process to be more efficient.

The six 21st Century skills addressed in this study, (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation, are important facets of STEM education. This study documented the extent to which each of the 21st Century skills were present in both lesson plans and in student work at seven exemplar ISHSs. Given that the schools in the study were highly regarded, understanding the structure and student outcomes of lessons could provide a model for teachers and teacher educators. With that in mind, the study was driven by the following research questions:

  1. 1.

    To what extent do teacher lesson plans at exemplar ISHSs exhibit 21st Century learning practices as measured by the 21st Century Learning Design Learning Activity and Student Work Rubrics?

  2. 2.

    Do teacher lesson plans and student work samples from exemplar ISHSs show differences in rubric scores by grade level?

During the analysis of these questions, a third research question emerged regarding the duration of lessons. The question and rationale can be found in the data analysis section.


This study is part of a larger multiple instrumental case study of eight exemplar ISHSs. The larger study (Opportunity Structures for Preparation and Inspiration in STEM; OSPrI) examined the common features of successful ISHSs (Lynch et al., 2018; Lynch, Peters-Burton, & Ford, 2014). OSPrI identified 14 critical components (CC; Table 2) that successful ISHSs possess (Behrend et al., 2016; Lynch et al., 2015; Lynch, Means, Behrend, & Peters-Burton, 2011; Peters-Burton, Lynch, Behrend, & Means, 2014). Three of the 14 critical components involve the application of 21st Century skills in the classroom. This study addresses these three critical components: (a) CC1: STEM focused curriculum for all, (b) CC2: reform instructional strategies and project-based learning, and (c) CC3: integrated, innovative technology use.

Table 2 The 14 critical components of an inclusive STEM high school

Cross-case analysis of the eight schools found similarities in how the schools addressed two specific critical components: CC1: college-prep, STEM focused curriculum for all and CC2: reform instructional strategies and project-based learning. From these two critical components, curriculum and instruction, four themes emerged: (a) classroom-related STEM opportunities, (b) cross-cutting school level STEM learning opportunities, (c) school-wide design for STEM opportunities, and (d) responsive design (Peters-Burton, House, Han, & Lynch, 2018). The theme of classroom-related STEM opportunities was characterized by the expectation that teachers act as designers of the curriculum and look beyond the typical textbook for resources. While designing the curriculum, teachers took a mastery learning approach and provided students multiple opportunities to master the material. Through the use of collaborative group projects, summative projects, culminating projects, and interdisciplinary studies, the schools demonstrated a cross-cutting school level approach to the STEM learning. School-wide STEM opportunities included a rigorous curriculum, incorporating engineering classes and/or engineering design thinking, emphasizing connections between the curriculum and real-world examples, as well as building strong collaboration between teachers. Finally, these ISHSs had systems such as data-driven decision making and supports for incoming ninth graders built into their schools as a responsive design. In summary, these schools worked to improve students’ 21st Century skill such as collaboration, problem solving, information and media literacy, and self-directed learning (Lynch et al., 2018).

Research design

This study was designed as a conversion mixed methods approach (Tashakkori & Teddlie, 2003) in that qualitative data were transformed into quantitative data using established rubrics. Document analysis was used as a tool to identify occasions of evidence within lessons plans and student work products related to the identified 21st Century skills (Krippendorff, 2012). In this conversion approach, the 21st Century skill demonstrated qualitatively in the documents was scored using the rubrics, ergo integrating qualitative and quantitative methods in the analysis.

Participating schools

The eight exemplar ISHSs for this study came from the same quintain as used by the OSPrI project (Lynch et al., 2018). Because this origin project was a cross-case analysis and the IRB did not allow for school to school comparison, the data collected from individual schools was aggregated as one data source. Protocol for inclusion in the OSPrI study was that the school had no academic admission requirements, self-identified as a STEM school, was in operation for grades 9 through 12, and intentionally recruited students typically underrepresented in STEM. For more information on the demographics of the schools and the selection process, see Lynch et al., 2018. Of the eight schools that were in the original OSPrI project, seven provided teacher lesson plans and/or student work samples during the school visit. All schools have given permission to use their actual names. The sample size from each school was inconsistent, therefore, we treated the data set as one combined group that included all seven schools.

Data sources

Student work samples and teacher lesson plans were collected during OSPrI site visits to the seven schools, which were each visited once between 2012 and 2014. Researchers requested paper copies of typical lesson plans and student work that resulted in an average performance from the lesson plan that was observed at all eight ISHSs during the site visits. Because this was a convenience sample, not all teachers submitted lesson plans, and only a few teachers submitted the student work products related to those lessons. Unfortunately, few parents consented to release student work products. As a result, 67 teacher lesson plans and 29 student work samples were collected from seven of the eight schools. We decided to keep the student work products in the descriptive portion of the analysis, but not the inferential analysis in the study because this is a unique opportunity to gain even a small insight into student work from STEM schools that were considered exemplary and served students who are typically underrepresented in STEM. Table 3 describes the content matter and grade level(s) associated collected teacher lesson plan and corresponding student work product.

Table 3 Number of student work samples and teacher lesson plans by content matter and grade level


Each teacher lesson plan was analyzed using the 21st Century Learning Design (21CLD) Learning Activity Rubric and each student work product was analyzed using the 21st Century Learning Design Student Work Rubric (SRI International, n.d.-a; SRI International, n.d.-b). These instruments were found to be valid and reliable for use in high school classrooms, and Shear et al., 2010 reports the details of the development and validation of the rubrics. Although the student work products were related to the teacher lesson plans, they were analyzed independently according to the protocol of the 21CLD rubrics. The 21CLD Activity Rubric and the 21CLD Student Work Rubric were designed by Microsoft Partner’s in Learning with a collaboration between ITL Research and SRI International (SRI International, n.d.-a; SRI International, n.d.-b). These two 21CLD rubrics were the result of a multi-year project synthesizing research-based practices that promote 21st Century skills (Shear et al., 2010). The rubrics, each 44-pages in length, are available online for public use ( The 21CLD rubrics assess teacher lesson plans or student work products on six metrics aligned with 21st Century skills: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation (SRI International, n.d.-a; SRI International, n.d.-b). Collaboration, knowledge construction, and use of ICT score ratings range from one to five while real-world problem solving, self-regulation, and skilled communication score ratings range from one to four.

Data analysis

The teacher lessons and student work samples were assessed on (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of ICT for learning, and (f) self-regulation using the 21CLD Learning Activity and the 21CLD Student Work Rubrics respectively. Examples of excerpts from teacher lesson plans and student work products for each category can be found in Table 4. Two raters were used to establish interrater reliability. Both raters have a background as secondary science teachers and were trained on the use of the rubric. One rater has a terminal degree in education and the other rater is a doctoral student in education. The two raters met and discussed the rubric scores until the interrater reliability was 100%. Once consensus scores were established, tests for assumptions, descriptive, and inferential statistics were run.

Table 4 Examples of teacher lesson plans as assessed with the 21CLD learning activity rubric

During the analysis of research questions one and two, unique trends of short-term and long-term lesson plans were noted. From this, a third research question emerged from the analysis:

  1. 3.

    Are there differences in the 21 CLD Learning Activity scores of short-term lessons and long-term lessons?

The 21CLD Learning Activity and the 21CLD Student Work Rubrics required a lesson to be long-term order to assess self-regulation. The rubric defined long-term as “if students work on it for a substantive period of time” (SRI International, n.d.-a, p. 32). From our reading of the lesson plans, lessons that were scheduled for three or more days met the criterion of a substantive period of time, while lesson that were scheduled for 1 or 2 days did not meet this criterion. For the purposes of this study, we decided to refine the definition of long-term to be a lesson lasting three or more class periods and a short-term lesson lasting less than three class periods. The analyses for all research questions separated lessons into long-term and short-term in order to clarify the category of self-regulation.


The data were checked for normality, skewness, and outliers; only the teacher lesson plans met all assumptions for an ANOVA (comparison of grade levels) and t test (long-term versus short-term). Due to the small number of student work samples collected (see Table 6), the data related to student work did not meet the assumptions needed to run a t test therefore was not included in this analysis.

Overall rubric scores

To answer the first research question, a descriptive analysis was run for each of the six categories on the rubric and the total score (found in Tables 5 and 6). The average score for all teacher lesson plans was less than 2 for all six categories (out of a total of 4 or 5). Likewise, overall student work sample averages scored below 2 except on the category of Knowledge Construction. Table 6 also shows the median score for long-term student work sample categories to better describe central tendencies of the data. Figure 2 shows the distribution of total rubric scores for all teacher lesson plans. Seventeen of the 67 lessons scored a 6, the lowest possible score. Only 16 of the 67 lessons scored higher than 13 points, half of the total possible points. Out of those 16 scoring over 50%, only three lessons scored 20 points or more out of the possible 27.

Table 5 Average rubric scores for lesson plans
Table 6 Average rubric scores for student work samples
Fig. 2
figure 2

Distribution of total 21CLD rubric scores for all lessons

Figure 3 illustrates the quantity of 21st Century skills found in each lesson. Nearly 75% of the teacher lesson plans included at least one 21st Century skill in the lesson and 67% addressed two or more 21st Century skills. Although most of the lessons at the ISHSs introduced multiple 21st Century skills, the overall scores for the quality were low.

Fig. 3
figure 3

Distribution of number of 21st Century skills addressed in a lesson

21st Century learning by grade

To answer the second research question, an ANOVA was conducted to compare lesson scores by grade level. There were no statistically significant differences between grade level scores for the total rubric score. Data were separated into short-term and long-term lessons by rubric category. There were no significant differences in short-term lessons by grade level (Fig. 4). However, there were significant differences across grades for long-term lessons. Total rubric score for grade 12 lessons were significantly higher than grade 9 (p = 0.023) and grade 11 (p = 0.032). Difference in total rubric scores for grade 12 lessons were approaching significance with grade 10 (p = 0.063). As seen in Fig. 5, category scores for long-term learning activities have small differences in 9th, 10th, and 11th grades but peaks noticeably in 12th grade. The exception to this trend is use of ICT which peaks in 11th grade.

Fig. 4
figure 4

The average rubric metric scores for short-term lessons, sorted by grade level for the lesson

Fig. 5
figure 5

The average rubric metric scores for long-term lessons, sorted by grade level for the lesson

Long-term versus short-term assignments

To answer the second research question, a t test with Bonferroni correction was performed to compare long-term and short-term lessons for each of the categories. A statistically significant difference was found between short-term (N = 35) and long-term (N = 32) lessons on total score, knowledge construction, use of ICT, self-regulation, and skilled communication (Table 7). The effect sizes for these categories as calculated by Hedges g (Lakens, 2013) were all above 0.8 indicated large effect size (Table 7). In all of those categories, long-term lessons scored higher than short-term lessons (Table 5). The category of real-world problem solving was approaching statistical significance with the t-score not showing significance [t = − 2.67, p = .001] but a statistically significant confidence interval [− 1.23, 0.003] and a medium effect size (Table 7).

Table 7 Independent samples test comparing short-term to long-term lesson plans


21st Century skills

Overall, the teacher lesson plans collected at the ISHSs showed evidence of addressing 21st Century skills. Nearly 75% of the lessons included at least one 21st Century skill with 67% addressing two or more. Although the majority of lessons addressed multiple 21st Century skills, the rubric scores for these lessons were low because they addressed these skills at a minimal level. For example, a minimal level of collaboration would be instructions to form a group. A high level of collaboration would include defining roles, explicit instructions on how to share responsibility, and evidence of interdependence. Only five lessons showed evidence of multiple 21st Century skills implemented at the highest level, as measured by the 21CLD Learning Activity Rubric.

While assessing the lesson plans, we noted that more explicit instructions in the teacher lesson plans would have resulted in higher rubric scores. Placing students in groups, structuring peer feedback, and having students design a final project for a particular audience are three small changes not seen frequently in the lesson plans that are articulated in the Lesson Plan rubrics to encourage multiple 21st Century skills. When students work in groups, they improve their collaboration and communication skills while constructing knowledge and solving problems (Care et al., 2016; Shear et al., 2010). When teachers incorporate peer feedback into their lesson, students engage in collaboration. Peer feedback also gives students the opportunity to revise their work based on feedback, increasing self-regulation (Shear et al., 2010; Zimmerman, 2000). When students design their final project for a specific target audience, rather than simply displaying their knowledge for the teacher, they work on their skilled communication processes (Claro et al., 2012; van Laar et al., 2017; Warin et al., 2016). In summary, placing students in groups, structuring peer feedback, and having students design a final project for a particular audience provides opportunities for students to practice 21st Century skills.

When lessons addressed more than one 21st Century skill, they usually demonstrated the use of collaboration or communication in real-world problem solving and knowledge construction (Care et al., 2016; Carpenter & Pease, 2013). Thirty-three lesson plans in which real-world problem solving or knowledge construction was evident, 31 showed evidence of collaboration or communication. Similarly, 13 of the 18 student work samples showed evidence of collaboration or communication when real-world problem solving or knowledge construction was practiced. The results from the indirect measures of the rubric build support for a conceptual model connecting the components of 21st Century skills (Fig. 1). There was some evidence demonstrating the support that collaboration and communication have for knowledge construction and real-world problem solving.

The findings of this study point to the likelihood of self-regulation being connected to other 21st Century skills. Each time self-regulation was present in a teacher lesson plan, there was evidence of at least one other 21st Century skill in that lesson. Seventeen of the 23 lesson plans addressing self-regulation included at least three other 21st Century skills, showing evidence that self-regulation is a skill that is related to knowledge construction and real-world problem solving. Our findings reflect the findings of other researchers, in that self-regulation guides the students’ individual connections, reflections, and revisions between knowledge construction and real-world problem solving (Brown et al., 1983; Carpenter & Pease, 2013; Flavell, 1987; Shear et al., 2010).

Evidence from the lessons showed that there was no consistent connection to the use of ICT and the presence of the other 21st Century skills. ICT was seen in both low-scoring lessons as the sole 21st Century skill, as well as in high-scoring lessons in tandem with multiple other 21st Century skills. As in our model, technology is a tool to help facilitate but is not necessary in the development of the other 21st Century skills (Koh et al., 2015; Shear et al., 2010). After examining the data, our model remained unchanged for all 21st Century skills and their relationship to each other.

Grade level differences

Overall, there were no statistically significant differences in the total 21CLD scores across grade levels. This is consistent with the missions of the ISHSs in this study to shift responsibility for learning to the students by weaving 21st Century skills throughout high school grade levels (Lynch et al., 2017). When looking at trends in long-term projects, there was a jump in total 21CLD score for 12th grade. Again, this aligns with the participating schools’ goals of creating an environment where students have a more independent learning experience during their senior year internships, college classes, and specialized programs CC1 (Lynch et al., 2018). This is consistent with the goal of many of the schools to have the students work independently during their senior year either by taking college classes, completing an internship, or taking a career specific set of classes.

Short-term vs. long-term lessons

The data showed that long-term lesson planning had significantly higher scores on the rubric as compared to the short-termed lessons. This difference is consistent with the literature regarding the need for students to have time to develop and practice skills (Lynch et al., 2017; NGSS Lead States, 2013). The extended time allows students to monitor and reflect on their progress while working toward self-regulation of the skill (Carpenter & Pease, 2013; English & Kitsantas, 2013). To truly become self-regulated, students need repeated supported attempts to be able to do it on their own (Zimmerman, 2000).

Although not significant, collaboration was the only rubric metric where the short-term lessons averaged a higher collaboration score than the long-term lessons. Evidence from the lessons show students worked in pairs or groups, but infrequently shared responsibility, made decisions together, or worked interdependently. This leads to the possibility that incorporating the higher levels of collaborations is difficult, even in long-term projects. In addition, evaluating the higher levels of collaboration is difficult to make based solely on documents. Observations would be required to evaluate how the students within the group were interacting with one another.


Because this study used data collected as part of a larger study, there were several limitations. The work collected is a snapshot of the work students were doing at the time of the observation and does not allow for a clear longitudinal look at student growth over time. As stated before, the small student work sample limited what we were able to do with the analysis.

By only analyzing paper copies of the student work, it was not possible to determine a true collaboration score for many of the projects. Higher levels of collaboration such as sharing responsibility, making decisions together, and working interdependently require observation or more detailed notes from the students or teachers. Some lessons may have scored higher in the metric of collaboration had the student interactions been observed or noted.


This study confirmed the presence of all identified 21st Century skills in the lesson plans at the selected exemplar ISHSs serving underrepresented students in STEM: (a) knowledge construction, (b) real-world problem solving, (c) skilled communication, (d) collaboration, (e) use of information and communication technology (ICT) for learning, and (f) self-regulation. In light of the patterns that emerged from the rubrics, we posit that in the lesson plans communication and collaboration are the core 21st Century skills that facilitate knowledge construction and real-world problem solving, while student self-regulation creates efficiencies resulting in improved knowledge construction and real-world problem solving. We also saw in the lesson plans that ICT provides tools to support communication and reflection which leads to knowledge construction and real-world problem solving. To further develop knowledge about how 21st Century skills addressed in lesson plans help to support student work, our model can be a hypothesized starting point to investigate interactions.

While teachers were successful at including 21st Century skills into lessons, very few lessons practiced higher levels of those skills. This could be an indication that high levels of 21st Century skills are difficult to teach explicitly at the high school level. Future studies may investigate why teachers are not frequently incorporating higher level 21st Century skills into their lessons to answer questions as to whether teachers feel that (a) they need more training on incorporating 21st Century skills, (b) students need more practice and scaffolding to build up to higher levels of 21st Century skills, or (c) they need more time for long-term projects to work on the higher level skills.

The use of the 21CLD rubric is a tangible way for teachers to self-assess the level of 21st Century skills in their lessons. Self-evaluation helps encourage reflection, promote professional growth, and recommendations for new aspects of lessons (Akram & Zepeda, 2015; Peterson & Comeaux, 1990). This can also help teachers make the instructions for the development of 21st Century skills more explicit in their lesson. In conducting a self-evaluation, teachers may realize that they do not have a deep understanding of the characteristics of 21st Century skills. If teachers are new to incorporating these skills into their lessons, the teachers may need time to learn the skills themselves before they can incorporate them into their lessons (Yoon et al., 2015). Further studies may examine how teachers use the 21CLD rubric to improve their lesson.

Students need time to grapple with and learn new skills (Lynch et al., 2017; NGSS Lead States, 2013). While we were able to see evidence of higher rubric scores for 21st Century skills for 12th grade students in the lesson plans, due to the convenience sampling of lesson plans and student work samples, we were not able to look at how students’ 21st Century skills were built over time. There is a desire to better understand how ISHSs successfully develop these skills. This includes how schools incorporate and build the 21st Century skills (a) within multiple lessons in one course, (b) across multiple classes over the course of a school year, and (c) throughout the students’ entire high school sequence. Future research may look at a longitudinal study that follows one student’s work over an entire school year to see how the 21CLD scores change. In addition, future studies may also look at how the short-term projects build the skills needed for the students to incorporate higher levels of 21st Century skills in long-term projects.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.



21st Century Learning Design


Critical component


Information and communication technology


Inclusive STEM high school


National Assessment of Educational Progress


Next-generation science standards


Opportunity Structures for Preparation and Inspiration in STEM


Partnership for 21st Century Learning


Programme for International Student Assessment


Science, technology, engineering, and mathematics


Technology and engineering literacy


  • Ainley, J., Fraillon, J., Schulz, W., & Gebhardt, E. (2016). Conceptualizing and measuring computer and information literacy in cross-national contexts. Applied Measurement in Education, 29, 291–309.

    Article  Google Scholar 

  • Akram, M., & Zepeda, S. J. (2015). Development and validation of a teacher self-assessment instrument. Journal of Research and Reflections in Education, 9(2), 134–148.

    Google Scholar 

  • Behrend, T. S., Peters-Burton, E. E., Hudson, C., Matray, S., Ford, M., & Lynch, S. J. (2016). STEM High School Inventory. [Measurement instrument]. Retrieved from

  • Bell, C. V., & Pape, S. J. (2014). Scaffolding the development of self-regulated learning in mathematics classrooms. Middle School Journal, 45(4), 23–32.

    Article  Google Scholar 

  • Boekaerts, M. (2016). Engagement as an inherent aspect of the learning process. Learning and Instruction, 43, 76–83.

    Article  Google Scholar 

  • Brown, A. L., Bransford, J., Ferrara, R., & Campione, J. (1983). Learning, remembering, and understanding. In P. H. Musen (Ed.), Handbook of child psychology (Vol. III, pp. 77–166). New York: Wiley.

    Google Scholar 

  • Bybee, R. W. (2013). The case for STEM education. Arlington: NSTA press.

    Google Scholar 

  • Care, E., Scoular, C., & Griffin, P. (2016). Assessment of collaborative problem solving in education environments. Applied Measurement in Education, 29, 250–264.

    Article  Google Scholar 

  • Carpenter, J. P., & Pease, J. S. (2013). Preparing students to take responsibility for learning: The role of non-curricular learning strategies. Journal of Curriculum & Instruction, 7(2), 38–55.

    Article  Google Scholar 

  • Carroll, M. (2015). Stretch, dream, and do—A 21st century design thinking & STEM journey. Journal of Research in STEM Education, 1(1), 59–70.

    Google Scholar 

  • Chinn, C. A., & Malhortra, B. A. (2002). Epistemologically authentic inquiry in schools: A theoretical framework for evaluating inquiry tasks. Science Education, 86, 175–218.

    Article  Google Scholar 

  • Claro, M., Preiss, D. D., San Martin, E., Jara, I., Hinostroza, J. E., Valenzuela, S., Cortes, F., & Nussbaun, M. (2012). Assessment of 21st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59, 1042–1053.

    Article  Google Scholar 

  • Eisenhart, M., Weis, L., Allen, C. D., Cipollone, K., Stich, A., & Dominguez, R. (2015). High school opportunities for STEM: Comparing inclusive STEM-focused and comprehensive high schools in two US cities. Journal of Research in Science Teaching, 52(6), 763–789.

    Article  Google Scholar 

  • English, M. C., & Kitsantas, A. (2013). Supporting student self-regulated learning in problem- and project based learning. Interdisciplinary Journal of Problem-Based Learning, 7(2), 127–150.

    Article  Google Scholar 

  • Erdogan, N., & Stuessy, C. (2015). Examining the role of inclusive STEM schools in the college and career readiness of students in the United States: A multi-group analysis on the outcome of student achievement. Educational Sciences: Theory & Practice, 15(6), 1517–1529.

    Article  Google Scholar 

  • Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners. The role of noncognitive factors in shaping school performance: A critical literature review. Chicago: University of Chicago Consortium on Chicago School Research.

    Google Scholar 

  • Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. Weinert & U. R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 21–29). Hillsdale: Erlbaum.

    Google Scholar 

  • Johnson, C. C., Peters-Burton, E. E., & Moore, T. J. (Eds.). (2016). STEM road map: A framework for integrated STEM education. New York: Routledge.

    Google Scholar 

  • Koh, J. H. L., Chai, C. S., Benjamin, W., & Hong, H. Y. (2015). Technological pedagogical content knowledge (TPACK) and design thinking: A framework to support ICT lesson design for 21st century learning. Asia-Pacific Education Researcher (Springer Science & Business Media B.V.), 24(3), 535–543.

    Article  Google Scholar 

  • Krippendorff, K. H. (2012). Content analysis: An introduction to its methodology (3rd ed.). Los Angeles: SAGE Publications, Inc..

    Google Scholar 

  • LaForce, M., Noble, E., King, H., Century, J., Blackwell, C., Holt, S., Ibrahim, A., & Loo, S. (2016). The eight essential elements of inclusive STEM high schools. International Journal of STEM Education, 3(21), 1–11.

    Article  Google Scholar 

  • Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863.

    Article  Google Scholar 

  • Lynch, S. J., House, A., Peters-Burton, E., Behrend, T., Means, B., Ford, M., Spillane, N., Matray, S., Moore, I., Coyne, C., Williams, C., & Corn, J. (2015). A Logic model that describes and explains eight exemplary STEM-focused high schools with diverse student populations. Washington DC: George Washington University OSPrI Project Retrieved from

    Google Scholar 

  • Lynch, S. J., Means, B., Behrend, T., & Peters-Burton, E. (2011). Multiple instrumental case studies of inclusive STEM-focused high schools: Opportunity Structures for Preparation and Inspiration (OSPrI). Retrieved from

    Google Scholar 

  • Lynch, S. J., Peters-Burton, E. E., Behrend, T., House, A., Ford, M., Spillane, N., Matray, S., Han, E., & Means, B. (2018). Understanding inclusive STEM high schools as opportunity structures for underrepresented students: Critical components. Journal of Research in Science Teaching., 55(5), 712–748.

    Article  Google Scholar 

  • Lynch, S. J., Peters-Burton, E. E., & Ford, M. (2014). Building STEM opportunities for all. Educational Leadership, 72(4), 54–60.

    Google Scholar 

  • Lynch, S. J., Spillane, N., House, A., Peters-Burton, E., Behrend, T., Ross, K. M., & Han, E. M. (2017). A policy-relevant instrument case study of an inclusive STEM-focused high school: Manor New Tech High. International Journal of Education in Mathematics, Science and Technology, 5(1), 1–20.

    Article  Google Scholar 

  • McCabe, A., & O’Connor, U. (2014). Student-centered learning: The role and responsibility of the lecturer. Teacher in Higher Education, 19(4), 350–359.

    Article  Google Scholar 

  • McNutt, M. (2013). Improving scientific communication. Science, 342, 13.

    Article  Google Scholar 

  • Means, B., Wang, H., Viki, Y., Peters, V. L., & Lynch, S. J. (2016). STEM-focused high schools as a strategy for enhancing readiness for postsecondary STEM programs. Journal of Research in Science Teaching, 53(5), 709–736.

    Article  Google Scholar 

  • National Research Council. (2013). Monitoring progress toward successful K-12 STEM education: A nation advancing? Washington, DC: National Academies Press.

    Book  Google Scholar 

  • National Science Board. (2016). Science and engineering indicators 2016. (NSB-2016-1). Arlington: National Science Foundation.

    Google Scholar 

  • NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.

    Google Scholar 

  • Organisation for Economic Co-operation and Development. (2018). PISA 2015 results in focus. Retrieved from

    Google Scholar 

  • Partnership for 21st Century Learning. (2016). Framework for 21st century learning. Retrieved from

    Google Scholar 

  • Peters, E. E. (2010). Shifting to a student-centered science classroom: An exploration of teacher and student changes in perceptions and practices. Journal of Science Teacher Education, 21(3), 329–349.

    Article  Google Scholar 

  • Peters, E. E., & Kitsantas, A. (2010). The effect of nature of science metacognitive prompts on science students’ content and nature of science knowledge, metacognition, and self-regulatory efficacy. School Science and Mathematics, 110, 382–396.

    Article  Google Scholar 

  • Peters-Burton, E. E., House, A., Han, E., & Lynch, S. (2018). Curriculum and instruction at inclusive STEM high schools. Journal of Research in STEM Education, 4(2), 193–212.

    Google Scholar 

  • Peters-Burton, E. E., Lynch, S. J., Behrend, T. S., & Means, B. B. (2014). Inclusive STEM high school design: 10 critical components. Theory into Practice, 53(1), 64–71.

    Article  Google Scholar 

  • Peterson, P. L., & Comeaux, M. A. (1990). Evaluating the systems: Teachers’ perspectives on teacher evaluation. Educational Evaluation and Policy Analysis, 12(1), 3–24.

    Article  Google Scholar 

  • President’s Council of Advisors on Science and Technology. (2010). Prepare and inspire: K-12 education in science, technology, engineering, and math (STEM) for America’s future. Washington, DC: Executive Office of the President.

    Google Scholar 

  • Prettyman, S. S., Ward, C. L., Jauk, D., & Awad, G. (2012). 21st century learners: Voices of students in a one-to-one STEM environment. Journal of Applied Learning Technology, 2(4), 6–15.

    Google Scholar 

  • Shear, L., Novais, G., Means, B., Gallagher, L., & Langworthy, M. (2010). ITL research design. Menlo Park: SRI International Retrieved from

    Google Scholar 

  • SRI International. (n.d.-a). 21CLD learning activity rubrics. Retrieved from

  • SRI International. (n.d.-b). 21CLD student work rubrics. Retrieved from

  • Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social & behavioral research. Thousand Oaks: Sage.

    Google Scholar 

  • Texas Education Association (n.d.). Texas science, technology, engineering and mathematics initiative (T-STEM). Retrieved from

  • Tofel-Grehl, C., & Callahan, C. M. (2014). STEM high school communities: Common and differing features. Journal of Advanced Academics, 25(3), 237–271.

    Article  Google Scholar 

  • U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015a). The nation’s report card: 2015 mathematics and reading assessments. (NCES No. 2015136). Retrieved from

    Google Scholar 

  • U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. (2015b). The nation’s report card: Technology and engineering literacy. (NCES No. 2016119). Retrieved from

    Google Scholar 

  • van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 77, 577–588.

    Article  Google Scholar 

  • Voss, J. F., & Post, T. A. (1988). On the solving of ill-structured problems. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 261–285). Hillsdale: Lawrence Erlbaum.

    Google Scholar 

  • Warin, B., Talbi, O., Kolski, C., & Hoogstoel, F. (2016). Multi-role project (MRP): A new project-based learning method for STEM. IEEE Transactions on Education, 59(2), 137–146.

    Article  Google Scholar 

  • White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16(1), 3–18.

    Article  Google Scholar 

  • Williams, P. J., & Mangan, J. (2016). The effectiveness of using young professionals to influence STEM career choices of secondary school students. Journal of Research in STEM Education, 2(1), 2–18.

    Google Scholar 

  • Yoon, S. A., Anderson, E., Koehler-Yom, J., Klopfer, E., Sheldon, J., Wendel, D., Schoenfeld, I., Scheintaub, H., Oztok, M., & Evans, C. (2015). Designing curriculum and instruction for computer-supported complex systems teaching and learning in high school science classrooms. Journal of Research in STEM Education, 1(1), 4–14.

    Google Scholar 

  • Zimmerman, B. J. (1998). Developing self-fulfilling cycles of academic regulation: An analysis of exemplary instructional models. In D. H. Schunk & B. J. Zimmerman (Eds.), Self-regulated learning: From teaching to self-reflective practice (pp. 1–19). New York: The Guilford Press.

    Google Scholar 

  • Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). San Diego: Academic Press.

    Chapter  Google Scholar 

Download references


Publication of this article was funded in part by the George Mason University Libraries Open Access Publishing Fund.


This work was conducted by the OSPrI research project, with Sharon Lynch, Tara Behrend, Erin Peters-Burton, and Barbara Means as principal investigators. Funding for OSPrI was provided by the National Science Foundation (DRL 1118851). Any opinions, findings, conclusions, or recommendations are those of the authors and do not necessarily reflect the position or policy of endorsement of the funding agency.

Author information

Authors and Affiliations



Both authors contributed equally to this manuscript. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Stephanie M. Stehle.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stehle, S.M., Peters-Burton, E.E. Developing student 21st Century skills in selected exemplary inclusive STEM high schools. IJ STEM Ed 6, 39 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: