Engineering design learning for high school and college first-year students in a STEM battlebot design project

Abstract

Background

There is a worldwide trend to include engineering design in high school curricula as a bridge course to higher-level STEM education and to increase high school students’ interest in STEM fields. This study used a battlebot design curriculum to compare engineering design learning between high school and college first-year students and then proposed suggestions for curriculum planning that promoted the continuity of learning between different levels of engineering design education.

Results

This study used the creative product analysis matrix (CPAM) and lag sequential analysis (LSA) to explore the possible similarities and differences between the two groups’ understanding of engineering design. The results show that college first-year students were significantly better than high school students in CPAM, but the two groups were similar in their reflections on engineering design behaviors, indicating that the noncumulative learning results must be taken seriously.

Conclusions

Higher-order engineering design thinking skills take a longer time to develop than technical skills. For both high school and college first-year students, it is important to enhance their higher-order engineering design thinking skills to promote higher engineering design performance. Moreover, high school students could be provided with convenient processing tools and easy-to-use, hands-on techniques to increase their technical skills. Educators from institutions of higher education and K-12 schools should work together to develop pedagogical models that provide rigorous, well-rounded education and outstanding engineering design instructions to most effectively cultivate STEM talent.

Introduction

Engineering design is a critical element of engineering education and a competency that students in STEM fields need to acquire (Atman et al., 2007; Lin et al., 2021). In addition to courses in higher education, STEM-based engineering curricula is promoted by many countries in K-12 education (Chien & Chu, 2018; Cruz et al., 2021) because engineering design experiences have played an increasingly substantive role in precollege students’ STEM education and career preparation in recent years (Crismond & Adams, 2012). K-12 education studies have found that technology-learning activities based on engineering design enhance STEM learning (Lin et al., 2021). For instance, one study reported students’ responses of designing and constructing a paper bridge that could withstand an optimal load. The results of the study showed that student sketches indicated an awareness of problem constraints, an understanding of basic engineering principles, and the application of mathematics and science knowledge (English & King, 2019). Engineering curricula help students effectively integrate STEM knowledge through the development of engineering designs, practices, and production skills (Bybee, 2010; Wicklein, 2006). STEM-based engineering design activities can help students form associations between theory and practice. This type of curriculum assists students in forming connections to their future careers, motivates them to learn, and facilitates their interest and integration of STEM subjects (Kwon, 2016).

Connecting individuals’ existing engineering practices to their formal experiences is quite essential (e.g., Wilson et al., 2013). As proposed by constructivist theories of knowledge, individuals gain new comprehension by building upon the know-how they already possess as well as their preconceptions (Bransford et al., 2000). If new content has few connections to what students already know, student motivation will be lower, and effective learning is less likely to occur (Eisenkraft, 2010). Thus, we suggest that investigating the understanding in engineering design among different levels of students are necessary, for they are the basis for increasing the learning connections between different levels of design expertise. Many studies have compared engineering design learning among different levels of college students and experts and highlighted key differences between these populations to draw attention to areas that may benefit from targeted instruction and to provide powerful representations of the findings for enabling continued discussion and reflection (Atman, 2019; Atman et al., 1999, 2005, 2007; Douglas et al., 2015). Few studies have focused on comparing K-12 education with college education (Lammi & Gero, 2011; Strimel et al., 2018). High school engineering curricula contain exploratory and preparatory elements that have been recognized as bridging courses connected to college engineering courses (Fan & Yu, 2017). It is essential to clarify the consistencies and discrepancies in the knowledge and skills of high school and college students receiving the same curriculum to improve the development of STEM-based engineering design curricula. Moreover, very few studies have explicitly operationalized or represented the most important characteristic of engineering design, the iterative nature (e.g., Adams & Atman, 2000), let alone how this understanding emerges at different levels of design expertise.

One primary objective of the Taiwanese technology curriculum throughout 12-year compulsory education is to ensure that students possess basic competencies in engineering design before they enter higher-level STEM-related institutions (Ministry of Education, 2018). Taiwan technology curriculum guidelines further indicate that engineering curricula in high schools should include thematic courses on mechanics and structure, electromechanical integration, computer programming, and robotics development with the incorporation of digital tools such as 3D printing and laser cutting machines to complete project-based engineering designs (Ministry of Education, 2018). Therefore, this study aims to develop a battlebot design course that follows the Taiwan K-12 technology curriculum guidelines to explore the learning of high school and college first-year students in regard to engineering design.

By understanding students’ initial engineering design competencies and how students with different education levels practice engineering design, this study makes it possible to explore the strengths and weaknesses of high school and college students regarding the integration of STEM knowledge into engineering design practice. It is possible to determine the expected design skill level for high school students entering college as well as to understand the path from novice to expert designers to reinforce the convergence of high school and college engineering. This can help to eventually foster a STEM-related workforce that meets national development standards.

Theoretical framework

Engineering design

Multiple researchers have attempted to define engineering design, so there are multiple definitions of it. Furthermore, engineering design is not a prescribed procedure but includes various open, fluid, deliberate, and iterative activities with the purpose of finding a solution that arises as a response to a need or problem. The iterative process differs by discipline and the type of project focus for professional engineers, and this variation is also seen in existing curricular approaches to supporting engineering design.

While there are a multitude of proposed definitions of engineering design, the educational community has sought to identify a number of common core behaviors of engineering design that are frequently performed iteratively. A previous study argued that engineers may need to define problems, generate solutions, and evaluate the solutions before integrating them back into the overall engineering design (Sheppard et al., 2009). Another study divided engineering design into five activity phases: problem definition, conceptual design, preliminary design, detailed design, and design communication (Dym & Little, 2013). The two studies mostly focused on activities related to the early stage of design concept development. The National Center for Engineering and Technology Education (NCETE) emphasizes eight essential activities of engineering design: need identification, problem/specification definition, search, design development, analysis, decision, prototype testing and solution verification, and communication (Childress & Maurizio, 2007). Hynes (2012) also proposed eight engineering design activities: identifying needs or problems, researching those needs or problems, developing possible solutions, selecting the best possible solution, constructing a prototype, testing and evaluating solutions, communicating solutions, and redesigning. These two studies included more design activities. Based on her research, which began in 1999, Atman (2019) analyzed the data from 177 individuals who solved 401 separate engineering design problems. Ten engineering design activities were identified: need identification, problem definition, gathering information, generating ideas, modeling, feasibility analysis, evaluation, decision-making, communication, and implementation. Comparatively, Atman provided a more holistic and precise scheme that included more detailed categories of design activities to assess students’ engineering design learning.

The engineering design process has critical effects on design quality (Adam & Atman, 2000; Sun et al., 2014). Thus, it is essential to clarify the iterative design processes that students at different educational levels follow and their misconceptions and inefficient habits of mind; thus, more assistance strategies for learners could be developed (Atman & Bursic, 1998; Crismond & Adams, 2012; Wind et al., 2019).

One of the common methods to examine individuals’ iterative design processes is the think-aloud protocol (Strimel et al., 2018). Its procedure is used to collect a person’s actions while solving a predetermined design task, along with their own verbal interpretation of their thought processes as they perform those actions (Atman et al., 2007). In addition to employing concurrent tasks, other types of think-aloud paradigms have also been proposed (Alemdar et al., 2017; Heras et al., 2020; Schindler & Lilienthal, 2019). For instance, Alemdar et al.’s (2017) think-aloud interview asked students to read and select a response while verbalizing their thinking. This method is valuable for measuring one’s understanding while responding to multiple-choice assessment items, but it might not be applicable when individuals engage in real design behaviors. Furthermore, Heras et al. (2020) arranged activities after STEM courses requiring the pupils to reflect on their learning throughout the courses with their team members. To examine a student’s learning process in solving a problem, Schindler and Lilienthal (2019) first recorded a video as the participant solved the problem, and then the participant was required to reflect on his thinking using this video, which induced a deeper level of recall. These methods all allow researchers to better understand what specific processes students engage in when they are designing their work.

Lag sequential analysis (LSA) (Bakeman & Quera, 2011) is a perfect technique to examine the data of the think-aloud protocol for revealing how students iteratively perform core design behaviors (self-iterative behaviors) and transfer core behaviors (sequential transition behaviors), given that the engineering design process is an activity that is frequently performed iteratively (Atman, 2019). The more significant the patterns of behavior that the students have in their self-iterative and sequential transition behaviors, the more complete the student's cognition of the design process will be. LSA identifies and visualizes learners’ behavioral trajectories by detecting whether the sequence between each behavior has been significantly achieved. It has often been used to investigate learners’ knowledge construction in a collaborative mode (Chiang et al., 2014; Hou, 2015; Lin et al., 2013) and involves recording behavioral performance and compiling behaviors into sequential, continuous, and time-series data. For example, Lin et al. (2013) adopted LSA to identify several patterns of knowledge construction behaviors in a collaborative augmented reality system and a traditional 2D simulation system.

Comparison of various engineering design educational levels

Learning must be based on previous experience (Bransford et al., 2000), so studies would need to target issues that bridge different learners in different contexts (Crismond & Adams, 2012; Strimel et al., 2018). These studies contain three categories and often explore the similarities and differences between different educational levels of students to propose teaching strategies to improve the learning performance of different levels of students.

The first category of the studies compared engineering design learning among different levels of college students and experts (Atman et al., 1999, 2005, 2007; Douglas et al., 2015). Studies found that the performance of college seniors was significantly better than that of first-year students in many design activities, such as gathering more information (Douglas et al., 2015), producing higher-quality designs, considering more alternative solutions, and transitioning more frequently between design steps (Atman et al., 1999, 2005). However, they also had similarities, such as the fact that they all incorporated a complete modeling process (Atman et al., 1999) and that they all allocated insufficient effort to the design activities of project realization and needed more attention to the final steps for the production of a quality product (Atman et al., 2005). Since one of the goals in engineering education is to qualify students to become future expert engineering designers (Carmona Marques, 2017), a study directly compared expert engineers and college seniors (Atman et al., 2007) and found that both groups had similar design quality and activity transitions, but the major differences between them were that the experts had better problem scoping and information gathering skills than the college seniors.

The second category of the studies, which were relatively few, compared the similarities and differences between K-12 and college students (Lammi & Gero, 2011; Strimel et al., 2018). Lammi and Gero (2011) conducted design cognition studies of two groups of students: high school juniors and seniors who had taken preengineering courses and sophomore university students in a mechanical engineering department. The results suggested that these students shared commonalities in design, although high school students rarely analyze their design ideas. Another study investigated the design cognition and performance results of secondary and postsecondary engineering students while they engaged in an engineering design task (Strimel et al., 2018). The results of their study revealed that students with secondary engineering experiences achieved significantly higher scores than those without for the given design scenario. Improved design performance was also found to be significantly correlated with more time spent employing the mental processes of analyzing, communicating, designing, interpreting data, predicting, and questioning/hypothesizing.

The third category of the studies investigated the design behaviors of K-12 students with or without engineering design experience (Grubbs, 2016; Kannengiesser et al., 2015; Mentzer et al., 2015). Mentzer et al. (2015) showed that the more experience engineering design students have, the more cognitive efforts they engage in for idea generation, feasibility analysis, and decision-making. Similarly, Grubbs (2016) identified that secondary students with engineering design experiences expended considerably more cognitive effort when proposing solutions to engineering problems than those without these experiences. However, Kannengiesser et al. (2015) found no statistically significant differences in design thinking between students with and without secondary engineering experience. We supposed that the inconsistency findings of these studies may come from whether educators explain and emphasize the engineering design process and related key activities in the curriculum and whether students focus only on the implementation of hands-on activities following the progress of the course and ignore understanding the design process.

In general, most studies confirmed that higher-level individuals in the learning of engineering design should be more familiar with and sophisticated in some design activities than lower-level individuals. However, as suggested by Grubbs (2016), more investigations of high school students’ understanding of the learning/design processes are needed (Grubbs, 2016). Hence, we seek to compare two learning groups to highlight the challenges and weaknesses that high school students will bring with them as college first-year students. The results of the aforementioned studies could be the basis for discussion after the comparison of the two student groups in engineering design learning. Therefore, college-level engineering educators should be aware of the background knowledge and skills that first-year students have so that educators and educational programs can best meet the needs of their students.

Creativity in the design process

Creativity often plays key roles in sophisticated processes in engineering design education (Dorst, 2011; Forbes, 2008). Several studies have examined how individuals’ creativity is influenced during the design-based learning process (Jia et al., 2021; Khamhaengpol et al., 2021; Ozkan & Topsakal, 2021; Shen et al., 2021). However, instead of examining how individuals creatively designed their works, these studies investigated individuals’ creativity through self-report questionnaires (Shen et al., 2021), divergent thinking tests (Ozkan & Topsakal, 2021), or merely the completeness of the products (Jia et al., 2021; Khamhaengpol et al., 2021).

To assess the creativity of products with accuracy and validity, the creative product analysis matrix (CPAM) proposed by Besemer and Treffinger (1981) has been a popular method for decades. CPAM could not only assist raters’ observation of products but also focus raters’ attention on product properties. Moreover, its usefulness has been confirmed (Kudrowitz & Wallace, 2013; O’Quin & Besemer, 2006; Wei et al., 2015).

CPAM is a three-factor model consisting of novelty, resolution, and elaboration/synthesis. Composing the three factors yielded nine facets (Tsai, 2016). First, novelty concerns the originality of the concepts underlying a product and the methods used to present it. It comprises two facets: originality and surprise. Originality evaluates the different degrees to which a student team’s design work compares to similar existing products, and surprise evaluates whether a student team’s design work has unexpected effects. Second, resolution concerns how well the product fits within its context. The factor is closely related to a product’s usefulness; logic, value, usefulness, and understandability are four facets under this factor. Logic evaluates whether the design or solution follows rational thinking. Value evaluates whether the work meets the design requirements. Usefulness evaluates whether the work or solution has obvious practical application effects. Understandability measures the degree to which the work or solution is self-explanatory and easy to understand. Third, elaboration/synthesis pertains to esthetic and stylistic perspectives. It comprises three facets, including organic qualities, elegance, and craftsmanship. Organic qualities evaluate whether the students’ design work is complete and running well, elegance evaluates the meticulousness of various aspects of the design work, and craftsmanship evaluates the maximum performance of the design work after adjustment and correction.

Methods

Since examining the product and process are two common approaches for assessing students’ understanding in engineering design (Wind et al., 2019), we developed a teaching experiment that incorporated a battlebot design course in this study. For the product, the performance of the student’s battlebot design was examined and served as the quantitative data. For the process, the student’s reflection practice was revealed and served as the qualitative data. We chose an embedded design approach (Creswell & Plano Clark, 2011) to provide a comprehensive picture of high school and college first-year students’ engineering design learning. In this approach, qualitative data were added to the quantitative strand to address different purposes in the study and enhance the overall design.

Participants

Using convenience sampling, 33 11th grade high school students taking a living technology course and 33 first-year university engineering students taking a basic design course were recruited as participants in this study. Since the learning of engineering design at the high school level was not emphasized in Taiwan until the “Curriculum Guidelines of the 12 Year Basic Education” were put into action at the end of 2019, the high school students in this study had never taken an engineering design-related course before this course. The college first-year students had taken engineering-related courses, including introductions to computers, computer image processing, and electronic circuits, and had experience conducting engineering design from these courses. Each student was free to join two other learners as a team, so 22 battlebot teams were formed from two courses, with 11 teams in each course. The students were informed that a battlebot design project was involved in the previously mentioned courses as teaching experiments for the present study. After the courses, the three groups with the highest CPAM scores in both courses participated in the activity of reflective practice. They were chosen because we supposed that these participants, who design battlebots of better quality, would be better able to observe their design activities and then reflect on them compared to the middle- and bottom-scoring teams, who might still be trying hard to manipulate the design tools. Furthermore, no reward was provided for participation, and students were free to inform researchers at any stage not to include their data in this study.

The battlebot design course

An 8-week teaching course on battlebot design was developed for this study. The course included knowledge related to mechanics and structure, electromechanical integration, and computer programming. Furthermore, 3D printers coupled with free 3D digital modeling software and laser cutting machines were used to assist in the production of prototypes. During the first 4 weeks, students were instructed to design the wheels of a battlebot and test the wheel function on various terrains. In weeks 5–8, students had to design the shape of a battlebot for the balloon-popping battle. In the 5-min balloon-piercing battle, each team had one balloon on their battlebot. They were asked to try and pop other teams’ balloons while keeping their own balloon from being popped. A team was disqualified from the battle if their balloon popped. The student team that remained in the battle with the most popped balloons won. The courses had two 50-min classes per week for a total of 800 min, which served as a project-oriented, hands-on curriculum for students to experience engineering design within specific technological settings.

Both courses were taught by the two researchers who authored this study to ensure the consistency of the teaching progress and quality. The engineering design involved in the courses was based on the study of Atman (2019), in which students experienced three stages of design activities. Table 1 shows details of the battlebot design curriculum, and Fig. 1 shows the mechanical parts of the battlebot and the class scenarios conducted in this study.

Table 1 Content descriptions of the battlebot design course

Fig. 1

Class scenarios conducted in this study

Data collection and analysis

Quantitative data: creative product analysis matrix

To investigate students’ understanding of engineering design, CPAM (Besemer & Treffinger, 1981) was adopted to systematically evaluate the students’ battlebot designs. Under the three factors of novelty, resolution, and elaboration/synthesis, nine variables were scored on a five-point Likert scale, where 1 represents a low score and 5 a high score. Two researchers were blinded to evaluate the high school and college first-year students’ works in terms of wheel design, balloon-popping device design, balloon protection device design, and body design. To estimate interrater reliability, the intraclass correlation coefficient (ICC) was adopted. The model assumed that one set of raters examined all designs in the dataset with two-way random effects (Shrout & Fleiss, 1979). The interrater reliabilities (ICCs) ranged from 0.86–0.99, demonstrating almost perfect agreement, according to Montgomery et al. (2002).

Qualitative data: reflective practice

As another way to investigate students’ understanding of engineering design, reflective practice was conducted for the three top-scoring teams after the battlebot engineering design course. Two researchers from the present study guided the reflective session. The students were asked to discuss the following aspects: the requirements of design activities to complete a battlebot design, the key activities required to perform engineering design, and how the design activities are related. Simultaneously, their discussions were recorded. Afterward, the students listened to their audio recordings as an aid for reflection and were asked the final question: “Is there anything that you would like to add or modify?” Researchers prompted and encouraged students to articulate their thought processes and engage in extended descriptions and discussions of their thinking with respect to the design activities without being concerned about the correctness of their answers. The prompting continued until the students had no further clarification or justification to offer for their responses. There are two reasons for adopting this approach. First, we collected the students’ reflective data rather than “actual” design behaviors since these high school students had never taken an engineering design course; thus, we believed it would be more reasonable to let them experience the design process at least once, without adding an additional cognitive demand of asking them to think aloud while engaging in design or the possible pressure that could come with video recording. Therefore, rather than capturing their design processes in real time, we asked the students to reflect on their design processes. Second, we interviewed the students together rather than individually because they participated in the 8-week course and designed their work with their team members, making it easier for them to reflect on their design processes together. Furthermore, familiar people and contexts could reduce their anxiety during reflective practices. Each team had an approximately 15-min reflective practice. The reflective practices were audio-recorded, transcribed into verbatim manuscripts, and then coded to investigate students’ behavioral patterns.

The students’ responses to reflective practice were analyzed using LSA (Bakeman & Quera, 2011) to examine their collaborative engineering design behavior and to explore the differences between high school and college first-year students in the construction of reflection on engineering design. Three steps were included: establishing a coding scheme, defining behavioral codes, and analyzing time-series data (Bakeman & Gottman, 1997). For the first two steps, after adapting Atman’s (2019) coding scheme and behavioral codes, ten engineering design behaviors belonging to three major design stages were identified: (1) problem scoping—identifying needs, problem definition, and gathering information; (2) developing alternative solutions—generating ideas, modeling, feasibility analysis, and evaluation; and (3) project realization—decision-making, communication, and implementation. Table 2 shows the coding scheme for the engineering design behaviors. Two researchers completed the coding of student teams’ dialogs, and the interrater kappa reliability coefficient was 0.85 (p < 0.001) between the two coders, showing a high consistency.

Table 2 Coding scheme of engineering design behavior

For the last step, analyzing time-series data, the frequency and ratio distribution of students’ behaviors were analyzed, and LSA was conducted to obtain one-way z scores that were used for the analysis and explanation of the continuous significance of each sequence on the students’ reflection on their various behaviors. A z score higher than 1.96 indicated that two analyzed behaviors had a significant sequential relationship (Bakeman & Quera, 2011). When the z score was higher than 1.96 for the correlation between the same behavior in a sequence, it was defined as a significant self-iteration behavior. When the z score was higher than 1.96 for the correlation between different behaviors, sequential behaviors were confirmed. Finally, a behavioral transfer diagram of the statistically significant sequential relationships of the engineering design behaviors based on the z scores was produced.

Results

Student’s creative product

Table 3 shows the means and standard deviations of the CPAM scores for the two student groups. The Mann–Whitney U test (α = 0.05) was applied to find significant differences in the scores between the two student groups, and the results showed that college first-year students’ scores were better than those of high school students in all three dimensions.

Table 3 Means and standard deviations of the CPAM scores for the two student groups

The dimension of novelty contains two variables: originality and surprise. The college first-year students (Mdn = 4.50, mean rank = 16.32, n = 11) had significantly higher scores in the dimension of novelty than the high school students (Mdn = 3.00, mean rank = 3.00, n = 11), U = 7.50, z = – 3.59, p < 0.001, r = 0.76, indicating a large effect size. According to Cohen’s (1988) conventions, small, medium, and large effects are defined for r values of 0.1, 0.3, and 0.5, respectively. Figure 2 shows one high school student team’s design work and one college first-year team’s design work. The high school students’ design work had some variability in the shape of the wheels, and there was no special body design or decoration. The balloon-piercing and protective devices only used bamboo sticks, which were provided by the teacher. Consequently, none of the physical features of the high school students’ design work was a particularly novel design idea. College first-year students focused more on the overall body design by using 3D printed parts to cover the mechanical components and create more complicated and delicate artwork.

Fig. 2

Comparison of the novelty of the design works of a high school students and b college first-year students

In the dimension of resolution, the four variables of logic, value, usefulness, and understandability were included. The college first-year students (Mdn = 4.50, mean rank = 15.64, n = 11) had significantly higher scores in the dimension of resolution than the high school students (Mdn = 3.63, mean rank = 7.36, n = 11), U = 15.00, z = – 3.02, p = 0.003, r = 0.74, indicating a large effect size. Figure 3 shows one high school student team’s design work and one college first-year student team’s design work. The high school students’ works were straightforward and function-oriented for piercing others’ balloons and protecting their own balloons. Most of the high school student design works in this study used a similar strategy by attaching bamboo sticks all around the body of the battlebot, which displayed a convergence in the functional design of the battlebots. Moreover, the high school students’ design work had incomplete wheel functions for handling different terrains, and the wheels could fall off the battlebot during battle. In contrast, college first-year students used bamboo sticks as a device for piercing balloons in the design work, and other materials were also used to form the body shape, which not only acted as bionics but also created specific functions for protecting balloons. This design strategy of “form follows function” made the shape of the battlebots functional and easy to understand. Furthermore, the wheels designed by college students rolled on different terrains more smoothly than those designed by high school students. Overall, college students were significantly better than high school students in the dimension of resolution when designing battlebots.

Fig. 3

Comparison of the resolution of the design works of a high school students and b college first-year students

The dimension of elaboration/synthesis contains three variables: organic qualities, elegance, and craftsmanship. The college first-year students (Mdn = 4.50, mean rank = 17.00, n = 11) had significantly higher elaboration/synthesis scores than high school students (Mdn = 3.00, mean rank = 6.00, n = 11), U = 0.00, z = – 3.99, p < 0.001, r = 0.85, indicating a large effect size. Figure 4 shows one high school student team’s design work and one college first-year student team’s design work. The design work of high school students was in poor processing condition. There was a considerable amount of melted hot glue on the wooden boards, and the combined method using tape was also very rough. The college first-year students’ design works were not only handled properly regarding the combination of different materials but also left no obvious traces of processing. The designs were more complete and functioned well. In this regard, it can also be stated that high school students were not sufficiently mature in their production and integration skills, whereas college students were able to use composite materials and processing methods appropriately, were more proficient in production skills, and presented more complete design work.

Fig. 4

Comparison of the elaboration/synthesis of the design works of a high school students and b college first-year students

Reflective practice of engineering design behavior

The audio recording files collected 181 and 205 dialogues from high school students and college first-year students, respectively. In total, 386 dialogues were coded as engineering design behaviors based on Atman’s work (2019).

The frequency of the engineering design behaviors is shown in Fig. 5. The overall engineering design behavior frequency distributions of the two student groups were similar. The design behaviors were mostly concentrated on the design activity of modeling (above 30%) and less active in decision (DEC), communication (COM), and implementation (IM) (under 5%).

Fig. 5

The frequency of engineering design behavior of the two groups

Table 4 shows the percentages of self-iterative and sequential transition behaviors for high school and college first-year students. Based on the z scores, the behavioral diagrams of the two groups’ engineering design behaviors are illustrated in Fig. 6.

Table 4 Percentages of the self-iterative and sequential transition behaviors for high school and college first-year students

Fig. 6

The behavioral transfer diagrams of the two student groups

Ten behaviors were observed in this study, which means that there should be 100 specific engineering design behaviors, including 10 self-iterative behaviors and 90 sequential transition behaviors. Only 51 behaviors were coded among the high school students, and 50 behaviors were coded among the college first-year students, meaning that nearly 50% of the behaviors were not recorded for either group of students.

Among high school students, eight self-iterative behaviors were significant, the self-iterative behavior in evaluation was not significant, and the self-iterative behavior in decision-making was completely missing. Only two sequential transition behaviors of EVAL → GATH and GATH → PD were significant. With similar amounts of self-iterative and sequential transition behaviors, college first-year students showed seven significant self-iterative behaviors, though the self-iterative behaviors of gathering information, decisions, and communication were not significant, and three sequential behavior transfers of COM → GEN, MOD → FEAS, and DEC → IM were significant.

Discussion

This study developed the same course, the same instructors, and the same materials for high school and college first-year students and compared the two groups’ understanding of engineering design through their creativity in the designs and reflective practice of engineering design behaviors. The results showed that the college first-year students’ design work was significantly better than that of high school students in all aspects of the CPAM. However, the self-iterative and sequential transition behaviors between the two groups were similar. Most student teams tended to recall and describe the same engineering design behavior in a more detailed way, which led to the representation of significant self-iterative behavior. In contrast, student teams seldom reflected on the sequential relationship between two different design behaviors, thus most sequential transition behaviors were not significant.

For the two groups’ creativity, one of the main reasons for their differences should be attributed to experience in physical and mathematical modeling ability, as previous studies demonstrated (Chien & Chu, 2018; Mentzer et al., 2014). In this study, digital tools such as 3D printing were provided to students to complete project-based engineering designs. Studies have found that digital tools can help students rapidly produce tangible models to realize innovative ideas and visualize abstract concepts, which helps facilitate student learning (Verner & Merksamer, 2015). According to the models made by high school students and our observations during the teaching process, students can achieve variable wheel designs and materialize wheels through 3D printing because the software used to create the wheel shape designs is relatively simple. However, once they started working with more complex shapes, their insufficient skills limited their creative performances. In contrast, college first-year students were relatively proficient in digital tools, so they used more sophistication with digital tools in wheel design and in making models with complex appearances. For high school students, incorporating easy-to-use digital tools into practical engineering design courses is a way to help them combine theories and ideas about design, output, and revision processes, as proposed by Eisenberg (2013). However, if it is not possible to incorporate simple digital tools, the more traditional hand tools/materials (e.g., cardboard, Styrofoam, and clay), which are easily and quickly available, could also be adopted to make creative models before actual production. It is noteworthy that the use of easily available materials to make the joints of product parts more robust and increase usability depends on hands-on experience to achieve this better accuracy. Therefore, it is necessary to strengthen basic hands-on techniques for high school students. This depends on the more detailed time allocated to teaching, and the teacher should also have higher standards for increasing hands-on abilities. The model-making ability seems unrelated to design, but indeed it does affect students’ creative design potential, the production of usable product models, and the design of works with exquisite appearances. Once the high school students are less restricted by technical issues, their balloon piercing and protective devices are restricted by the use of bamboo sticks provided by the teacher, they can be more creative in designing and decorating their works.

For the two groups’ reflective practice of engineering design behaviors, based on the foundation of Adams and Atman’s (2000) work that explicitly operationalizes or represents iterative activity, we determined that both groups mentioned limited iterations and transitions. This unexpected result contradicts several studies, which indicated that individuals with more engineering design experience displayed better problem scoping and information gathering skills (Atman et al., 2007), greater performances in design activities (Atman et al., 1999, 2005; Douglas et al., 2015), higher design scores (Strimel et al., 2018), and more cognitive effort at idea generation, feasibility analysis, and decision-making (Grubbs, 2016; Mentzer et al., 2015) than those with less experience. However, it is consistent with the findings of Lammi and Gero (2011) and Kannengiesser et al. (2015) that all participants showed similar and limited thinking about the engineering design processes. When students ignore the importance of self-iterative and sequential transition behaviors, their concepts of these processes will be vague (Mesutoglu & Baran, 2020), and it will be impossible for them to achieve the core of engineering design—acquire the necessary abilities in the design process and achieve innovative design results, as proposed by Khot et al. (2020). Accordingly, engineering design education should focus more on helping high school and college first-year students understand the important characteristics of constantly repeating various design activities to improve design quality. In addition to accumulating an understanding of engineering design from the experience of design project practices, more teaching strategies, such as reinforcing the reading of the engineering design textbook (Atman & Bursic, 1996), might be used so that students can understand the meaning of every engineering design activity. College first-year students can therefore accumulate engineering design knowledge while also learning technical skills, and high school students can have more curriculum content focused on building a solid concept of engineering design before entering college instead of experiencing many new technology tools only. Nonetheless, there is no denying that the results might be influenced by the think-aloud paradigms. This present study adopted the reflective approach rather than the concurrent one that was adopted by previous studies, thus the students might not be able to convert their behaviors during the design process to long-term memory in such a short time and reflect on them. As students may not develop mature reflective judgment, their epistemological assumptions and their ability to evaluate knowledge claims and evidence and to justify their claims and beliefs would not change.

Generally, the difference between the design works of the two groups came from various hands-on abilities and experiences instead of their understanding in the engineering design. In other words, college first-year students received much training on the growth of technical skills, but this did not mean that they had a more mature understanding of engineering design. At first glance, our finding might seem inconsistent with Strimel et al.’s (2018) study, which determined that secondary students had significantly better design cognition and thus achieved higher design rubric scores than postsecondary students. Nonetheless, it might be attributed to the tasks used in the two studies—students were instructed to create conceptual designs in Strimel et al.’s study, but they were asked to design 3D prototypes that demanded more technical skills in the present study. Although different technical skill levels demand different levels of importance of understanding in engineering design, a student’s understanding, which involves higher-order thinking (Fan & Yu, 2017), such as problem solving (Dym et al., 2005) and critical thinking (Mosely et al., 2018), is still thought to be the key to creating designs of higher quality. Improving students’ understanding of engineering design, which could nourish their higher-order thinking skill sets, is important for entering the workplace and fitting into a team. If individuals rely only on technical skills rather than the high-level thinking skill sets required in engineering design, their future development will be limited.

Conclusions and implication for teaching practice

K-12 STEM education focuses on promoting scientific inquiry and engineering design by integrating science, technology, engineering, and mathematics into a curriculum to promote the innovation of problem solving (Kennedy & Odell, 2014). Many researchers have highlighted that few investigations of high school students’ thinking during designing have been undertaken (Crismond & Adams, 2012; Hynes, 2012; Lammi & Gero, 2011). It is important to analyze the impacts of engineering experiences on students’ understanding in engineering design to help facilitate the learning bridge between high school and college levels. The results of this study form an evidence-based foundation for clarifying high school students’ design experiences and can also be used to assist higher education in developing more effective first-year college programs based on students’ prior practices. According to the discussion between the results of this study and those of empirical research, we found that the noncumulative learning of engineering design between high school and college first-year students must be taken seriously. Therefore, we suggest the following: (1) the study shows that with the same learning context, high school students were not that different from college students in terms of the design process (with the implication being that high school students are just as capable of learning engineering design as college students, so we should invest in teaching engineering design to high school students); (2) both groups of students demonstrated relatively low levels of self-iterative and sequential transition behaviors in their reflections—and this was surprising given the design of the course; and (3) the undergraduates (first-year students) were more proficient with design tools than the high school students, which allowed them to demonstrate more creativity (with the implication being that we should think about experiences for high school students and college students to learn to use tools).

Limitations and future research

This study has some limitations that should be addressed in the future. First, since the high school students were complete novices in engineering design, we selected only the three top-scoring teams’ reflective data rather than the “actual” design behaviors to avoid muddying the data with further cognitive demands of thinking aloud while designing or the pressure that might accompany video recording. In future studies, a design task could be provided for novice participants after the courses were completed, and their real design behaviors could be recorded and analyzed. Moreover, the top-, middle-, and bottom-scoring teams’ working processes for the design task could also be further analyzed and compared. Second, we developed an 8-week teaching course on battlebot design and examined its effects on the performances of high school and college first-year students. However, the learning of engineering design is obtained through incremental and repeated practice, which takes time. Future studies could adopt a longitudinal approach to investigate students’ real design behaviors and clarify how they accumulate experience and expertise in the domain of engineering design. Furthermore, future research on the repeated process could also determine what behaviors are often ignored by students.