An exploratory study of informed engineering design behaviors associated with scientific explanations

Abstract

Background

Design and science inquiry are intertwined during engineering practice. In this study, we examined the relationship between design behaviors and scientific explanations. Data on student design processes were collected as students engaged in a project on designing energy-efficient buildings on a blank square city block surrounded by existing buildings using a computer-aided design program, Energy3D, with built-in solar energy simulation capabilities. We used criterion sampling to select two highly reflective students among 63 high school students.

Results

The main data sources were design replays (automatic playback of student design sequences within the CAD software) and electronic notes taken by the students. We identified evidence of informed design such as problem framing, idea fluency, and balancing benefits and trade-offs. Opportunities for meaningful science learning through engineering design occurred when students attempted to balance design benefits and trade-offs.

Conclusions

The results suggest that design projects used in classrooms should emphasize trade-off analysis and include time and resources for supporting trade-off decisions through experimentation and reflection. Future research should explore ways to visualize patterns of design behavior based on large samples of students.

Background

Engineering and technological design has been considered to be a component of K-12 education curricula and standards in a number of countries since the 1970s. Many industrialized countries such as Australia, Great Britain, New Zealand, and the US have implemented reforms for engineering education in K-12 schools to promote technological literacy as well as competencies to create and evaluate design solutions (Cajas 2001; Lachapelle and Cunningham 2014). While engineering design has been a part of the National Science Education Standards in the US (National Research Council 1996), it has gained a new sense of importance worldwide with the increased number of studies examining engineering in association with science learning, twenty-first century skill development, and academic motivation (Brophy et al. 2008; Penner et al. 1997; Sadler et al. 2000; Purzer et al. 2014) and with the publication of a number of reports such as A Framework for K-12 Science Education (National Research Council 2012), the Next Generation Science Standards (Achieve 2013), and STEM Integration in K-12 Education (National Academy of Engineering and National Research Council 2014).

Scientific inquiry vs. engineering design

Given that engineering is a relatively new subject as compared to science for many educators, curriculum developers, and policy makers, there are many attempts to understand features of scientific inquiry and engineering design and how they might relate or interact with each other (Lewis 2006; National Research Council 2012; Schunn et al. 2012). Lewis (2006) provides a brief history on the push for inquiry in educational standards and identifies key differences and similarities between inquiry and design. He notes that inquiry was first viewed as an intellectual process - the attitudes and activities associated with discovery (Bruner 1961). Over time, definitions of inquiry focused on the intellectual processes peculiar to particular aspects of science that were crucial to understanding the nature of science. As such, scientific inquiry became to be understood as more than a technique (i.e., a process or inquisitive stance) but as the content and practices of science (i.e., the role of scientific theory in generating new lines of investigation and how science is done). Scientific inquiry represents the diverse ways in which scientists study the natural world and develop explanations based on the evidence derived from research (National Research Council 1996, p. 23). While scientific inquiry involves observations, posing questions, and examining various sources of information to see what is already known, there is no single scientific method (Penner et al. 1997), and while scientific investigations may share a common set of deductive or inductive reasoning activities, there is ‘no fixed sequence of steps that all scientific investigations follow’ (National Research Council 1996, p. 3). Instead, inquiry is characterized as a fluid process in which the outcomes are not absolute, but rather tentative.

Similar to scientific inquiry, design is understood to be a complex cognitive activity (Cross 2001; Lawson and Dorst 2009) where there is no one generalized design method (Dubberly 2004) although variations of methods share many elements such as posing, generating, evaluating, and choosing solutions (Lawson and Dorst 2009). Designers tackle and resolve ill-defined or wicked problems in which solutions are neither right nor wrong but rather sufficient (Cross 2006; Goel and Pirolli 1992). This uncertainty requires that designers construct or frame problems, which involves taking into account the problem context and the needs of diverse stakeholders. This occurs through identifying constraints or boundaries on the space in which the designer searches for a solution. Constraints cover a large and often under-determined space from objective parameters (governed by natural or mathematical principles) to subjective values (Goel and Pirolli 1992; Jonassen 2000). As such, designers need to consider trade-offs to select among a set of plausible solutions and may need to revisit early problem formulations or backtrack on design decisions as needed.

Since design problems are often not amenable to exhaustive analysis, the designer uses abductive reasoning to impose conjectures to manage ambiguity and make solutions possible (Dorst 2011; Kolko 2010). They also rely on other cognitive reasoning tools such as analogical reasoning to search for solutions and manage ambiguity (Ball and Christensen 2009; Paletz et al. 2013; Daugherty and Mentzer 2008), visual thinking to amplify the inventive uses of mental imagery (Fish and Scrivener 1990; Goldschmidt 1991; Goldschmidt and Smolkov 2006), and representational thinking to translate abstract requirements into concrete ideas and communicate these ideas to others (Bucciarelli 1996; Cross 2006). Overall, the design process may be characterized as a goal-directed and iterative activity where the mode of problem solving is described as solution-focused (as compared to problem-focused) since a designer learns about the problem through proposing solutions and synthesizing ideas.

As Lewis (2006) describes, notable divergences between engineering design and scientific inquiry include purpose (starting points that emphasize pure versus applied dimensions), the role of constraints (constraints in design are intrinsically woven into design reasoning), the role of trade-offs (science has no parallel to this form of reasoning), and the role of context (design problems are shaped by context whereas science problems seek to transcend context). Two central ways design converges with Scientific inquiry include (a) reasoning processes such as analogical reasoning as navigational devices to bridge the gap between problem and solution and (b) uncertainty as a starting condition that demands expenditure of cognitive resources in the form of search strategies, mental models and visual representations to rehearse and communicate thought, and decision-making via testing and evaluation (Lewis 2006). As stated earlier, both design and inquiry are often referred to as procedures, although no generic procedure exists, and are taught as content in their own right, although they must draw upon a base of knowledge to formulate and solve problems.

Another key point of commonality between engineering design and scientific inquiry is that both emphasize learning by doing. For example, a key performance dimension of design is learning while designing (Crismond and Adams 2012). This includes learning by doing and reflecting, from brainstorming and prototyping, through iteration and feedback from failure, by noticing and troubleshooting, as well as drawing and dialoging with ideas, materials, and people (Adams et al. 2003; Crismond and Adams 2012; Lawson and Dorst 2009; Schön 1993). Similarly, scientific inquiry is characterized as a process of transformative learning that allows expression of both teacher and student voices (Lewis 2006).

While much has been written about comparing design and scientific inquiry, there are limited studies focusing on how K-12 students engage in engineering design and scientific inquiry simultaneously while solving complex and ill-defined problems (Svihla and Petrosino 2008; National Research Council 2009). In this study, we used a computer-aided design software that captured student design activities allowing us access to fine-grained data on students’ explanations and actions. More specifically, we asked:

How are secondary school students’ design behaviors associated with scientific explanations in informing design decisions?

Conceptual framework

Engineering and science manifest and interact to support learning

Learning one (e.g., engineering) through the other (e.g., science) represents an underlying philosophy integrating scientific inquiry and design in K-12 education. Previous research suggests that engineering design is an effective approach to supporting science learning (Apedoe et al. 2008; Hmelo et al. 2000; Kolodner 2002; Kolodner et al. 2003; Doppelt et al. 2008; Schnittka and Bell 2011). Well-documented and effective approaches for which design provides a vehicle for learning science include Learning By Design (LBD) (Kolodner 2002), Model Eliciting Activities (MEA) (Zubrowski 2002), and Knowledge and Skill Builders (Burghardt and Hacker 2004). As an example, the LBD model of constructing and testing real devices provides students with opportunities to test their conceptions and discover the holes in their own knowledge as well as predict the performance of their designs (Kolodner 2002).

Parallel to accounts of design approaches to enhance science learning are those that use science as a vehicle for prompting design. For example, Lewis (2006) describes two activities that ask students to develop design ideas by applying science concepts they learned previously. The first activity by Snetsinger et al. (1999) involves application of electrical and mechanical energy concepts to the design of wind turbines. The second activity by Subramanian (1999) applies thermal expansion principles into the design of bimetallic household products. Despite these examples, there are concerns about the so-called ‘design-science gap’ (Vattam and Kolodner 2008). A challenge is that such learning is not easily amenable to traditional assessment methods. In classroom practice, critical aspects of science learning tend to be set aside (Apedoe and Schunn 2013) as students focus on aesthetic features or on building and construction as opposed to experimentation, analysis, and justification of design decisions.

Using a conceptual framework that characterizes the interaction between scientific inquiry and design, this study offers an emergent theory of how secondary students learn and apply science concepts through engineering design. The study also offers tools for understanding how engineering and science manifest and interact and the kinds of learning associated with this interaction. As shown in Figure 1, this framework is situated in a shared space of convergence - learning while designing. This space includes learning science through design (where design is the vehicle for learning science), learning design through science (where science serves as vehicle for prompting a design task), and learning design. For this study, learning science focuses on specific concepts that are relevant to the design task. Both learning science and learning design goals are essential and in alignment with the NGSS (Achieve 2013).

Figure 1

Science and design knowledge informing design decisions.

The intersection of science and design knowledge marks the nature of the interaction between science inquiry and design, making and explaining knowledge-based decisions. Crismond and Adams (2012) describe this as using knowledge of physical laws, how things work, methods of construction, everyday knowledge, application of known cases, and knowledge gained from design revisions and experiments. The framing illustrates the ways science inquiry and design occur concurrently feeding one another. For example, simulation data collected through experimentation can inform the design of specific features. Similarly, the design of multiple prototypes with comparable features can guide scientific inquiry with the goal of evaluating the performance of these alternatives. Together, these activities guide the explanation of relationships among design criteria as well as help inform decisions about making trade-offs among these criteria.

Informed design behaviors

To qualitatively characterize patterns of student design behaviors, we used Crismond and Adams (2012) Informed Design Learning and Teaching Matrix. The design of this matrix was based on a meta-literature review of more than 50 studies and synthesized key performance dimensions of informed designing and behaviors that distinguish novice and expert designers. Our goal was to explore how these behaviors manifest at the interactions between science and engineering. One of these key performance dimensions is ‘using design strategies effectively’. This is defined as knowing when and how to use a range of design practices and strategies, and being open to altering a design approach or decision to accommodate new information and time constraints. These key performance dimensions were translated into nine observable design strategies that were further translated into nine ‘contrasting set’ statements that could act as a guide for noticing how beginning vs. informed designers do the selected strategies.

As shown in Table 1, the matrix describes a two-step learning trajectory from ‘beginning designer’ (that depicts assumptions about prior knowledge and skills of learners as they enter a learning progression) to ‘informed designer’ (that depicts what learners are expected to know and do by the end of the progression). Here, an informed designer is one whose level of experience includes some formal training in design, and level of competence lies somewhere between that of the novice and expert designer. Collectively, the nine statements represent key elements of design learning that make up effective practice, and each statement describes the signs of beginning designers’ less effective design approaches as compared to how informed designers would more effectively perform the same strategy. For example, for the pattern of ignore vs. balance benefits and trade-offs, beginning designers are prone to ignore complexity and trade-offs and make design decisions without weighing all options or attend only to pros of favored ideas or cons of lesser approaches. In contrast, informed designers use words and graphics to display and weigh both benefits and trade-offs before selecting a solution. For the pattern, haphazard or linear vs. managed and iterative designing, beginners design in haphazard ways or do design steps once and in a linear manner. Informed designers do design in a managed way, improving on ideas iteratively through feedback and using strategies multiple times as needed, in any order.

Table 1 Informed design patterns (Crismond and Adams 2012 )

Methods

Research participants and classroom context

This exploratory study was conducted at a large public high school located in the Northeast United States. Sixty-three students were enrolled in the engineering track during the spring 2013 semester in five courses that comprised the engineering track. Our intent was to target the students with the richest reflective notes from all students in the sample. We reviewed notes from students who completed at least two unique designs. In reviewing each student’s designs, we looked at richness of reflection notes over their three designs. These reflective notes included scientific explanations and provided information on students’ actions and the reasoning associated with these actions.

For this exploratory study, we were purposeful in our selection of the cases. We wanted to select students from the same section to account for a similar course experience. The subsample that met all selection criteria ultimately included two senior high school students with the richest reflective notes from a course, named capstone design. Such reflectivity suggests that these students were at a place more than average in the beginning-informed designer spectrum. Each student, given the pseudonyms of Andrew and Brent, generated two design solutions addressing a design challenge described in the following section.

Design challenge

The design problem, the solar urban design challenge (Xie et al. 2014b), asked students to design energy-efficient, passive solar buildings on a blank square city block surrounded by existing buildings of different heights (see Figure 2).

Figure 2

Solar urban design challenge.

The design challenge, described in Figure 3, involves the use of design practices to consider solar radiation as it varies over a day and over seasons, analyze simulation data, consider constraints, make trade-offs, and optimize solutions (e.g., minimize the energy required to heat a building in the winter and minimize the energy required to cool a building in the summer). The Energy3D environment allows verification or testing of design performance within the software (Xie et al. 2014a).

Figure 3

City block design challenge.

Students used a computer-aided design (CAD) software, Energy3D, to design solutions for the built environment based on solar energy and heat transfer concepts (i.e., science knowledge) addressing the following engineering principles (i.e., design knowledge) outlined in the NGSS (Achieve 2013).

• HS-PS3-3. Design, build, and refine a device that works within given constraints to convert one form of energy into another form of energy.

• HS-ETS1-1. Analyze a major global challenge to specify qualitative and quantitative criteria and constraints for solutions that account for societal needs and wants.

• HS-ETS1-3. Evaluate a solution to a complex real-world problem based on prioritized criteria and trade-offs that account for a range of constraints, including cost, safety, reliability, and esthetics as well as possible social, cultural, and environmental impacts.

• HS-ETS1-4. Use a computer simulation to model the impact of proposed solutions to a complex real-world problem with numerous criteria and constraints on interactions within and between systems relevant to the problem.

Data sources

Design replays

The Energy3D software collected logs of student design actions in the background as students were sketching buildings, conducting experiments, collecting data using simulations, and taking reflective notes. These learner data were used to reconstruct the entire design process with all important actions restored for analysis. The snapshots (computer models, not images) were played back just like running a slide show. Student notes (text strings) to explain or reflect on their designs were replayed synchronously to their design actions such that we can coordinate the analyses of what they did and what they thought.

The computer recorded every action of the students. The datasets collected were large in that each design challenge required 5 to 7 h of classroom time to complete and summed up to approximately 20 megabytes at the end of a design project for a single active student. The replay of students’ design actions can be sped up to provide a compressed view of the full design process, similar to time-lapse photography. In addition, these design replays can be stopped and restarted with one keystroke to better analyze each design action when needed.

Figure 4 presents the simulation of solar path (also called virtual heliodon), which students use to view the path of the sun. Another simulation allows viewing of the daily radiation heat map on all building surfaces. These features are helpful in understanding shadows related to both heating and cooling and enable students to visualize warmer and cooler parts of the building in multiple seasons. Figure 4 shows system capabilities in assisting students in balancing benefits and trade-offs from both a heating and cooling perspective as well as taking different seasons into account.

Figure 4

Energy3D simulations (a) analysis for December (b) analysis for July (c) detailed view of the quantitative analysis for December.

Electronic notes

The Energy3D system includes a note tool that allows students to enter electronic notes while designing. During the classroom implementation, students were encouraged to note their design process, decisions, and thinking and were instructed to take notes ‘like an engineer’ such as reasons for creating or selecting design features, describing tests to conduct about the performance of the current design, documenting performance and interpreting test results, and making plans for next steps. The student design rationales, collected through electronic notes within the system, were used in conjunction with the process data to gain a much clearer picture of student actions, learning, and reflection.

Data analysis

Because we were interested in the interaction between science inquiry and design, which was defined as making and explaining knowledge-based decisions in Figure 1, design decisions were our unit of analysis. To characterize observable patterns of student design behaviors within the video playbacks and student notes, we developed and used the informed design coding protocol based on the Informed Design Learning and Teaching Matrix (Crismond and Adams 2012). Finally, we used Valkenburg’s (1998) representational system to visually display students’ decision-making over time in terms of observable student design patterns. This also provided a mechanism for connecting macro-level iterations that addressed system-wide performance to micro-level iterations that addressed local issues.

Development of the informed design coding protocol

Each set of design decisions (moves), with student notes as further evidence when available, was mapped against the Informed Design Learning and Teaching Matrix (Crismond and Adams 2012). Table 2 includes examples of these behaviors as exhibited by Andrew and Brent. We did not consider pattern d, deep drawing, and modeling, from the Informed Design Learning and Teaching Matrix, due to the fact that the given project uses a CAD system without the explicit use of other forms of representations.

Table 2 Informed design coding protocol

Each student’s design replays were reviewed in 10-s increments, which corresponded to 6 to 7 min of the actual class time, similar to time-lapse photography. Prior studies show data from such low-fidelity approaches are compatible with high-fidelity approaches where the observer is physically co-located with the student observed (Baker et al. 2006; Zhang et al. 2014). In addition, this approach allowed researchers to see subtle actions that are not obvious to the human eye. While designing, students typically made CAD design moves at a pace that allows these steps to be accelerated for analysis.

Within each 10-s increment, student design actions (moves) were reviewed and coded using the action categorization scheme presented in Figure 5. This categorization was used in combination with student notes to understand design episodes when the student was working within one goal or frame of the problem (see Figure 6). For example, within a frame, a student may write notes in which he reflects upon or considers the actions he has been taking in relation to a goal such as revising the roof while reflecting about increasing shade and decreasing the energy profile in the summer. Next, these chunks of time with similar goals served as input for characterization and visualization of the intersection of scientific explanation and design processes. Together, these actions formed a coherent design episode.

Figure 5

Action visualization protocol.

Figure 6

Brent’s full design process.

Valkenburg’s (1998) representational system was then used to visually display students’ decision-making over time in terms of observable student design patterns. This also provided a mechanism for connecting macro-level iterations. Figure 5 shows the symbols that were used to create visual representations of the design episodes based on the informed design coding protocol. Each design episode started with a goal, represented with an oval. The triangles are representative of moves or student actions such as making, editing, or changing roof shape. When a connection to a science concept is made, this was indicated with a blue dot. Student reflections are shown with a diamond.

Results and discussion

The analysis of Andrew’s and Brent’s design notes and design actions, resulted in a series of coherent activities reflecting informed design behaviors. Brent’s full design process is used for this illustrative example to show how he made connections to science concepts. Andrew exhibited similar patterns in his design process as well. In Figure 6, all design episodes for Brent are represented in sequential order, composed of six episodes. All seven patterns of informed design behaviors were evident in both Andrew and Brent’s design processes: problem framing, doing background research, idea fluency, balancing benefits and trade-offs, diagnostic troubleshooting, managed and iterative designing, and reflective design thinking.

As illustrated in Figure 6, the use of science concepts appeared in four episodes (episodes 1, 2, 3, and 5) along with trade-off analysis he engaged in. Where there was no trade-off analysis, there were no explicit connections to science. Reflective design thinking was also evident in these episodes. Episodes 4 and 6 were the only two instances where Brent did not make a science connection. These episodes also did not include any written reflections or evidence of an explicit trade-off analysis. During idea fluency, when Brent affluently generated and tested a large number of ideas, there was no evidence of reflective thinking and consequently no scientific explanations.

Throughout his design process, Brent made connections to two science concepts: heat transfer and seasonal solar path. As illustrated in Episode 1 in Figure 6, he conducted experiments that resulted in scientific explanations on the relationship between building geometry (more specifically surface area) and the solar energy gains. In episodes 1 and 2, he recognized that the larger the surface area the greater the heat gain was. He also associated surface area with specific geometric shapes. For example, he justified building a parabola-shaped building (i.e., a U-shaped building) to increase the amount of solar heat energy transferred to the building. As his reflective notes in episodes 3 and 4 show, Brent made explicit efforts to establish a trade-off between heating in winter and cooling in summer.

Hence, it appears that exhibiting design behaviors of balancing benefits and trade-offs through experimentation was more conducive to students making connections to science concept during engineering design. According to Jonassen (2012), the trade-off analysis is a rational model of decision-making and an avenue for metacognitive awareness that opens opportunities for learning (Jonassen 2012). In our study, we similarly found evidence of systematic experimenting, scientific explanations, and decision-making at times when trade-off was to be made. Moreover, in authentic engineering design practice, reasoning through benefits and trade-offs of different design alternatives is an important component of decision-making (National Research Council 2012). In balancing benefits and trade-offs, students practice weighing options from experimentation and making decisions rather than focusing solely on the features of their design solutions.

The actions associated with trade-off analysis did not occur in a continual form. In fact, there was evidence of macro iterations in Brent’s full design processes starting from episode 1 leading to episode 6. These macro iterations occurred between idea fluency and trade-off analysis and exposed managed and iterative designing. In episodes 1, 2, and 3, Brent’s focus was on trade-off analysis with some evidence of knowledge building, he engaged in idea fluency in episode 4. In episode 5, he went back to trade-off analysis and then moved back to idea fluency in episode 6. In the informed design coding protocol, idea fluency represents the student working with a large variety of ideas (i.e., building height, width, roof characteristics, or location) with the ability to work with one idea element at a time (i.e., building height) or several elements at a time. Idea fluency represents exploration of design alternatives in greater breadth and/or depth. In the exploratory cases of Andrew and Brent, there were no explicit evidence for science connections during idea generation and testing; however, macro iterations were evident between generating multiple designs through idea fluency and balancing benefits and trade-offs. It is possible that idea fluency allowed students to apply their new understandings into designs or gain information about their designs that are later tested more systematically through trade-off analysis. The visualization of the design episodes of Brent allowed us to better understand students’ design goals through their design processes and reflective notes.

This exploratory study concludes that students explicitly explore and develop scientific explanations when balancing design benefits and trade-offs. Such decision-making through trade-off leads to opportunities for meaningful science learning. There are two critical variables that are likely to influence this phenomenon. First, learning becomes apparent through reflective design and reflective notes that illustrate the negotiation of benefits and trade-offs. Hence, without the reflective thought process or structures that reinforce the expressing of such thinking, learning may not be possible. Second, people in general, not just students, struggle with evaluating trade-offs (Scholten and Sherman 2006), demonstrating that such decision-making is not necessarily a naturally easy process (Papadouris 2012). Without well-constructed design problems, most students cannot consider or fully engage with the complexity of weighing trade-offs in making design decisions.

Conclusions

As noted in the introduction, while engineering design has been considered to be a component of K-12 education curricula and standards for decades, there is little research on how the students learn to apply science through design. In this study, we attempted to untangle and deconstruct two high school students’ engineering practices as intertwined activities between engineering design and scientific explanations. Analyzing design behaviors as a series of design episodes provided ways to visualize student design processes, which was facilitated by the Energy3D system. Student design replays and reflective notes provided evidence that characterized engineering and scientific practices. As a result, we present a framework for explaining this intersection and an emergent theory of engineering-science integration.

This exploratory study included two highly reflective high school students who were able to engage in divergent and convergent behaviors. Future research should explore patterns for a more diverse group of students and ways to visualize these patterns with large samples of students. In conclusion, the results suggest that for science learning to occur, design projects used in classrooms should allow and emphasize trade-off analysis and include time and resources for supporting trade-off decisions through experimenting, data gathering, and reflection. Abilities to conduct trade-off analysis should be a desired learning objective in its own right and should be appropriately scaffolded for students.