Scaffolded team-based computational modeling and simulation projects for promoting representational competence and regulatory skills

Abstract

Background

This study posits that scaffolded team-based computational modeling and simulation projects can support model-based learning that can result in evidence of representational competence and regulatory skills. The study involved 116 students from a second-year thermodynamics undergraduate course organized into 24 teams, who worked on three two-week-long team-based computational modeling and simulation projects and reflected upon their experience.

Results

Results characterized different levels of engagement with computational model-based learning in the form of problem formulation and model planning, implementation and use of the computational model, evaluation, and interpretation of the outputs of the model, as well as reflection on the process. Results report on students’ levels of representational competence as related to the computational model, meaning-making of the underlying code of the computational model, graphical representations generated by the model, and explanations and interpretations of the output representations. Results also described regulatory skills as challenges and strategies related to programming skills, challenges and strategies related to meaning-making skills for understanding and connecting the science to the code and the results, and challenges and strategies related to process management mainly focused on project management skills.

Conclusion

Characterizing dimensions of computational model-based reasoning provides insights that showcase students’ learning, benefits, and challenges when engaging in team-based computational modeling and simulation projects. This study also contributes to evidence-based scaffolding strategies that can support undergraduate students' engagement in the context of computational modeling and simulation.

Introduction

Model-based reasoning is central for scientists, engineers, and mathematicians to draw upon foundational scientific or mathematic principles to model, simulate, and understand systems (Aurigemma et al., 2013; Lehrer & Schauble, 2006). Model-based reasoning involves cognitive processes, such as analogical reasoning, visual imagery, representation, and thought experiments, with the goal of constructing new conceptual structures (Nersessian, 2002a). Such conceptual structures are often represented as models used to understand or explain an ill-known phenomenon or system (Ifenthaler & Seel, 2013). Thus, to promote the effective use of models in education, instructors must apply model-based learning, which involves teaching strategies that engage students in model-based reasoning processes (Clement & Rea-Ramirez, 2008). Such model-based learning aims to engage learners in co-construction processes building on instructor-generated and student-generated model elements (Hallström & Schönborn, 2019; Hallström et al., 2023). Models, once expressed as external representations, can assist students in making sense of phenomena or systems (Develaki, 2017; Johnson-Laird, 1995). External representations can take the form of diagrams, equations, and computer simulations (Magana et al., 2020; Nersessian, 2009). Learning the uses of these representations is referred to as acquiring representational competence (Stull et al., 2016). Thus, to develop model-based learning, students must be (a) exposed to the process of creating, testing, revising, and using models that reflect their understanding of phenomena or systems (Malone et al., 2018; Schwarz & White, 2005), (b) supported so that they develop representational competence as an ability to understand, create, and flexibly utilize multiple types of models (Avargil & Piorko, 2022), (c) engaged in metacognitive processes for monitoring and evaluating progress (Cox, 2005; Lyon & Magana, 2021), and (d) guided through regulatory processes while working with others (DiDonato, 2013; Garrison & Akyol, 2015).

Students pursuing majors in STEM are expected to engage in a specific form of model-based learning—computational model-based learning—where, in addition, modeling and simulation skills are applied to various problem-solving contexts across disciplinary curriculums in foundational and advanced courses (Nersessian & Patton, 2009; Sun et al., 2006). Empirical research has identified that students benefit from their conceptual learning when engaging in computational model-based learning (Malone et al., 2018). However, since the 80s, education research has identified over and over the complex nature of learning computational modeling and simulation skills (e.g., Cheah, 2020; Mayer, 1985). In recent years, discipline-based education researchers have begun to delineate critical points and key challenges in the development of students' computational skills to support model-based learning processes in the context of modeling and simulation (Magana et al., 2020; Turner et al., 2011).

Specific challenges in undergraduate STEM education are centered on developing computational model-based learning (Magana et al., 2017, 2020). This is attributed in part to students needing to integrate programming skills with mathematical knowledge and disciplinary concepts (Magana et al., 2013, 2017). Specifically, prior research indicates that in addition to programming challenges, certain practices, particularly those that demand the integration of differing representations, i.e., physical, mathematical, and algorithmic, such as those that occur in modeling and simulation, run significant risks of causing "cognitive overload" (Magana et al., 2013; Vieira et al., 2016b). Even advanced postgraduate engineers in research settings face significant difficulty in the abstraction and transfer of knowledge between physical and algorithmic representations (Madamanchi et al., 2018). A significant limitation in our evolving understanding of the progression of undergraduate students in computational and model-based skills is that, to date, most studies have been limited to students in K-12 settings (Hurt et al., 2023). Fortunately, work in engineering and physics education has started to document effective ways for delivering computational model-based instruction through scaffolded pedagogy (e.g., Fennell et al., 2020; Lyon & Magana, 2021; Medeiros et al., 2018; Odden et al., 2019).

As part of our educational initiatives in the context of higher education, we are focusing on the need to integrate computational model-based learning sooner and more often in the undergraduate STEM curriculum. Specifically in the context of engineering majors, it is important for students to develop computational model-based reasoning because bodies of accreditation, such as ABET and the Washington and Dublin Accords, have consistently identified as an outcome for students to possess an ability to use the computational techniques, skills, and modern engineering tools necessary for engineering practice. However, research has identified that the integration of computational model-based learning is more common at the graduate level, with fewer opportunities for undergraduate students to practice those skills (Aurigemma et al., 2013; Magana & Mathur, 2012). More recently, educators and discipline-based education researchers in physics education (e.g., Mashood et al., 2022), biology education (e.g., Helikar et al., 2015), and engineering education (e.g., Ortega-Alvarez et al., 2018), among others, have documented the opportunities and challenges of integrating computational model-based learning in the context of at the undergraduate level. Research suggests that the integration of certain forms of scaffolding (i.e., strategies that can help learners succeed in solving problems that are otherwise too difficult for them to achieve) could effectively support students in successfully completing their projects (Quintana et al., 2004).

Specifically, in this study, we characterized and evaluated undergraduate students' evidence of computational model-based learning while completing scaffolded computational modeling and simulation projects working in teams. In addition, we identified how that engagement resulted in (a) representational competence in terms of the code representing the computational model and the output of the code and (b) regulatory processes students applied to overcome learning challenges and working with others as they approached the project solution as a team. This study posited that scaffolded computational modeling and simulation projects could facilitate computational model-based learning, resulting in representational competence in the form of mappings between models and regulatory skills to overcome learning and collaboration challenges. To test this conjecture, the research question for this study is: How do scaffolded team-based computational modeling and simulation projects engage students in model-based learning, resulting in representational competence and regulatory skills?

Background literature

In an inquiry context, learning includes engaging in processes of creating models embodying aspects of theory and data, evaluating models using criteria, such as accuracy and consistency, revising models to accommodate new theoretical ideas or empirical findings, and engaging in discussions about models and the process of modeling (Schwarz & White, 2005). While often the models created take the form of drawings (e.g., Quillin & Thomas, 2015), models can also take the form of mathematical formulations (Lehrer & Schauble, 2003), or computer models (e.g., Araujo et al., 2008). When combined with computer modeling, inquiry learning can be enhanced by providing tools to represent ideas formally, configuring and running a model to compute the values of the variables as they develop over time, interpreting the outputs of models by noticing the course of events predicted by the model, evaluating the calculated values by comparison with predictions or data collected in an experiment, and iteratively revising the model to improve it based on data or new knowledge (Borkulo et al., 2008; Magana et al., 2020).

In the context of STEM education, inquiry learning is often complemented with problem-solving processes. In such instances, modeling is accompanied by simulation processes to iteratively revise systems. That is, while modeling involves producing a representation of the inner workings of a phenomenon or system, simulation involves operating the model by exploring and reconfiguring it iteratively in cyclic steps that allow individuals to make predictions about and alter the systems under study (Shiflet & Shiflet, 2014). However, in addition to inquiry and problem-solving processes, engaging in modeling and simulation may also require the application of computational skills (Mashood et al., 2022) and mathematical skills (Penner, 2000).

Although the benefits of integrating computational modeling and simulation processes at the undergraduate level can result in meaningful forms of learning (Chao, 2016), the integration of multiple knowledge and skills also involves learning difficulties (Magana et al., 2013; Vieira et al., 2017). Such difficulties include: (1) the level of interaction with real-world systems makes these mathematical models inflexible or fragile, (2) models based on calculus may be beyond the mathematical capabilities of learners, and (3) highly abstract mathematical models are far from the student’s everyday knowledge of the world (Penner, 2000). Thus, new pedagogical and scaffolding strategies are needed to support the effective integration of computational modeling and simulation, resulting in effective learning and engagement individually and with others.

Computational model-based learning

Model-based reasoning refers to the practice of engaging in analogical reasoning, visual imagery, representation, and thought experimenting (Nersessian, 2002a), with the goal of making sense of phenomena or systems through different forms of external representations (Johnson-Laird, 1995; Zwickl et al., 2015). To promote model-based reasoning, educators must engage learners in social processes of model-based learning involving creating, testing, revising, and using externalized models (e.g., diagrams, graphs, drawings, mathematical expressions, and computational simulations) (Nersessian, 2008; Zwickl et al., 2015). The acquired ability of students to understand, develop, and flexibly utilize multiple types of representation models is referred to as representational competence (Avargil & Piorko, 2022; Stull & Hegarty, 2016). In this context, representational competence involves learners' ability to (a) abstract physical phenomena into some form of representational model alluding to a target system or phenomenon, (b) identify assumptions, simplifications, and limitations, (c) connect those models to known principles, concepts, or data, and (d) adapt them effectively during problem-solving episodes (Nersessian, 2002b; Zwickl et al., 2015). As learners engage in model-based learning, they apply and, at the same time, acquire or refine their existing knowledge (Edelsbrunner et al., 2023) and representational competence (Magana et al., 2020). Representational competence involves learners' ability to translate among multiple representations and delineate conceptual mappings between them (Stieff et al., 2016). Specifically, for analyzing a problem or phenomenon and formulating a model, individuals need to identify the variables of a model and the relationship while creating an initial representational model. It is also necessary to determine variables and relationships between variables and determine equations and function interpretations of those variables, where prior knowledge relevant to the phenomenon to be modeled is critical (Löhner et al., 2005). To determine the model’s behavior, physical laws are determined by the underlying equations (Sins et al., 2005). Such equations and functions need to be mapped into an algorithm and implemented in the model using different methods, techniques, and computational tools (Shiflet & Shiflet, 2014). Equations are solved numerically and programed those into functions embedded within an executable simulation (Magana et al., 2017). Validation and verification processes also require inspection of representations to determine whether the model is consistent with data obtained from experiments or with theoretical descriptions of the expected behavior (Sins et al., 2005). Finally, once the model is evaluated, individuals must execute the model and interpret the outputs of the model. In this process, prior knowledge influences the quality of the models students construct (Sins et al., 2005), and new knowledge is achieved through the processes of creating, connecting, and refining representations (Nersessian, 2002b).

However, in the context of computational modeling and simulation activities, research has documented the challenges students may experience when engaged in computational model-based learning. Specifically, students experience challenges in identifying mathematical formulations, as those are commonly based on differential or partial differential equations (Penner, 2000). Students also experience challenges in mapping mathematical equations into algorithmic representations and algorithmic representations into some form of code representing a computational model (Magana et al., 2017). These challenges have not only been attributed to insufficient prior conceptual understanding (Zwickl et al., 2015), but also to limited programming skills (Vieira et al., 2016a, 2016b). Students may experience difficulties regarding algorithmic logic, the use of multiple computational platforms to program, and the lack of non-technical skills, such as group work (Santos et al., 2020).

To overcome challenges during model-based learning, students must apply regulatory skills (Munahefi & Waluya, 2018). One type of regulatory skill is self-monitoring, which involves students assessing their learning and learning processes (Hawe et al., 2019). This regulatory skill is an important initial stage for students to evaluate their progress and make necessary changes to guide their course of learning more effectively (Zimmerman & Paulsen, 1995). Another type of regulatory skill needed during model-based learning is cognition regulation (Tee et al., 2021). Specifically, cognition regulation involves one’s proactive selection and use of various cognitive strategies, such as critical thinking skills and elaboration strategies, to support reasoning abilities (Tee et al., 2021). While critical thinking skills involve the use of strategies to devise alternative solutions, questions, reflection, and mathematical thinking, elaboration skills involve the use of strategies to extract and summarize main ideas, connecting, relating, and applying knowledge by pulling together the acquired information from different resources to facilitate reasoning (Tee et al., 2021).

Thus, evidence-based strategies are needed to help learners engage in conceptual mappings informed by prior knowledge as well as regulatory skills that help them identify alternative solutions and connect relevant ideas. Evidence-based strategies must engage learners in (a) implementing iterative modeling cycles promoting conceptual knowledge and disciplinary practices (Rittle-Johnson et al., 2001), (b) building models that make possible the re-representation, manipulation, and refinement of knowledge (Chandrasekharan & Nersessian, 2015), (c) making explicit connections between knowledge and modeling experiences to create arguments in the form of explanations of why and how (Zhang et al., 2006), (d) probing the model behavior to identify causal relations by making some attributes of the phenomenon or system more salient than others (Chandrasekharan et al., 2012; Izsak, 2004), and (e) facilitating collaboration and regulatory processes to engage learners in the social processes involved in model-based reasoning (Liu et al., 2018; Sabourin et al., 2013; Şen et al., 2015).

Evidence-based strategies for teaching computational model-based learning.

It is clear that learners need guidance informed by evidence-based strategies that support them in socially engaging in computational model-based learning. By doing so, learners can make explicit connections between multiple forms of representations needed during the computational modeling and simulation processes, as well as the application and acquisition of representational competence and regulatory skills. Computational model-based learning can be best delivered through modeling and simulation practices (Magana, 2024). In doing so, it is important to consider (a) delivering these practices within the context of disciplinary courses so that students apply and at the same time reinforce scientific or engineering knowledge, (b) sequencing the activities from simple to complex, and practice skills in a variety of situations, and (c) implementing elements of the sociology of learning (Magana, 2024). Regarding the sociology of learning, it is important for computational model-based learning to occur in the context of realistic tasks and in accordance with communities of practice. Thus, it is important to engage students in collaborative and cooperative forms of learning that closely resemble professional practice and that can also support the development of teamwork skills (Collins et al., 2018).

Research on learning computational modeling and simulation also suggests that students need support outside of the lecture in the form of scaffolding (Magana, 2024). Scaffolding refers to evidence-based strategies that can help learners succeed in solving problems that are otherwise too difficult for them to achieve (Quintana et al., 2004). Scaffolding is based on the zone of proximal development, which refers to the idea that learning is most effective when learners actively engage in tasks they can do with some guidance (Vygotsky & Cole, 1978). There may be tasks that learners can perform with no help and tasks that they simply cannot do. Learning activities should be designed so learners take advantage of the tasks that they can complete on their own but are moved toward activities or tasks they cannot complete by providing guidance in the form of scaffolding (Quintana et al., 2004). By presenting scaffolded learning challenges and then gradually decreasing those, learners can acquire new knowledge and skills as learners become more capable (Wood et al., 1976). Scaffolding also promotes motivation, as it can make the solution to the posed problem achievable for the students (Deci et al., 2017). There are different forms of scaffolding that differ in their (a) purpose, such as for supporting procedural, conceptual, metacognitive, or strategic processes; (b) form facilitated interaction, such as static or fixed guidelines or dynamic interaction methods as a response to learners’ actions; and (c) sources such as peers, teachers, or technology (Kim & Hannafin, 2011). These could take the form of individual activities, such as exercises, worked-out examples, live coding, visualizations, or prompts (Luxton-Reilly et al., 2018; Sanchez-Pena et al., 2022). Scaffolds can have multiple functions, including reducing the degrees of freedom of a task, demonstrating how to approach a task, showing directions, and making critical features salient (Kim & Hannafin, 2011). Thus, scaffolding is needed to support computation practices in science and engineering among novice learners, particularly those that effectively engage students (Magana et al., 2019; Psycharis et al., 2014).

Recent empirical studies in engineering and science education have started to characterize specific scaffolding in assisting student learning in the context of computational model-based learning (Magana, 2024). Specifically, the scaffolding includes sense-making, articulation, process management, and reflection, as proposed by Quintana et al. (2004). Specifically, (1) sense-making scaffolding supports ways to understand real-world phenomena or experiences and transform those into formal representations; (2) articulation scaffolding supports ways of making thinking explicit, synthesizing explanations, and creating arguments; (3) process management scaffolding supports the use of disciplinary practices, problem-solving, and strategic approaches, and (4) reflection scaffolding, supports the reviewing, reflecting on, and evaluating results, including elements of collaboration.

For instance, sense-making scaffolding, in the form of worked-out examples consisting of step-by-step sample solutions, has been identified as highly effective. In addition, articulation scaffolding in the form of in-code comments as an explanation strategy has been identified as an effective approach to engaging students with the worked-out examples (Vieira et al., 2017). Project templates have also been identified as useful process management scaffolds to guide students through the problem-solving processes, such as the modeling and simulation cycle (Lyon et al., 2022; Vieira et al., 2016b). When guided through processes using project templates, students can structure their work. For instance, students can be asked to express their understanding of the problem explicitly during the initial problem recognition phase. During this phase, the students can be guided to create a plan to work toward a solution. The student uses verbal and mathematical representations for this purpose. During the problem-framing phase, the students create a solution in the form of an algorithm instantiated as a program. In the problem synthesis phase, the student completes the plan by evaluating the solution according to both instructor-provided and student-generated criteria. The implementation of test cases is one of the most challenging phases for students and one of the most valuable for those who go on to programming practice in any context (Vieira et al., 2015).

Finally, to engage in individual learning and collaborative problem-solving, students need to also engage in metacognitive and regulatory processes, so they use elaboration and learning strategies and apply critical thinking skills (Tee et al., 2021). Effectively managing model-based reasoning requires extra metacognitive processes to regulate the complexity of modeling tasks while also working on them with others (Oyelere et al., 2021). Thus, an argument has been made that supporting regulatory, co-regulatory, or metacognitive processes is also needed for supporting computational model-based learning while working individually and with others (Magana et al., 2019), where reflection scaffolding, in the form of reflective practices, is one of the primary mechanisms for promoting these processes (Arigye et al., 2024; Jaiswal et al., 2021; Peters & Kitsantas, 2010).

Methods

This study used a multimethod approach to answer our guiding research question: How do scaffolded team-based computational modeling and simulation projects engage students in model-based learning, resulting in representational competence and regulatory skills? A multimethod approach was deemed adequate for the study as it allowed us to flexibly characterize and interrelate evidence of model-based learning, representational competence, and regulatory skills, as promoted through our learning design. Multimethod studies also support triangulation by incorporating multiple data sources and analytical approaches (Vivek & Nanthagopan, 2021).

Context and participants

The context of this study was a second-year thermodynamics course titled Thermodynamics in Biological Systems II offered at a large Land Grant Public Midwestern University. This course was deemed adequate for conducting this investigation for various reasons. First, thermodynamics is a course required for most engineering students whose topics have been characterized as difficult to understand (Abdullah et al., 2013); second, the integration of computational methods has been identified as a feasible pedagogical approach for teaching thermodynamics (Vieira et al., 2018); and third, there is a need to expose undergraduate students, especially in their second and third year of their undergraduate majors, to computational methods and practices, such as modeling and simulation (Magana & Mathur, 2012). In their first year of their undergraduate major, engineering students are introduced to programming concepts, while in their last year, they apply those skills in the context of their capstone courses. However, there are fewer opportunities for students to practice computational modeling and simulation practices in the middle years and in the context of their disciplines.

The course is required for students majoring in biomedical, agricultural, and biological engineering. The major objective of the course is to understand and exploit basic principles of thermodynamics as they apply to biological systems and biological processes. Prerequisites of the course include an introductory thermodynamics course where students learned about (a) the fundamentals of steady-state mass and energy balances for reacting and non-reacting processes, including multiple unit operations emphasizing living systems and bioprocessing, (b) the applications of the first law conservation of energy to biological systems, energy conversion systems, and the environmental impacts of energy production and (c) the development of engineering problem-solving skills via MathCad and MatLab software. A second prerequisite includes a mathematics course where students learn about multivariate and vector calculus. Also, in their first year, students were required to take an introductory course where they learned and applied basic programming concepts and constructs via the MATLAB programming language. This introductory course also introduced best practices for teamwork skills, and students completed a team-based project.

The objectives of the thermodynamics course are for students to understand basic principles of mass, energy, and entropy balance equations; apply these basic principles to solve problems in biological and biomedical engineering; gain knowledge of the main factors that determine numerical values of physical properties associated with bioprocesses; and solve biological and biomedical problems for thermodynamic processes using computer code. This is a required course for all majors in biomedical and biological engineering. Students typically enroll in this course during the second year of their undergraduate studies. Exceptions occur when students participate in voluntary or extracurricular programs such as co-ops or long-term internships. These students have been exposed to related foundational courses, such as (a) a first-year engineering course where they learn about the problem-solving process and the use of scientific computation in this context, (b) mathematics courses where they learn linear algebra and differential equations that are used within this course, and (c) in the case of biological engineering students, an introductory thermodynamics course where they are introduced to the basic thermodynamic principles.

The participants were 116 undergraduate students pursuing majors in biomedical, agricultural, and biological engineering. According to institutional data, in 2021–2022, 56% of the students were women, and 44% of the students were men. The majority of the students were White 67%, followed by Asian 21%, International 10%, more than two races 6%, Hispanic or Latino 3% and Black or African American 2%. Given this is a required course for the major plan of study, the participants were representative of the program as a whole. The students were organized into a total of 24 teams, each with four or five members. The students were randomly assigned to their teams, and the teams remained the same throughout the duration of the semester. Thus, the same team was consistent across all projects, and the three projects were submitted as a team.

Learning design

The learning design for this study aimed to promote computational model-based learning that would guide students to enact educational practices that support the development of representational competence and the application of regulatory skills in the context of computational modeling and simulation of biological systems. For this, three two-week-long computational modeling and simulation projects were implemented throughout the semester. Specifics on how the scaffolding was delivered in the context of the computational modeling and simulation projects are presented in Table 1.

Table 1 Specifics of sense-making, articulation, process management, and reflection scaffolding

To deploy the computational modeling and simulation projects and, at the same time, embed the pedagogical supports described in Table 1, we used computational notebooks. The computational notebooks provided detailed explanations, guidance, and scaffolding throughout the project solution. Computational notebooks are web-based solutions that allow for authoring and executing code within a single document (Wang et al., 2019). Computational documents enable the delivery of programming content that is more accessible for teaching and learning by supporting the visualization of code execution, along with other forms of multimedia, including text, images, and videos (Granger & Pérez, 2021). Computational notebooks can also provide a learning advantage for embedding computational pedagogy (Sanchez-Pena et al., 2022).

The three projects were conceived by emphasizing the four educational practices supporting the development of model-based learning by exposing learners to the processes of formulating, implementing, using, and evaluating models. All three projects were scaffolded in a similar manner. The implementation of scaffolding is shown in Figs. 1, 2, 3, 4. Specifically, Fig. 1 provides an example of process management scaffolding aimed at guiding the students to define an initial plan in the form of a procedure and to recall the domain knowledge they would need to approach the implementation of their solutions. In Fig. 2, sense-making scaffolding was provided as guidance to the learners in the form of a template of code, providing some direction in the variables and functions needed to implement the solution. This form of scaffolding also elicited students to comment on the code to explain their thinking and decision-making processes.

Fig. 1

Sample of process management scaffolding consisting of prompts aimed at eliciting the analyzing the problem and planning and formulating a model

Fig. 2

Sample of sense-making scaffolding consisting of templates of code and in-code commenting to guide learners through the process of model implementation

Fig. 3

Sample of articulation scaffolding consisting of prompts that elicited students’ explanations of the outputs of the computational models

Fig. 4

Sample of reflection scaffolding consisting of prompts that elicited students’ reflections on their learning processes, the challenges they experienced in their learning, and working in teams

Once the model was implemented, the next step for the students was to use the model to solve the biological problem. For this, students were prompted to explain their understanding of the outputs of the computational models in terms of the biological phenomenon. For this, Fig. 3 presents an example of articulation scaffolding that prompted students to explain the meaning of the graphs the models generated and to translate those observations into an explanation of the phenomenon in question and a solution to the biological problem. Finally, students were also provided with reflection scaffolding that had two purposes. One was for students to engage in a metacognitive reflection of what they learned, identify the challenges students experienced in solving a particular project, along with the problem-solving strategies they implemented, and reflect on their team regulation processes by describing what worked, what didn’t work, and what can be improved in the next project or iteration. Examples of the reflection scaffolding are shown in Fig. 4.

Procedures and data collection methods

The specific first-time implementation of the scaffolding methods and the corresponding data collection occurred in the spring of 2022. The data collection methods consisted of student-generated team-based solutions to the scaffolded computational modeling and simulation projects that allowed us to characterize students’ implementations of their model-based learning and the resulting outcomes in terms of representational competence and regulatory skills. Thus, instruments were defined so that each captured (a) students' products and processes for evaluating, creating, and communicating their models and outputs of their models, (b) students’ representations and the interpretations of such representations, and (c) individual reflections and team-based retrospectives describing the application of their regulatory processes implemented to complete the projects.

Three team-based scaffolded computational modeling and simulation projects were implemented throughout the semester to fulfill one of the primary course objectives of applying basic principles of mass, energy, and entropy balance equations for solving biological and biomedical problems for thermodynamic processes using computer code. Table 2 presents specifics of the disciplinary and computational learning objectives for each of the projects. The titles of the projects were Could You Outrun a Dinosaur (P1), Toxin-Antitoxin System Design (P2), and Chemical Reactor Stability and Sensitivity (P3). The projects also increased in their level of difficulty in terms of the disciplinary and computational learning objectives. Each project was introduced following a similar instructional approach. Each project followed in-class lectures introducing the technical concepts required to complete the projects. When projects were assigned, a lecture was delivered that provided a description of (a) the context in which they were being asked to apply the technical concepts from lectures and (b) the full tasks they needed to complete within the project. Students had two weeks to complete the project outside of class. During that two-week period, students had an opportunity to attend TA and instructor office hours to receive project-specific help. The instructor provided opportunities during class times for project questions/clarifications based on patterns that the instructional team observed during office hours.

Table 2 Disciplinary and computational learning objectives for the course

Project 1 was assigned on week two of the semester, Project 2 was assigned on week five of the semester, and Project 3 was implemented on week 11 of the semester. As shown in Table 2, the projects increased in their level of difficulty in terms of both disciplinary learning objectives and computational learning objectives. Since the projects were submitted as a team, there were 24 submissions per project, with a total of 72 projects analyzed.

For characterizing model-based learning, we qualitatively analyzed students’ solutions to the projects involved observations of how students planned for their models, implemented and evaluated their models, used the models and interpreted the outputs of the models, and reflected on the process as shown in Figs. 1, 2, 3, 4, respectively. For characterizing representational competence, we evaluated students’ alignments between (a) the implementation of the model and their explanations of the model in the form of in-code comments and (b) the outputs in the form of graphs generated by their models and their explanations of the outputs in terms of the biological system represented. To characterize the regulatory processes, we qualitatively analyzed and coded students’ strategies to manage their learning challenges and teamwork processes as reported in their individual and team-based reflection questions listed in Table 3.

Table 3 Individual and team-based reflection questions

Data scoring and analysis methods

Computational model-based learning The data analysis procedure was initiated with a qualitative content analysis of the solutions that students submitted as a team, which consisted of the computational notebooks that embedded the scaffolding. The analysis started at the end of the semester, once all grades were submitted and all the dataset was compiled into teams. The projects were analyzed sequentially; that is, we first analyzed all 24 submissions corresponding to Project 1, then moved on to Project 2, and once completed, to Project 3. Each project was analyzed in full for a given team. For each project submission, the analysis started by chunking the data. To guide the process of chunking the data (Hsieh & Shannon, 2005), we used a directed content analysis approach where the definitions of model-based learning processes were used as guidance for the initial codes or, in this case, chunks of data (refer to Table 4). For this, we made a distinction between the portions of the submission corresponding to descriptions associated with the planning of the project, implementation of the model, interpretation of the outputs of the model, and the reflection processes that were submitted individually and also as a team, all within the same computational notebook.

Table 4 Definitions of model-based learning processes that guided the chunking of the data

We completed the chunking project for all 24 teams for Project 1. Once we identified the relevant portions of the project submissions aligned to how scaffolding supported students’ model-based learning, we analyzed the samples qualitatively to explore similarities and differences between the samples of solutions with the goal of identifying patterns in the data. Following this process, we aimed to identify groups consisting of teams with similar outcomes resulting from computational model-based learning. We then selected representative extreme cases for further description in our findings with the goal of characterizing how students within teams demonstrated evidence of their modeling processes. For this, we present excerpts of students’ solutions showcasing contrasting cases for Project 2. We opted to present samples from Project 2 in order to favor the readers’ understanding of the interconnection of ideas within a single context. Specifically, we showcase how the scaffolding methods embedded within the computational notebooks elicited model-based learning processes that resulted in students’ evidence of (a) analyzing the problem and planning and formulating the model, (b) implementing and evaluating the model to solve the problem, (c) using the model and interpreting the outputs generated after executing or running the model, and (d) reflection processes reflecting regulatory strategies students implemented individually and as a team.

Representational competence Because of the identified similarities and differences in students’ project submissions, we started to document those through a scoring rubric to capture and evaluate students' representational competence. For this, we paid particular attention to how students working in teams explained the connections between the models they generated and their understanding of such models. For this, an initial version of the rubric was created by two investigators who jointly analyzed the data. One investigator had extensive expertise in education research and assessment, and the second investigator, in addition, had disciplinary expertise in the domain of thermodynamics. The original version of the rubric was based on previous studies evaluating students' performance on their computational solutions to engineering problems (i.e., Shaikh et al., 2015; Vieira et al., 2016a, 2016b). As the scoring process started, the rubric was iteratively revised based on the characteristics of students' responses to the project. The rubric was flexible enough so all three projects could be scored with the same rubric. The revised and final version of the rubric is presented in Table 5.

Table 5 Scoring rubric for the projects for evidence of representational competence

Once the two investigators agreed that the rubric was representative of both the epistemic aims and the data itself, a subset of the assignments was reviewed by the two researchers and the rubric to evaluate the clarity of wording and the differentiation of the scoring categories. Once agreement (> 80%) was reached between the two researchers, all disagreements were resolved, and the disciplinary expert completed the scoring of the dataset. The interpretation for the scores assigned with the rubric presented in Table 4 was (a) between 1.0 and 2.0, the level of achievement in a particular skill was considered as below basic; (b) between 2.1 and 3.0, the level of achievement for a particular skill was considered as basic; (c) between 3.1 to 4.0, the level of achievement for a particular skill was considered as proficient; and (d) between 4.1 to 5.0 the level of achievement for the corresponding skill was considered as advanced.

Regulatory skills Regarding the regulatory strategies students reported they implemented, which were captured through the reflection prompts, they were analyzed qualitatively. Recall that to capture such strategies, we prompted students to reflect individually on their learning processes at the end of each project submission. The prompts aimed to identify what challenges students experienced in solving their projects, such as characterizations of self-monitoring, and the strategies they used to overcome such challenges, such as characterizations of cognition regulation, as shown in Table 6. Also, a team retrospective was written and submitted jointly by all team members along with each project submission. The team retrospective included prompts to characterize the challenges students experienced as a team and what strategies they implemented to improve the teamwork performance for their next project submission.

Table 6 Categories for characterizing learning challenges and strategies to address them

Although we primarily focused on self-monitoring and cognition regulation as two overarching inductive categories, our goal was to engage in deductive processes, so we describe in detail how these played out in the context of solving the scaffolded computational modeling and simulation projects.

These reflection prompts and team retrospectives were analyzed following a conventional content analysis approach, followed by a summative content analysis approach (Hsieh & Shannon, 2005). That is, during the conventional content analysis approach, we first generated codes derived directly from the text data, and then those codes were compared, contrasted, and logically organized, resulting in a set of categories with meaning (Vaismoradi et al., 2016). In this stage of the analysis, we identified exemplars of representative quotes of the categories (Braun & Clarke, 2006). Then, the categories were summatively counted and visualized to identify and report overall patterns (Hsieh & Shannon, 2005).

Researchers’ positionalities

The positionality of the researchers is highly interdisciplinary, multicultural, and diverse. The research team consisted of (a) three females, two of them being faculty members and, another one being a graduate student, and (b) two males, both of them graduate students. The expertise of the team is a combination of engineering education research, computational science and engineering, and biomedical, mechanical, and computer engineering. The team also represents different academic positions, such as education researchers, engineering educators, faculty, and graduate and undergraduate students learning about research. Finally, our multicultural background also brings international perspectives from Africa, Asia, Latin America, Europe, and North America. The identities and roles, as well as the diversity in the team, have been reflected in the decisions implemented in the learning design and the research process.

Specifically, in the design process, our experiences as instructors or faculty and our own experiences as students in different countries informed the scaffolding design to be human-centered and culturally sensitive. In the research process, our expertise was critical, as the domain is highly complex and interdisciplinary. Thus, the analysis of the data required a high level of collaboration and learning among the team members. Finally, our multiple roles as faculty, students, and researchers allowed us to better protect students’ privacy and confidentiality by establishing procedures where the course instructor would not be involved in the analysis but would inform the interpretation of the findings as a whole. The course instructor also brought in the practical perspective of what is feasible in the classroom and the learnings needed to improve the learning experience for the students in future implementations.

Our roles in the project were as follows: a senior discipline-based education researcher worked with the faculty member who was also the course instructor on revising the projects to incorporate the scaffolding and plan for the data collection. The initial discussions determined what was feasible in the context of a semester in terms of the number of projects, the amount of work required for each project, and additional work accommodating for data collection. Once the plan was formed, the senior researcher worked with two of the graduate students. One graduate student had just completed his dissertation investigating computational thinking in the context of modeling and simulation in engineering education, and the second graduate student, with a programming and data science background, had just started her doctoral program. The initial team worked together to revise the projects, ensuring they were technically and pedagogically sound. A third graduate student with a background in mechanical engineering and a deep understanding of thermodynamics joined the team once the data was collected. His disciplinary expertise and the expertise of the senior graduate student in biological engineering were essential to the data analysis procedures. The undergraduate student assisted with data processing, data analysis, and data visualization tasks.

Ethical and trustworthiness considerations

Before initiating the research activities, we obtained an Institutional Review Board approval from our university. The study was approved as exempted by the Institution’s Review Board under protocol No. IRB-2021–1702, since it involved research conducted in the classroom that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn or the assessment of educators. Specific ethical considerations consisted of ensuring students’ privacy and confidentiality. For this, we conducted the data analysis once the semester was completed and the grades were submitted. Also, the data analysis was conducted by the discipline-based education researchers in the team, leaving out the course instructor at this stage.

Multiple strategies were implemented to ensure the study's trustworthiness. To ensure the study's credibility, we implemented triangulation processes through three rounds of data collection within the same semester, where multiple data sources and repeated measures helped us identify the observed patterns (Stahl & King, 2020). To promote the study’s transferability (Gunawan, 2015), we provided thick descriptions of the projects, the deployment of the projects within the course, and samples of students’ work from at least one project where observations and explanations from two teams were used to inform the process of creating and refining the scoring rubric. Finally, the reproducibility of the study was ensured by a consensus-making process between two researchers (Watts & Finkenstaedt-Quinn, 2021).

Two researchers followed a consensus-making process to approach the analysis of the project data regarding the identification of computational model-based learning and representational competence. For this, the first researcher, trained in qualitative analysis, generated the first version of the chunking (and coding) of the observations for computational model-based learning within a subset of the data consisting of ten students’ submissions corresponding to Project 1. Based on this initial analysis, the second researcher generated an initial version of the rubric to score representational competence guided by samples of rubrics from previous research and iteratively refined it according to observations found in the actual data. When an initial rubric was developed, the first researcher performed an audit trial by testing the rubric on a subsample of the data the second researcher already scored. In this way, the two researchers engaged in an iterative process of scoring the data, improving the scoring rubric, and coming together to validate their observations. The second researcher then, after a waiting period of time, recoded the same data to ensure dependability. Once the second researcher was satisfied with the recoding process, he proceeded to score the rest of the entire dataset by himself, but regularly consulting with the first researcher when unsure of how to code specific observations.

Two other researchers followed a similar consensus-making approach to analyzing the reflection data that characterized the regulatory skills in terms of the experienced challenges and the applied strategies for addressing the challenges. However, instead of chunking the data this time, the first researcher established initial codes with corresponding definitions. These codes were shared with the second researcher, who coded the same initial subset. The researchers then met to discuss initial codes and revised them based on discrepancies. The two researchers decided on the final application of the coding scheme for the entire dataset. Then, the two researchers divided the data into about equal parts and coded it independently. Once they coded the full dataset separately, the two researchers came back together once again and jointly defined the categories. Once the codes and categories were fully analyzed, tables were used to organize the data into teams in order to report final insights. The selection of the representative quotes was also done in collaboration between the two researchers to ensure the selected quotes truly characterized the definition of the category.

Results

The results are organized into three main subsections. The first sub-section provides in-depth descriptions of how computational model-based learning resulted from teams of students engaging in solving the scaffolded computational modeling and simulation projects. The second sub-section provides the results of students’ levels of representational competence, and the third sub-section delivers qualitative descriptions of the regulatory skills students reported that were applied to support their learning processes and team coordination processes.

Computational model-based learning

In the first stage of the analysis, our goal was to characterize how students working in teams evidenced aspects of computational model-based learning. Through our qualitative analysis, we identified two major groups. One group (n = 16) consisted of Teams 2, 5, 8, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22, 23 and 24 whose solutions evidenced a meaningful engagement with computational model-based learning, and another group (n = 8) consisted of Teams 1, 3, 4, 6, 7, 9, 15, and 18 whose solutions evidenced an engagement with computational model-based learning more superficially. To exemplify differences between the two groups of teams, we purposefully selected artifacts produced by two teams: Team 1 as an example of a team that engaged with computational model-based learning more superficially, and Team 2, as an example of a team that demonstrated a more meaningful engagement with computational model-based learning.

To demonstrate the range of possible outcomes generated by the two groups of teams, this sub-section is organized into four main stages describing how computational model-based learning resulted from students’ engagement with the scaffolded computational modeling and simulation projects. For this, we focused on Project 2, which deals with students creating models to analyze a set of gene regulatory modules for the purpose of neutralizing extracellular toxin at varying production rates. Once the models were created, a second goal was for the students to implement the model in a computer-programmable environment to characterize and describe the biological interactions and system dynamics. To approach their solutions, students had to (a) determine a system of differential equations describing the dynamics of the biological system and build a corresponding model, (b) use the computational model to predict what the dynamics of receptor, toxin, and anti-toxin levels are over time, (b) improve their models by including one or two regulatory modules for activation of anti-toxin production, and (c) optimize the system by finding a combination of to three gene regulatory modules that can meet the design criteria.

Analyze the problem, plan, and formulate the model. For this portion of the project, students were guided through the initial stages of the modeling process to identify a preliminary procedure or plan. The guidance took the form of process management scaffolding that intended for students in a team to consider and recall the disciplinary knowledge they would need to approach the project solution and consider an initial approach for solving the problem. Figures 5 and 6 depict two samples of planning approaches provided by two different teams on Project 2.

Fig. 5

Team 1’s plan for approaching the problem in the form of a flowchart

Fig. 6

Team 2’s plan for approaching the problem in the form of differential equations and explanations

While the planning approach for Team 1 focused on using a flowchart to describe the implementation of a solution model at the code level, Team 2 focused on formulating a stepwise procedure required to solve sets of coupled differential equations that represent a solution model plan. Team 1’s flowchart illustrates the process workflow with a decision point that depends on the transient behavior of the represented biological phenomenon. In Fig. 6, Team 2 considered obtaining numerical solutions based on the model’s initial conditions and included an expectation to generate new output plots from potential outcomes of planned optimization and sensitivity analysis.

Team 1’s flowchart-based plan provides the students with a high-level conceptual view of the solution model, while Team 2’s algorithm-based plan presents the precise mathematical framework that includes ordinary differential equations (ODEs) and execution steps for the modeling and simulation plan. A reflection from a student in Team 2 suggests some merit to the team’s algorithm-based plan when asked about the project’s most beneficial aspects. The student noted that “Being able to see how each of the different steps of the project affected the outcome made it easier to understand how the different parts of the system affected each other over time.”

While a detailed flowchart was not an explicit requirement for this project, Team 1’s flowchart lacked important workflow components like input model parameterization, initial conditions, result outputs, decision points for model testing, and validations (sensitivity analysis). These omissions may potentially limit the flowchart’s ability to effectively convey the project’s methodology sequence to students. Team 2’s algorithm-based plan may also have posed a daunting challenge to students who might have been averse to outlines of mathematical representations and who did not have any form of visual or scaffolded outlook on the solution procedures. Overall, formulating a modeling plan of action by embedding the mathematical equations into the flowchart may better support students in building clear and concise visual models of the problem and solution workflow.

Implement and evaluate the model to solve the problem. Comparing Teams 1 and 2 model implementation approaches along with the respective impacts on solution models for one of the project tasks is observed in Figs. 7 and 8. The figures show how the teams began the model solution implementation in a Python code environment where initial values and input parameterization were set, along with an array of evenly spaced time vectors and a number of time steps used for the simulation. The teams followed the next steps by defining a function that takes in two arguments representing concentration values of selected biochemical components used to solve the sets of coupled mass balance ODEs. The teams went further to introduce the “odeint” numerical technique in Python’s SciPy library that allows for a stepwise approximation of unknown functions to compute and store the first incremental solutions to the ODEs.

Fig. 7

Team 1’s model implementation and corresponding code comments explaining the model

Fig. 8

Team 2’s model implementation and corresponding code comments explaining the model

The initial solution sequence used by both teams was overall similar because of the scaffolded code templates provided for the project. However, Team 2 proceeded to use a ‘for loop’ to iterate through each biochemical component over time to calculate varying toxin production rates and, calculate allowable thresholds, and generate results in the form of graphical representations. Team 1’s implementation appears to lack the ‘for loop’ essential for iterating through the biochemical components, hence limiting the model's flexibility and analytical capabilities. In contrast, the presence of the "for loop" in Team 2's code enabled students to systematically study the various gene modules while considering the impact of varying toxin production rates. That is, Team 2's model promoted a more efficient and organized approach to parameter exploration, which was suitable for robust sensitivity analyses and optimization studies compared to Team 1’s model, which required duplicating functions to handle computational experiments.

In Fig. 7, it can be observed that Team 1 employed an inaccurate step value of 100 over a 100-s time vector range, resulting in a relatively large simulation time step value of 1.01 s. In contrast, Team 2 used 1000 steps over the same 100 s range leading to a finer simulation time step of 0.1 s. This discrepancy explains the errors in Team 1’s model output results, specifically with the anti-toxin profile levels. Team 1's choice of a larger time step (1.01 s) resulted in some loss of important details when capturing rapid transient changes in the system, as seen in Fig. 9. The instructors for the project recommended a specific time step size value of 1001 and emphasized that it should not be less than 1000 steps. Regrettably, Team 1 did not adhere to this recommended step size in their implementation.

Fig. 9

Team 1’s model result output and interpretation

In the final stages of implementing the model using code, both teams used the provided code templates for extracting values from the solution matrix and creating the necessary plots. However, each team made some slight adjustments to the graph generation code. While these modifications did not impact the computational results produced by the implemented model, they did have a noticeable effect on the appearance and features of the generated graph plots. These adjustments primarily influenced aspects, such as the axis scale size, plot line colors, titles, axis labels, and plot legends. Team 1's modifications to the graph code resulted in plots that lacked proper title text and axis labels, which is an essential component for understanding the content and context of the graphs. This omission could have potentially made it more challenging for viewers to interpret the visualized data accurately. In contrast, Team 2's adjustments to the graph code were more successful in producing graph plots with accurate and expected features like titles, labels, and legends, enhancing the clarity and interpretability of the graphical representations of the model's results.

Model use and interpretation of the model outputs The explanations offered by Team 1 and Team 2 in Figs. 9 and 10, respectively, share similarities in their attempts to shed light on the behavior of gene module combinations under low toxin production rates. Both explanations agree on the system's oscillatory nature and its inability to stabilize at low toxin levels. Furthermore, they provide biologically grounded interpretations, emphasizing how concentrations of toxin, anti-toxin, and receptors interact to influence system dynamics. Nonetheless, as mentioned earlier, the discrepancy in the computational time step of Team 1’s model is a crucial factor contributing to the visible mismatch in spike levels observed in the anti-toxin profile. This mismatch becomes evident when comparing Team 1's model to Team 2's model and the actual solutions of the system model.

Fig. 10

Team 2’s model result output and interpretation

It appears that the students in Team 1 did not fully grasp the significance of the time step error as a potential contributor to the model's output discrepancies, as this factor is not explicitly explained by the students in Fig. 9. This lack of comprehension can be attributed to two key reasons. First, the students may not have an in-depth understanding of the fundamental biomechanics underlying the system's dynamics. To know what's inaccurate, one needs to know what's accurate by having a solid grasp of the underlying principles. To underscore the importance of the students' background knowledge, it's notable that both teams in their result discussions omitted to cite the influence of a significant underlying variable, known as the hill function term. In the students’ reflections below, some students have raised these concerns:

Student A: “My biggest struggling was understanding where hill functions fit within the differential equations and how to correctly incorporate them”

Student B: “I think the most challenging thing I met during the project is to reflect "whether our model" is successful or not. To be more specific, after making sure that our code is correct, the results of our models are not that "ideal". Thus, this took a lot of time to think about it.”

Second, the students may not have acquired sufficient expertise in computational modeling and simulation to monitor the numerical error margin during the convergence process and to recognize the trade-offs between model accuracy and computational costs (memory consumption and computational speed) during the simulation time. In this case, the student’s computational simulation experience holds paramount importance because, in situations where students may identify the plot errors in Fig. 10, they might still struggle to connect these errors to the time step parameter in the code, especially when the code execution appears to be successful. The ability to bridge the gap between the observed discrepancies and the underlying computational parameters is a crucial skill in ensuring the accuracy and reliability of computational models.

Report on the solution of the problem and reflect on the process While there was not a formal report for the project solution, as the computational notebook that embedded the indications and guidance was the actual final deliverable, students had an opportunity to reflect on the process. Students reflected individually on their learning and their team processes collectively. The samples selected for reflecting on the process for Project 2 focused on students’ individual reflections on their learning. Figures 11 and 12 present the individual reflections for Team 1 and Team 2, respectively, regarding their learnings in Project 2.

Fig. 11

Team 1’s individual reflections on team members’ learnings

Fig. 12

Team 2’s individual reflections on team members’ learnings

In terms of skills acquired by members of Team 1 and Team 2, both groups reported that they had to develop a range of skills to accomplish the project. While there are some similarities in the skills mentioned, such as improving python coding skills and working with equations through building differential equations, mass balance equations, or translating biological systems into mathematical models, there are also differences in the specific skills emphasized by each team. Team 1 focused mainly on technical and mathematical skills. They mentioned developing proficiency in creating differential equations, implementing functions in Python, debugging code, and using mathematical methods like Euler's Method. Team 2, on the other hand, highlighted a broader set of skills. They mentioned communication skills, patience, and the ability to understand and translate code from pseudocode into Python. Therefore, while Team 1 prioritized the technical aspects of the project, such as mathematical modeling and coding proficiency, Team 2 recognized not just the technical aspects but also the interpersonal and problem-solving skills. This further highlights the complex nature of these computational modeling projects for students working in teams.

Regarding the aspects of this project that were most beneficial for students’ learning, both teams reported gaining valuable insights from their experiences in mostly similar and some different ways. Both groups emphasized the significance of practical, hands-on learning. They mentioned the value of seeing how changes in code and parameters affected the project outputs. They both appreciated the visual learning aspect, with students noting the importance of graphs, plots, and visualizations for their understanding. They also both expressed an appreciation for the project setup structure in the computational notebooks that guided them through the progression of the project from simpler to more complex aspects, which enhanced the building of foundational knowledge and end-to-end holistic project understanding. With no stack difference between the two teams, some of Team 1’s responses focused on the technical and coding aspects of the project, like the development of Python skills, integration, hill functions, and code execution, while Team 2’s responses also highlighted the importance of visualization and conceptual understanding of the biological model beyond the coding aspect.

Representational competence

Representational competence was operationalized as students’ ability to connect the computational representations of their models, either in the form of code or plots, to biological phenomena. For this, the rubric presented in Table 4 was used to score the accuracy of the models the students generated as the solutions for each of the three projects implemented throughout the semester and students’ explanations of such models and their outputs. The first two criteria were aligned with the computational learning objectives of the course, which focused on creating computational models and students’ ability to recognize the behavior of such models by explaining their functionality via in-code commenting. The second two criteria were aligned with the disciplinary learning objectives of the course, focused on generating the proper solution to the biological problem in the form of plots and interpreting those accurately via explanations. Specifically, we identified and measured that the computational model executed and worked properly, the graphical representations or output were accurate, and that their explanations of the behavior of the system in the form of in-code commenting were accurate, as well as the explanations of the output of the computational model regarding students' interpretations.

Table 7 presents the descriptive statistics for each of the three projects organized by the four constructs of a computational model, graphical representations, meaning-making of the code, and explanations of output. As can be observed, overall, students performed at the advanced level in terms of the ability to create a computational model representing a biological system. However, the ability to properly create graphical representations is overall more consistent with a proficient level of achievement. Regarding the understanding of the code in terms of in-code comments and the explanations of the interpretations of the output of the computational models, the student's achievement was at an advanced level for Project 1 but at a proficient level for Projects 2 and 3.

Table 7 Projects' performance scores for evaluating representational competence

We performed a second analysis at the team level to explore if the teams had similar patterns in their performance regarding (a) the implementation of the model and the understanding of the inner workings of the model via in-code comments and (b) the use of the model to solve the biological problem and the explanations of the outputs of the model. Table 8 presents the average scores for each of the criteria across all the projects and the overall team performance throughout the semester.

Table 8 Projects' performance scores for evaluating representational competence by team

We identified performance patterns where Teams 17 and 21 performed relatively higher across all projects and all criteria, while Teams 6 and 4 performed relatively lower across all projects and all criteria. We also identified that Teams 5, 8, 12, 13, and 14 performed relatively higher in the criteria focused on creating computational models and commenting the code properly but somewhat lower in criteria focused on using the models and interpreting the outputs of the model to solve the biological problem. In contrast, Teams 1, 15, and 19 performed relatively lower in the criteria focused on evaluating the computational models and explaining those via in-code commenting but relatively higher in criteria focused on using models and interpreting the outputs of those models to solve the biological problem. The rest of the teams performed on average across all criteria.

Regulatory skills

Regulatory skills were operationalized as self-monitoring, describing instances involving students assessing their learning and learning processes in the form of experienced learning challenges individually and as a team, and cognition regulation in the form of evidence of students' application of strategies aimed at overcoming learning challenges and teamwork strategies and skills for working with others. To characterize students’ processes, we performed a qualitative analysis focusing on students' individual and team-based reflections submitted along with each project submission. Individual team members reported learning challenges, while collaboration challenges were reported as a team. Regarding self-monitoring regulation, we identified three main categories: (a) challenges and strategies related to programming skills, (b) challenges and strategies related to meaning-making skills for understanding and connecting the science to the code and the results, and (c) challenges and strategies related to process management mainly focused on project management skills. Each of these challenges is described below, along with the cognition regulation strategies students implemented in order to overcome such challenges.

Programming challenges The first theme addresses the team's challenges regarding the programming requirements of the projects. Specifically, teams struggled with transforming the mathematical equations into computational representations, generating the required plots, learning the programming language's syntax, and troubleshooting and debugging the code. Some team members also experienced difficulties reading someone else’s codes, especially when not commented. One student expressed their challenges with Python syntax: "One challenge for me was figuring out the necessary syntax to code things correctly in Python.” Another one indicated the complexity of learning concepts simultaneously: “My biggest challenge was working through both the code and the overall thermodynamics because as the system got more complex, the equations became harder to solve and understand.” Another student highlighted the challenges related to visualizing and plotting “I found the plotting of the functions to be the most challenging because the functions can be hard to visualize.” The first four columns in Table 7 summarize the experienced programming challenges reported by team members. As can be observed in the patterns presented in Table 7, almost every team had difficulties deriving the equations and dealing with aspects of programming syntax.

To overcome the challenges, students implemented various cognition regulation strategies like seeking help from others through consultations with teammates, teaching assistants, or instructors or engaging in independent study by utilizing online resources, textbooks, and class materials. A student is quoted as seeking help from various resources: "I overcame the challenges I faced by asking my group members for help when I didn’t understand and reading through the notes from lecture when there was confusion about how to begin finding a mathematical solution.” Another student utilized the provided class resources, “The professor provided lots of resources that I could use to brush up on my Python skills and reference throughout the project,” while other students consulted each other like this one “I overcame the problem by brainstorming with other students from my group and using a lot of research power on the internet to gain a deeper understanding of the concepts at work in this problem.” Table 8 summarizes the strategies teams implemented to overcome their programming challenges. As can be observed in the patterns presented in Table 8, peer learning and feedback, attending office hours, and debugging code were the strategies implemented by almost every team.

Meaning-making skills The second theme addresses the team’s difficulties in interpreting and reflecting on how the computational models work and what the model results mean. The theme is derived from some teams reporting challenges with understanding the engineering tasks asked and understanding how the overall model system works. This is seen in students’ difficulty with interpreting the provided code blocks and writing in-code comments that connect engineering with programming. One student said, “The most challenges I met in the project is to understand what is going on in the system to explain the result from my code.” Another explained the technical difficulties as follows: “I initially had some trouble finding how all of the components of the system came together into the mass balance equation. The biggest challenge for my task personally was understanding how the ODEs needed to be set up in order to accurately model the function of the enzymes.” Another wondered about the relevance of the inputs to the problem task: “I struggled with knowing why different values input in Part 6 were significant in the context of the problem.” Finally, this student explained the challenges of understanding the graphical outputs:

“The main challenges encountered in completing this project were related to taking a step back after our graphical displays were working properly to ask if they make sense. I needed to discuss with my teammates to understand why the shape changes when we leave a steady state assumption.”

The last three columns in Table 9 summarize the experienced meaning-making challenges reported by team members. As can be observed in the patterns presented in Table 9, the most common challenge experienced by team members involved generating the plots and understanding the underlying science needed to understand the system as a whole.

Table 9 Self-monitoring regulation regarding programming and meaning-making challenges

The cognition regulation strategies used to overcome the challenges detailed in the theme were similar to those used in the first theme about programming. For example, consultations with teammates, teaching assistants, or instructors or engaging in an independent study by utilizing online resources, textbooks, and class materials. Other students expressed that examining the code and analyzing how it works was beneficial, especially with this theme. One student reported, “The questions made me examine the code and dig into the WHY, which was beneficial to my learning. I went through each section and tried to understand the fundamentals.” Another student said, “We had to find outside information as well as seek additional help from office hours as well as TAs. We required finding more insight into the specific parameters in order to understand how to analyze the graphs and plots,” and another reported that “By brainstorming with classmates and reading up on the topic myself for a better understanding.” Table 10 summarizes the strategies teams implemented to overcome their meaning-making challenges. We noticed that the strategies teams implemented to overcome their programming challenges were similar or the same as those students implemented to overcome their meaning-making challenges, where peer learning and feedback and attending office hours were the most frequent strategies implemented by almost every team. This pattern is understandable as the experienced programming and meaning-making challenges were directly related to model-based reasoning processes where students had to create, use, evaluate, or interpret models, which are highly interconnected processes.

Table 10 Cognition regulation strategies for overcoming programming or meaning-making challenges

Process management skills The third theme addresses the students’ difficulties in the process management of the project in terms of planning, communication, division of labor, time management, teamwork coordination, and using the Google Colab technology. Regarding coordination and time management, a student reported, “Not only was the coding itself difficult, but it was hard to work through our varying schedules and find the time for all of us to meet and complete the project.” Concerning the division of labor, a student reported:

“I thought the most challenging part of this project was properly coordinating how the work could be distributed among five people with very different schedules. We struggled to find an appropriate meeting time, and I believe this may have reduced teamwork.”

Table 11 summarizes the challenges the teams experienced regarding their process management. It appears the most common issues reported by most teams were poor project management, clashing schedules to meet and engage in collaborative or individual work, and technical issues experienced with Google Colab.

Table 11 Self-monitoring regulation regarding process management challenges

The teams handled this challenge using various cognition regulation strategies like asynchronous coding in Google Colab at different times, scheduling regular asynchronous and synchronous meetings, assigning internal deadlines to tasks for each team member, discussing the division of labor, and increasing team collaboration. Regarding team coordination, a student reported:

“As the team coordinator for this project, I created a secondary document to track the progress of each section and suggested that the teamwork in order. Team members worked expeditiously to work through every part of the code so that the next person could start working on theirs.”

Regarding the division of labor, a student reported, “teamwork strategies that worked included separating different tasks for the people who were available.” Table 12 depicts the strategies the teams reported implemented for overcoming process management challenges. Planning for the division of labor and improving communication processes were implemented by about half of the teams. Also, to overcome the technical challenges with Google Colab, students created schedules and rotations to work on the computational notebook asynchronously.

Table 12 Cognition regulation strategies for overcoming process management challenges

We further explored the data to identify possible patterns between teams’ computational model-based learning, representational competence, and regulatory skills. No significant patterns were identified regarding commonly experienced challenges or typically implemented strategies related to performance between the meaningful or superficial forms of engagement with computational model-based learning. Overall, teams reported about seven experienced challenges and six strategies implemented to overcome challenges across programming, meaning making, and process management skills. We also explored if a relationship existed between the number of reported challenges and strategies and overall performance, and no significant patterns were identified. However, we offer some qualitative observations here. It appears that Teams 17 and 21, who overall performed at the advanced level across all three projects and were part of the meaningful engagement with the computational model-based learning group, reported experiencing an average number of challenges and fewer than the average strategies for overcoming the challenges they experienced. We also identified that teams that performed at the advanced level regarding the implementation of the model and the understanding of the inner workings of the model via in-code comments, but at the basic level regarding the use of the model to solve the biological problem and the explanations of the outputs of the model (i.e., Teams 5, 8, 12, 13, and 14), on average experienced more challenges, about nine, and reported implementing about eight strategies to overcome those challenges. These five teams were also in the group with teams that engaged meaningfully in computational model-based learning. The teams that were, on average, proficient across the board reported experiencing an average number of challenges, about six or seven, but mentioned they implemented about five strategies to overcome those challenges, which was lower than the average.

Discussion and implications

Our overall findings revealed that our learning design consisting of computational modeling and simulation collaborative projects scaffolded through d processes and delivered via computational notebooks resulted in enactments of computational model-based learning. Specifically, sense-making and process management scaffolding guided students through enactments of (a) analyzing the problem and planning models to approach the solution of the problem, (b) implementing and evaluating the models to solve the problem, (c) using the model and interpreting the outputs of the model, and (d) reporting on the solution of the problem and reflecting on the process. Such enactments are critical for students to promote model-based learning (Develaki, 2017; Lehrer & Schauble, 2006). However, based on qualitative observations from our characterizations of computational model-based learning, students experienced these processes differently, even when the scores of their representational competence were comparable. Specifically, while one group of 16 teams, such as Team 2, engaged more meaningfully in the computational model-based learning process, another group of 8 teams, such as Team 1, appeared to be less engaged in in-depth model-based reasoning. Thus, this finding suggests that some teams may need more scaffolding or feedback through the modeling and simulation cycle.

A second outcome of the study focuses on ways to support students' development of representational competence. Specifically, articulation scaffolding supported students in their computational model-based learning processes while creating and using computational models, thus resulting in forms of representational competence. Students were guided in processes of self-explanation as they articulated details of the implementation of their models in the form of in-code comments and the explanations of the outputs generated by their models explaining the biological phenomenon. Findings suggest that the articulation scaffolding was useful in helping students achieve working solutions for the biological problem posed to them, as evidenced by the overall level of representational competence for all teams and all projects ranging between the proficient and advanced levels of performance (over 3.5, on average). We can also conclude that students performed at the advanced level in tasks associated with creating the models and explaining the models with an overall average performance of 4.3 and over, as shown by students’ understanding of all or most aspects of the code, as evidenced by their in-code commenting. It can also be concluded that students performed at a proficient level in tasks associated with creating proper outputs of their models and explaining those outputs in terms of the biological phenomenon. These findings also suggest that providing students with templates of code and test cases was an effective strategy to guide learners through the steps of implementing a solution. This is consistent with previous studies, which have identified that test cases and templates can be an effective strategy for the sense-making of complex programming topics (Denny et al., 2019; Hislop & Ellis, 2009), as well as educational evidence-based teaching practices such as the use-modify-create approach, commonly implemented in learning programming contexts (Lee & Malyn-Smith, 2020; Lee et al., 2011). Our results contribute to this body of research by elaborating on additional scaffolding, such as reflection and articulation scaffolding, which can be useful additions to help students better implement, use, and learn from computational models.

When comparing the students’ level of representational competence in computational modeling, graphical representation, code interpretation, and model output explanation skill categories, we observed that the overall performance in the graphical representation scored the lowest, averaging just over 3.7, when contrasted with the other three skill categories, all of which averaged over 4.1. It's important to highlight that, despite students receiving code scaffolds for generating plots, certain teams encountered difficulties when adapting the provided code to create precise graphs. They struggled to add just a few lines of code to ensure accuracy while maintaining crucial details, such as axis scale size, plot line colors, titles, axis labels, and plot legends. These findings suggest that students need clear guidelines to familiarize themselves with the significance of producing descriptive graphical plots. Research suggests that students may need direct instruction or scaffolding in graph-reading and graph-generating, which can result in graphical literacy skills (Witkow et al., 2022). For instance, students may benefit from tailored code scaffolding aimed at enhancing students' graphical representation skills, along with explanation guides that highlight key domain-specific variables students must elaborate upon when making argumentations about the model results. Through the reflection data, we also identified that some students experienced difficulties properly explaining the outputs of their models. This finding is consistent with prior research that suggests that students may be able to produce good plots while having little understanding of the underlying physical meaning of the plot (Nixon et al., 2016). This finding can be attributed in part to the struggles students often experience in connecting programming tasks with their real-world contexts (Magana et al., 2013; Ryoo, 2019). Thus, in future implementations of the course, we will implement and further investigate the usefulness of providing students with an argumentation framework and corresponding training that will help them align claims with evidence and supported reasoning by leveraging model results, often in the form of plots (McNeill et al., 2006).

A third outcome of the study consisted of identifying students' reported regulatory skills that were made explicit through the reflection scaffolding. Specifically, students reflected on their learnings, the challenges experienced in the process, individually and as a team, and strategies they implemented to overcome those challenges. We identified specific themes that characterized students’ experienced difficulties, along with the strategies the students used to overcome those challenges. While no evident patterns were identified regarding the relationship between representational competence, challenges, and regulatory skills, it appears that teams who consistently implemented computational models at the advanced level also identified more challenges experienced in their reflections and implemented more regulatory strategies to overcome those challenges. Learning outcomes have been positively associated with metacognitive and regulatory strategies (Guo, 2022). However, since this observation is based on reflection data, there is no way to verify if these strategies were truly enacted and if they were indeed effective. Future course implementations could consider formal instruction that can promote co-regulated learning skills and the effectiveness of such instruction or scaffolding in students’ improvement of their metacognitive processes (e.g., Udosen et al., 2024). Taking a co-regulated learning lens can support learners not only in their planning, monitoring, and evaluating processes but also in the central role of social transactions (Rich, 2017). Such social transactions can “emphasize shared control of learning and the role more advanced others play in scaffolding novices’ metacognitive engagement” (Rich, 2017, p. 1100).

Also, while two of the major reported challenges were related to learning the concepts and skills of the course (i.e., programming models and explanations of the outputs), a third challenge was related to teamwork coordination and project management skills. For instance, from the qualitative data, it was observed that students experienced challenges with time and project management skills. Research has previously identified that time and project management skills often directly correlate to student performance in the classroom (Adams & Blair, 2019). Thus, future implementations of this course will consider providing students with additional process management scaffolding to aid student performance.

Research has consistently reported that teamwork and project management skills need to be explicitly taught, as just having students work in groups may not translate into becoming teams (Wheelan et al., 2020). Although the students were provided with process management scaffolding to provide some planning at the beginning of the semester and at the beginning of each project, more scaffolding or training was needed in order to guide them through the implementation and completion of the project. The team contracts signed at the beginning of the semester may have helped students decide on the communication medium and coordinate potential meeting times. However, more guidance is necessary to help students define internal deadlines and potential guidelines for role assignments and the distribution of work and timing within each of the project’s submissions (De los Ríos-Carmenado et al., 2015).

Implications for STEM education research, teaching, and learning

The first implication of this study focuses on studying model-based learning in the context of team-based scaffolded computational modeling and simulation practices. Operationalizing model-based learning in the context of computational modeling and simulation opens up two aspects of discipline-based education research. First, we can investigate computational model-based learning by focusing on people’s knowledge and skills (Chinn et al., 2014). By looking at model-based learning this way, we can consequently design, implement, test, and further revise educational practices that can support students in the form of computational model-based learning (Seel, 2017; Simarro & Couso, 2021).

A second implication of the study contributing to the body of knowledge in discipline-based education relates to the delineation of computational modeling and simulation practices, specifically characterizing them in terms of model-based learning and their relation to representational competence and regulatory skills. Specifically, computational model-based learning occurs when learners (a) engage in the process of creating and using computational models and generating explanations of the computational models and the outputs of such models and (b) use regulatory or co-regulatory learning strategies when approaching the solution to the computational modeling and simulation projects. This finding is significant as previous research has mainly characterized model-based learning as the processes of using, creating, evaluating, or revising models (Johnson-Laird, 1995; Nersessian, 2008; Schwarz & White, 2005).

A third implication of this study is that students, as they work in teams in solving computational modeling and simulation assignments, can develop some levels of representational competence and regulatory skills. Thus, regarding teaching and learning, it is important to provide students not only with scaffolding to support students in the development of their representational competence but also learning strategies for promoting the development of self-regulated and co-regulated learning (Hadwin et al., 2018). For instance, guidelines from self-regulated learning (Zimmerman, 2000) can provide students and their teams with strategies and project management methods for engaging in processes for (a) planning, including goal setting and strategic planning, (b) performance monitoring, including deliberate self and peer instruction, time management, and metacognitive monitoring, and (c) self-reflection including self-evaluations of the work achieved and the processes involved for achieving that work. Furthermore, it is also important that students develop socially shared or co-regulated learning behaviors as they engage in problem-solving processes, share common goals, and achieve those goals through interactions with peers (Hadwin & Oshige, 2011). By doing so, students would not only improve their own self-regulated learning skills but also develop effective teamwork skills (Aggrawal et al., 2023; Tsai, 2015). If students can effectively develop self-regulated and co-regulated learning skills when scaffolding fades, the literature suggests they may be able to continue to succeed in completing their computational projects while working in teams (VanDeGrift et al., 2011).

Conclusion, limitations, and future work

The main contribution of this study is twofold. First, this study provided scaffolding guidelines in the form of sense-making, articulation, process management, and reflection, embedded in team-based computational modeling and simulation projects that promoted computational model-based learning. Second, and more importantly, for discipline-based education research, this study characterized different levels of students' engagement, more meaningful and more superficial, of computational model-based learning in the context of modeling and simulation projects operationalized as evidence for (a) analyzing the problem, planning and formulating a model, (b) implementing and evaluating the model to solve the problem, (c) using the model and interpreting the outputs of the model, and (d) reporting on the solution of the problem and reflecting on the overall process. Characterizing model-based learning in the context of computational modeling and simulation represents important contributions that showcase students’ benefits and experienced challenges when engaging in model-based learning and working with others to solve computational projects. Third, the study described how computational model-based learning promoted different levels of (a) representational competence in the form of working models and accurate outputs of the models and corresponding explanations and interpretations and (b) regulatory skills that students reported they applied to overcome learning and collaboration challenges.

Despite its contributions, this study suffers from limitations associated with intervention studies with no control condition in that its design limits the generalizability of the findings to similar contexts. In this regard, the study had limited information regarding students’ backgrounds, such as information regarding students’ socioeconomic backgrounds or regarding non-traditional students, thus further limiting the generalizability of the findings. Also, findings associated with regulatory processes rely on self-reported data, which may or may not have been enacted by the teams or that were enacted but not reported or noticed by team members. Similarly, the reflection data may have been influenced by team dynamics, including team size, as well as the cultural or demographic backgrounds of team members. However, the rich environment of this intervention and the thorough analyses provide a unique context that contributes to our understanding of computational model-based learning. Thus, the findings have potential transferability to similar contexts. Theoretical considerations for future research involve a deeper understanding of model-based learning, among other cognitive activities, and their applications to using computational modeling and simulation practices for solving ill-structured problems. Practical considerations for future research involve investigating the effect of promoting co-regulated learning skills and gradually removing scaffolding.