An investigation of verbal episodes that relate to individual and team performance in engineering student teams

Abstract

Background

Being an effective team member is one of the key twenty-first century skills and a fundamental proficiency required for jobs and work settings in an increasingly global economy. Collaborative problem-solving in team settings is a critical practice in science, technology, engineering, and math (STEM) fields as an effective teaching method that is found to promote outcomes associated with individual student learning and the quality of team solutions. However, the social and discursive dynamics of a team can impact these outcomes. The primary goal of this study was to examine the different types of verbal episodes (questions, conflict, and reasoning episodes) in engineering student teams and how these verbal interactions related to the individual and team performances.

Results

Our results showed that different verbal episodes played a significant role on students’ individual success and team performance. Students spent most of their time on question episodes, followed by reasoning episodes, and less frequently so on conflict episodes. The linear combination of question, conflict, and reasoning episodes was significantly related to students’ individual achievement scores. Specifically, the number of question episodes was the best predictor of individual performance. And among six subcategories of question episodes, three (open question, verification answer, and elaborated answer) were significantly correlated with the individual achievement scores. Furthermore, according to the regression model, elaborated answer category was significantly related to individual achievement compared to all other categories. In terms of team performance, results showed that reasoning episodes were significantly related to team performance, and the calculation subcategory of reasoning episodes was the best predictor of team performance. Finally, we found that the teams with more balanced participation among team members performed significantly better than the teams with unequal participation among team members.

Conclusions

These results suggest different verbal episodes are related to individual and team outcomes. Discourse in teams is critical not only for the improvement of academic achievement for individuals but also for the development of better team solutions. Moreover, balanced participation from all team members is important, suggesting teams should be formed, monitored, and supported to ensure balanced participation. Our results recommend that educators should monitor team interactions and promote verbal exchanges that promote student learning and positive team outcomes.

Introduction

Teams play a critical role in organizations, companies, and academic institutions. As businesses and organizations become increasingly global and complex, the required tasks, challenges, and solutions also become more team-based. As a result, engineers are often expected to work in team projects (Borrego, Karlin, McNair, & Beddoes, 2013; Cohen & Bailey, 1997; Dossick & Neff, 2011). They need to communicate with other team members as they exchange their knowledge and insights, give and receive feedback, and negotiate to reach decisions. Similar to other STEM (science, technology, engineering, mathematics) professionals, engineers are required to have a wide range of technical and professional competencies to solve problems collaboratively and efficiently in their work settings. Not surprisingly, communication is one of the most critical skills that many engineers count as important (Katz, 1993; Passow, 2012; National Association of Colleges and Employers, 2014). In fact, engineers spend a significant amount of their work time on communication (Darling & Dannels, 2003). Accordingly, the Accreditation Board for Engineering and Technology (ABET (2017) state that engineering graduates must have “… (d) an ability to function on multidisciplinary teams, (e) an ability to identify, formulate, and solve engineering problems, …(g) an ability to communicate effectively.” Similarly, a survey from more than 5000 engineers reported that teamwork, communication, and problem-solving were the most important skills for engineering in the workplace (Passow, 2012).

To ensure that engineering students develop these competencies, engineering schools have implemented approaches and developed curriculum that promote problem-solving skills in teamwork settings (Anwar, Menekse, Heo, & Kim, 2018; Dym, Agogino, Eris, Frey, & Leifer, 2005; Felder & Brent, 2003; Froyd, Wankat, & Smith, 2012). However, solving problems in a time-constraint situation and often with insufficient data is challenging even for experienced engineers (Katsikopoulos, 2012; Sha & Panchal, 2014). Thus, the problem-solving processes can become even more challenging for team members when they need to make decisions about ill-structured problems. In order to understand what kind of challenges students experience in team settings, there is a need for studies that analyze the process data (or verbal data) while students are working in teams.

Although prior studies have examined teams and team-based learning of engineering students (e.g., Huang, Shih, & Lai, 2011; Terenzini, Cabrera, Colbeck, Parente, & Bjorklund, 2001), most of these studies relied on students’ survey responses or other measures without exploring the verbal data that exposes how teams actually collaborate. There are a few studies that have examined verbal data in engineering student teams (e.g., Purzer, 2011; Eggert, Joshi, Mehrotra, Zastavker, & Darer, 2014; Menekse & Chi, 2018; Tonso, 2006). Some studies have shown that group interactions and discourse processes can facilitate learning (e.g., Purzer, 2011; Chi & Menekse, 2015; Menekse, Higashi, Schunn, & Baehr, 2017). However, these interactions may also reduce successful collaboration (Kuhn, 2015). Therefore, it is important to study the types of verbal interactions and their relationship to individual achievement and team performances. The primary goal of this study is to examine the relationship between verbal interactions that occur in teams and how these verbal interactions relate to the individual achievement and team performance. More specifically, the present study focused on the following questions:

• 1. How do different types of verbal episodes relate to individual student achievement?

• 2. How do different types of verbal episodes relate to team performance on an engineering design project?

• 3. How does the distribution of individual contributions among team members relate to team performance?

Relevant literature

Collaborative learning and problem-based learning

Recent reform efforts and reviews of high-quality classroom instruction highlight the role of collaboration and student learning (Chi & Menekse, 2015). Recommendations incorporate collaboration during authentic disciplinary activity, which in STEM fields involve argumentation and evidenced-based explanation. Engaging in collaborative groups provides cognitive benefits and deepens conceptual understanding given opportunities to explain, question, justify, and negotiate, with benefits for learning and achievement outcomes. However, there are challenges to obtaining meaningful collaboration, which translate into group variation in learning outcomes. There are challenges to maintaining joint attention and on-task behavior, equitable participation among group members, coordination and regulation of common goals, and responding to disagreement by legitimizing alternative perspectives (Menekse, 2012; Menekse & Chi, 2018).

Collaborative learning is often crucial for achieving conceptual knowledge while solving problems (Van Boxtel, Van der Linden, & Kanselaar, 2000). In the study of Van Boxtel et al. (2000), the authors defined the conceptual knowledge as a learning process where students embellish, organize, and refine the knowledge and develop the ability to use scientific concepts. They argued that collaborative learning settings for team projects facilitate elaborating conceptual understanding.

The collaboration among team members is also the main structure for the problem-based learning (PBL) activities (Hmelo-Silver, Chinn, Chan, & O’Donnell, 2013). PBL is one of the examples that facilitate active learning as students construct knowledge by themselves and solve real-world problems based on their experiences (Barrows, 1986; Cockrell, Caplow, & Donaldson, 2000; Hmelo-Silver, 2004; Jonassen & Hung, 2008; Quinn & Albano, 2008; Savery, 2006; Streveler & Menekse, 2017; Wood, 2006). Barrows’ (2002) definition of PBL included that the learning characteristically occurs in a small group, face to face, with active discussion among the learners. Advocates of PBL point out that “soft skills” such as collaboration and self-directed learning can be acquired through problem-based learning experiences (Hmelo-Silver, Duncan, & Chinn, 2007). Barron (2000) observed that the success of teams of middle school students working on a math-focused problem-solving activity hinged upon collaborative behaviors (or the lack thereof). We might therefore surmise that PBL both depends upon and teaches collaborative work skills.

Problems that are used in PBL settings are ill-structured which means they are vaguely defined and can have multiple solutions. This can especially be challenging for students who do not have much prior knowledge related to the given task, as they often do not know where to start. Thus, collaborative learning is one of the fundamental elements of PBL since students can reduce cognitive load by sharing each team member’s expertise via verbal interaction (Cockrell et al., 2000; Hmelo-Silver et al., 2013). In other words, the interaction with team/group members is a key resource to solve tasks much more efficiently.

Simply having students work in teams, however, is not sufficient to access the benefits of collaborative learning in PBL as the nature of collaborative learning is a complicated mechanism. In fact, some previous research studies found the cases where collaborative learning was not always productive (Kuhn, 2015; Pai, Sears, & Maeda, 2015). For instance, Pai et al. (2015) reported the difference on students’ performance such as learning transferability in their meta-analytic study that investigated the effect sizes of teamwork from previous studies. Chi and Menekse (2015) conducted a research study to explore the reason that discrepancy occurred among effect sizes about collaborative learning in small groups. They studied dialog patterns of dyads and found that students were engaged in three different types of dialog patterns as constructive, active, and passive based on the ICAP framework (Chi, 2009; Menekse, Stump, Krause, & Chi, 2013). Chi and Menekse (2015) argued that students could promote learning when both partners are constructive. On the other hand, there was no benefit of collaboration, beyond individual learning, when partners in dyads were either passive-active, passive-constructive, or active-active.

Educators and administrators in engineering programs have also adapted PBL approaches with team projects in their courses (Prince, 2004; Yadav, Subedi, Lundeberg, & Bunting, 2011). Yadav et al. (2011) compared the effect of PBL on team learning versus traditional lectures on individual learning. They collected data from electrical engineering undergraduate students for their conceptual understanding of certain concepts and perceptions of learning. They found that students’ performance was significantly improved during the two PBL phases and the difference, was vastly greater compared to when students were introduced to the traditional lectures. Accordingly, PBL can be suitable to train engineering students to solve ill-structured problems that they would face in the real work settings (Christensen & Schunn, 2007; Dixon & Johnson, 2011).

Understanding the team interaction is important to design effective engineering course curricula with PBL as it can affect both individual achievement and team performances. Therefore, researchers should investigate team discourse more thoroughly that could hinder or promote successful teamwork as well as individual performance. However, we observed that discourse data have not been investigated as much in the engineering education field regarding team work.

The role of verbal discourse in teams

One key process in engineering is collaboration. Successful collaboration requires effective communication, coordination, trust, and respect among team members, as well as individual and team level responsibility and accountability to achieve certain tasks. Literature has shown the significant role of peer interactions and verbal communication for knowledge construction (Goldman, 2018; Mezirow, 1997). However, some studies have shown that achieving successful collaboration is challenging, and working in small groups is not always beneficial in terms of group performance and individual learning (e.g., Chi & Menekse, 2015; Barron, 2003). Dillenbourg and Hong (2008) argued that the lack of the elaborated explanations, mismatch in mutual regulations of cognitive processes between group members, low quality of arguments, and the nonexistence of negotiation of meanings reduces the effectiveness of collaborative learning.

Verbal discourse characteristics in team settings are critical and often strongly relate to the overall team performance (Chinn, O’donnell, & Jinks, 2000). Prior studies have found that types and frequency of verbal discourse during team interaction affect the collaborative learning and decision-making process (e.g., Salter & Gann, 2003). Saab, Joolingen, and Hout-Wolters (2005) investigated the relationship between communicative activities and discovery learning process. They found a significant relationship between these variables. Indeed, team discourse, captured in the form of verbal episodes, can be seen as the most critical set of data that can be used to predict student learning outcomes. Thus, it is important to accurately analyze verbal episodes so that we could have more thorough understanding of team interactions.

A significant number of research studies have been conducted on engineering students’ team processes and team problem-solving. Some of these studies investigated team-related constructs to find how teamwork influenced learning outcomes or other variables such as engagement without analyzing process or verbal data (Fila & Loui, 2014; Hsiung, 2010; Hsiung, 2012; Huang et al., 2011; Mishra, Ostrovska, & Hacaloglu, 2015; Pasha-Zaidi et al., 2015; Terenzini et al., 2001; Zafft, Adams, & Matkin, 2009; Zhou, Luo, Du, & Kolmos, 2010). On the other hand, a few other studies specifically focused on exploring team interactions through discourse analysis by collecting and analyzing audio and/or video-based verbal data (Donath et al., 2005; Haller, Gallagher, Weldon, & Felder, 2000; Kittleson & Southerland, 2004; Ponsa Asensio, Román Jiménez, Arnó Macià, & Pérez Soriano, 2015; Whitman et al., 2005).

For example, Haller et al. (2000) investigated the social dynamics by discourse analysis to find how team members learned from each other in an engineering class. They observed that students were engaged in two types of interaction in group settings: transfer-of-knowledge sequences and collaborative sequences. They also found that problems in team dynamics mostly occurred during transfer-of-knowledge sequences since team members either discouraged other team members’ contribution or had to assume their roles for the task. In another example, Purzer (2011) performed a mixed-methods study to find the relationship between types of discourse actions, self-efficacy, and student achievement. Purzer collected the data from 22 first-year engineering students in an introduction to engineering design course. Purzer conducted the discourse analysis from six teams to explore the verbal exchanges among team members during the project. The results showed a moderate positive correlation between post self-efficacy and the level of engagement in support-oriented discourse.

While some studies focused on verbal interactions of engineering teams, there is still much to be understood about how engineering students learn and interact in team settings. Verbal data contains rich information that is helpful for researchers to understand students’ discourse patterns in team settings. In prior studies of engineering students’ teams, some researchers have studied verbal discourse but did not explore the relationship among the quality of team interaction, team performance, and individual performance except Purzer’s (2011) study. Thus, more studies are needed to explore team verbal discourse to understand how it is related to engineering students’ individual achievement and team level success.

In this present study, we explored how engineering students’ verbal episodes in team settings relate to their individual achievement as well as their team performances on an engineering design project. In addition, we studied how the distribution of individual team member contributions during teamwork affect their team level success.

Method

Participants

Data were collected from seventy-three first-year engineering students at a large public university in the United States. Sixty-four of them were male and nine were female students. Three or four students were assigned to teams at the beginning of the semester based on their availability to allow common meeting times. These students continued to work with the same team members on different assignments and projects throughout the semester. There were nineteen teams in total and the age range of students was 18–19. Sixteen of these teams had four student members and three teams had three student members.

Data sources

Collection of verbal data

All 19 teams were asked to work on a problem-solving assignment requiring analysis and decision on a system that would reduce the energy consumption of a town library with the least cost. Students were provided with three options as installing solar panels, installing a green roof, or making no changes to the existing design. Based on their decision and justification, all teams were responsible to make a recommendation to the client. The solar panels option represented a solution with the least energy consumption. The green roof option created the least carbon dioxide emissions with an added esthetic benefit. And the no change option represented the lowest cost, and thus the most economical option.

Each team was asked to create a document outlining their recommendation and reasoning behind it. Specifically, teams were asked to describe their problem scoping, specify a plan and/or process, explain the formulae for total system cost, construct a graphical representation of ten-year cost for all current and new systems, and finalize the decisions and justifications in 60 min. All teams were video-recorded during this task, so we collected a total of 19 h of video data. They also produced reports indicating calculations, data for each design option, and their decision with a rationale. The task was graded by the instructional team as part of a team exam, and hence students were incentivized to work on the task and do well. This team problem-solving activity happened during week 11 in a 16-week-long academic semester. Prior to this assignment, the same teams also worked on multiple different assignments; therefore, team members had a history of working together on similar assignments. The “Appendix” section shows an exemplary episode of student discussions.

We decided to video record the team problem-solving activity at week 11 in order to use this verbal data as a predictor for future team performances and individual student performances. Also, the team problem-solving activity was a time-limited assignment and each team had to complete all aspects of the assignment within 60 min. And this was the only time-limited team assignment during the semester. Therefore, it provided equal time and resources to all teams to work on the same assignment. For other assignments, teams had opportunities to meet out of class times and to complete before coming to classes. Therefore, we did not have any control or process data on how teams were interacting while working on other team assignments.

Outcome measures

Individual performance

Students’ exam scores were used as a measure of their individual performance. The exam covered specific skills needed for the problem-solving task such as analyzing data using Microsoft Excel and were created by the instructional team and graded by teaching assistants with no involvement from the research team. As we described in the “Collection of verbal data” section, students worked on to choose a system that would reduce the energy consumption of a town library with the least cost as a team assignment. Similarly, the individual exam required students to make decisions based on economic and environmental analysis for different scenarios individually. The individual exam was administrated after the team problem-solving assignment that was videorecorded.

Team performance

All student teams were asked to work on an engineering design project in the last 4 weeks of the semester. This project was different than the problem-solving assignment that we collected video data from all teams. The focus of the project was to design a commuter system for university campus and surrounding city to reduce the energy consumption of the existing system. In addition, projects needed to identify an innovative approach to reducing energy consumption and greenhouse gas emission. Each team was expected to assess the energy efficiency and usability of the current system and propose a more efficient and innovative system that is feasible in 2020. Each team project was evaluated based on six criteria: (1) problem scoping (10 points), (2) criteria & constraints (10 points), (3) concept generation (20 points), (4) concept reduction (20 points), (5) design & value proposition (20 points), and (6) final design (20 points). Each team received a score on a scale of 100 based on these six criteria. Similarly, the final design project was created by the instructional team and graded by instructors and teaching assistants with no involvement from the research team. There was no video or audio data for this engineering design project.

Question-conflict-reasoning coding protocol

The question-conflict-reasoning coding protocol was developed based on the synthesis of three highly-cited protocols from previous research studies regarding collaborative learning. These three protocols were as follows: (1) the conceptual knowledge elaboration scheme (Van Boxtel et al., 2000), (2) the different communicative activities (Saab et al., 2005), and (3) the dialog utterances (e.g., Erkens & Janssen, 2008). We found three major categories common in these three protocols, which were questioning, conflict, and reasoning episodes. In addition, each episode had multiple categories (11 subcategories in total). Table 1 provides detailed descriptions of each category and examples associated with that category. By using this coding protocol, we analyzed students’ interactions within teams while they were working on exploring the best system to reduce the energy consumption and cost of a town library. These three verbal episodes with 11 subcategories captured most of the different interactions among team members. Since students were working on a time-limited and graded team assignment, all teams were working hard to complete it within the time frame. Also, all team interactions were videotaped, thus there was very little off-task behaviors.

Table 1 The question-conflict-reasoning coding protocol

Two raters scored nine team videos individually for verbal episodes. Each rater worked on identifying the number of different verbal episodes in each of these nine videos and then we compared the number of episodes across two raters. Based on the absolute agreement across raters, the intra-class correlation was .93 for the nine videos that both raters scored, which indicates an excellent agreement between raters (Heyman, Lorber, Eddy, & West, 2014). After this step, two raters discussed the differences in their ratings and finalized the scores. The rest of the videos was scored by one of these two raters. The verbal episodes were calculated at the individual level. Hence, each student had a score for each verbal episode and subcategory.

Question episodes

Segments for question episodes begin with a question and end with an answer. Questions could take the form of a statement when a question is implied. However, the episode may extend if a follow-up question is asked regarding or requesting further information about the previous answer. The question episodes were classified into six categories: (1) verification questions, (2) search questions, (3) open questions, (4) verification answers, (5) short answers, and (6) elaborated answers. For example, questions by any team member to their peers that were asking to verify a specific information or an action were coded as verification questions.

Conflict episodes

Segments for conflict episodes begin with a response to a statement or action that presented conflicting opinions or understanding. These segments generally end when the conflicting opinions or understandings were resolved and agreed upon. The conflict episodes include two categories as conflicts elicited and conflicts elaborated. Statements for the conflicts elicited category display disagreements with a peer’s statement. And, statements for the conflicts-elaborated category refer to the arguments that clarify and conclude the conflicting ideas.

Reasoning episodes

Segments for reasoning episodes begin with a statement concerning a particular topic or concept that conveys useful information for others. The episode was extended when additional statements from the same person or others served to supplement the initial statement. The segment could end in three ways: (1) no responses from others regarding the topic before a new topic is initiated, (2) general agreement among team members, or (3) with a conflicting statement rather than supplementing statement (which begins a “conflict episode”). The reasoning episodes include three categories as observational, calculation, and procedural. The observational category refers to the statements that are inferred from given text, information, or data. The calculation category refers to the statements that include calculation procedures initiated and completed by team members by using given information or data. And the procedural category refers to the statements that indicate propositions about the course of action through reasoning or hypotheses.

Table 2 shows the total number, mean, and standard deviation of verbal episodes and associated categories. Data shows students primarily engaged with the categories related to question episodes. On average, each student involved approximately 20 question episodes during 60 min of teamwork data that we collected for each team. Based on the categories, students generated more search and verification type questions rather than the open type questions. Accordingly, students generated short and verification type answers rather than the elaborated type answers. Furthermore, on average, the smallest number of verbal episodes were related to the conflict type episodes.

Table 2 The total count, means, and standard deviations for each verbal episode and related subcategories

Distribution of team participation

We were interested in understanding how the distribution of individual team members’ contributions to teamwork relate to their team performances. In other words, how the equal or unequal individual contributions among team members predict team success for the engineering design project. First, we calculated dispersion scores for each team by using total verbal episodes for each team. We used the variance-to-mean ratio formula (VMR), which is a commonly used index for dispersion scores. The actual formula is the ratio of the variance (σ²) to the mean (μ). Accordingly, the relatively lower VMR index indicates a more equal contribution from all team members. And the relatively higher VMR index indicates an unequal contribution from team members. For example, in a hypothetical team with all four team members having equal number of verbal episodes, the VMR index would be equal to zero since the variance would be zero.

Next, we created two groups as homogenous versus heterogeneous teams by using the median split method on the VMR indices. The homogeneous teams refer to the ones that individual team members have relatively similar number of verbal episodes, whereas the heterogonous teams represent the ones that individual team members have relatively different number of verbal episodes. As a result, we created two groups in which one of the groups included relatively more homogeneous teams in terms of contribution to teamwork, and the other group included relatively more heterogeneous teams.

Analyses and results

We conducted three sets of analyses. The first set of analyses focused on the relationship between verbal episodes and individual student performance. The second one focused on the relationship between verbal episodes and team performance. And the third one focused on the dispersion of team members’ participation during collaboration and how the dispersion of teams relates to their team performance.

Verbal episodes and individual student performance on exams

We conducted regression analysis to explore how students’ involvement in different verbal episodes predicts their individual exam performance. First, we used three verbal episodes (question, conflict, reasoning) as the predictors, while the dependent variable was students’ individual exam performance. And the linear combination of the three verbal episodes was significantly related to the students’ individual exam scores, F(3, 69) = 2.85, p < .05. The sample multiple correlation coefficient was .33, indicating 11% of the variance of the students’ individual exam scores can be accounted for by the linear combinations of verbal episodes exchanged in their teams.

Next, we conducted stepwise regression to identify the best predictors among the different types of verbal episodes. The results showed that the best model kept the question episodes and excluded conflict and reasoning episodes. In other words, the number of questions asked and/or answered by students was the best predictor for the students’ individual exam scores, F(1, 71) = 8.57, p < .01, R² = .11.

Finally, we explored which subcategories of the question episodes are the best predictors for the students’ individual exam scores. We conducted another stepwise regression by using the six categories of the question episodes (verification questions, search questions, open questions, verification answers, short answers, and elaborated answers). Results showed that the number of elaborated answers was the most significant among these six categories, F(1, 71) = 6.78, p < .01, R² = .09. Table 3 presents the bivariate and partial correlation of all six categories with the students’ individual exam performances. The bivariate correlation, also known as Pearson product-moment correlation, indicates the strength of linear relationship between two variables. In addition, the partial correlation indicates the strength of linear relationship, while controlling the effects of additional variables. The results of bivariate correlations showed three categories (open questions, verification answers, and elaborated answers) were significantly correlated with the exam performance; however, only the partial correlation between the elaborated answers and the exam performance was significant. Based on these regression and correlation analyses, we can conclude that the elaborated answers category was significantly related to students’ individual exam scores.

Table 3 The bivariate and partial correlation coefficients of the question episode categories with students’ individual exam scores

Verbal episodes and team performance on final project

As described in the methods section, all teams worked on a final engineering design project in the last four weeks of the semester. The focus of the project was to design a commuter system for the university campus and surrounding city to reduce the energy consumption of the existing system. We used team scores on this design project as the outcome measure to evaluate how students’ verbal episodes relate to their success on team design projects. We first conducted correlation analyses, by using three verbal episodes (question, conflict, reasoning) and team scores on the engineering design project. The correlation showed that there was significant correlation between the number of reasoning episodes and team performance, r(19) = .48, p < .05. Next, we conducted a bivariate linear regression by using the reasoning episodes as the predictor and the team score as the outcome measure. The results showed that two variables were linearly related such that the team scores increased with the increase in the number of reasoning episodes as represented with the regression model:

Team Score = 1.48 Reasoning Episodes + 62.36

Furthermore, we were interested in which categories of the reasoning episodes are the good predictors for team performance. We conducted stepwise regression to identify the predictors among observation, calculation, and procedure categories. The results showed the best regression model kept the calculation category and excluded the observation and procedure categories, F(1, 17) = 6.21, p < .05, R² = .27. In other words, the number of calculation-related episodes was the best predictor for team performance on the engineering design project. Calculations were important to this task as it took a significant amount of the time for students to make the calculations before making a recommendation based on cost and carbon dioxide emissions from different solutions.

Distribution of team participation and team performance on final project

As we described in section 3.4, we were also interested in understanding how the distribution of individual team members’ contributions to teamwork relate to their team performance on the final project. We used the VMR index for the dispersion scores and we created two groups as homogenous versus heterogeneous teams by using the median split method on the VMR indices. We conducted a one-way ANOVA to understand the differences between these two groups based on their team score on the engineering design project. The results showed the homogeneous teams performed significantly better than the heterogeneous teams, F(1, 17) = 5.49, p < .05. The η² was .24, indicating the dispersion factor accounting for 24% of the variance of the team scores on the engineering design project.

Discussion

In this study, we explored the different types of verbal episodes (questions, conflict, and reasoning episodes) in engineering student teams, and how these verbal interactions related to the individual and team performances. For the first research question, we investigated how different verbal episodes relate to individual student achievement and the results showed the question episodes were the best predictor among all three verbal episodes. And among all six subcategories for the question episodes, the elaborated answers subcategory was significantly related to students’ individual exam scores. The second research question focused on examining the relationship between verbal episodes and team performance. Results showed the reasoning episodes were significantly related to team performance, and the calculation subcategory of reasoning episodes was the best predictor of team performance. For the third research question, we focused on how the distribution of individual contributions among team members relates to team performance. We found that the teams with more balanced participation among team members performed significantly better than the teams with unequal participation among team members.

We also found students spent most of their time on question episodes, followed by reasoning episodes, and less frequently so on conflict episodes. This finding is similar to Purzer’s (2011) finding that conflict (labeled as challenge-oriented discourse in Purzer’s (2011) study) was the least observed. These results are not surprising since questions are central to students’ learning processes. Prior studies have shown that ability to generate questions and to provide meaningful answers are crucial to understanding (e.g., Graesser, McMahen, & Johnson, 1994; Watts, Alsop, Gould, & Walsh, 1997). Both generating and answering questions involve monitoring and regulation of knowledge. According to Graesser et al. (1994), there are four types of questions asked in conversations: (1) questions to address knowledge deficit, (2) questions to monitor common ground, (3) questions for coordinating social action, and (4) questions for controlling conversation. While we observed all four types of questions in our dataset, the first category, knowledge deficit, was the most common among all. This can also be explained by the fact that students were given a limited time to work in team settings. And they spent most of their time for seeking related information to solve the ill-structured problem.

The result that the elaborated answer category was the best predictor for individual exam scores indicates students who provided detailed answers to their peers’ questions are performing better than the other students in their teams. In another study, Purzer (2011) found a similar pattern that links elaborated answers to individual achievement through self-efficacy. This result can also be explained by the learning by teaching paradigm (Biswas et al., 2005). Prior studies have shown that teaching others is an effective method to learn (e.g., Menekse & Chi, 2018; Chi, Siler, Jeong, Yamauchi, & Hausmann, 2001; Renkl, 1995; Rosenshine & Meister, 1994). Constructing elaborated responses help students to structure and organize their own knowledge. In addition, students who try to provide elaborated responses typically develop a deeper understanding of the domain. However, this type of other directed learning environment significantly increases the learning outcomes of “teaching/tutoring” students more than “learning/listening” partners (Menekse & Chi, 2018). As our results show, the students that provided detailed responses to the questions from their team members performed significantly better than the other team members. Another explanation is that students who are high performers engage in more elaboration. Our regression model shows a relationship; however, it is possible that students’ prior abilities might also be impacting their discourse in teams.

In terms of team performance, our results showed that the reasoning episodes were significantly related to team performance on the final engineering design project. As previous studies have shown explicit reasoning and time spent on comprehensive data gathering help teams to minimize potential errors (Hong, Doll, Nahm, & Li, 2004). For example, Tschan et al. (2009) explored the role of explicit reasoning in medical teams to reduce the number of incorrect diagnoses. They found teams that spent more time on explicit reasoning performed better than the other teams in terms of avoiding incorrect diagnoses.

Based on our coding scheme, the reasoning episodes included three categories: observational, calculation, and procedural. The calculation category was the best predictor of team performance. So, this finding indicates that the calculation related to coordination and communication within team members had a significant effect on team level success on the engineering design task. Since all teams were asked to justify their decisions based on evidence, results show successful teams spent considerable amount of time on tasks related to calculation, and this effort resulted in fruitful success. Also, calculations were important since students were asked to compare costs, savings, and payback period of the old and proposed new system. In addition, students needed to conduct an environmental analysis to compare carbon dioxide emissions of different systems.

Finally, we were interested in understanding how the dispersion of verbal episodes among team members relates to team level performance. Results showed the teams with more equal participation among team members performed significantly better than the teams that had unequal participation among team members. In prior studies, the equitability of team participation can be evaluated in two common ways: by analyzing turn-taking behaviors and by examining the distribution of verbal episodes. As prior studies argued that the distribution and frequency of turn-taking influence the contributions among team members and hence is important for successful interactions and effective team level outcomes (Chi & Wylie, 2014). More frequent turn-taking allows more equitable contributions from all team members and “make it easier for learners to incorporate their partners’ understanding and adjust their own mental model due to dynamic revisions of knowledge” (Menekse & Chi, 2018, p. 3). Chi and Menekse (2015) argued that ideal case of interactive learning environments requires joint dialog patterns in that all partners make substantive contributions to the topic. Likewise, our results provided evidence that teams with better participation and contribution from all team members performed better than other teams that did not have an equal participation among team members.

Instructional implications

Based on the results from this study, and prior team studies, there are some important pedagogical implications. First, results showed the balanced participation among team members is critical for team and individual success. As, other studies also suggested, instructors and/or instructional designers need to provide training and opportunities for team members, so students can realize and experience that their individual contributions to discussions are crucial for team and individual success. Perhaps, this finding is also related to research on team size. While the team size was not considered in this study, one may argue that in large teams (i.e., more than five members) individual contributions may not be as effective as in small teams (e.g., Lowry et al. 2006).

In regard to verbal episodes, we found that question episodes are important for individual performance. So, it is essential for students to be scaffolded to ask meaningful questions, as well as to generate responses to others’ questions in their interactions with their team members. In other words, students need to pay attention to what their team members are asking and how they can best address these questions, which are also related to the idea of balanced participation within team members. So, when all team members participate in generating and responding the questions, this process naturally creates a balanced participation among team members as well. Similar to first pedagogical implication, students need to receive explicit training on the importance of generating and responding to questions within team settings.

We also found the calculation-related episodes were significantly related to team performance. Calculations were important for the team assignment since students were asked to compare costs, savings, and payback period of the old and proposed new systems. In addition, students needed to conduct an environmental analysis to compare carbon dioxide emissions of different systems. As a pedagogical implication, this finding underlines the importance of calculations in engineering. The calculations are at the core of all engineering disciplines, and the fundamental engineering equations and related calculations can facilitate students’ understanding. Therefore, engineering students need continuous instruction on how to apply the knowledge of mathematics and science into engineering problems. In addition, students need opportunities and feedback on how to analyze and interpret data, as well as how to communicate and disseminate their findings based on these calculations.

Limitations

It is important to note some of the limitations of our study. The first limitation is that we had a fairly small sample size, especially at the team level (i.e., 19 teams). Accordingly, we did not use a multilevel approach for the analysis. Multilevel analysis addresses the problems that researchers encounter in the mixed factorial analysis of variance (Raudenbush & Bryk, 2002). On the other hand, multilevel analysis requires relatively large sample sizes. However, the coding for verbal data is labor and time intensive, and verbal coding for studies with large sample sizes requires resources to hire several well-trained coders. As a result, conducting a multilevel analysis is difficult for team learning studies that involve verbal data coding.

Another limitation was that the verbal data we used was from a separate problem-solving assignment and this assignment was not part of the final engineering design project that students worked on for the last 4 weeks of the semester. While teams received their team performance scores based on the engineering design project, we did not have any verbal data during team members’ collaboration for the engineering design project. On the other hand, this problem-solving assignment was right before the engineering design project, on week 11 in a 16-week-long semester, and it provided a detailed snapshot of how students collaborate in a team environment. Also, we chose to videorecord the team problem-solving activity at week 11 because this activity was a time limited assignment and each team had to complete all aspects of the assignment within 60 min. And this was the only time-limited team assignment during the semester which gave teams equal amount of time to work on it. For other assignments, teams had multiple weeks and they had been working on tasks out-of-class time as well; therefore, it was not feasible for us to keep track of 19 teams in order to have a complete picture of how they collaborate certain aspects of assignments. However, we believe it would be more beneficial to have multiple verbal data points during team collaborations to understand how team dynamics and interactions change over time.

Furthermore, team formation-related factors could be other limitations. In this study, teams were formed at the beginning of the semester based on students’ schedules and their available times, so they can meet out of class times to complete assignments. Based on prior research on team formation, the gender diversity of team members (e.g., Bear & Woolley, 2011), cultural diversity of team members (e.g., Nathan & Lee, 2013), students’ attitudes toward collaboration (e.g., Ruiz Ulloa & Adams, 2004), and social loafing (e.g., Lam, 2015) could play a significant role on team performances. Additionally, instructor and students’ interactions throughout the semester could also play a role on students’ academic success (e.g., Luo, Fan, Menekse, Wang, & Litman, 2015) However, we do not have such data in this study.

Conclusion

Past research indicates that peer interaction is critical not only for the improvement of academic achievement but also for social and emotional development (Hartup, 1989). Teams in general allow students to develop social skills such as helping others, sharing, taking turns, showing respect, and working collaboratively. And students’ social and discursive practices during collaborations could play a substantial role in team harmony and success. Verbal discourse in teams is critical not only for the improvement of academic achievement for individuals in teams but also for the development of better team solutions. It is important that such verbal discourse also includes questions and elaborated responses, which are necessary to promote individual student academic achievement. Student team interactions must also promote verbalization of calculations to promote quality of team outcomes. It is through such verbalization that calculations can be checked and agreed by team members. Engineering educators should monitor discourse and specially reinforce those that are critical for positive performance. Moreover, balanced participation from all team members is critical, suggesting teams should be formed and monitored to support balanced participation in verbal discourse. Future research is needed to explore how the interventions, designed to promote specific verbal episodes, can promote student academic achievement and teamwork outcomes.

Appendix

Exemplary episode of student discussions

Speaker A: Are we just gonna, are we just gonna compare the cost or something?

Speaker B: No, apparently, we have to decide, we have to use uh, one second. Let me read. You have to use cost as your main criteria, but you also have to consider carbon dioxide emission.

Speaker A: Okay. So is that a calculated [inaudible 00:14:29], obviously. [inaudible 00:14:34].

Speaker B: Okay how, do you know the area of the [inaudible 00:14:35]? Uh, is it the same?

Speaker C: Here, look right here. Okay. [crosstalk 00:14:42].

Speaker B: Are you reading that from the data sheet? Okay.

Speaker C: Yeah.

Speaker D: 1300 square feet.

Speaker B: So garden material is five dollars.

Speaker D: Wait a second. Is that meters or feet?

Speaker B: That’s ...

Speaker A: Meters.

Speaker D: Meters, so you have got to convert it to feet.

Speaker B: No way.

Speaker D: Okay. One meter is [crosstalk 00:14:55]. Oh, oh, oh, oh.

Speaker B: We should be using Excel as well?

Speaker D: Hold on, hold on, hold on, hold on. Okay.

Speaker B: Because Excel calculation would be zero.

Speaker D: Okay.

Speaker B: Just one sec.

Speaker D: I am just going to write it out, whatever you tell em.

Speaker B: So um, it’s just five dollars per square feet, as your garden material.

Speaker D: One sec. [crosstalk] That’s 1300 square, feet a square.

Speaker B: [inaudible 00:15:24]. Cost estimation.

Speaker D: [inaudible 00:15:29].

Speaker B: Stop reading your text.

Speaker D: I am not reading anything. It just keeps on vibrating right there.

Speaker B: Create [inaudible 00:15:47].

Speaker D: So we got 13,985 usable square feet. Wow.

Speaker B: One sec. Usable square feet.

Speaker D: Okay, usable square feet. Right here.

Speaker B: 1300. What’s that building area?

Speaker C: That’s the area of the building [crosstalk].

Speaker B: Consumption is that, daily consumption is per year. Okay fine. 13,000.

Speaker D: Wait, so this 2000 is regardless of size, right?

Speaker B: Um, this? Yeah. That’s total.

Speaker D: It’s just 2000 [crosstalk 00:16:37]. Okay, cool.

Speaker B: 1300, that’s in meter square. Meters. let us see. How much, how many meters is one feet?

Speaker D: Um, 3.2. I got here, 13,986.

Speaker B: No, you have to show all the calculations.

Speaker D: Oh, okay, so...

Speaker B: 1300 m square.

Speaker D: One meter is 3.28 ft.

Speaker B: 3.28 square?

Speaker D: What?

Speaker D: Square.

Speaker D: No, one meter is 3.28 ft. Not square.

Speaker B: But we are converting areas. It’s meters squared.

Speaker D: Yeah. It’s meters squared. So squared, 1300. Then times 3.28, then square that. To get feet squared.

Speaker B: Then you might as well, okay.

Speaker D: Yeah, you probably have a short cut for that.

Speaker C: Okay. At the end we want net cost.

Speaker B: 3.28, right?

Speaker D: 3.28 ft.

Speaker B: Okay.

Speaker D: [inaudible] I am just going to put this here. I do not know what you are doing there.

Speaker B: 13985.

Speaker D: I am just putting the costs one by one.

Speaker B: Um, yeah, I mean I am just recording.

Speaker D: Okay.