Open Access

Case studies of a robot-based game to shape interests and hone proportional reasoning skills

  • Louis Alfieri1Email author,
  • Ross Higashi1,
  • Robin Shoop2 and
  • Christian D Schunn1
International Journal of STEM Education20152:4

DOI: 10.1186/s40594-015-0017-9

Received: 3 June 2014

Accepted: 29 January 2015

Published: 20 February 2015

Abstract

Background

Robot-math is a term used to describe mathematics instruction centered on engineering, particularly robotics. This type of instruction seeks first to make the mathematics skills useful for robotics-centered challenges, and then to help students extend (transfer) those skills. A robot-math intervention was designed to target the proportional reasoning skills of sixth- through eighth-graders. Proportional reasoning lays the foundation for further progress within mathematics. It is also necessary for success in a number of other domains (engineering, science, etc.). Furthermore, proportional reasoning is a life skill that helps with daily decision making, planning, etc. However, it is a skill that is complex and often difficult for students. Previous attempts to design similar robot-math activities have struggled to focus students’ attention on key mathematics concepts (in complex engineering domains), and to motivate students to use the math properly. The current intervention was designed with these challenges in mind. This intervention centers on a computer-based 3D game called Expedition Atlantis. It employs a game design that focuses student attention on a specific proportional reasoning task: students calculate correct quantities of wheel rotations to move the robot to desired locations. The software also offers individualized tutorials. Whole-class discussions around daily word problems promote further application of proportional reasoning outside the robot programming context. The 1-week intervention was implemented by three teachers at different schools with varying levels of ability among students.

Results

Overall, within-participant comparisons revealed that the intervention was successful in improving the number of correct responses, the number of problems attempted, the proportions of correct responses, students’ interest in robotics, and students’ valuing of mathematics within robotics from pre- to post-test. Further analysis of teachers revealed that the two class sections of special education benefited most. Consideration was given to the qualities of the implementation that might have led to these enhancements.

Conclusions

The success of this intervention suggests that robot-math activities might be successful when focused on a few target skills and when designed with individualized tutorials/prompts that motivate proper skills. Further investigations of student and implementation characteristics would help to refine these interventions further.

Keywords

Robot-math Gaming to learn Proportional reasoning Special education

Findings

Background

Robotics is being taught in over 35,000 formal and informal education settings in the USA (FIRST 2013; REC 2013) and a number of educators and researchers have highlighted the potential use of robotics lessons to reinforce students’ mathematical understanding (Benitti 2012; Vollstedt et al. 2007). One popular premise for doing so is that students are more likely to recognize the math as relevant because of its direct, applicable utility to a concrete and contextualized task (Doppelt et al. 2008; Mubin et al. 2013). This instructional approach of teaching mathematics within the context of robotics is commonly referred to as robot math. Robot math’s cross-disciplinary integration of engineering, technology, and mathematics highlights for students how these different disciplines necessitate and facilitate each other. Such instruction stands in contrast to more traditional mathematics classes, which might prompt mathematical reasoning only for the sake of understanding numerical relationships, and merely describe it as occurring in a context. Although the current paper focuses exclusively on robot math, such integrated activity-based instructional approaches might be appropriate for STEM (science, technology, engineering, and math) education more broadly as the lines between these disciplines increasingly disappear.

When engaged in a robotics activity, students might first encounter a mathematical problem within a concrete context such as movement programming, and leverage that context to begin sense-making at a higher level both faster and sooner. Through successful early encounters with problems of a mathematical nature, students could gain both experience and confidence in using robot-contextualized math (those concepts/solutions common to robotics) within that single, well-defined context. They might then extend that understanding to mathematics more generally (by increasing the level of abstraction of their mental models) and consequently, see how these solutions also connect to everyday math-related situations (generalize their solution strategies and/or models in order to transfer them to other situations; e.g., math classes, science classes, or even everyday life).

However, successful mathematics learning can be difficult for students to achieve through robotics. Investigations of educational robotics (e.g., Silk and Schunn 2008) have revealed that learners encounter difficulties learning mathematics when too many topics are incorporated into a single lesson. Moreover, subsequent studies (e.g., Silk et al. 2010; Silk 2011) found that (a) the majority of teams in a middle-school robotics competition “guessed and checked” their ways through the programming portion of the task rather than use mathematics, and (b) the success of teams that did attempt to use mathematical solutions was highly dependent upon the way in which the mathematics were integrated into the team’s overall problem-solving process. Finally, extended investigations of mathematics in robotics sometimes suffered from low engagement and thus no interest development in robotics or mathematics, especially when the same basic robotics task needed to be revisited many times to deepen the mathematical solution.

Robotics lessons designed to reinforce mathematical understanding (i.e., robot math instructional methods) could therefore benefit from the following points of refinement: (1) narrow the number/variety of mathematical concepts included within the lesson, (2) actively discourage the guess-and-check method, (3) improve the use of mathematical solutions, and (4) provide a robotics context that is sufficiently varied to maintain engagement. The current study investigates whether a robot programming game using those strategies can produce better student outcomes.

In particular, we explore whether a robot programming game is a platform that can facilitate robot math by affording teachers with the opportunity to capture students’ attention and engage them in an activity that is enjoyable while also highlighting for them the utility of math both within the game and more generally. The research questions included (1) whether this approach is effective in achieving math learning gains, and (2) whether the approach also raises students’ interests in robotics, mathematics, or their awareness of the importance of mathematics in robotics. The intervention was designed to be implemented within the kind of classroom typical of robotics education settings: elective technology classes. And so, also of interest was the fidelity of teachers’ implementation and whether gains would be found in this elective type of classroom environment.

Supporting at-risk classrooms and teachers not certified in mathematics

Implementing robotics activities in elective technology classes to facilitate mathematics learning can be challenging because teachers in those classes are rarely certified in mathematics instruction. Therefore, we designed activities that required little support from teachers by including the basic mathematics strategies in the instructions to students within the game. Furthermore, to allow for differentiated support for students of varying mathematics ability, we developed the robot math activities using the model eliciting activity (MEA) design principles (see Table 1; Lesh et al. 2000; Reid and Floyd 2007). Activities designed with these principles would challenge more advanced students while remaining approachable to weaker students.
Table 1

Principles for MEA design

Principle

Explanation

Reality principle

Students can make sense of the situation based on their experience.

Model construction

The task creates a need for a mental model to be constructed, modified, extended, or refined.

Model documentation

Students are required to explicitly reveal how they are thinking about the problem.

Self-evaluation

The problem statement includes rubrics enabling students to judge for themselves whether their solution is acceptable.

Generalizable model

The model should not only work on the specific problem but should be sharable and usable in other situations.

Simple prototype

The problem should be as simple as possible given the instructional goals.

An additional benefit of robot math is its potential appeal as an “alternative” method for classrooms of learners who are considered at-risk of not learning the mathematics skills essential for testing and success using mainstream curriculum. For example, students grouped under the umbrella of special education might struggle with schoolwork for a variety of reasons (e.g., poor self-efficacy, poor reading ability, or attention disorders; Hampton and Mason 2003; Pastor and Reuben 2008; Zimmerman and Martinez-Pons 1990) and sometimes non-traditional methods or multi-modal methods of instruction can be more effective (Kroesbergen and Van Luit 2003; Moreno and Mayer 2007). Because of the potential for robot math to engage students in a genuinely motivating way by conveying useful mathematics concepts (grounded within robotics instead of described abstractly), these at-risk classrooms might have the most to gain from such activities. The engagement of otherwise at-risk students paired with differentiated student supports within the game (hints, tutorials, etc.) could maximize the potential for student success. Furthermore, the varying levels of ability within such classrooms can be handled by the differentiated supports within the game, which include difficulty settings that can be adjusted to increase/decrease the complexity of calculations (e.g., the inclusion of decimal numbers).

The challenges of developing effective activities

While there are many salient mathematics concepts that can be found within robotics activities typically experienced at the middle and high school levels, we focused on proportional reasoning. Proportional reasoning is a pervasive concept in mathematics (National Research Council 2001) that involves scale, rate, and conversion of units and has been identified as “the capstone of childrens elementary arithmetic and the cornerstone of all that is to follow” (Lesh et al. 1988). Proportional reasoning is also pervasive in robotics movement planning, as wheels and limbs typically move distances and angles that are proportional to motor commands.

In Expedition Atlantis, we targeted this single key mathematical concept for intense intervention and used principles of MEAs to guide our material design (see Table 1). Prior efforts with this approach had already revealed design issues that we attempted to avoid or mitigate. For example, one such unit, robot synchronized dancing (RSD), challenged students to program two robots with different-sized wheels and wheel bases in order to allow them to move (dance) in synchrony. A unique sequence of dance movements programmed into one robot had to be adapted to synchronize a second (differently sized) robot. This overall task was divided into subtasks to scaffold the demands of the challenge (Apedoe et al. 2008): (1) synchronizing the distance traveled, (2) synchronizing robots’ speeds, and (3) synchronizing their turns. As students worked, they were asked to create and revise a general technique that would allow for the synchronization of more robots of different sizes (i.e., a generalizable model of the solution). Additionally, students solved math-focused abstraction bridges (Brown and Clement 1989) that prompted students to solve robot math word problems. These problems were initially analogous to the calculations of distance and speed required for the challenge, but progressed toward everyday proportional reasoning scenarios to prompt students to generalize the proportional reasoning skills that they had developed within the robot challenge to more traditional situations, thereby developing a more general understanding.

The first version of RSD fell short in implementation because students spent too much time on developing the base dance. Instead of tackling the mathematics involved in the challenge, initial lessons had to teach the necessary programming language commands, and students spent too much time working on the esthetics of the dance. Initial revisions to the module mitigated both issues by providing the base dance to students, but encountered another layer of implementation challenges revolving around the inaccuracies and inconsistency of robotic systems in physical operation. In particular, the real-world noisiness of physical robotics created a distraction: the teacher needed to facilitate additional discussions about why averaging across multiple trials were required, why precise measurement was necessary, etc. (Silk 2011).

As a result of these challenges, the current intervention uses a game-like robot simulator called Expedition Atlantis that operates entirely in a “virtual world,” thus eliminating the mathematical challenges of stochastic performance and measurement error. It also limits the scope of programming to the selection of the most mathematically vital parameter: how many wheel rotations the robot needs to move or turn to achieve the desired movement outcome. These differences reduce the burden on teachers and students to first learn programming before learning the math, as well as highlight proportional reasoning without requiring attention to statistics and measurement concepts.

Furthermore, to reduce the chances of guess-and-check methods (what students do when they are avoiding the mathematics), Expedition Atlantis includes automatic tutorial prompts after the player repeatedly answers incorrectly. With these built-in tutorials and prompts for students to develop their skills independently, teachers are not expected to do much more than guide discussions of the solutions to accompanying word problems (abstraction bridges). The abstraction bridges written for the RSD intervention (Silk 2011) were adapted to the challenges within Expedition Atlantis. The content of the abstraction bridge problems initially center around the Expedition Atlantis game, then gradually shift to other contexts. As before, the intent is for students to develop proportional reasoning skills by first creating models of the proportional relationships within the game, then extending those models to everyday situations over time.

On the basis of prior results and the additional issues addressed in Expedition Atlantis, we hypothesized that we would find increases in proportional reasoning test scores from pre- to post-test, as well as increases in measures of students’ interests and values pertaining to mathematics, robotics, and the use of mathematics in robotics.

Methods

Participants

The intervention was implemented by three teachers at different schools within urban/suburban districts in the northeastern USA; 116 students participated. Teachers were financially compensated for their time, which included a 2-h professional development session before implementation was scheduled to begin. Teachers at all three schools implemented the intervention in two of their class sections ranging from grade 6 through 8. Table 2 lists the numbers of students and their mathematics levels by section.
Table 2

Sample of students across three locations

Context

6th graders

7th graders

8th graders

School A (public)

   

Special education section 1

 

15

 

Special education section 2

  

15

School B (private)

   

Mixed section 1

10

7

6

Mixed section 2

8

8

5

School C (public)

   

Average section

22

  

Accelerated section

20

  

Schools were selected in order to explore the impact of the intervention across different classroom contexts. Two of the schools were public, one private. Both class sections at school A (the first public school) were labeled as special education. These students were considered to be at-risk of not passing state standardized tests. At school C (the second public school), one of the class sections was labeled as accelerated and the other section was at a standard level. Because school B (the private school) included a mix of grade levels (6 through 8) within its sections, ability levels varied within each section but all were around average for the students’ respective grade levels. School B was in an urban district as compared to the more suburban settings of the public schools (A and C). The teachers at schools A and C were not certified in mathematics but the teacher at school B was certified in secondary (grades 7–12) mathematics.

At the end of a pre-test, students were asked if/when they would complete an algebra course and to rate their levels of experience with robots, among other questions on a survey of prior experience. Survey responses indicated that the majority of students (n = 66) planned to take a full algebra course in the eighth grade (including all 11 eighth-grade students, who were already doing so). The 20 students within the accelerated sixth-grade math class would take an algebra course in seventh-grade and the 30 students within the special education classes would take such a course in ninth grade. The majority of students (n = 99) had little or no experience working with robots. Only four students reported having extensive experience with robots and removing these students from analyses did not significantly alter the results.

Materials

Curriculum

At the center of the intervention was an interactive simulator game (Expedition Atlantis) requiring students to utilize their proportional reasoning skills in order to program a robot to navigate through a 3D aquatic environment. Figure 1 provides an overview of the entire game (its display, example problems from each chapter, etc.).
Figure 1

Chapters within Expedition Atlantis.

Students interact with the game by filling in parameters to control the robot’s motors during a series of maneuvers. These parameters are expressed in units internal to the robot, such as motor/wheel rotations. However, each parameter exposed in this way is proportional to a tangible in-game goal quantity, such as distance the robot travels. Students are given a goal in each level of the game that (implicitly) requires them to relate the two. For instance, because wheel rotations are proportional to the distance the robot travels, one level tasks students with making the robot move a specified distance by determining the corresponding number of wheel rotations to make it do so. A correct answer advances the student in the story, while an incorrect answer prompts the software to begin offering progressively stronger hints, eventually directing students through an additional series of activities that use a building-up approach to illustrate the physical relationship between the number of wheel rotations and the distance traveled, using the values from the student’s current problem.

The game contains five substantively different scenarios, spread over four chapters, each raising an additional complication for the robot. The first level involves only forward movement; the second introduces turning; the third scenario combines moving and turning; the fourth introduces physical changes to the robot that alter the underlying rate values (e.g., a larger wheel goes farther in the same number of rotations); and the fifth scenario requires students to plot out their own movement sequences in order to navigate a maze-like environment, and subsequently solve the individual movement-value problems that the sequence generates. See Figure 1 for another description of individual chapters, explanations of the display, and example problems students had to solve.

When used in a classroom setting, teachers could leverage each new scenario as an opportunity to examine, discuss, and reason about the nature of the proportional relationships and how best to describe and apply them in the game context.

Each school was provided with access to the program for installation on student computers. In addition to a 2-h professional development session prior to implementation, teachers received binders that explained the purpose of the game and its accompanying activities.

Each day’s activities were accompanied by worksheets that prompted students to complete proportional reasoning practice problems called Abstraction Bridges. Our intention was to have these problems help students to generalize the mathematical skills and knowledge gained within the game. On the first day, students completed problems that were very similar to calculations needed within the game (e.g., moving and turning the robot; near transfer). On each successive day, problems on the worksheets became increasingly dissimilar in theme; then in form, from the calculations within the game until on the last day(s), problems were completely unrelated to robot movement (e.g., plow sizing, paint mixing, etc.; i.e., far transfer). See Table 3 for examples of both near and far transfer practice problems.
Table 3

Abstraction bridges

When encountered

Name of problem

Problem

Following day 1

Robot movements

Your robot moves 10 m in 4 wheel rotations.

How many wheel rotations does it take to move 30 m?

Following day 2

A new robot

You just purchased a new robot! The instructions say that for every 8 motor rotations programmed in the robot, it turns 5°.

How much will your robot turn if it uses 56 motor rotations?

Following day 3

Going on a trip

You are traveling to visit a friend in another state. Your car has an 18 gal gas tank and costs $48.00 to fill up. You will use 54 gal of gasoline in your travels.

How much will you need to spend on gasoline?

Following day 4

Paint mixing

When we arrived at the robot garage, we were informed that we could customize our robots with custom colors. A gallon of red, 2 gal of white, and a gallon of blue paint will give us one particular color that we want.

If the job is going to take 24 gal of paint total, how many gallons of each color will we need?

 

Density of tire spikes

We have to travel across ice with our ice tires. Ice tires have little spikes that dig into the ice so the wheel doesn’t slide. Our ice tires are 6 m but Goodtire 3000 makes much smaller ice tires. Our ice tires have more spikes but are less dense.

If each type moves 12 m, which one will create more spike marks on the path?

The game allowed teachers to decide which difficulty levels were most appropriate for students. This setting influenced the difficulty of the calculations (e.g., the presence or absence of decimal numbers), but not the order of robotics topics (e.g., straight movement was always followed by turning). Similarly, difficulty settings did eventually influence areas directly relevant to the target topic of proportional reasoning (e.g., the inclusion of non-unit base rates; 2 rotations = 3 m) at higher settings. Figure 1 provides prompts at different difficulty levels for comparison. Teachers initially set all students’ difficulty to a common level based on the mathematics level of the class, and then directed individual students to increase or decrease their individual difficulty levels based on their rate of progress through the game.

Proportional reasoning test

There were two versions of the test that were administered roughly equally as either pre- or post-tests within classrooms (adapted from Silk 2011; Weaver and Junker 2004). The two versions were comparable in content (i.e., mostly far transfer, proportional reasoning problems some of which could be solved without complex mathematical calculations) and contained the same number of question prompts. See Table 4 for example questions. Students received one version as a pre-test and the other as a post-test. Prior work using this assessment failed to find reliable pre/post improvements with similar populations in several control groups and in some of the robotics-related interventions (Silk 2011). Therefore, it is unlikely that any gains from pre to post in this study would simply be the result of increased familiarity with the questions, a Hawthorne effect (i.e., gains simply from experiencing novel instructional approaches), or typical ongoing mathematics instruction during this time period. Because the current study could not investigate students’ transfer of potentially gained math skills in other subjects’ classes, the questions on pre- and post-tests were chosen to be outside of robotics and in contexts more like everyday life.
Table 4

Example questions from the proportional reasoning test

Version 1

Version 2

Question

Question

1) There are 36 poker chips, 12 blue (B) and 24 red (R).

6) Frances has an enlarging machine that can enlarge or reduce a photograph to any size while keeping the same shape. If an original photograph is 2 in wide by 2.4 in long and Frances wants to enlarge it to be 5 in wide, how long will it be?

Arrangement 1

Arrangement 2

4B 4B 4B

3B 3B 3B 3B

8R 8R 8R

6R 6R 6R 6R

What changed between the first and second arrangements?

What did not change?

Show another arrangement of the poker chips that preserves the same relationship.

5) John is 15 years old. John’s father is now three times as old as John is.

11) It takes Vanessa 9 h to paint 15 chairs. How long will it take her to paint 20 chairs?

How old is John’s father now?

When John is 20, how old will his father be?

14) The ratio of 5 to 4 is the same as:

13) Which statement is correct?

a) The ratio of 6 to 5

a) 2/5 = 1/4 = 3/6

b) The ratio of 5/4 to 1

b) 2/5 = 4/25 = 16/625

c) The ratio of 10 to 8

c) 2/5 = 6/15 = 4/10

d) b and c

Prior experience survey

At pre-test, students were also asked to answer questions about their grade level, algebra-taking plans, and familiarity with robots.

Motivation survey

At both testing times, students rated their personal interests in and values of mathematics and robotics on a five-point scale from NO! (strongly disagree) to YES! (strongly agree). Students rated statements about mathematics interest (four items; Cronbach’s alpha: pre = .74; post = .82), robotics interest (four items; Cronbach’s alpha: pre = .73; post = .76), and the value of math in robotics (four items; Cronbach’s alpha: pre = .66; post = .80). In prior work using these scales (for a summary, see Silk 2011), robotics interventions either tended to increase motivation or proportional reasoning but not both. Again, that a number of prior robotics interventions showed no pre/post gains on these measures rules out gains from simple test-retest effects or Hawthorne effects.

Design and procedure

Pre- and post-tests were administered by the teachers in the presence of the experimenter on the days immediately preceding and following the roughly 5-day intervention. An experimenter was present in all classrooms throughout implementation to ensure the materials were being implemented, assess student reactions, and handle technological obstacles (e.g., game freezing). To maintain the authenticity of the implementation, the experimenter imposed few restrictions or recommendations. Teachers led abstraction bridge discussions as they would in the absence of the experimenter and students worked individually or in pairs as they traditionally would. The experimenter met with the teacher before and/or after each class to discuss the day’s activities, the subsequent day’s activities, and any other concerns that the teacher had. However, teachers raised few questions as to how to proceed with the intervention. Most discussions with teachers centered around students’ progress and prompted feedback from teachers about the intervention more generally. Whereas stricter guidance would have increased the amount of experimental control, it potentially could have made the class seem unusual to students and consequently suspect. For these reasons, it seemed best for this first study of the unit’s implementation to allow the variations inherent to different teaching styles and classroom environments, which would further allow us to explore preliminarily which variations were most effective.

Before the first day of implementation, students were pre-tested. The intervention was then implemented across subsequent days; students engaged in game play and then received the day’s practice problems to complete either at the end of class or for homework. Except for the first day of game play in which students immediately began with game activities, all other days began with the teacher reviewing the practice problems (abstraction bridges) with the class in the form of a class discussion and modeling (either by classmates or the teacher) of the correct solution(s). Reviewing solutions to a problem required about 5 to 10 min. After the review session, game play resumed at whichever chapter the student had left off. Following the intervention, post-tests were administered.

It warrants noting that the current investigation was not designed to directly compare outcomes between schools. Instead, it was designed to explore the variety of ways that teachers implemented the intervention and the effects of that implementation on learning outcomes. That is, this investigation examines effects within three purposely-diverse sites as case studies. The examination of these three implementations provided insight into the efficacy of the teacher’s guide, the fidelity of implementation, and the efficacy of the intervention following that implementation.

Results and discussion

Pre- and post-tests were scored by the experimenter, providing partial points for correct reasoning even if calculated incorrectly (e.g., rounding error). Two test scores were created: raw scores of the number of correct responses (out of 17), and proportions of answers correct out of the number attempted by each student because students sometimes skipped questions. Changes in student interest in mathematics and robotics, and recognized value of math in robotics were also scored to investigate whether participation in the intervention changed students’ opinions.

The proportional gains from pre- to post-test were subjected to an ANCOVA that included ability level as a fixed effect and pre-test number correct and pre-test number attempted as covariates. Because ability level, location/teacher, and grade level were all highly contingent (contingency coefficients of 0.8, p < .001), we chose to focus analyses only on ability levels. The ANCOVA’s corrected model for proportional gains was significant, F(4, 100) = 9.07, p < .001 with a marginal effect of class category, F(2, 100) = 2.92, p = .06. However, there still seemed to be reason to suspect that there were differences in implementation at different locations - particularly when comparing students of traditional ability levels. Therefore, the data were analyzed with pre/post paired-sample t-tests, first including all locations, and then, follow-up analyses for each location separately. This was done for both sum scores and the proportions correct, as well as mathematics-, robotics-, and value-related self-ratingsa. Effect sizes to capture pre/post change were then calculated by using groups’ pre- and post-test measures’ means, standard deviations, and correlations between measures (in order to adjust the effect-size for a within-subjects design).

All three schools

Overall, we found that across locations/teachers, the intervention improved scores on proportional reasoning (both sum scores and proportions correct). See Table 5 for a presentation of all measures, and Figure 2 for effect sizes. We also see that there is a marginal (p = .061) increase in the number of prompts attempted by students from pre- to post-test. Thus, the large change in overall number correct reflects changes in both number attempted and accuracy within those attempted.
Table 5

Pre/post gains in proportional reasoning and interests (statistically significant gains in italic)

Measure

Pre-test

 

Post-test

  
 

M

SD

M

SD

t statement

Sum score

6.74

3.50

7.97

3.64

t(104) = 4.20, p < .00

School A

6.14

3.00

9.71

2.09

t(27) = 6.80, p < .00

School B

8.89

3.62

9.50

3.73

t(37) = 1.31, p = .20

School C

5.08

2.59

5.23

2.70

t(38) = 0.41, p = .69

Proportion correct

.56

.22

.62

.21

t(104) = 2.85, p < .01

School A

.52

.19

.66

.15

t(27) = 3.83, p < .01

School B

.68

.20

.68

.18

t(37) = 0.00, p = 1.0

School C

.46

.20

.53

.25

t(38) = 1.44, p = .16

Prompts attempted

12.27

4.08

13.08

4.38

t(104) = 1.90, p = .06

School A

11.93

3.62

15.00

2.98

t(27) = 4.57, p < .00

School B

13.37

4.09

14.11

3.93

t(37) = 1.37, p = .18

School C

11.44

4.24

10.69

4.65

t(38) = −0.91, p = .37

Student interest in mathematics

2.20

0.96

2.21

1.09

t(104) = 0.16, p = .87

School A

1.97

1.05

2.06

1.20

t(27) = 0.86, p = .41

School B

2.17

0.77

2.29

0.94

t(37) = 0.96, p = .35

School C

2.36

1.03

2.24

1.14

t(38) = −0.93, p = .36

Student interest in robotics

2.48

0.74

2.84

0.82

t(104) = 4.84, p < .00

School A

2.29

0.58

2.57

0.78

t(27) = 2.40, p = .02

School B

2.63

0.80

2.96

0.72

t(37) = 2.96, p = .01

School C

2.48

0.75

2.90

0.91

t(38) = 3.00, p < .01

Student values math in robotics

2.46

0.64

2.81

0.83

t(104) = 4.29, p < .00

School A

2.28

0.68

2.52

0.81

t(27) = 1.50, p = .15

School B

2.49

0.65

2.95

0.72

t(37) = 3.55, p < .01

School C

2.53

0.61

2.83

0.91

t(38) = 2.24, p = .03

Figure 2

The effect sizes for skill development and interest.

We also see significant increases in student robotics interest and students’ values of mathematics for use in robotics from pre- to post-test. There was not a significant increase in students’ general ratings of mathematics interest, again ruling out a Hawthorne effect in which students broadly show change as a result of exposure to a new environment.

To test whether the mathematical gains depended upon initial attitudes toward mathematics or robotics, correlations were computed between pre-test interest or values ratings (for mathematics, robotics, mathematics in robotics, or the combined average of all three) and gains in proportional reasoning. None of these correlations was statistically significant (Pearson r = −.09, −.11, .03, and −.09, respectively).

Individual schools

The patterns of results across our three locations indicated some variability in effects but generally, patterns were consistent. As can be seen in Table 5, marginal or non-significant differences were found between pre- and post-test scores at the schools B and C but the special education classrooms at school A did show gains large enough to be statistically significant on their own. On average, classes at all locations increased in their interests in robotics, and classes at two of the three locations increased in their value of mathematics for use in robotics. After only several days of game play, these are considerable gains. Furthermore, every setting saw gains on at least two measures.

Observations of classroom environments

All three teachers maintained high levels of implementation fidelity across days. None of the three teachers deviated from the pace or purposes of the materials as they had been prepared. Students worked largely independently at individual computers at schools A and C but paired up to play the game at school B. Across all three locations, few if any students asked for help in playing the game. Teachers and the researcher circled to tackle more technical problems (computers freeze, graphics malfunction, etc.), which were the few times students requested assistance. Teachers had little to do but walk around and monitor student progress while students played the game. Their part in this intervention was mainly to lead discussions of the abstraction bridge problems.

In this capacity, the three teachers behaved somewhat differently. As the abstraction bridge discussions were unscripted, teachers led students through solution strategies as they thought appropriate.

Discussions of the abstraction bridges at schools B and C included presentations of multiple solution strategies (e.g., the base rate or segment strategy, the T chart strategy described below, or the scale factor strategy described belowb) by students who displayed and explained their reasoning behind the solution. Teacher B (i.e., at school B) decided who would present her/his strategy after asking students what their answers were. A first student would provide an answer and others would agree or disagree by a show of hands. The teacher would then typically ask the first (or multiple students) to show how (s)he solved the problem. Consequently, all strategies led to similar answers (some rounding variations) and all were generally correct - as selected by the teacher. In comparison, teacher C chose students either because they volunteered or because he wanted to challenge specific students. This led to student presentations of both correct and incorrect strategies and answers, which the teacher then labeled as correct or incorrect after their presentation. Consequently, more students presented different strategies at school C (both correct and incorrect) than did students at school B. Furthermore, the time spent on discussions of these problems were generally longer at school C than at the other schools (precise time measurements unavailable) because a number of incorrect responses needed to be explained before a correct solution was presented. This also slightly decreased the amount of time spent playing the game.

One of the strategies presented at school C included an approach not seen at the other two schools: the T chart. The T chart is a method of increasing two numbers incrementally through addition on both sides (e.g., doubling 18 gallons and $63.00 until $63.00 is close to $192.50 in order to figure out how many gallons of gas were purchased for $192.50). Answers following from this strategy are approximate at best. In comparison to other solution strategies, this approach seemed inefficient but was observed in both the traditional and accelerated classrooms at school C.

Discussions at school A surrounded the use of only a single strategy - scale factor. As at the other schools, students here were asked to present their solutions either because they volunteered or because the teacher encouraged a student personally. On the first day of abstraction bridge discussions, a student identified and explained her scale factor strategy and the other students were quick to recognize it from a previous class. Subsequently, all students adapted the same scale factor strategy - or a closely similar variant. This allowed discussions to focus on (1) the relationships between numbers (e.g., labels such as meters and rotations and the relationship between number of rotations and distance traveled in meters), (2) what each number represented at each stage of the solution (e.g., the distance per single rotation, the cost per gallon of gas, etc. - numbers calculated in the process of reaching the final answer), and (3) the details of working through this mathematical procedure (e.g., rounding, multiplying without a calculator, etc.). These three components were present in all discussions at school A but were less commonly present in discussions at the other two schools. Later problems elicited discussions as to how to apply the scale factor strategy to these more complex “life math” problems - as referred to by teacher A.

Thus, the particular gains observed in school A might not be due to student characteristics (special education) alone, but also to teacher A’s organization of abstraction bridge discussions. It may be that students benefit most when the number of strategies presented to solve problems is kept low and when only correct strategies are highlighted. Furthermore, teacher A elicited explanations of these strategies in particular ways: asking students what each number represents, the purpose of each mathematic calculation, etc. However, given the various differences between school contexts, no strong inferences can be made about the effects of the teacher strategies.

Conclusions

Overall, the 1-week intervention increased students’ proportional reasoning skills from pre- to post-test (particularly at school A), as well as their interest in robotics (all three schools) and their value of mathematics for use within robotics (particularly at schools B and C) as inferred from within-participant comparisons. In past studies using physical robots without the additional structures provided by the Expedition Atlantis game, limited mathematics gain was generally found. In the one prior case in which significant gains were found using the physical robots, the students were from an advanced private school and the instructor was from the research team, and therefore able to provide many complex supports for the students that would be difficult for traditional technology educators to provide effectively. Finally, the prior case with mathematics gains did not show gains in interest in mathematics or robotics (Silk 2011). Thus, the current outcomes were generally more positive, especially given the implementation in challenging contexts without traditional technology instructors.

We believe these improved outcomes to be the result of four major features of the intervention. The first feature was a focus on a single mathematics topic: proportional reasoning. This allowed students to concentrate on this skill instead of attempting to learn all of the mathematics potentially involved within robotics. The second was a simplification of how the robot needed to be programmed. Instead of requiring students to learn the syntax of programming, they could directly enter their calculations for the number of wheel rotations required for a given destination. The third feature was a curbing of students’ propensity for guess-and-check approaches. After a few incorrect guesses, students’ attention was drawn to particular pieces of information on screen (e.g., the distance for a given number of rotations) needed to solve the calculation correctly. After a few more incorrect guesses, students were assisted by a pop-up tutorial within the game that demonstrated how to think about the problem, how to imagine the distances, and how to calculate the number of rotations needed. Although most students who encountered the tutorial were grateful for the personal assistance, some students were frustrated because they had reached the tutorial when their guess-and-check approaches were insufficient. Nevertheless, these students were observed to be less likely to guess in the future. Lastly, the fourth feature was the inclusion of abstraction bridge word problems that facilitated a generalized application of proportional reasoning to other domains, thereby preparing students to use this reasoning for analogous problems and/or in daily life. Thus, these initial findings would predict that robot-math activities designed with similar features would also likely be successful.

Again, the findings reported here are case studies of three locations with conclusions drawn from within-participant comparisons. Further investigations are necessary to tease apart these features of the intervention - perhaps including a comparison to a meaningful baseline control group (no instruction, practice only, traditional instruction, etc.).

Supporting teachers not certified in mathematics and at-risk classrooms

When schools were examined individually, the greatest gains were found at school A where both class sections had been classified as special education (at-risk) and the teacher was not certified in mathematics. We had hoped of course that our intervention would improve scores of all students but were particularly interested in its effects within special education classrooms because many of these students are considered to be at-risk of failing benchmark tests in mathematics. We hypothesized that this “alternative” method of teaching proportional reasoning may be successful because of its non-traditional, multi-modal approach that makes the math skills acquired transparently useful (within the game immediately, and later within daily situations as discovered in discussions of abstraction bridges; Kroesbergen and Van Luit 2003; Moreno and Mayer 2007). Furthermore, the student supports present within the game (highlighting of important information followed by tutorials) could effectively scaffold the calculations to make them accessible to students at varying levels of ability. And indeed, students at school A benefited considerably from the intervention.

However, these initial findings require further investigation as to the influences of student and/or teacher characteristics. It is difficult to distinguish the two with the current sample because only teacher A was responsible for at-risk classes. Moreover, teacher A was exceptional in fostering students’ understanding of the abstraction bridges. After the first student who presented a solution reminded her classmates of the scale factor strategy by using it successfully within the problem, all students adapted that strategy to meet the demands of subsequent problems. The scale factor solution strategy had been introduced in a previous class and so all students were familiar with it. This allowed teacher A to focus on the mathematics content handled by the strategy instead of on the selection of a proper strategy. She mediated discussions by focusing on the relationships between numbers (through the use of labels such as meters and rotations - e.g., the relationship between numbers of rotations and distance traveled in meters), on what each number represented at each stage of the solution (e.g., the distance per single rotation, the cost per gallon of gas, etc. - numbers calculated in the process of reaching the final answer), and on the details of working through this mathematical procedure (rounding, multiplying without a calculator, etc.). Thus, teacher A was not limited to discussions as to which mathematical procedure was best because that procedure was already a more-or-less routinized practice of students. She could instead explain the mathematical underpinnings of that practice as it related to the problems, thereby focusing on the content of proportional reasoning skills. Again, further investigations could disaggregate the effects of the intervention, the teacher, and other factors to better understand each.

The game at the center of the intervention seems well designed for its purpose and our results suggest focusing further development on the activities surrounding the game. If after some further research teacher A’s approach to abstraction bridge discussions is found to be successful with other students at other locations, professional development training and the teacher manual might be revised to help teachers adopt such an approach. Whereas it seems unlikely that students will all have a shared previous experience of using a particular solution strategy (as was the case at school A), teachers can nonetheless approach discussions in much the same way - by highlighting the relationships between the numbers, requiring students to explain what each number represents at each stage of the solution, and fostering proper mathematical habits in rounding, multiplying, etc. This might seem like an intuitive approach to discussing mathematics problems but our evidence suggests otherwise. Perhaps the more common intuition is to present as many solution strategies as possible and to allow students to decide which they prefer of those decidedly correct. This is not to imply that allowing students to present incorrect solutions should be considered unproductive. In fact, teachers might find them to be teachable moments that can highlight how the strategy is unsuccessful for the problem at hand - not only in regard to the final calculation but also to the incorrect/meaningless numbers generated at each stage of the solution. Whereas scripting teachers’ contributions seems like a rigid approach, explaining to teachers what we have found to work (whether within their manuals and/or as part of professional development) could foster these behaviors in their own implementations.

In summary, this preliminary study suggests that the design of the Expedition Atlantis intervention, which included four main features that were informed by the findings in previous robot-math interventions (e.g., RSD), was successful in fostering both students’ proportional reasoning skills and personal interests/values as determined by within-participant comparisons. However, further investigations of student and teacher characteristics seem warranted to determine the influences/interactions of both in order to be able to maximize the potentials of robot-math activities.

The current effort represents but one type of integrated STEM instruction, namely technology and engineering used to teach mathematics. Within that type of STEM instruction, it provides a model for how technology and engineering could be used more often to teach mathematics: focusing on one kind of mathematics, developing model eliciting activities centered on that mathematics, using rich game-like scenarios to maintain engagement, and abstraction bridges to generalize the learning. Similar methods could be used in other forms of STEM education - engaging students with application-focused, real-world activities that do not center the to-be-learned skills as the ultimate goals but instead necessitate and motivate (particular, repeatedly practiced) skills as tools.

Endnotes

aBecause some students scored as outliers on a number of measures (number correct on the pre-test, self-rating of familiarity with robots, pre-test ratings of interests or value), those students were removed from analyses to assess their influence. This was true of 33 of our 116 students but when removed, analyses revealed the same pattern of results. Consequently, they remained included. This is true both of the previously presented ANCOVA and the subsequently presented t-tests.

bThe base rate strategy label is one added by the authors. Students most often did not provide a name for their strategy - unless otherwise noted in the case of T charts and scale factors.

Declarations

Authors’ Affiliations

(1)
Learning Research and Development Center, University of Pittsburgh
(2)
National Robotics Engineering Center, Carnegie Mellon University

References

  1. Apedoe, XS, Reynolds, B, Ellefson, MR, & Schunn, CD. (2008). Bringing engineering design into high school science classrooms: the heating/cooling unit. Journal of Science Education and Technology, 17(5), 454–465. doi:10.1007/s10956-008-9114-6.View ArticleGoogle Scholar
  2. Benitti, FBV. (2012). Exploring the educational potential of robotics in schools: a systematic review. Computers & Education, 58(3), 978–988.View ArticleGoogle Scholar
  3. Brown, DE, & Clement, J. (1989). Overcoming misconceptions via analogical reasoning: abstract transfer versus explanatory model construction. Instructional Science, 18, 237–261.View ArticleGoogle Scholar
  4. Doppelt, Y, Mehalik, MM, Schunn, CD, Silk, E, & Krysinski, D. (2008). Engagement and achievements: a case study of design-based learning in a science context. Journal of Technology Education, 19(2), 21–38.Google Scholar
  5. FIRST. (2013). The science of celebration: 2013 annual impact report. Retrieved from http://www.usfirst.org/annual-report.Google Scholar
  6. Hampton, NZ, & Mason, E. (2003). Learning disabilities, gender, sources of efficacy, self-efficacy beliefs, and academic achievement in high school students. Journal of School Psychology, 41, 101–112. doi:10.1016/S0022-4405(03)00028-1.View ArticleGoogle Scholar
  7. Kroesbergen, EH, & Van Luit, JEH. (2003). Mathematics interventions for children with special education needs: a meta-analysis. Remedial and Special Education, 24(2), 97–114. doi:10.1177/07419325030240020501.View ArticleGoogle Scholar
  8. Lesh, R, Hoover, M, Hole, B, Kelly, A, & Post, T. (2000). Principles for developing thought-revealing activities for students and teachers. In AE Kelly & RA Lesh (Eds.), Handbook of Research Design in Mathematics and Science Education (pp. 591–646). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  9. Lesh, R, Post, T, & Behr, M. (1988). Proportional reasoning. In J Hiebert & M Behr (Eds.), Number Concepts and Operation in the Middle Grades (Vol. 2, pp. 93–118). Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  10. Moreno, R, & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309–326. doi:10.1007/s10648-007-9047-2.View ArticleGoogle Scholar
  11. Mubin, O., Stevens, C. J., Shahid, S., Al Mahmud, A., & Dong, J. J. (2013). A review of the applicability of robots in education. Journal of Technology in Education and Learning, 1. Retrieved from http://roila.org/wp-content/uploads/2013/07/209-0015.pdf
  12. National Research Council (2001). Adding it up: Helping children learn mathematics. In J. Kilpatrick, J. Swafford, & F. Bradford (Eds.), Mathematics Learning Study. Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: Nation Academy PressGoogle Scholar
  13. Pastor, PN, & Reuben, CA. (2008). Diagnosed attention deficit hyperactivity disorder and learning disability: United States, 2004–2006. Vital Health Statistics, 10(237), 1–15.Google Scholar
  14. REC. (2013). Inspire: Annual report 2013. Retrieved from www.roboticseducation.org/about/.
  15. Reid, K, & Floyd, C (2007). The tsunami model eliciting activity: Implementation and assessment of an interdisciplinary activity in a pre-engineering course. Proceedings of the 2007 American Society of Engineering Education National Conference. Retrieved from http://www.asee.org/search/proceedings?search=session_title%3A%22Engineering+in+Middle+Schools%22+AND+conference%3A%222007+Annual+Conference+%26+Exposition%22
  16. Silk, EM (2011). Resources for learning robots: Environments and framings connecting math in robotics (Doctoral dissertation). Retrieved from ACM Digital Library.(AAI3485771)Google Scholar
  17. Silk, EM, Higashi, R, Shoop, R, & Schunn, CD. (2010). Designing technology activities that teach mathematics. The Technology Teacher, 69(4), 21–27.Google Scholar
  18. Silk, EM, Schunn, CD (2008). Core concepts in engineering as a basis for understanding and improving K-12 engineering education in the United States. Report submitted to the National Academy of Engineering Committee on Understanding and Improving K-12 Engineering Education in the United States. Retrieved from http://elisilk.net/research/SilkSchunn2008a-NAE-FinalDraft.pdf
  19. Vollstedt, AM, Robinson, M, & Wang, E. (2007). Using robotics to enhance science, technology, engineering, and mathematics curricula. In Proceedings of American Society for Engineering Education Pacific Southwest annual conference, Honolulu: HawaiiGoogle Scholar
  20. Weaver, R, Junker, B. W (2004). Model specification for cognitive assessment of proportional reasoning. Department of Statistics, Paper 171. Retrieved from http://repository.cmu.edu/statistics/171
  21. Zimmerman, BJ, & Martinez-Pons, M. (1990). Student differences in self-regulated learning: relating grade, sex, and giftedness to self-efficacy and strategy use. Journal of Educational Psychology, 82(1), 51–59. doi:10.1037/0022-0663.82.1.51.View ArticleGoogle Scholar

Copyright

© Alfieri et al.; licensee Springer. 2015

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.