The effects of educational robotics in STEM education: a multilevel meta-analysis

Abstract

Educational robotics, as emerging technologies, have been widely applied in the field of STEM education to enhance the instructional and learning quality. Although previous research has highlighted potentials of applying educational robotics in STEM education, there is a lack of empirical evidence to investigate and understand the overall effects of using educational robotics in STEM education as well as the critical factors that influence the effects. To fill this gap, this research conducted a multilevel meta-analysis to examine the overall effect size of using educational robotics in STEM education under K-16 education based on 30 effect sizes from 21 studies published between 2010 and 2022. Furthermore, we examined the possible moderator variables of robot-assisted STEM education, including discipline, educational level, instructor support, instructional strategy, interactive type, intervention duration, robotic type, and control group condition. Results showed that educational robotics had the moderate-sized effects on students’ STEM learning compared to the non-robotics condition. Specifically, educational robotics had moderate-sized effects on students’ learning performances and learning attitudes, and insignificant effects on the improvement of computational thinking. Furthermore, we examined the influence of moderator variables in robot-assisted STEM education. Results indicated that the moderator variable of discipline was significantly associated with the effects of educational robotics on STEM learning. Based on the findings, educational and technological implications were provided to guide future research and practice in the application of educational robotics in STEM education.

Introduction

With the rapid development of science and technology, educational robotics, as emerging technologies that combine different digital techniques (e.g., mechanical manufacturing, electronic sensors, artificial intelligence), have been applied in multiple educational contexts to enhance the instructional and learning quality. Specifically, in the field of STEM education, which highlights the integration of science, technology, engineering, and mathematics, educational robotics are usually used to mediate and assist the instructional and learning process, which is named the robot-assisted STEM education in prior research (Atman Uslu et al., 2022; Augello et al., 2020; Evripidou et al., 2020). In recent years, the emergence of educational robotics has attracted wide attention and previous research has revealed the potentials of applying educational robotics in STEM education, such as promoting students’ learning performance (Okita, 2014), arousing their interest (Chin et al., 2014), and cultivating their computational thinking (Chalmers, 2018). However, although multiple studies revealed the benefits of applying robotics in STEM education, some of them also reported the ineffectiveness or negative effects of using robotics in assisting STEM education (e.g., Berland & Wilensky, 2015; Keren & Fridin, 2014). Therefore, to guide the research and practice of robot-assisted STEM education, it is essential to further examine and verify the effects of educational robotics in STEM education. More importantly, since the complexity of robot-assisted STEM education (Xu & Ouyang, 2022b), except the technological element of educational robotics, other elements might also influence the effects of using educational robotics in STEM education, such as human subject (e.g., instructors, students), information (e.g., discipline knowledge), medium (e.g., interactive types), and external environment (e.g., intervention durations). To achieve a high-quality STEM education, the application of educational robotics should also take careful consideration of these complex factors (Byrne & Callaghan, 2014; Ouyang & Jiao, 2021). Meta-analysis approaches have been conducted to examine the effects of educational robotics on STEM education (e.g., Batdi et al., 2019; Kazu & Kurtoglu, 2021; Mustafa et al., 2016). For example, Sapounidis et al. (2023) conducted a systematic review and meta-analysis to investigate the application effects of educational robotics in STEM education in primary school. However, existing review and meta-analysis works about robot-assisted STEM education (e.g., Atman Uslu et al., 2022; Sapounidis et al., 2023; Zhong & Xia, 2020) mainly focused on the application of educational robotics in one of the disciplines (e.g., mathematics, physics) or educational levels (e.g., primary school). Moreover, these studies did not holistically examine the moderator variables that might influence the effects of educational robotics on STEM education. To fill these gaps, the current research conducted a multilevel meta-analysis to holistically investigate the field of robot-assisted STEM education. Specifically, we aimed to evaluate the effects of educational robotic applications in STEM education, including how educational robotics impact on students’ learning performances, learning attitudes, and computational thinking (CT). Furthermore, we holistically explored the moderator variables of robot-assisted STEM education (i.e., discipline, educational level, instructor support, instructional strategy, interactive type, intervention duration, robotic type, control group condition) and examined how these factors influenced the effects of using educational robotics in STEM education. Based on our findings, educational and technological implications were provided to improve the research and practice of robot-assisted STEM education.

Literature review

Educational robotics in STEM education

With the rapid development of computer science and technologies, robot, which combines the techniques of mechanical manufacturing, electronic sensors, and artificial intelligence, has been applied in different fields to help humans reach automatic, efficient, and adaptive life (Wang et al., 2018). Educational robotics, as the application of robot technique in education, is recognized as an innovative learning tool to change learning environment, transform instructional and learning processes, and create new educational ecology (Atman Uslu et al., 2022; Evripidou et al., 2020; Yueh & Chiang, 2020). Compared to other learning tools and techniques (e.g., animated characters, virtual agents), educational robotic is usually designed as entity of animal, vehicle, human, or in different shapes and sizes, which can directly interact with students and enrich their learning experience (Atman Uslu et al., 2022; Mubin et al., 2013). In the last decade, educational robotics have been widely used in different learning contexts, such as language education (Van den Berghe et al., 2019), special education (Tlili et al., 2020) and STEM education (Sophokleous et al., 2021), with a goal to enhance educational quality.

STEM education requires students to master multidisciplinary knowledge (e.g., science, technology, engineering, mathematics) and solve open-ended, ill-structured problems in the real world. Traditional instructor-directed strategies might fail to help students achieve a high quality of learning effects in STEM education because it mainly highlights the efficiency of knowledge conveying and memorizing rather than students’ understandings (Sapounidis & Alimisis, 2020; Xu & Ouyang, 2022b). For example, under traditional lecturing modes, students might have difficulty in understanding the complex STEM concepts and have few opportunities to develop higher-order thinking (e.g., problem-solving ability, creativity, CT) (Bers, 2021). Hence, the robot-assisted STEM education, as an integrated field of educational robotics and STEM education, highlights the utilization of educational robotics to empower the instructional and learning process of STEM education. The emergence of educational robotics can offer opportunities for students to manipulate objects, understand relevant theories and concepts, and solve problems during the learning processes (Atman Uslu et al., 2022; Evripidou et al., 2020). Therefore, educational robotics are usually used in STEM education to introduce students to complex knowledge and concepts (Okita, 2014), promote their motivation and engagement (Chin et al., 2014; Kim et al., 2015), and cultivate their cognitive thinking and abilities (Chalmers, 2018).

The effects of educational robotics on STEM education

As an emerging technology, the positive effects of educational robotics have been verified in the field of STEM education (Anwar et al., 2019; Atman Uslu et al., 2022). First, increasing evidence showed that educational robotics had positive effects on improving students’ learning performances in STEM education. Rather than gaining knowledge from instructors passively, students can understand the complex concepts in STEM-related disciplines through learning with educational robotics (Ferrarelli & Iocchi, 2021; Mohamed et al., 2021). For example, Ferrarelli and Iocchi (2021) found that the application of computer programming robotics significantly improved high school students’ understandings of the physical principles. Mohamed et al. (2021) used a kind of educational robotics, namely Kodockly, to teach programming and found it had positive impact on the programming performances of young children aged from 6 to 11. Second, previous research found that educational robotics had potential to promote students’ learning attitudes of STEM education (Gomoll et al., 2016; Jaipal-Jamani & Angeli, 2017). Learning attitude refers to students’ attitudes to learning as well as their perceptions, beliefs, and interests during STEM learning. As physical entity, educational robotics in STEM education can serve as a role of a learning companion, to motivate students’ learning interest (Anwar et al., 2019; Mitnik et al., 2009). For example, Leonard et al. (2016) found that fifth- through eighth-grade students’ self-efficacy and STEM attitudes enhanced when learning in the combined environment of educational robotics and gamification. Merino-Armero et al. (2018) found that third-grade students who learned with robotics showed higher level motivation than those who learned through pencil-and-paper. Third, previous studies have claimed that educational robotics contributed to the development of students’ computational thinking in STEM education (Chen et al., 2017; Eguchi, 2014; Sarıtepeci & Durak, 2017). Computational thinking (CT) refers to the cognitive ability to solve problems in the most efficient and effective ways through organizing and analyzing data logically (Relkin et al., 2021; Xu et al., 2022). Since robot-assisted activities in STEM education can provide suitable contexts for problem-solving for students, it can promote the development of students’ higher-order thinking, especially CT (Gomoll et al., 2017). For example, Ioannou and Makridou (2018) found that robot-based instruction facilitated eighth-grade students’ development of algorithmic thinking as well as computational thinking.

Despite multiple studies revealing that the usage of educational robotics had positive effects on STEM education, the applications of educational robotics in STEM education also existed some challenges. Educational robotics sometimes may distract students from STEM learning, increase their cognitive loads, and had negative learning effects (Berland & Wilensky, 2015; Keren & Fridin, 2014). For example, Berland and Wilensky (2015) found that eighth-grade students in course supported with physical robotics gained lower programming skills and CT scores rather than those students who worked in courses supported with virtual agents. Keren and Fridin (2014) also highlighted that it was difficult for robotics to play active roles in mathematics education without instructor’s assistance and guidance. Therefore, since the complicated effects of educational robotics, it is essential to further investigate the effects of applying educational robotics in STEM education, to better aid the educators, researchers and technical developers to integrate the robotics techniques and STEM education.

Moderator variables that influence the effects of educational robotics

Since the complexity of educational system, multiple moderator variables exist when applying educational robotics in STEM education context. Specifically, robot-assisted STEM education can be viewed as a complex system that arises from the interactions between technologies (i.e., educational robotics), human subjects (e.g., instructors, students), information (e.g., discipline knowledge), mediums (e.g., interactive types), and external environment (e.g., intervention durations) (Byrne & Callaghan, 2014; Xu & Ouyang, 2022a). Except the technology element of educational robotics, the multiple and complex factors in STEM education (e.g., instructor, student, medium, information, environment) also play integral roles in robot-assisted STEM education (Byrne & Callaghan, 2014; Xu & Ouyang, 2022b). Therefore, to deeply understand the effects of educational robotics on STEM education, the holistic principle needs to be taken into considerations to identify the potential moderator variables, as well as their influences upon the robot-assisted STEM education.

Recently, researchers have started to consider the potential factors and elements that might influence the effects of educational robotics on STEM education (e.g., Sapounidis & Alimisis, 2020; Woo et al., 2021). For example, Sapounidis and Alimisis (2020) highlighted some important educational considerations that might influence the application effects, such as the role of age, student collaboration, and teacher instruction. Woo et al. (2021) also proposed some technical and procedural problems that might affect the implementation of educational robotics in classroom, including the length of deployment, the autonomy of robotic action, and the interactive type. However, these existing works mainly discussed the potential factors that might influence robot-assisted STEM education from the theoretical level, without enough empirical evidence to reveal the specific effects of these moderator variables. Hence, how these complicated moderator variables influence the application of educational robotics in STEM education is still unknown. Therefore, it is important to holistically explore the possible moderator variables that influenced the effect of robot-assisted STEM education.

Previous and current research

In recent years, various reviews and meta-analyses have been conducted to explore the application of robotics in the field of education. For example, Atman Uslu et al. (2019) conducted a systematic review to examine the application of educational robotics. The results clarified that educational robotics had potentials to promote students’ higher-order thinking skills, social skills, learning performances, and affective characteristics. Wang et al. (2023) conducted a meta-analysis and revealed a moderate and positive effect (g = 0.57) of educational robotics on students’ learning outcomes. Talan (2021) used a meta-analysis approach to investigate the effects of educational robotic applications on students’ academic achievement. However, the results showed that the effect size was at a low level (g = 0.385).

Furthermore, some reviews and meta-analyses started to focus on the applications of educational robotics in specific STEM education contexts. Specifically, we located three prior studies that focused on robot-assisted STEM education, including two systematic reviews (Karim et al., 2015; Zhong & Xia, 2020) and one meta-analysis (Sapounidis et al., 2023). Karim et al. (2015) reviewed the research of robot-based mathematics and physics learning and found that educational robotics contributed to reshape K-12 STEM education. Zhong and Xia (2020) revealed that educational robotics generally played active roles in mathematics education in a systematic review. Additionally, Sapounidis et al. (2023) used systematic review and meta-analysis methods to investigate the integration of educational robotics and STEM education in primary school. The findings showed that educational robotics have positive effects on students’ knowledge (g = 0.528), skills (g = 0.600), and attitudes (g = 0.287). However, this research had a limitation that it only examined the effect sizes of using educational robotics on primary school students’ STEM learning and did not examine the influences of possible moderator variables in robot-assisted STEM education.

Overall, existing literature reviews and meta-analyses about educational robotics in STEM education mainly focused on one of the disciplines (e.g., mathematics, physics) or educational levels (e.g., primary school). In addition, although previous results preliminary revealed the positive effects of educational robotics, the effect sizes varied (e.g., minor effect size, moderate effect size). Moreover, previous meta-analyses did not holistically examine the moderator variables that might influence the effects of educational robotics on STEM education. Therefore, although prior research contributed to the understanding of educational robotics, there is still a lack of meta-analysis to examine the effects of educational robotics in STEM education context holistically.

To fill these gaps, the main purpose of this research is to examine the effects of educational robotics in the STEM education context, in order to guide educators, instructors, researchers, and technical developers for future practice and research in robot-assisted STEM education. Specifically, we conducted a multilevel meta-analysis to gain a comprehensive understanding of the application effect of educational robotics in STEM education as well as the moderator variables that might influence the application effects. Specifically, three research questions (RQs) were proposed:

RQ1: What is the overall effect size of using educational robotics in STEM education?

RQ2: What are the average effect sizes of educational robotics on students’ learning performance, learning attitudes and computational thinking (CT) in STEM education?

RQ3: What are the moderator variables of robot-assisted STEM education and how these variables influence the effects of using robotics in STEM education?

Methods

In order to explore the application effects of educational robotics in STEM education, we conducted a meta-analysis from 2010 to 2022, following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) principles (Moher et al., 2009).

Study searching and screening

Searching strategy

To locate the empirical studies of robot-assisted STEM education, several major publisher databases were selected: Web of Science, Taylor & Francis, Scopus, IEEE, Wiley, and ACM (Guan et al., 2020). Filters were used to the empirical research and peer-reviewed articles in the field of education and educational research from January 2010 to December 2022. Additionally, snowballing approach, was also utilized (Wohlin, 2014) to find the articles that were not extracted in database search through citation checking. At the stage of snowballing, Google Scholar was used to manually searching specific articles based on their titles.

Identification of search keywords

The searching strategies were prosed according to the specific requirements of bibliographic databases. To locate the research that applied educational robotics in STEM education, three types of keywords were used as the search terms of article titles in each database, including keywords related to robotics, STEM disciplines, and educational context. Specifically, the following search keywords were used: (“robot*”) AND (“STEM” OR “technology” OR “math*” OR “science” OR “physics” OR “engineering” OR “chemistry” OR “biology” OR “programming” OR “geography”) AND (“learning” OR “education” OR “teaching” OR “class” OR “course” OR “school” OR “student” OR “grade”).

Eligibility criteria

The eligibility criteria were proposed to screen the empirical studies that focused on the applications of educational robotics in STEM education. Based on the research objectives and questions of the current meta-analysis, the following inclusion criteria were adopted: (1) the research should be in the field of STEM education with the support of educational robotics; (2) the research should be an experimental or quasi-experimental design with experimental and control groups; (3) the research should report outcomes related to STEM learning (i.e., learning performance, learning attitude, CT); (4) the research should report sufficient data for effect-size calculations (e.g., sample sizes, means, standard deviations, or t, F values); (5) the research should ensure homogeneity between experimental and control groups through either the pretests or the randomized controlled trials; (6) the research should be published in English; (7) the research should be full-text available.

Searching and screening procedure

The screening process involved the following procedures: (1) removing the duplicates; (2) screening the articles through titles and abstracts based on the eligibility criteria; (3) reading the full texts to further screen the articles based on the eligibility criteria; (4) utilizing the snowballing to further locate the articles; and (5) including the final filtered articles in meta-analysis. In addition, all articles were stored and screened through the Mendeley software (see Fig. 1).

Fig. 1

The selection flowchart used based on PRISMA (Moher et al., 2009)

Specifically, a total of 758 studies were located as the result of the first round of database searching (n = 741) and subsequent snowballing through Google Scholar (n = 17). There was a total of 563 studies after the duplicates were removed. Then, through reviewing the research titles and abstracts, 483 studies were removed based on the eligibility criteria. The selected studies were examined by the first author to determine whether they were suitable for the purpose of this systematic review. Another researcher was invited to independently review approximately 30% of the articles to confirm the reliability and an inter-rater agreement of 93% was achieved. After that, the full text of articles was reviewed by the first author to verify if the studies met all the criteria for inclusion in this meta-analysis. Finally, a total of 21 studies were included for the meta-analysis (see Fig. 1).

Data extraction and coding

To address the research questions, a coding scheme was proposed to search for and identify comparable features and moderator variables of robot-assisted STEM education among the included studies. First, we coded the basic information of each study, including the article’s title, the author names, and the publication year. Second, we coded the main content of each study, including the sample sizes, outcome variables (i.e., learning performance, learning attitude and CT) and moderator variables that might influence the effects of robot-assisted STEM education. Specifically, the outcome variable of learning performance included students’ academic achievements, learning task performance, conceptual knowledge gains in STEM learning. The outcome variable of learning attitude included students’ learning motivation, interest, and perception in STEM learning.

Eight types of moderator variables were coded according to previous research (Byrne & Callaghan, 2014; Lee & Lee, 2022; Zhang et al., 2021): (1) discipline (i.e., science, technology, engineering, mathematics, cross-disciplinary); (2) educational level (i.e., kindergarten, primary school, middle school, high school, higher education); (3) instructor support (i.e., support, no support); (4) instructional strategy (i.e., problem-based learning, project-based learning, game-based learning, lecturing); (5) interactive type (i.e., one-to-one, in groups); (6) intervention duration (i.e., ≤ 1 day, > 1 day and ≤ 1 month, > 1 month); (7) robotic type (i.e., programming robot, social robot); and (8) control group condition (i.e., traditional instruction, other technology).

Two raters completed the coding procedure of the 21 included articles. First, 50% of articles were coded by two coders independently to calculate coding reliability. The Krippendorff’s (2004) alpha reliability of all the coding results was 0.84 among two raters at this phase. Second, after the reliability was ensured, the other articles were coded independently by two raters. Two raters discussed together to reach a consensus when there were conflicting coding results.

Statistical analysis approaches

Effect size calculation

In meta-analysis, Cohen’s d is usually used to calculate the effect sizes (Cohen, 1988). However, Cohen’s d may result in bias and overestimates the effect sizes in small-scale studies (Hedges, 1981). To compensate for the vulnerability of Cohen’s d, Hedges’ g (also known as Cohen’s unbiased d) is used to calculate effect sizes in meta-analysis with respect to small sample sizes, such as sample sizes that are smaller than 50 (Hedges & Olkin, 1985). Therefore, Hedges’ g was selected to measure the mean weighted effect sizes in this study (see Eq. 1). The quantifiable results of all the effect sizes were calculated and converted into the scale-free effect sizes. The 95% confidence interval (CI) for Hedges’ g was used to examine the significant differences. According to Cohen’s criterion of Hedges’ g, 0.2 indicates a minor effect, 0.5 indicates a moderate effect, and 0.8 indicates a large effect (Cohen, 1988). The Comprehensive Meta-Analysis (CMA) software (Borenstein et al., 2013) was used to calculate the individual effect sizes of the included studies.

$$\text{Hedges}^{\prime}\text{s}\,g = \, J*\left( {\text{M}_{{{\text{Exp}}}} {-} \, \text{M}_{{{\text{Control}}}} /\text{SD}_{{{\text{Pooled}}}} } \right),$$

(1)

where J = 1 − 3 / (4 df − 1), df = n_Exp + n_Control− 2.

Multilevel meta-analysis

After calculating the individual effect size, this study examined the average effect sizes of educational robotics on students’ learning performance, learning attitude and CT in STEM education. Moreover, the overall effect size of all the outcome variables was also calculated. Seven studies in our dataset reported more than one outcome variable or one sample size (i.e., Berland & Wilensky, 2015; Brown & Howard, 2014; Kurniawan et al., 2018; Mohamed et al., 2021; Nugent et al., 2014; Rodríguez Corral et al., 2016; Sáez-López et al., 2019). We decided to calculate them as different effect sizes separately. Therefore, we detected 30 effect sizes from 21 studies.

Due to the nested structure of the data in the meta-analysis, we applied a multilevel meta-analysis approach to deal with the dependency of effect sizes (Houben et al., 2015; Spruit et al., 2020). Compared to the traditional meta-analysis model, the multilevel meta-analysis model is developed to handle the hierarchical structure of data and avoid “double-counting” studies in the meta-analysis (Van den Noortgate et al., 2013). This approach can adjust the variance within each study according to the number of reported effect sizes, while maintaining the separation between moderators.

Specifically, a three-level meta-analysis model was used to calculate the average effects sizes of educational robotics on students’ STEM learning (Cheung, 2014). In the three-level meta-analysis model, the sampling variance for the observed effect sizes was set as Level 1, the variance between effect sizes within the same study was set as Level 2, and the variance between the studies was set as Level 3. The method of restricted maximum likelihood (REML) was used to estimate the variances in the model (Hox, 2010). Moreover, a log-likelihood-ratio test was conducted to compare the deviance of the full model to the deviance of the models excluding one of the variance parameters, which helped us determine whether significant variance exists at Level 2 and 3 (Assink & Wibbelink, 2016). R package metafor was used to conduct the multilevel meta-analysis (Viechtbauer & Viechtbauer, 2015).

To further detect associations between different moderator variables and effect sizes, the moderator analysis was further performed (Bloch, 2014). All the moderator variables were categorical moderators in this meta-analysis. Multivariate models were used to conduct the moderator analysis, where all the subgroups of moderator variables were inputted to examine the moderate effects on robot-assisted STEM education (Spruit et al., 2016). Moderator analysis was only performed in case each category of the potential moderator variables was filled with at least three effect sizes (Spruit et al., 2016, 2020). Therefore, the moderator variable categories without enough effect sizes were not included in the moderator analysis (i.e., engineering in discipline, kindergarten in educational level). Furthermore, omnibus tests were used to test the statistical significance of moderator effects. Furthermore, separate effect sizes were conducted for each subgroup as ad hoc analyses for understanding the influence of each moderator variable.

Publication bias

Since studies with positive findings were more likely to be published, we checked if there was any publication bias in this meta-analysis (Franco et al., 2014; Thornton & Lee, 2000). Funnel plot, Egger’s test (Bowden et al., 2015) and Rosenthal’s Fail-safe N test (Orwin, 1983) were used together to detect publication bias. First, the funnel plot is a qualitative and visual method to evaluate the possible publication bias. If the amount of effect of each study does not distribute in the funnel plot symmetrically, it indicates the publication bias might exist (Moher et al., 2009). Second, the Egger’s test is a statistical method used to plot the effect size estimates against their standard errors (Bowden et al., 2015). Third, Fail-safe N estimates the number of unpublished studies as 5 k + 10 (k is the number of effect sizes retrieved) and the greater the Fail-safe N value is, the smaller the publication bias is (Rosenberg, 2007).

Results

The overall effect size of using educational robotics on STEM learning

The meta-analysis consisted of 21 robot-assisted STEM education studies with 30 effect sizes, which included a total of 2433 participants. The Hedges’ g of each effect size was calculated (see Table 1). Furthermore, a three-level meta-analysis model was used to calculate the overall effect size of educational robotics on STEM learning (see Table 2). The results of multilevel meta-analysis showed a mean effect size of 0.488 (SE = 0.19; p < 0.05; 95% CI was 0.094–0.882). The results showed that educational robotics had significantly positive and moderate effects on STEM education, compared to the instructions without the application of educational robotics.

Table 1 The outcome variables, measurements, sample sizes, and effect sizes of the included studies

Table 2 Mean effect sizes of educational robotics on STEM learning

In the three-level meta-analysis model, the result of the Cochran’s Q value was 92.839 (p < 0.001) and the I² value was 71.13%. The results indicated that there was a high heterogeneity among the included studies (Higgins et al., 2003; Sedgwick, 2015). Specifically, the I² of Level 2 was 37.52%, the I² of Level 3 was 33.61%, which revealed that there was a significant variation of effect sizes within unique studies (Level 2) and between unique studies (Level 3). Therefore, the use of multilevel meta-analysis was necessary in this research. In addition, the likelihood ratio test comparing models with and without between-study variance showed that the significant variance was presented at the between-study level (σ² _{Level 3} = 0.31, SE = 0.55, p < 0.001). The variance between the effect sizes within studies was also significant (σ² _{Level 2} = 0.34, SE = 0.58, p < 0.001).

The average effect sizes of using robotics on students’ learning performances, learning attitudes and CT

To further explore the effects of educational robotics on varied learning aspects, this research examined the average effect sizes of specific outcome variables, including students’ learning performances, learning attitudes (i.e., learning perception and learning interest), and CT skills. First, 14 effect sizes from 12 included studies (N = 1082) reported students’ learning performances in the robot-assisted STEM education. The multilevel modeling analysis found a mean effect size of 0.665 (SE = 0.18; p < 0.01; 95% CI was 0.275–1.054), which indicated that educational robotics had a moderate-to-large effect on improving students’ learning performances in STEM education (see Table 2). Second, 12 effect sizes from 10 included studies (N = 900) reported students’ learning attitudes in the robot-assisted STEM education. The multilevel modeling analysis found a mean effect size of 0.497 (SE = 0.24; p < 0.05; 95% CI was 0.023–1.017), which indicated that educational robotics had a moderate effect on influencing students’ learning attitudes in STEM education (see Table 2). Third, 4 effect sizes from 4 included studies (N = 451) reported students’ CT skills in the robot-assisted STEM education. Since no hierarchical structure of data existed, the random effect model was used to calculate the mean effect size. The random effect model analysis found a mean effect size of 0.079 (SE = 0.101; p = 0.434; 95% CI was − 0.119–0.277), which indicated that educational robotics had a minor and insignificant effect on students’ CT in STEM education (see Table 2).

Moderator analysis

The moderator variables of 30 effect sizes were coded from the 21 included studies (see Table 3). Furthermore, the multilevel analyses were conducted to examine the moderator effects of educational robotics on students’ STEM learning (see Table 4). Specifically, among all the moderator variables, discipline was significantly associated with variability in robot-assisted STEM learning effects (F(3, 25) = 3.9205, p < 0.05). However, educational level, instructor support, instructional strategy, interactive type, intervention duration, robotic type and control group condition were insignificantly associated to the students’ learning effects (p > 0.05).

Table 3 The moderator variables of the included studies

Table 4 The overall results of moderator analysis

Discipline

Regarding the discipline, most of the studies were conducted in the discipline of technology (k = 10), followed by cross-disciplines (i.e., more than one discipline) (k = 8), mathematics (k = 6), science (k = 5), and engineering (k = 1). The category of engineering was not included in the moderator analysis, because it included less than 3 effect sizes. Significant difference was detected between the weighted effect sizes of different disciplines (F(3, 25) = 3.9205, p < 0.05), which indicated that discipline significantly moderated the effects of educational robotics on STEM learning. Specifically, technology discipline had a large effect on robot-assisted STEM education (g = 0.947, p < 0.01), while the cross-disciplinary courses had a nearly moderate effect on robot-assisted STEM education (g = 0.414, p < 0.05). In addition, small effect sizes were found in the mathematics discipline and the effect size was not statistically significant (mathematics: g = 0.376, p = 0.204). In addition, science discipline had a negative and insignificant effect size (g = − 0.613, p = 0.488). Therefore, the applications of educational robotics had larger effect sizes in technology discipline and cross-disciplinary, and smaller effect sizes in science and mathematics disciplines.

Educational level

Regarding the educational level, most of the studies were conducted in primary school (k = 9), followed by middle school (k = 8), higher education (k = 8), high school (k = 4) and kindergarten (k = 1). The category of kindergarten was not included in the moderator analysis, because it included less than 3 effect sizes. No significant difference was detected between the weighted effect sizes of different educational levels (F(3, 25) = 2.5690, p > 0.05), indicating that the effects of educational robotics on STEM education had no difference across educational levels. Specifically, the effect size of educational robotics on STEM education was at the large level for students in higher education (g = 1.052, p < 0.01) and at the moderate level for students in primary school (g = 0.518, p < 0.05). In addition, the effect size of high school was at the large level without statistical significance (g = 0.824, p = 0.061). However, educational robotics had a negative and insignificant effect size for students in middle school (g = -0.189, p = 0.729).

Instructor support

Regarding the instructor support, among the included studies, instructors supported students in robot-assisted STEM education in most of the cases (k = 19), and they sometimes did not provide support to students (k = 11). No significant difference was detected between the weighted effect sizes of instructor supports (F(1, 28) = 0.1951, p > 0.05), which indicated that the effects of educational robotics on STEM education had no difference across instructor supports. Specifically, the effect size of educational robotics on STEM education was significantly larger in the instructor-supported cases (g = 0.546, p < 0.05) than the cases where the instructors did not provide supports to students (g = 0.312, p = 0.351).

Instructional strategy

Regarding the instructional strategy, most of the studies were conducted through the problem-based learning mode (k = 13), followed by the project-based learning mode (k = 8), lecturing (k = 6), and game-based learning (k = 3). No significant difference was detected between the weighted effect sizes of different instructional strategies (F(3, 26) = 0.5636, p > 0.05). This result indicated that the effects of educational robotics on STEM education had no difference across instructional strategies. Specifically, the game-based learning had a significantly large-level effect size (g = 1.029, p < 0.001) on robot-assisted STEM learning. The project-based learning had a significantly moderate-level effect size (g = 0.555, p < 0.05). However, both the problem-based learning and lecturing had insignificant effect sizes (problem-based learning: g = 0.440, p = 0.277; lecturing: g = 0.025, p = 0.964).

Interactive type

Regarding the interactive type, most of the included studies reported student’s interactive types with robotics (k = 28). Among the 28 effect sizes, the interactive type between students and robotics was one-to-one in 13 effect sizes and the interactive type was within groups in other 15 effect sizes. No significant difference was detected between the weighted effect sizes of different interactive types (F(1, 26) = 0.0027, p > 0.05), which indicated that the effects of educational robotics on STEM education had no difference across interactive types. The effect size of robotics on STEM education was at the moderate level for students who interacted with robotics in groups (g = 0.455, p < 0.05). No significant effect size was identified in the one-to-one interactive type (g = 0.523, p = 0.336).

Intervention duration

Regarding the intervention duration, included studies were mostly conducted in the duration length of “> 1 month” (k = 15), followed by “≤ 1 day” (k = 7), and “> 1 day and ≤ 1 month” (k = 7). No significant difference was detected between the weighted effect sizes of different intervention durations (F(2, 26) = 1.2442, p > 0.05), which indicated that the effects of educational robotics on STEM education had no difference across intervention durations. Specifically, the duration length of “> 1 month” (g = 0.672, p < 0.001) had the largest effect size on robot-assisted STEM learning. No significant effect sizes were identified in the duration length of “≤ 1 day” (g = 0.263, p = 724), and “> 1 day and ≤ 1 month” (g = − 0.040, p = 0.955).

Robotic type

Regarding the robotic type, programming robots (i.e., robotics that specifically designed as learning tools for students to design and operate them with programming languages) were mostly used in the include studies (k = 24), while social robots were rarely used (k = 6). No significant difference was detected between the weighted effect sizes of different robotic types (F(1, 28) = 0.9438, p > 0.05), which indicated that the effects of educational robotics on STEM education had no difference across robotic types. Specifically, the effect size of programming robot on STEM education was identified at the moderate level (g = 0.569, p < 0.05), while the effect size of social robot was small and insignificant (g = 0.025, p = 0.964).

Control group condition

Regarding the control group condition, the control group condition was reported as the traditional instruction without educational robotics in 23 effect sizes. The other 7 effect sizes reported that they used other technologies (e.g., virtual agent) in control groups. No significant difference was detected between the weighted effect sizes of different control group conditions (F(1, 28) = 2.2099, p > 0.05). This result indicated that the effects of educational robotics on STEM education had no difference across control group conditions. Specifically, the effect size of educational robotics on STEM education was at the moderate level when the control group condition was the traditional instruction (g = 0.614, p < 0.01). The effect size was negative and insignificant when the control group condition was set as other technologies (g = − 0.026, p = 0.970).

Publication bias

To check for the publication bias, a funnel plot was firstly used to visually present the distribution of effect sizes in this meta-analysis. In the funnel plot, the distribution of the effect sizes was not in a perfectly symmetric pattern around the summary effect, which indicated that the publication bias may exist. Therefore, the quantitative statistical approaches (i.e., Egger’s test, fail-safe N) were further used to test the possible publication bias. First, a non-significant p-value (p = 0.0964) was examined in the Egger’s test for a regression intercept, which indicated there was no evidence of publication bias in this meta-analysis. Second, the fail-safe N = 530 (k = 30), suggested that 530 studies should be added to the meta-analysis before the cumulative size effect would become statistically insignificant. Overall, the results implied that publication bias to some extent existed in our data, but there was no indication of a strong bias.

Discussion and implications

Addressing the research questions

Although educational robotics have attracted wide attention recently, there is a lack of literature review work to holistically examine the effects of robot-assisted STEM education. To gain a comprehensive understanding of the integration of educational robotics and STEM education, this research conducted a multilevel meta-analysis to examine the application effects of educational robotics in STEM education as well as the influence of relevant moderator variables. Specifically, to address the first research question, we included 21 robot-assisted STEM education studies with 30 effect sizes to examine the overall effect size of educational robotics on STEM education through a multilevel approach. A moderate-level and positive effect (g = 0.488) of educational robotics on students’ STEM learning was shown in the current research. Furthermore, to address the second research question, we also examined the effects of educational robotics on students’ learning performance, learning attitude and computational thinking in STEM education. Specifically, educational robotics both had moderate impacts on students’ learning performances (g = 0.665) and learning attitudes (g = 0.497) in STEM education. In addition, this meta-analysis found that the robot-assisted STEM education could not significantly improve students’ CT.

To address the third research question, we further used multilevel analyses to explore the potential moderator variables (i.e., discipline, educational level, instructor support, instructional strategy, interactive type, intervention duration, robotic type, control group condition) in the robot-assisted STEM education as well as their moderator effects. Among all the moderator variables, only discipline was found to be significantly associated with robot-assisted STEM learning effects. Specifically, the applications of educational robotics were more effective in technology discipline and cross-disciplinary, and less effective in science and mathematics disciplines. Furthermore, first, we found that the educational level of higher education was more suitable to apply educational robotic techniques than other educational levels. Second, robot-assisted STEM education had more positive effects when instructors provided supports to students, compared to the absence of instructors. Third, educational robotics had relatively greater effects on STEM learning when combining with game-based learning and project-based learning than other instructional strategies. Fourth, students’ group-level interaction resulted in better learning outcomes than one-to-one interaction when applying educational robotics in STEM education. Fifth, among all the intervention durations (i.e., from ≤ 1 day to ≥ 1 month), the best intervention duration for robot-assisted STEM education was more than one month. Sixth, the results found that the use of programming robotics had more positive effects than social robotics in STEM education. Finally, compared to using other technologies, the effect size of using educational robotics in STEM learning was larger when the control group condition was set as the traditional instruction without technological supports.

Consistent with the findings of previous meta-analyses (e.g., Sapounidis et al., 2023; Wang et al., 2023), the current research also revealed the positive application effects of educational robotics. However, the overall effect size of educational robotics in this research is at a moderate level (g = 0.488), which is different when comparing to prior studies. The different scopes of previous meta-analyses might cause this discrepancy. For example, Wang et al. (2023) mainly focused on the effects of educational robotics in the whole educational context rather than specific STEM education context (g = 0.57), while Sapounidis et al. (2023) merely focused on the effects of educational robotics on primary school students’ STEM learning (g = 0.428). In addition, from an analytical perspective, the current study used a multilevel meta-analysis approach to calculate the combined effect sizes and conduct the moderator analysis. Hence, compared to previous meta-analyses that mainly used traditional approaches (e.g., fixed effect model, random effect model), the analytical procedure in this study might better deal with the hierarchical structure of data and avoid “double-counting” studies (Van Den Noortgate et al., 2013). To sum up, the current research used a multilevel meta-analysis approach to expand the findings of previous meta-analyses in the field of robot-assisted STEM education (e.g., Sapounidis et al., 2023; Talan, 2021; Wang et al., 2023), which guided the following educational and technological implications.

Educational implications

From an educational perspective, the emergence of educational robotics might influence other educational elements (e.g., instructor, student) in STEM education, to finally affect the instructional and learning process and performance. First, regarding the effects of educational robotics, our findings showed that students benefited from robot-assisted STEM education, which mainly promoted their learning performances and attitudes. Confirming previous studies’ findings (e.g., Atman Uslu et al., 2022; Karim et al., 2015; Zhong & Xia, 2020), we yielded reliable and accurate inferences to indicate that educational robotics had great potential to reshape STEM education. However, inconsistent with previous research that highlighted the role of educational robotic in promoting students’ CT (Sapounidis et al., 2023; Zhang et al., 2021), we found the insignificant effects of educational robotics on students’ CT. Since there are few empirical studies investigating the effects of educational robotics on students’ CT during STEM learning, the current meta-analysis only included 4 related research to calculate the average effect size. Therefore, future research is supposed to further examine how educational robotics influence students’ CT in STEM education. In addition, CT is a higher-order thinking associated with problem-solving ability, logical skill and creativity (Li et al., 2020; Wing, 2006), which may cannot be enhanced merely through using educational robotics. Hence, the robot-assisted STEM education is supposed to be combined with ill-structured problems and tasks, to allow students to solve problems and develop CT during the learning processes (Wang et al., 2022; Xu et al., 2022). Supporting this viewpoint, we found that project-based learning and game-based learning had more positive effects on robot-assisted STEM education than traditional lecturing mode. Therefore, when educational robotics are applied in STEM education, the instructional process should also shift from the instructor-directed to student-centered learning (Xu & Ouyang, 2022b). Instead of directly conveying knowledge, instructors should utilize educational robotics to facilitate students’ learning experience and work as facilitators to provide guidance and support students. Furthermore, although previous research claimed that robotics could free human beings from redundant work or even replace them in the field of education (DeCanio, 2016), our findings indicated that instructor’s involvement was important and irreplaceable in robot-assisted STEM education. In addition, the findings of our moderator analysis showed a more positive result for technology discipline than other disciplines. Hence, when applying educational robotics in STEM education contexts, educators and instructors should pay attention to the connection between robotic curriculum and specific STEM disciplines, such as science, mathematics, and engineering (Zhang et al., 2021). For example, in a science course, learning activities may focus on using robotics to explore scientific concepts, such as the principles of motion and energy. In a mathematics course, programming robotics can be operated to solve mathematical problems, such as measuring angles or calculating distances. Overall, although educational robotics can bring opportunities to promote the quality of STEM education (Chalmers, 2018; Chin et al., 2014; Okita, 2014), we cannot overstate the role of technology and overlook the role of pedagogy (Selwyn, 2016). Pedagogy should remain the foundation of STEM education, while the emerging technologies, such as educational robotics, are supposed to act as supplements to enhance students’ learning experience (Friedman & Deek, 2003). Therefore, to reach the goal of high-quality STEM learning, future robot-assisted STEM education should also focus on instructor involvement and pedagogical design to create student-centered learning experience.

Technological implications

From a technological perspective, since educational robotic is an emerging technique in STEM education, future development requires a better fit between robotic technologies and STEM education contexts. First, compared to computer programming robot, social robot was less applied in STEM education context, and had lower-level impact on students’ STEM learning. To explain this finding, programming robots mainly focus on the programming training and operation (Calinon, 2009), while social robots are usually designed to convey knowledge or answer students’ questions, and more likely to be used in language education and special education (Belpaeme et al., 2018). However, instead of programming robots, previous studies have revealed that social robots had the potential to enhance students’ learning interest and experience in STEM education though human–robot communication and interaction (e.g., Shiomi et al., 2015; Yang et al., 2018). Therefore, future design and development of social robots can focus on STEM-related disciplines and learning contexts. Moreover, advanced artificial intelligence (AI) techniques can be also added into the robotic designs, enabling robotics to adapt to the needs and abilities of students and provide adaptive feedbacks and supports (Chu et al., 2022; Ouyang et al., 2023). Second, in this research, we found that educational robotics were more effective in high school and higher education than other education levels (e.g., primary school, middle school). Due to the complex functions of educational robotics, most of the educational robotics might be more suitable for older students (e.g., students in higher education) than younger students (e.g., students in primary school). Therefore, more advanced techniques, such as 3D printing and virtual reality, can be integrated with the design of educational robotics, to help younger students easily understand knowledge during STEM learning (Belpaeme et al., 2018; Zapata-Cáceres & Martin-Barroso, 2021; Zhang et al., 2021). Considering younger students’ cognitive load during robot-assisted STEM learning, the ease of use is also one of the essential points during the design process of educational robotics (Law, 2019; Xu & Ouyang, 2022b). Third, regarding the interactive types with educational robotics, we found that group-level interaction performed better than one-to-one interaction. Based on the social, cultural, and situated perspectives of learning (Vygotsky, 1978), students’ collaboration during robot-assisted learning might help them emerge collective intelligence and generate high-quality learning outcomes (Ouyang et al., 2023). However, in this research, we found that most of the educational robotics were individual-oriented, that did not support multiple students to work, operate or interact at the same time (e.g., Ferrarelli & Iocchi, 2021; Ioannou & Angeli, 2016). Hence, collaboration-oriented educational robotics are suggested to be developed as a future direction of robot-assisted STEM education, in order to empower students’ collaboration during interacting with robotics. For example, future educational robotics can consist of collaboration-oriented features such as shared control of robotics, collaborative data analysis, and group decision-making algorithms.

Conclusions, limitations, and future directions

The application of educational robotics in STEM education, as an emerging trend, has attracted wide attention in the field of education. Previous review works contributed to the understanding of educational robotics, but few of them focused on the STEM education contexts. In addition, although multiple research has revealed the benefits of educational robotics in STEM education, the ineffectiveness or negative effects in robot-assisted STEM education were also reported in some cases. Therefore, to gain a comprehensive understanding of the application effect of educational robotics in STEM education, we conducted a meta-analysis to holistically examine the effects of robot-assisted STEM education as well as the moderator variables that might influence the application effects. Specifically, we used a multilevel meta-analysis approach to examine (a) the overall effect sizes of educational robotics on students’ STEM learning; (b) the average effect sizes of using educational robotics on students’ learning performance, learning attitude, and computational thinking in STEM education; and (c) the possible moderator variables as well as their influence on robot-assisted STEM education. Based on our findings, we proposed educational and technological implications for future practice and development of robot-assisted STEM education.

Three limitations existed in the current research, which led to future research directions. First, we located articles from the best-known scholar databases with the keywords relevant to robot-assisted STEM education. However, since it is an interdisciplinary and technology-dependent field, some articles that only highlighted the technology rather than the STEM education context might be missed during the searching process. In addition, the unpublished articles might be ignored in the current research, which might inflate the total effect size (but under control in the publication bias). Therefore, future review works can further adjust the searching criteria to avoid this problem. Second, due to the criteria of meta-analysis, this research excluded the quantitative studies that involved non-experimental/quasi-experimental designs as well as qualitative studies. However, these studies may also consist of valuable information for us to understand the effects of robot-assisted STEM education. Hence, although meta-analysis can provide powerful evidence for summarizing and synthesizing the effects of educational robotics on STEM education, future review works can use it in conjunction with other methods (e.g., systematic review). Third, we mainly focused on students’ learning performance, learning attitude, and computational thinking during the robot-assisted STEM education; the effects of educational robotics on other cognitive abilities and higher-order thinking (e.g., spatial ability, reasoning ability, creativity) can be investigated in future meta-analysis. To sum up, the potentials and challenges of educational robotics for enhancing STEM education were revealed in this meta-analysis, to guide instructors, educational practitioners, policymakers, and technical developers in promoting the development of future STEM education.