Evidence of STEM enactment effectiveness in Asian student learning outcomes

Abstract

This study used a systematic review and meta-analysis as a method to investigate whether STEM enactment in Asia effectively enhances students’ learning outcomes. Verifiable examples of science, technology, engineering, and mathematics (STEM) education, effectively being applied in Asia, are presented in this study. The study involved 4768 students from 54 studies. Learning outcomes focused on the students’ academic learning achievement, higher-order thinking skills (HOTS), and motivation. The analysis results of effect sizes showed that the STEM enactments in Asia were effective at a moderate level (0.69 [0.58, 0.81 of 95% CI]) of improving students’ learning outcomes. Sequentially, the effectiveness of STEM enactment starts from students’ higher-order thinking skills, moves to students’ academic learning achievement, and ends with the motivation. In addition, STEM enactments in Asia were carried out with several variations where STEM integrated with project-based learning was preferred. The recommendations of this study include a combination of the learning approach, learning orientation, and duration of instruction, all of which contribute to the STEM enactment effectiveness and maximize results in STEM education. Some practical implications, such as the central role of the teacher during the STEM enactment, are extensively discussed. This study supports that STEM education is a universally crucial tool which effectively prepares students from various national and cultural backgrounds, across Asia, toward improved learning outcomes.

Introduction

The role of science, technology, engineering, and mathematics (STEM) education in terms of students’ learning outcome is a central topic for the educational field. However, STEM education is a very broad term (Baran, Bilici, Mesutoglu, & Ocak, 2016; Bybee, 2013; Hsu, Lin, & Yang, 2017). Therefore, in this current study, STEM education (enactment) refers to teaching, learning, and integrating the disciplines and skills of science, technology, mathematics, and engineering in STEM topics, with an emphasis on solving real-world problems. Indeed, STEM education focuses on hands-on activity (Cameron & Craig, 2016; Yildirim & Turk, 2018) to prepare students in facing the developments of a new competitive era. In STEM learning activities, soft skills such as problem-solving, higher-order thinking skills, and collaborative work are the main focuses on which students’ learning is geared toward (Li, Huang, Jiang, & Chang, 2016; Meyrick, 2011).

STEM activities in the classroom endeavor to improve the quality of the learning process (Meyrick, 2011), as well as learning outcomes (Adam, 2004; Cedefop, 2017). Student-learning outcomes vary in areas, including academic learning achievement, attitude, motivation, and higher-order thinking skills. Moreover, some studies said that the learning process and learning outcomes might differ on many factors, such as the subject of study, learning duration, or even kinds of environmental conditions (Marton, Alba, & Kun, 2014; OECD, 2018). Furthermore, a strong link between the quality of the learning process and outcomes from STEM education, which originated from the west, constitutes a fundamental reason for educators and policy-makers to apply the same principles in Asian countries (Khaeroningtyas, Permanasari & Hamidah, 2016; Yildirim, 2016).

Even though the eastern countries (Asia) and western countries (notably, the USA) have many differences such as in teaching and learning characteristics as well as their culture (Di, 2017; Hassan & Jamaludin, 2010; Lee, Chai, & Hong, 2019), both regions have similarities, primarily in terms of problems and challenges faced in the education field. The birth and development of STEM education in the west were motivated by the low interest of the younger generation in work related to the STEM field (Chesky & Wolfmeyer, 2015). This low-interest condition was also exacerbated by the increasing competitiveness of workplace and uncertain global world challenges (Chesky & Wolfmeyer, 2015). Indeed, this condition is also the same as that faced by most countries in Asia. The problem of low student interest in a subject related to STEM, the lack of interest for young people in STEM-related work, and the highly competitive global challenges of the world, are similar to what happened in the USA (Jayarajah, Saat, Rauf, & Amnah, 2014; Kim, Chu, & Lim, 2015).

New changes are needed for the teaching and learning process that can address the challenges faced by Asian countries. Therefore, it is not surprising that over the last decade, there has been a good deal of research done by researchers and teachers in Asia, especially related to STEM enactment in classrooms (Lee et al., 2019; Lutfi, Ismail, & Azis, 2018; Yildirim, 2016; Yıldırım & Altun, 2015; Yıldırım & Sevi, 2016). Currently, STEM enactments in Asia not only focus on extending STEM-related subjects and students’ interest but also on concerns about students’ twenty-first-century learning outcomes such as real-world problem-solving capacity, academic learning achievement, as well as higher-order thinking skills (Lee et al., 2019). STEM implementation in Asia is often accompanied by a learning approach or model (Suratno, Wahono, Chang, Retnowati, & Yushardi, 2020). An evaluation and current status of whether STEM education also has a good impact, specifically in terms of learning outcomes in the Asian region, is logically necessary.

Several extensive works on the effectiveness of STEM education on learning outcomes have been published (Jayarajah et al., 2014; Saraç, 2018; Yildirim, 2016). Research showed that STEM education is effective in improving students’ learning outcomes, such as academic learning achievement, student motivation, attitude, problem-solving skills (Saraç, 2018; Yildirim, 2016). Further research shows that more than two-thirds of publications in the STEM field come from America (Lee et al., 2019). Lee et al. also state that further research is needed to adjust the STEM education for the conditions faced by Asian countries. The statement indicates that an important consideration is how to redesign curricula in Asia in a way that accommodates STEM education. Another research conducted by Mustafa, Ismail, Tasir, Said, and Haruzuan (2016) reviewed effective strategies in integrating STEM education globally for many purposes, including student-learning outcomes. Based on this study, project-based learning was the most effective strategy to implement STEM education among Asian countries; especially studies were focused on students in the secondary setting. Furthermore, some studies have recently reviewed the trend of research in STEM education. The studies argued that research in STEM education is increasing in importance globally and being an international field (Li, Froyd, & Wang, 2019; Li, Wang, Xiao, & Froyd, 2020). However, none of the studies revealed the effectiveness of STEM enactment in the Asian sphere with all the characteristics inherent in said countries. It is crucial to delve into the effectiveness of STEM enactment in Asian countries, which from some aspects, are quite different. However, many problems faced in education have similarities to the western country, the USA, where STEM education originated. Moreover, that is important to know whether STEM education is a fundamental tool in Asia toward improved learning outcomes. Therefore, this current study will have considerable impacts and substantial contributions to the knowledge body of STEM education throughout the world.

Research focus

This study points out a systematic result of the review and a meta-analysis pertinent to how the impact of STEM enactment to Asian students’ learning outcomes. The main focus of learning outcomes under investigation is students’ academic learning achievement, higher-order thinking skills, and motivation. The key questions that guide this study are as follows:

• What is the portrait of STEM enactment in Asian countries in terms of region, subject, and education level?

• Do the STEM enactments influence students’ academic learning achievement, higher-order thinking skills (HOTS), and motivation in Asian countries?

• Under what circumstances and for what learning outcomes are STEM enactments more effective in Asian students?

STEM education and its significant development in Asian regions

STEM education has a very broad meaning. Therefore, many definitions were developed and discovered during the last two decades. Bybee (2013) states that STEM education can consist of a subject, intradisciplinary, interdisciplinary, or can be a particular discipline. Furthermore, Bybee (2013) and Sanders (2009) asserted that STEM education is a spectrum that focuses on solving real problems, which have an interdisciplinary nature at its core. Another opinion states that STEM education is a meta-discipline based on learning standards where teaching has integrated teaching and learning approaches, and where specific content is undivided, contemplating a dynamic and fluid instruction (Merrill & Daugherty, 2009). A more modern definition states that STEM education is an interdisciplinary teaching method that integrates science, technology, engineering, mathematics, and other knowledge, skills, and beliefs, in particular, to these disciplines (Baran et al., 2016; Koul, Fraser, Maynard, & Tade, 2018; Thibaut et al., 2018). Thus, STEM education is a term referring to teaching and learning in a STEM subject, which emphasizes problem-solving with real-world problems integrating many disciplines and other skills such as science, technology, mathematics, and engineering.

STEM education has been present for more than two decades (Timms, Moyle, Weldon, & Mitchell, 2018). The term STEM started from the term SMET (science, mathematics, engineering, technology), which came into existence in the 1990s (Chesky & Wolfmeyer, 2015). Some education experts from western countries (notably, the USA) initiated STEM education. This approach grew in popularity after the US government announced the plan to advance education into STEM education in 2009 (Burke & McNeill, 2011). STEM education is highly promoted in the USA to encourage the next generation into training within the fields of STEM. Furthermore, Burke & McNeill argued that another goal was to maintain the enthusiasm of the younger generation in their interest in STEM-related careers. However, the essential goal is that both students and the younger generation can face the competition of the new global world.

The rapid development and functional effects of STEM education programs in western countries have attracted the interest of many researchers and policy-makers from other countries (Sheffield et al., 2018; Timms et al., 2018), including Asia. Eastern countries face similar problems where there is a lack of interest from the younger generation in careers related to STEM (Jayarajah et al., 2014; Kim et al., 2015; Sin, Ng, Shiu, & Chung, 2017). Furthermore, Jayarajah et al. (2014) and Shahali, Halim, Rasul, Osman, & Zulkifeli (2017) exemplify Malaysia consistently registers lower numbers of citizens interested in science, engineering, and technology issues compared to the USA. As for the Malaysian population, it shows that more than one-third of the children clearly expressed a lack of interest in science and technology. Another researcher, Kim et al. (2015), asserts that in the last two decades, Korea has faced a problem in science and engineering education, which is students’ disinterest in science and math, even though their achievement in science and math is high. Another crucial reason is that STEM education promises as an appropriate tool for students in facing challenges and global competition (Kim et al., 2015; Meyrick, 2011; Yildirim, 2016).

Several parts of Asia, such as Western Asia, Eastern Asia, and Southeastern Asia, are now aggressively implementing and developing STEM education (Chen & Chang, 2018; Choi & Hong, 2015; Karahan, Bilici & Unal, 2015; Park & Yoo, 2013). Some countries such as Korea, Thailand, and Malaysia have focused on STEM/ STEAM education as an essential part of their education system (Cho, 2013; Hong, 2017; Hsiao et al., 2017; Kang, Ju, & Jang, 2013; Shahali, Ismail, & Halim, 2017). While in other countries in Asia, even though STEM education has not become a regular part of the education system, many researchers or teachers have enacted STEM education. Several review studies have pointed out that the trend of research on STEM education in Asia began in 2013. Today, STEM has become a phenomenon that attracts many people (Jayarajah et al., 2014; Lee et al., 2019). Therefore, during this booming stage in Asia, it is crucial to know the extent of the impact of STEM enactments, especially concerning the students’ learning outcomes.

The supporting of instructional strategies on STEM education

The implementation of STEM education is carried out in various ways throughout the world, including in Asia. Some learning approaches or learning models are combined and or juxtaposed with the STEM enactment (Chung, Lin, & Lou, 2018; Lou, Tsai, Tseng, & Shih, 2014). For example, the researchers used project-based learning, problem-based learning, or the 6E learning model in enacting STEM education. This combination is needed to strengthen the expected effect after STEM learning (Mustafa et al., 2016). Furthermore, the modification and or combination of STEM with learning approaches or models have a high potential in facilitating implementation and for achieving effective instruction (Martín-Páez, Aguilera, Perales-Palacios, & Vílchez-González, 2019; Mustafa et al., 2016). However, STEM learning may be implemented with or without other learning approaches (Chung, Lin, & Lou, 2018; Martín-Páez et al., 2019). Moreover, Jeong and Kim (2015) proposes that effective instruction occurs when students are given the learning opportunity to demonstrate, adapt, modify, and transform new knowledge to meet the needs of new contexts and situations. Successful implementation of instruction, of course, leads to the accomplishment of predetermined targets, in this case, improved student learning outcomes.

Ample studies suggest using the project-based learning (PjBL) approach to implement STEM education. Mustafa et al. (2016) investigated the dominant instructional strategies to promote the integration of STEM education at different institutional levels. Mustafa et al. argued that combined with project-based learning was the most effective way to implement STEM education. This assertion is reasonable because PjBL characteristics are quite similar to the integrated STEM approach (Siew, Amir, & Chong, 2015). Chiang and Lee (2016) said that the characteristics of PjBL are encouraging students to work cooperatively, developing students’ thinking skills, allowing them to have creativity, and leading them to access the information on their own and to demonstrate this information. Finally, Çevik (2018) revealed that a learning environment created with STEM-PjBL is vital for solving the complexity of critical concepts in STEM fields. Thus, the role of several factors, such as learning approaches (e.g., PjBL), learning models, and or modifying STEM itself, become critical elements that must be considered when implementing STEM education.

Students’ learning outcomes estimated on STEM enactment

Learning outcomes are the main target in a learning process, including on STEM enactment. Cedefop (2017) argued that students’ learning outcomes are all types of results expected during and after the learning process. Another researcher, Adam (2004), states that learning outcome is a teaching result, which is expected to be obtained by students after a learning process. Further, Adam stated that learning outcomes are usually expressed in the form of knowledge, skills, and or attitude. Slightly different, Gosling and Moon (2002) state that there is no precise way of defining or writing the meaning of such learning outcomes, but a learning outcome must be measurable. It can be concluded that a learning outcome is a result of the learning process. Consequently, learning outcomes can be various forms, depending on the purpose expected by a teacher.

In this study, the estimated learning outcomes after STEM enactments concentrated on academic learning achievement, higher-order thinking skills (HOTS), and motivation. Theodore (1995) defined students’ achievement as a measurable behavior in a standardized series of tests. HOTS is the ability to apply skills, knowledge, and values in reasoning as well as in reflection (Pratama & Retnawati, 2018; Wahono & Chang, 2019a). Indeed, such an ability is crucial to making decisions, solve problems, innovate, and create. In terms of practical application, HOTS includes students’ thinking ranked above level three, according to Bloom’s taxonomy (Baharin, Kamarudin, & Manaf, 2018). Finally, the students’ learning motivation defines as a process where the learners’ attention becomes focused on meeting their educational objectives (Christophel, 1990; Kuo, Tseng, & Yang, 2019). Therefore, the educational and developmental fields give strategic reasons for the focus on these particular skills. For instance, these skills have been related to twenty-first-century skills, future educational attainment, and participation in STEM careers later in life (Martín-Páez et al., 2019; Wahono & Chang, 2019b). Furthermore, HOTS can be used in STEM, and research verifies these abilities in STEM fields can be transferred to other learning fields (Lin, Yu, Hsiao, Chang, & Chien, 2018; Yıldırım & Sidekli, 2018). Moreover, the learning outcomes can be influenced by several external factors, including culture and learner characteristics.

Asian culture and characteristics of teaching and learning

Many factors may influence the effectiveness of learning outcomes in STEM learning. However, Han, Capraro, and Capraro (2015) explained that the two most important factors were the learning environment and the level of individual students. The learning environment can be either a classroom environment or a cultural environment. Based on the literature review, there are many definitions of culture. However, most general definitions include that culture is a combination of many things such as beliefs, values, and assumptions trusted and understood among society (Rossman, Corbett, & Firestone, 1988; Schein, 2010). It is widely accepted that the characteristics of a culture affect individuals’ social behavior (Hampden-Turner & Trompenaars, 1997; Hofstede, 2005). More specifically, when cultural influences are insignificant and less integrated into a learning activity, students will likely experience a misunderstanding that hinders interactions between students and teachers (Popov, Biemans, Brinkman, Kuznetsov, & Mulder, 2013; Popov et al., 2019). Many studies show that culture, ethnics, geographical position, gender, language proficiency, and/or a combination of these components have a significant influence on students’ learning success (Han et al., 2015; Konstantopoulos, 2009; Shores, Shannon, & Smith, 2010). Rodriguez and Bell (2018) mentioned that the instruction in the STEM learning should acknowledge some specific contributions of members from diverse cultures. Thus, culture holds a crucial role in the successful process of student learning in class. Therefore, highly probable that the Asian cultural characteristics and habits have a significant impact on students’ performance and learning outcomes by STEM enactment.

In general, in eastern education, students practice remembering concepts; this philosophy focuses mainly on learning and memorization within the teaching and learning process (Lin, 2006; Thang, 2004). The eastern education system is exam-oriented. Time (duration) is a fundamental factor in teachers’ performance (Tytler, Murcia, Hsiung, & Ramseger, 2017) as they must go over textbooks to prepare students for the final tests. As a result, students tend to memorize the facts in textbooks rather than understanding it due to time constraints. Thus, the situation creates positive competition among students and eventually triggers the efforts of students to obtain and understand the knowledge considered pivotal to achieving a good score in their examination. Eastern-culture education is more generally systematic, with a standardized syllabus and timetable, when compared to western-culture education (Hassan & Jamaludin, 2010; Tytler et al., 2017). However, it is undeniable that this type of character (rote learning, exam-oriented, and curriculum oriented) is one of the reasons many of the Asian countries score inside the top ten, in international tests (Marton et al., 2014; OECD, 2018). Therefore, in the case of STEM enactment, in-depth investigation, whether the time (duration) has a significant impact on the students’ learning outcome is paramount.

Moreover, Asian countries are very different from western countries, especially in their educational philosophy, which tends to be robustly laden with religious and cultural-centric elements (Hassan & Jamaludin, 2010). By contrast, the opinions on such characteristics of the eastern-culture education must be addressed carefully. However, any consequences of those educational characteristics in the implementation of STEM in Asia can be assumed, such as the main target of STEM enactments are not merely to attract student interest in the lesson or higher-order thinking skills, but also more to obtain a higher academic learning achievement. In terms of learning materials and processes, the consequences are seen from many STEM enactments that actively grappled to cultural values, i.e., identify halal products by augmented reality (Majid & Majid, 2018; Mustafa et al., 2016). We firmly believed that such consequences are unique, which led to the potential impact of STEM enactment outcomes in Asia. Therefore, the current research aims to prove that STEM enactments carried out in the past few years have generated a wide range of impacts, especially in Asia.

Method

Research model

This research applied a quantitative approach. A meta-analysis method was used to determine the effectiveness of STEM education for students’ learning outcomes in the Asian region. The meta-analysis method was operative in this study because it enabled an objective investigation of the effect of the independent variable on the dependent variable that is STEM education toward the student’s learning outcome, respectively. Cohen, Manion, and Morrison (2007) state that with a meta-analysis, researchers can evaluate, compare, or combine quantitative data obtained from previous experimental research studies to acquire more convincing and comprehensive results. We identified studies to include in the review, coded for potential moderators, and calculated and analyzed effect sizes.

Selection of studies

The data collection in this study was carried out over 3 months, from February to April 2019. In the screening, several databases, including Scopus, ERIC, ScienceDirect, and Google Scholar, were utilized as the primary search references. We collected the data in the form of journal papers, proceeding conferences, books, or dissertations. Conferences, books, and dissertations were also included as data sources, namely to capture and find what is called the “file drawer” for information, which might not be published in journals (Rosenthal, 1979). Most of the data sources were in English, but there were also some non-English ones. However, from these data sources, at least the title or abstract were in English. The following keywords were at work upon data collection, including the effect of STEM, the effect of STEM learning, the effect of STEM approach, STEM and learning outcomes, STEM and student achievement, STEM and student motivation, and STEM and higher-order thinking skills. When searching, all the keywords used were in English.

A multilevel screening was carried out by applying several criteria, as shown in Fig. 1. The first-level screening of the papers was geared to collecting research papers aimed to examine the effectiveness of STEM education, such as the effectiveness of STEM on academic achievement, motivation, and HOTS. The second screening was based on whether the data was collected from Asian countries or not. The third stage of screening was concerned with whether the study was qualitative, quantitative, or mixed-method research. At this stage, we applied quantitative and mixed-methods research. The last step dealt with whether the paper had the minimum quantitative data required for calculating an effect size, such as mean, standard deviation, variance, number of respondents, the value of t, and the value of F. The results obtained from the first stage were more than 283 papers, while those that satisfied the second-stage criteria were 86 pieces. In the third selection, there were 63 articles. Finally, at the ultimate stage, there were 54 studies (see Supplementary Materials for the list of reviewed articles).

Fig. 1

Process of studies selection

Concerning the quality of studies collected in this review, most of the studies came from research papers published by peer-reviewed journals and conferences. The studies were taken from journal papers (46), conference papers (6), book chapter (1), and a thesis (1). All the studies were carried out in the form of classroom-based research from Asian countries. The total participants involved in this study were 4768 students, or in other words, about 111 students in each study. Those studies included primary school students, secondary school students, or higher-education students. The number of countries involved in this study was ten countries, including Turkey, Israel, Uni Emirate Arab, Taiwan, Korea, China, Hong Kong, Malaysia, Indonesia, and Thailand.

Data coding

Coding in this study was done to make it easier to analyze the obtained data. The coding included several biographical features such as sample size, year of publication, region, topic or subject, education level, and the type of learning outcome. The year of publication in this search ranged from the publications in 2009 to those in 2019. This range allowed for a vast number of studies in the last decade to be investigated. In terms of the region, we divided the Asian region into five regions based on the United Nations. The region included Eastern Asia, Western Asia, Southern Asia, South-Eastern Asia, and Central Asia. The term “subject” here meant a name of discipline or a class where the STEM enactment took place in the data source. In this case, we focused on three groups, particularly science, mathematics, and technology or engineering subjects. For instance, a STEM enactment from Sarican and Akgunduz (2018) has a topic about force and motion, which is a sort of “science” subject source. Furthermore, we divided educational levels into three groups, namely higher education level, secondary education level, and primary education level.

Finally, we divided learning outcomes into three major groups, namely academic learning achievement (ALA), higher-order thinking skills (HOTS), and students’ motivation (Mo). ALA defined as students’ scores, from either the mean of pretest/posttest or only the mean of the posttest score. ALA was tested to get information regarding students’ content knowledge. Meanwhile, HOTS score was collected from HOTS subset codes such as problem-solving, design thinking, creative thinking, reflective thinking, and includes students’ thinking ranked above level three (level 4–level 6) according to Bloom’s taxonomy. The HOTS studies, in general, performed such as a creativity test (fluency, flexibility, originality, and elaboration), a score of analyzing, evaluating, and creating assessment tests. Then, we recognized the Mo score from the domain, namely student motivation or student interest. In general, students’ motivation was measured in the studies through a questionnaire, including intrinsic motivation, self-determination, self-efficacy, and grade motivation.

In doing so, a description of the measure or process on those variables (ALA, HOTS, Mo) in this current study are discussed. Inevitably, each outcome was measured differently among the studies reviewed. For instance, a HOTS study reported scores of students’ problem-solving abilities, whereas another study of HOTS reported a set score of students’ creative thinking, and even a study of HOTS had reported an effect size of what the article authors called “HOTS scores before and after an intervention.” To deal with this concern, we performed some technical works. For example, initially, as a primary resource, we collected all the existing effect size scores of ALA, HOTS, and Mo studies. In the situation where we could not directly find the effect size scores of the selected studies, we would collect other supporting data. We required the supporting data for calculating the effect size, namely standard deviation, mean score, number of respondents, the value of t, and the value of F. Finally, we computed and standardized the collected data by statistical software (see data analysis).

To address the third research question in this study, we coded three moderator variables that could contribute to the STEM enactment effectiveness, namely, approach or learning model, learning orientation, and duration of instruction. The coding was distilled from the theoretical review framework in the introduction part. For instance, several studies revealed that some learning approaches or learning models are combined and or juxtaposed with the STEM enactment (Chung, Lin, & Lou, 2018; Lou, Tsai, Tseng, & Shih, 2014). Likewise, the duration of instruction is a fundamental factor in teachers’ performance in Asia (Tytler, Murcia, Hsiung, & Ramseger, 2017). Eastern-culture education is more generally systematic, with a standardized syllabus and timetable, when compared to western-culture education (Hassan & Jamaludin, 2010; Tytler et al., 2017). Moreover, Asian countries tend to be robustly laden with religious and cultural-centric elements (Hassan & Jamaludin, 2010).

In terms of the approach or learning model, the authors coded each study, whether it was accompanied by another approach/learning model (present) or only STEM lesson without clearly the presence of other approaches (absent). The authors have coded learning orientation into two types, namely culture centric and universal oriented. The culture centric refers to the study, which much follows the unique characteristics of Asian students, such as strongly curriculum oriented, more systematic with standardized syllabus and timetable, or tends to be robustly laden with religious and local cultural elements. The universal oriented study refers to a freer lesson, the selected studies because the curriculum was not as strict, and or the themes on STEM lesson did not much emphasize unique themes, in particular, Asian countries. Finally, the authors coded the duration of instruction as a short or long period. The long duration refers to STEM enactment that was conducted by more than two-time class periods, and the short was conducted by only one-time class periods (2 h or less).

Publication bias

Another thing that needed to be clarified was how the researchers coded whether a study investigated the STEM enactment or not. In this case, the researchers referred to several works (Bybee, 2013; Li, Wang, Xiao, & Froyd, 2020; Martín-Páez et al., 2019). The researchers point out that there is not a fixed consensus in the literature about under what condition(s) learning was said to be STEM learning. However, in general, they (Bybee & Martin-Paez et al.) say that STEM learning emphasizes problem-solving with real-world problems involving many disciplines and other skills such as science, technology, mathematics, and engineering in integrated ways. Furthermore, this study focused on articles related to such STEM definitions, and/or at least, the authors in the paper mentioned that they used the STEM education approach (an integrated STEM). Moreover, we selected publications from 2009 to 2019, meaning that a vast number of STEM enactments by this time were included in the intended definition.

Concerning publication bias, we have met some difficulties in obtaining unpublished papers, especially in the research area of STEM enactment in Asia, in terms of its impact on learning outcomes. In terms of an alpha level significance (0.05), this current study shows, specifically, that more than 14% of the reported effects were not/less significant. These findings are consistent with the varieties in perspectives concerning the inferiority, superiority, or equivalence of STEM enactment for various learning styles. The condition that only 14% of the study was not a significant effect is not because of the file drawer studies remain unpublished due to the magnitude, significance, or direction of their effects, but rather because of other factors such as written in local language as well as the quality of the studies (McElhaney, Chang, Chiu, & Linn, 2015).

Data analysis

The data collected from various references, such as journals, books, proceedings, and dissertations investigating the effect of STEM enactment, were then analyzed using the meta-analysis method. Data were all aimed at accessing the same target, namely students’ learning outcomes (academic learning achievement, motivation, and higher-order thinking skills). The multitude of data was examined using the meta-analysis method for systematic and beneficial analysis. We argued that making quantitative data comparisons of various studies as one of the challenging and vital jobs in the world of research today.

A summary effect size (E.S.) using a random effect model value was the dependent variable in this study, while the independent variable was the STEM enactment in diversified ways and types. A random effect model assumes that the true E.S. varies from one study to the next, and the summary effect is our estimate of the mean of these effects (Pigott, 2012). Therefore, in this study, we do not want that overall estimate to be overly influenced by any of them. Meanwhile, in terms of potential moderator variables, a mixed-effect model was used. The mixed-effect model allows us to get a trade-off from the true E.S. In the moderator variable case, the trade-off from the true E.S. is vital due to the comparison between two sub-variables (e.g., short and long of the instruction duration). In doing so, the investigations of effect size and visualization were carried out using the Jeffreys’s amazing statistics program (JASP) version 0.11.1 program, especially by the Hunter-Schmidt method. This method was used due to the ability to estimate the variability of the distribution of effect sizes through a two-step process, namely subtracts to yield a residual variance and boosts by a function of the reliability and range restriction distributions (Hunter & Schmidt, 2004). To deal with the effect sizes for some studies reporting only F or t values, or even reported Hedges g, the authors used algebraic techniques (Lipsey & Wilson, 2001) as well. In social science, a common practice for overcoming this task is to calculate Cohen’s coefficient (Cohen, 2013). In this study, Cohen’s theory was determined by the difference between the average control group and the experimental group (see Eq. 1) or the difference between the average posttest score and the pretest score (Eq. 2) (Howell, 2016).

$$ d=\frac{{\overline{x}}_1-{\overline{x}}_2}{s_{pooled}} Sd=\sqrt{\frac{n_1+{n}_2}{n_1{n}_2}+\frac{d^2}{2\left({n}_1+{n}_2\right)}} $$

(1)

$$ d=\frac{{\overline{x}}_{\mathrm{post}}-{\overline{x}}_{\mathrm{pre}}}{s_p}=\frac{{\overline{x}}_{\mathrm{post}}-{\overline{x}}_{\mathrm{pre}}}{s_{diff}/\sqrt{2\left(1-r\right)}}\ {s}_d=\sqrt{\frac{1}{n}+\frac{d^2}{2n}\times 2\ \left(1-r\right)} $$

(2)

Let$\overline{x}$_i, S_i, and n_i be the sample mean, standard deviation, and size of the group I, while S_pooled, S_diff, r, and S_d be the pooled standard deviation, the differences of standard deviation between pre and post, the correlation between pre- and post-treatment score, and standard deviation of Cohen’s d.

When the calculated magnitude effect size was large, a classification was deployed in this meta-analysis method. In the current study, the authors used the classification level of (Sawilowsky, 2009). This classification system was a revised version of Cohen’s work in 1988. Thus, when the effect size was less than 0.20, it was considered very small, while when it ranged from 0.20 to 0.49, it was classified as small. The effect size, which ranged from 0.49 to 0.79, was at a medium level. A large level was evident from 0.80 to 1.19. Between 1.20 and 1.99 was classified at a very large level. A value over 2.0 was regarded to have a huge effect. A d coefficient of one indicates that the difference between two means is equal to the standard deviation (S.D.). If Cohen’s d is larger than one, the difference between two means is larger than one S.D. Anything larger than two means that the difference is larger than two standard deviations. This calculation afforded a uniform scale in expressing all possibilities that show a relationship between variables. Regarding the variability observed in this study, we have standardized the magnitudes between the differences in interventions and outcomes measured. The results of the study were summarized and combined systematically using a commonly termed the standardized effect size, namely the standardized difference in means.

Results

The main objective of this study was to investigate whether STEM education originating and developing from the western countries (the USA) also affected students learning outcomes in the Asian environment. Another aim was to investigate whether there is a specific factor that contributes to the effectiveness of STEM enactment. Finally, another aim was to know more about the development and the enactment of STEM education in Asian countries. As a result, in terms of effect size, this current study found varies or heterogeneity. The value ranged from negative (− 0.19; 95% CI = − 0.78 to 0.40) to positive effect (+ 2.81; 95% CI = 2.01 to 3.61) (see Supplementary Materials for the list of effect sizes, study features, and coding elements).

The general portrait of study

Based on the literature reviewed, the first publications to assess the effect of STEM education on the learning outcome in Asia began in 2013. This time was only 4 years after the advent of STEM by the US government in 2009. Nevertheless, the authors assume that STEM education studies in Asia began to gain traction long before 2013. However, many of those studies were qualitative research, or the studies were not directly related to students’ learning outcomes. Table 1 illustrates the descriptive analysis of STEM educations in Asia, especially those related to the students’ learning outcomes.

Table 1 Statistical analysis of the study characteristics

In this study, we found that three Asian regions substantially contributed to the implementation and development of STEM education. Table 1 also shows that the Asian countries have conducted most studies on STEM education and its impact on students’ learning outcomes, with East Asia being the biggest contributor (25 studies), followed by West Asia (16 studies) and Southeast Asia (13 studies). However, there were significant differences in results between the three regions (Q_.B. = 4.208, p < .05). Furthermore, the difference evinces that STEM education is significantly effective in Southeast Asia, as evidenced by its impact on the learning outcome, greater than that in other regions (E.S. = 1.211). This value is a combination of the value of academic learning achievement, higher-order thinking skills, and motivation.

In terms of the subject or topic guiding the implementation of STEM education in Asia, Science is the most widely researched. Conversely, mathematics is the least popular topic. However, there was no significant difference (Q_.B. = 0.638, p > .05) when the effect of STEM education on the learning outcome related to topic or subject matter was investigated. Also, related to the level of education, this study found that the level of secondary education (junior and senior high school) has been widely researched (28 studies). In contrast, the higher education level (college or university level) is the least researched area (10 studies). At the same time, the statistical analysis also showed no significant difference (Q_.B. = 2.880, p > .05), the effect of STEM enactment on learning outcomes in terms of education levels. Nevertheless, this difference suggests that STEM education tends to influence at secondary-level education (E.S. = 1.009) compared to the other two levels (primary and higher education level).

The effect of STEM enactment on students’ learning outcomes

In terms of student learning outcome, in line with the second research question, the investigated focused on academic learning achievement, higher-order thinking skills, and motivation. Furthermore, based on the analysis results, the summary effect of the overall effect size is 0.69 [0.58, 0.81 of 95% CI]. According to Sawilowsky (2009), this value is classified as a medium level of effect. Detailed results between the three types of learning outcomes (learning achievement, higher-order thinking skills, and motivation) can be seen in Figs. 2, 3, and 4.

Fig. 2

A forest plot of students’ academic learning achievement (ALA)

Fig. 3

A forest plot of higher-order thinking skills (HOTS)

Fig. 4

A forest plot of students’ motivation (Mo)

Academic learning achievement

This study assumes that academic learning achievement is crucial in Asian students, even for the students’ parents. The rationale of this statement is related to the culture and characteristics of education, which is embraced in Asian countries (Hassan & Jamaludin, 2010; Tytler et al., 2017). Thus, one of the objectives of this study was to determine whether the implementation of STEM enactment in Asian countries affected the students’ academic learning achievement. In this study, we analyzed academic learning achievements from 24 studies that met the criteria (see the criteria on the “Selection of studies” section). The results of the analysis and distribution are shown in Fig. 2. Figure 2 below is a forest plot of students’ academic learning achievement.

The forest plot shows black squares and whisker lines (see Fig. 2). The black squares indicate the magnitude of the STEM effect on academic learning achievement, whereas the whisker lines indicate the upper and lower limit of the value of the confidence interval. The vertical dashed line is a line that shows the position of the effect size with a zero value. Thus, the right area of the line is positive values, whereas the left area of the line shows a negative value of effect sizes.

In Fig. 2, there are 20 studies where the Cohen value of d is below 1.0, while the other four studies have an effect size of more than 1.0. In addition, it is also known that a study seems a different appearance from the others, namely a study from Han, Rosli, Capraro, and Capraro, (2016) with Cohen’s values d 0.28 [0.16, 0.40 of 95% CI]. The black squares with short whisker lines indicate that the study has a very small range of the confidence interval. The minimum value of the confidence interval was due to the huge sample size in the study. Overall, the effect of STEM enactment for students’ academic learning achievement was 0.64 [0.48, 0.79 of 95% CI]. This positive d value indicates that STEM education affects students’ academic learning achievement in Asia. In classifying effect size, the value of .64 belongs to the medium effect category.

Higher-order thinking Skills

The second objective of this research is to find out more about whether STEM education affects students’ higher-order thinking skills (HOTS). To address this question, Fig. 3 below is a forest plot from Cohen d analysis about 16 previous studies that helped provide sufficient details.

Figure 3 illustrates the spread of effect size from 16 studies on students’ higher-order thinking skills (HOTS). The analysis results of the forest plot illustrate ample information. One interesting insight is the summary effect of 1.02 [0.71, 1.32 of 95% CI]. According to Sawilowsky (2009), this value is classified as a large effect. However, the largest d value in the study is reaching 2.81 [2.01, 3.61]. The value of d (2.81) means that the effect size value is twice the standard deviation value, while the smallest d value is at .06 [− 0.45, 0.57]. At a glance, there is a considerable difference between the largest values, the data distribution pattern, and the summary effect. This state is due to a study, which is Han et al. (2016) study reports the highest magnitude. The highest magnitude occurred because the study includes the largest sample size (1187 people). A large sample size certainly affects the result of the summary effect.

Motivation

Another goal to be achieved in this study is to find out whether STEM education is effective in increasing student motivation in Asia. Figure 4 below illustrates the details of the data distribution from 14 previous researchers. The studies measure student motivation distributed across many topics, including science, mathematics, technology, and engineering.

The illustration of Fig. 4, designated by the forest plot, are normally distributed (p > .05). However, Cohen’s d value is spread from the smallest (− 0.08) to the largest d value (1.58). Furthermore, the figure indicates the summary effect value is 0.49 [0.32, 0.65 of 95% CI]. The summary effect value of .49 in the Sawilowsky classification is categorized as a medium effect. Therefore, the STEM enactment is Asia has a great impact on students’ motivation as well as two others (academic learning achievement and higher-order thinking skills).

Moderator variable of STEM enactment’s learning outcomes effectiveness

In addition to knowing the extent to which STEM enactment in Asia affects the students’ learning outcome that includes academic learning achievement, higher-order thinking skills, and motivation, this study also answers whether there are specific factors behind that effectiveness. In particular, this section addresses the research question about under what conditions and for what learning outcomes are STEM activities more effective in Asian students. Several potential variable moderators, such as approach or learning model, research design, learning orientation, and duration of instruction, were analyzed to address the research question.

As shown in Table 2, several moderator variables reveal identical results in terms of student academic learning achievement. STEM enactment has a significant effect on the approach or learning model variable (p = .037). The presence of an approach or learning model contributes better to the effectiveness of STEM enactment. Other moderator variables that also show significant results are learning orientation (p = .039). STEM enactment, which tends to be culturally centric, gives a different effect compared to what is only universal oriented. Also, the last moderator variable that addresses significant results is the duration of instruction (p = .016). In this variable, a longer time provides better effectiveness in terms of student academic learning achievement.

Table 2 Moderator variable analysis of academic learning achievement

Heterogeneous results in higher-order thinking skills, especially in terms of the potential moderator variable, are shown in Table 3. The factor, the duration of instruction, shows a significant result (p = .046). Furthermore, the variable duration of instruction shows that time (long duration) has a crucial role in increasing the higher-order thinking skills of students in STEM enactment. Unlike the case for the duration of instruction, the other two factors (approach or learning model and learning orientation) do not address any significant differences (p > .05). This condition proves that whether STEM is carried out, with or without another approach or learning model, and whether learning orientation tends to be cultural centric or universal oriented, the higher-order thinking skills of students have relatively the same effectiveness.

Table 3 Moderator variable analysis of higher-order thinking skills

The results that are quite different concerning the potential moderator variables affecting the effectiveness of STEM enactment are shown in Table 4. In Table 4, the table shows that no moderator variables have the potential to differ rather significantly in the motivation of students in Asia. The three moderator variables, namely approach or learning model, learning orientation, and duration of instruction, show identical results that there is no significant difference (p > .05). These results mean that whether STEM enactment is accompanied or not by other learning approaches, cultural centric or universal oriented, or done with short or long periods, the effect on students’ motivation tends to be the same.

Table 4 Moderator variable analysis of motivation

Discussion

The overview of STEM enactment in Asia

As a portrait of STEM enactment in Asia, this current study tends to focus on the three variables, namely region, subject, and education level. We found that Eastern Asia was the most contributed to STEM researches, especially those related to the impact on student learning outcomes. On the other hand, the difference evinces that STEM education is significantly effective in Southeast Asia, as evidenced by its impact on the learning outcome higher than that in other regions. The different effects among regions are mostly due to an interaction of some factors, such as the differences regarding the number of published studies and the differences in students’ learning outcomes baseline (Saraç, 2018; Yildirim, 2016). For instance, the result showed that students’ motivation and HOTS were proven higher than students’ academic learning achievement, which is mostly found in the studies on Southeast Asia (Lestari, Astuti, & Darsono 2018; Lestari, Sarwi, & Sumarti, 2018; Ismayani, 2016; Soros, Ponkham, & Ekkapim, 2018; Surya, Abdurrahman, & Wahyudi, 2018; Tungsombatsanti, Ponkham, & Somtoa, 2018). The baseline of Southeast Asia learning outcome is lower than in other regions due to the low quality of educational practice (OECD, 2018). Thus, this study suggests that those students with a lower baseline of higher-order thinking skills will benefit the most from the STEM enactments. In terms of education level, the result showed that most studies were conducted at the secondary education level. The condition of most studies conducted in STEM education from the secondary education level is in line with the resulting study from Saraç (2018). The only difference from Sarac’s study is that the reviewed subjects came from all over the world and did not focus distinctively on the Asian region. However, in terms of effect size, there was no significant effect appearing in this variable.

Furthermore, STEM education applications on mathematical topics or subjects are small in the number when compared to topics or subjects of science and engineering. This case is in line with the results of research from Saraç (2018). Sarac has found that the application of STEM education related to the learning outcome is still very limited in mathematics-related topics. The situation reflects that STEM education research on the other focuses, such as students’ attitudes (besides focusing on the learning outcome), is also lacking. This condition is because quite challenging to associate mathematics-related topics and STEM education. Wahono and Chang (2019a) revealed that, when utilizing the STEM education approach, teachers felt challenged in connecting subject matter topics. The characteristic of mathematics, which is fundamentally theoretical and abstract (Acar, Tertemiz, & Tasdemir, 2018; Sabag & Trotskovsky, 2013), represents a stark contrast to the characteristics of STEM education, which involves activity that is more physical. Thus, it represents a critical reason why STEM enactment of the mathematical topic has a small number. However, there is still a tremendous opportunity to apply STEM education to mathematical-related topics. Examining students’ learning outcomes through particular STEM activities in mathematics is one of the worth for next future research. As evidenced in this study, we found only eight studies in Asia related to mathematics and learning outcomes.

Impacts of STEM enactment on Asian students’ learning outcomes

The results of the meta-analysis in this study suggest that the outline of STEM education of students’ learning outcomes in Asian countries differs among variables. The results showed the effect of STEM enactment by order; those are effect sizes on students’ HOTS at a large level (1.02), meanwhile the academic learning achievement and motivation at a moderate level (0.64 and 0.49). This result is advantageous because HOTS generated more of an effect in Asia when compared to students’ academic learning achievement. As Martín-Páez et al. (2019) and Chang, Ku, Yu, Wu, and Kuo (2015) stated that, in general, STEM education has the potential to increase students’ interest and higher-order thinking skills. The more substantial effect of students’ HOTS and interest could be due to the nature of the learning tools and processes of STEM education, which are based on eastern cultures and emphasize hands-on activities (Hassan & Jamaludin, 2010). The characteristics of STEM education (real-world problem and problem-solving) represent excellent potential for increasing students’ HOTS. Higher-order thinking skills such as problem-solving, critical thinking, and creative thinking are the leading targets in STEM learning in Asia (Barak & Assal, 2018; Lee et al., 2019). Therefore, HOTS is a decisive asset for Asian students in coping with global competition and industrial revolution 4.0.

Moreover, the result of academic learning achievement showed that the highest value of effect size (1.86) is in the Majid and Majid (2018) study. Based on an advanced analysis (a sample case), the study indicated that the researchers deeply embraced the Asian cultural characteristics of education. The study was devoted to several learning topics, particularly about chemical properties, atomic theory, and periodic tables. This Majid and Majid study also provides an example of the application of augmented reality, which is a topic familiar to students in their daily life, namely, to identify halal products. The result showed that the highest effect size value of students’ motivation is in the study of Ugras (2018). Based on further analysis, this study indicated that the learning process was influenced by the habits that are commonly faced in that particular place (Turkey/Asia). Most of the themes carried out in the learning process using STEM, such as how to build a strong house to withstand an earthquake or other often-encountered themes from daily life by Asian students. Furthermore, the themes or topics (culture and real-world problems) are the central themes in STEM learning. Such learning conditions certainly could encourage students’ enthusiasm and motivation in learning.

Moreover, a large variation has found naturally in the effect size of the Asian student learning outcomes. This condition is logically influenced by several factors such as learning instruction quality (McElhaney et al., 2015) and how effective the learning instruction, in this case, STEM enactment, fits into the Asian culture and characteristics (Hassan & Jamaludin, 2010). Indeed, a fit and comfortable the instruction to the learner characteristics (i.e., much grappled to cultural values) has strongly supported gaining a better impact on the STEM enactment outcomes. Furthermore, this moderate effect indicates that STEM education is quite promising to prepare students to face unpredictable global competition in the future. However, of course, there are still numerous efforts required to maximize the impact of implementing STEM education in the Asian region, including trying to find the hidden factor behinds the effectiveness of STEM enactment in terms of students’ learning outcomes.

Potential factors contributing to STEM enactment

Therefore, another exciting result to discuss is the role of the moderator variables on the effectiveness of student learning outcomes. Based on the analysis of the academic learning achievement of learning outcomes, better results would be obtained if the STEM enactment is accompanied by an approach, learning model, or other methods. This result is in line with the research from Lee, Capraro, and Bicer (2019). They (Lee et al.) investigated the role of companion another approach or learning model, in increasing the effectiveness of STEM lessons in the classroom. Lee et al. found that STEM combined with another approach or method (e.g., project-based learning or 6E learning model) would be more effective when compared to STEM lessons without other combinations.

Furthermore, the integration of STEM enactment with another approach or learning model provides better direction and control in the achievement of learning objectives (Mustafa et al., 2016). Besides, the results of the present study also show that STEM enactment, which tends to be culture centric, was more effective than universal oriented. This result is probably because culture-centric learning is more in line with most of the characteristics of Asian students who tend to rote learning, curriculum orientation and exam orientation (Di, 2017; Hassan & Jamaludin, 2010; Lin, 2006; Thang, 2004; Tytler et al., 2017). Therefore, the characteristics are more helpful in terms of increasing students’ academic learning achievement. In addition, the duration of the instruction factor also shows one of the potential factors in influencing the student’s effectiveness in academic learning achievement. Longer times of STEM enactment show to be more effective than shorter times; this result makes sense because, with sufficient time, students could better absorb and gradually improve their academic learning achievement (Çevik, 2018; Sarican & Akgunduz, 2018).

On the other hand, different conditions were found at higher-order thinking skills and motivation for learning outcomes. The results of both learning outcomes show that only the duration of instruction is significant, especially at the higher-order thinking of learning outcomes. This result means that a long time has the potential to be more effective in increasing higher-order thinking skills for Asian students. Lestari et al. (2018) and Lin, Hsiao, Chang, Chien, and Wu (2018) stated that time played a vital role in honing students’ higher-order thinking skills such as problem-solving and creative thinking of a STEM education field. However, the duration of the instruction factor is not significantly different from the motivation of learning outcomes. Whether STEM enactment is done in a short or over a long period, student motivation is equally effective. The same conditions are shown in other factors such as approach or learning model and learning orientation. Furthermore, this condition indicates that whether there are other approaches involved in STEM enactment, and whether it is culture centric or universal oriented, STEM enactment will provide relatively the equivalent effectiveness, especially in higher-order thinking skills and student motivation. That is, higher-order thinking skills and motivation are very closely tied to its STEM enactment, not from the supporting factors. This reason is reinforced by the opinion of Chiang and Lee (2016) and Ugras (2018), which states that STEM lessons have a robust character to increase learning motivation and higher-order thinking skills of students.

Conclusion and practical implications

The results of this study indicate a propitious effect of implementing STEM education on students’ learning outcomes in Asia. The effect is evident in the students’ learning achievement, higher-order thinking skills, and motivation. We have also concluded that STEM education in Asia leads to a higher effect on students’ higher-order thinking skills, students’ learning achievement, and finally, motivation. Furthermore, STEM education constitutes the most promising teaching and learning innovation, especially to prepare students honing higher-order thinking skills as well as to attract students’ interest in learning, which is crucial in adapting to the competitive era.

Likewise, based on the results of this study, when implementing STEM teaching and learning within a classroom, several factors must be considered; first, teachers may combine STEM lessons with any teaching approach or learning model. For instance, the teachers can combine STEM teaching with the 6E learning model or project-based learning approach. The combination would give a strong direction for a teacher in realizing the lesson goal. Another suggestion is to involve the local culture in STEM lessons. Such involvement is crucial to academic performance and essential to culturally responsive pedagogy. Local culture can be in the form of the main lesson topics, enrichment material, the way of teaching and learning process, or even the use of localized languages and properties. Lastly, when applying STEM lessons, calculating the amount of time needed, then utilizing a sufficient amount of time toward application is fundamental. The study suggests more than 2 h, spread over two or more class periods, will assist students’ academic learning achievement and higher-order thinking skills. Indeed, these three factors are significant in maximizing STEM effectiveness in Asian student learning outcomes.

While the authors strongly recommend educators, and researchers, apply STEM education as a regular part of learning in Asian countries, a concern is that this study only involves 54 selected studies. We believe there are still other studies that are also related to STEM education and the effectiveness of students’ learning outcomes that were not identified. These limitations can be caused by several things, such as the language used in the title and abstracts written in languages other than English. Another limitation is that this study is more focused on the meta-analysis method that evaluates quantitative research, so we cannot ascertain whether the learning outcome obtained so far has anything to do with teacher attitudes and knowledge of STEM education or not. Also, concerning to calculation of effect size on the potential moderator variables, this current research is still a limited number of studies. A power analysis indicated that the sample size showed relatively weak results to obtain significant and substantial effects for the targeted variables. A larger number of studies are needed to verify result analysis as well as to continue future research. Nevertheless, we believe this research is a comprehensive, valid, and reliable starting point in providing up-to-date information about the conditions of STEM enactment in Asia.

Potential future research based on the results, discussion, and limitations of this study includes investigating Asian teachers’ perceptions (based on their philosophy and belief) and current knowledge concerning STEM education as well as how to apply the approach in different fields. This study serves as an inspiration for researchers to develop or modify STEM lessons, originating from western countries, into diversified STEM types and variances that comply with the cultural background and geographical conditions of each country. Moreover, an attempt to develop, implement, or modify STEM-related curriculum is also a promising future research opportunity.