The role of structured inquiry, open inquiry, and epistemological beliefs in developing secondary students’ scientific and mathematical literacies

Abstract

Background

The recommendation from national documents and reports to promote inquiry-related science activities has not been supported by recent studies, which have found the overall frequency of inquiry activities to be negatively associated with student learning outcomes. This study was inspired by such conflicting reports and aimed to clarify the associations of science-specific, inquiry-related activities and epistemological beliefs with students’ mathematical and scientific literacies.

Results

A secondary analysis of the database from the Programme for International Student Assessment 2015 of Australia (N₁ = 14,530) and Taiwan (N₂ = 7708) utilizing structural equation modelling revealed that these two countries exhibited similar data patterns. Results suggested that open-inquiry activities (such as debating and planning experiments) had a negative relationship with secondary students’ mathematical and scientific literacies. Structured inquiry learning (such as students explaining their ideas and teacher explaining how an idea can be applied to different phenomena) and epistemological beliefs about science were significant and positive predictors of student mathematical and scientific literacy performance.

Conclusions

The current study further highlights and provides empirical evidence that the teacher’s role in structured inquiry (especially pertaining to the relevance and applicability of these ideas) appears to be essential to the development of student literacy. Educational implications and recommendations are discussed.

Introduction

Research studies and policy documents have emphasized the importance of promoting inquiry learning so that students can practice problem-solving competency and skills for their future lives rather than receiving scientific knowledge directly from science teachers (Jerrim et al., 2019; National Research Council USA [NRC], 2012). It has been assumed that inquiry involves students in formulating a hypothesis and trying to answer a research question with data and evidence (Bell et al., 2005). Consequently, increasing students’ conceptual understanding and application of science knowledge (Lin et al., 2011) has potential to promote students’ interest in, beliefs about, and attitudes toward learning science while simultaneously decreasing their anxiety (Jack et al., 2014). Inquiry-related learning activities have been officially recommended as national science education standards in which science teachers are encouraged to plan their own inquiry-based teaching programme for students (NRC, 1996, 2000) and as an approach to mathematics education. The Australian mathematics curriculum rationale encourages teachers to support students in becoming active learners with inquiry (Australian Curriculum Assessment & Reporting Authority, n.d. para.4). However, mathematical inquiry has received less research attention than scientific inquiry and its definition demonstrates little consensus. This is problematic as research findings are quite disparate due to the breadth of inquiry as a pedagogy, with notions of inquiry ranging from open exploration to teacher-guided learning (Bereiter & Scardamalia, 1996). The inconsistency regarding the relationship between inquiry-related learning and student achievement among the science and mathematics literature reveals the need to take a close look at which specific inquiry-related learning activities (e.g., When learning school science, students are allowed to design their own experiments) are positively or negatively related to the student achievement outcomes of mathematical and scientific literacies defined by OECD (2016b). The following definitions reveal that scientific inquiry is essential for scientific literacy.

“Scientific literacy is the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to: • Explain phenomena scientifically—recognise, offer and evaluate explanations for a range of natural and technological phenomena. • Evaluate and design scientific enquiry—describe and appraise scientific investigations and propose ways of addressing questions scientifically. • Interpret data and evidence scientifically—analyse and evaluate data, claims and arguments in a variety of representations and draw appropriate scientific conclusions.” (p. 50)

“Mathematical literacy is students’ capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals in recognising the role that mathematics plays in the world and to make the well-founded judgements and decisions needed by constructive, engaged and reflective citizens.” (p. 28)

In addition to inquiry-related activities, epistemological beliefs have been identified as a significant determinant of student learning outcomes (Khaleghinezhad et al., 2012). The Programme for International Student Assessment (PISA) 2015 assessment framework included epistemological beliefs in the student questionnaire for the first time in assessing student scientific literacy (Organisation for Economic Co-operation & Development [OECD], 2017a). Recently, an initial finding from analysis of PISA 2015 national data for Taiwan revealed that epistemological belief about science is the most significant predictor of student scientific literacy performance (She et al., 2019). The current study extends this research agenda and adds to the literature by evaluating an international model of the relationship between students’ specific inquiry-related learning activities as well as epistemological beliefs and their learning outcomes of mathematical literacy and scientific literacy. Taiwan and Australia are chosen for comparison because of their different cultural traditions, language backgrounds, and learning environments (Wallace & Chou, 2001). In addition, Taiwanese secondary students have reported fewer opportunities for experiencing inquiry activities than Australian secondary students, but have shown higher scientific literacy based on PISA 2015 survey results (OECD, 2016a, 2016b). The data analyses and comparisons between Australia and Taiwan, two countries with distinct teaching practices, cultural traditions, and learning environments, have the potential to provide a robust explanation for differences in students’ scientific literacy across the two jurisdictions. The results of comparisons allow for enhanced understanding of how different learning activities or personal epistemological beliefs are beneficial to or supportive of secondary student-specific literacy.

This study intends to explore the following two research questions:

• 1. How do the perceived inquiry-related learning activities and epistemological beliefs vary between Australian and Taiwanese students?

• 2. How do correlational models of Australian and Taiwanese students’ specific inquiry-related learning activities and epistemological beliefs and their mathematical and scientific literacies compare?

Through research question 1, we understand the current situation of the inquiry-related learning activities and epistemological beliefs of Australian and Taiwanese students. Research question 2 allows us to understand how these two factors relate to mathematical and scientific literacies.

Review of literature

An inspection of several international reforms in K–12 science and mathematics curricula along with reviews of literature on science and mathematics education have illustrated the critical issues mentioned in the introduction. The following literature review addresses the classification and effectiveness of inquiry in science and mathematics education and the role played by students’ epistemological beliefs about science.

The classification of scientific inquiry and science learning

The NRC in the USA (2000) claimed that scientific inquiry is a necessary component of learning for the development of students’ scientific literacy. Despite the lack of consensus regarding how inquiry should be taught, considerable communality has developed among science educators and researchers regarding classification systems pertaining to scientific inquiry. For example, in the 1980s and 1990s, the level of openness of inquiry was classified into the following categories: 0 = research question, experimental procedures, and even expected results are provided to students; 1 = only research question and experimental procedures are provided; and 2 = only the research question is provided (e.g., Friedler & Tamir, 1984; Staer et al., 1998). Blanchard et al. (2010) define four types of inquiry including verification, structured, guided, and open inquiry. To date, it still is debatable regarding which type of inquiry is most appropriate for high school students.

The essential features and understanding of inquiry-based activities have not been widely implemented in secondary classroom teaching (Capps & Crawford, 2013). The major reasons why science teachers have not implemented inquiry-based teaching strategy in their classroom teaching range from a lack of teaching materials and professional development to a perception that not all of their students have sufficient background knowledge or ability to meaningfully engage in inquiry activities (Capps et al., 2016). Overcoming these challenges and dilemmas will require the identification of specific inquiry-related learning activities that are beneficial to student literacy and can be applied within the limitations of resources, time, and learning experience that exist in current classrooms (Chinn & Malhotra, 2002).

Moreover, inquiry has been recommended as one of the scientific and engineering practices (e.g., designing and implementing investigations, engaging in modelling, analysing and interpreting data, practising argumentation) involved in solving problems (Cairns & Areepattamannil, 2019). A comprehensive literature review of inquiry yielded a more fine-grained description of inquiry, dividing it into phases of orientation, conceptualization, investigation, conclusion, and discussion (Pedaste et al., 2015).

Research findings regarding inquiry-related learning activities in science learning

Research studies investigating the effectiveness of inquiry-based science teaching include quasi-experimental studies (e.g., Marshall et al., 2017), meta-analyses (e.g., Schroeder et al., 2007), and large-scale data analyses (e.g., Cairns & Areepattamannil, 2019). A majority of the quasi-experimental studies have shown that inquiry-related learning activities are significantly related to learning outcomes of conceptual understanding and scientific practices (e.g., Chen et al., 2014; Lin et al., 2011; Marshall et al., 2017). Furthermore, the positive effect of inquiry-related learning activities also has been documented by meta-analytic results (Furtak et al., 2012; Schroeder et al., 2007). These studies found effect sizes of inquiry-based teaching on student learning outcomes ranging from 0.31 to 0.65. On the other hand, when the pedagogical approach of inquiry is considered, Kirschner et al. (2006) and Mayer (2004) warned that an unguided approach normally does not work and may even have a negative effect on student learning outcomes. In contrast to the argument of Kirschner et al. (2006), Hmelo-Silver et al. (2007) posited that some effective approaches of inquiry learning are not minimally guided but rather provide scaffolding to reduce cognitive load in students. Similarly, Warfa et al. (2018) confirmed that group dialogue is one of the important links in inquiry-related teaching. It not only engages students in the multiple ways of representing chemical knowledge, but also exposes their views and facilitates participation in classroom discourse. The more critical questions for researchers to ask should concern the specific kinds of scaffolding activities that are most effective or that should be provided.

Most of the above studies examining the effect of inquiry-based science teaching were based on small sample sizes or country-specific data that might restrict the generalizability of findings (e.g., Tuan et al., 2005; Wang, 2020; Yang et al., 2019; Yang, Lee, et al., 2016; Yang, Lin, et al., 2016; Yang, Lin, et al., 2016; Yang, Lin, et al., 2016). Fortunately, large-scale international student assessments have provided a new lens for comparisons of educational systems across countries regarding effective teaching practices (Cairns & Areepattamannil, 2019). However, the results of large-scale assessments have shown inconsistent conclusions. Findings from the 2015 Trends in International Mathematics and Science Study (TIMSS) have indicated that inquiry-based instruction is positively correlated with student learning outcomes (Teig, 2021; Teig et al., 2018). However, a secondary analysis of PISA 2015 data from Taiwan (Liou, 2021) and 72 countries revealed a negative association between inquiry-based teaching and students’ scientific literacy (Gómez & Suárez, 2020). Similar results were found through analysis of PISA 2006 data, confirming that science students who report engaging in low levels of inquiry activities show above-average levels of science literacy (McConney et al., 2014). Another study based on PISA 2006 data asserted that guided inquiry learning was a significant predictor of students’ achievement. On the other hand, open-inquiry learning activities were significantly negatively related to students’ achievement (Kang & Keinonen, 2018). More recently, with the use of PISA 2015 data sets from six countries, Oliver et al. (2021) found a complex pattern of associations between students’ scientific literacy and component aspects of inquiry. They also reported a consistent, strong, and negative association between inquiry (inquiry-based science teaching, IBTEACH) and students’ scientific literacy across six Anglophone countries. Readers are reminded that the inconsistent conclusions from secondary analyses of large-scale assessment data could be due to the fact that these studies are measuring different learning outcomes. With the consideration of instructional quality and different types of inquiry activity, Kang (2020) found that the teacher–student relationship was the most potent predictor of scientific literacy and moderated the effects of guided and open inquiry on scientific literacy. Along similar lines, Cairns and Areepattamannil (2021) tested the contribution of each item to students’ science achievement scores and suggested that instructional activity with explicit, teacher-directed practices demonstrated a significant, positive association with science achievement. In this study, we hypothesized that the examination of inquiry-related activities in the PISA 2015 student questionnaire may provide additional insights to reconcile the inconsistent arguments in existing literature about the effectiveness of inquiry learning, specifically on student performance pertaining to scientific and mathematical literacies.

Research findings of inquiry-related mathematics learning

The results of studies on the effectiveness of inquiry-related mathematics learning activities on student learning outcomes are mixed, ranging from significant positive effects to no effects. Some empirical studies have shown that, compared to non-inquiry-related intervention, inquiry-related learning is a more equitable form of instruction and leads to greater affective and cognitive gains among students. For example, after conducting IBL of mathematics in 100 course sections, Laursen et al. (2014) found that, when compared to their non-IBL peers, IBL students earned as good or better mathematics outcomes, such as “cognitive gains in understanding and thinking; affective gains in confidence, persistence, and positive attitude about mathematics; and collaborative gains in working with others, seeking help, and appreciating different perspectives” (p. 409). However in contrast to these positive learning outcomes, de Jong et al. (2010) concluded that students who experienced traditional classroom teaching outperformed the inquiry class on procedural knowledge test items despite the fact that students in the inquiry class were supported by the implementation of computer-based simulations. Similarly, So (2013) examined 26 outstanding open-inquiry science project reports pertaining to how students identify and use mathematical procedures in their scientific inquiries. Her findings indicated that most of the project reports identified appropriate application of measuring volume, weight, temperature, voltage, and concentration but there was room for improvement with mathematical procedural knowledge that requires higher-order thinking skills such as analysing numerical data and interpreting graphs and tables. The learning outcomes reported in the above studies differ across the studies (i.e. do not focus solely on mathematical competencies) and were conducted in different countries. This inconsistency, which is similar to that found in the review of inquiry for science learning, justifies the consideration of the associations of inquiry-related learning activities with science and mathematics learning achievement through analysis of large-scale datasets and the use of international cross-cultural comparisons.

Epistemological beliefs in learning science and mathematics

Epistemological beliefs are referred to as beliefs about the nature of knowledge and ways of knowing. The PISA 2015 assessment framework (OECD, 2017a) posits that epistemological beliefs include science as an evolving subject and explains how individuals justify knowledge. For example, on one hand, a student with a naïve epistemological belief of knowledge likely will believe that knowledge is something absolute and unchanging while, on the other hand, a student with sophisticated epistemological beliefs will believe that knowledge is constantly evolving (Cano, 2005). Research studies have documented that students’ epistemological beliefs are associated with their values of scientific inquiry (Fleener, 1996; Hofer & Pintrich, 2002; Peffer & Ramezani, 2019). Theoretically, Schommer (1993) asserted that epistemological beliefs have a significant effect on learning outcomes, and that those students who believe that knowledge is not absolute or unambiguous will achieve better academic achievement. Furthermore, research studies have found significant relationships between learners’ epistemological beliefs and their learning approach and achievement (Cano, 2005).

Despite the wealth of case studies that have described the relationship between epistemological beliefs and learning achievement (Acar, 2019; Cano, 2005), explorations of the relationship in large-scale international comparisons are relatively limited. Initial findings of a significant relationship between students’ epistemological belief about science and their scientific literacy performance (She et al., 2019) inspire us to extend the investigation of the correlation model for cross-cultural comparisons. We believe that the related findings can enable science educators to better understand the role and implications of epistemic belief in learning science. In addition to the aforementioned inquiry-related learning activities, we investigated the possible synergistic predictive effect of epistemological beliefs on literacy in science and mathematics.

Inquiry-related learning activities and epistemological beliefs

A few studies have explored the relationship between the inquiry activities experienced by students in the science classroom and their epistemological beliefs, but these studies have not yielded consistent results. Tsai (1999) analysed 25 eighth graders in Taiwan and found that students with constructivist epistemological views perceived actual laboratory learning environments as less open-ended and less integrated, and tended to explore deeply the involved concepts of laboratory activities, resulting in a richer understanding. The students with empiricist epistemological views paid more attention to “doing” experiments in accordance with science textbooks. However, another study of 68 fifth graders conducted by Wu and Wu (2011) found that most students’ epistemological beliefs were still naïve after engaging in 5 weeks of inquiry activities. Therefore, the relevance of inquiry-related learning to epistemological beliefs is not yet clearly established.

Hypothetical model to be tested

Drawing from the review of literature and theoretical foundations (e.g., Schommer, 1993) discussed earlier, the purpose of this study is to examine the relationships among participants’ inquiry-related learning activities, epistemological beliefs, and scientific and mathematical competencies (Fig. 1). We focus on students’ competencies as defined by PISA 2015. This model hypothesized that students’ experience with inquiry-related science learning activities and epistemological beliefs have predictive effects on their scientific and mathematical competencies. It will serve as the central framework for the data analysis and interpretation.

Fig. 1

Theoretical model of inquiry-related learning activities, epistemological beliefs, scientific and mathematical competencies. The dashed line indicates that there may be a potential correlation between the two variables

Methods

Secondary analyses of the PISA 2015 data, an international assessment conducted by the OECD aiming to measure 15-year-old students’ mathematics and science literacy, were conducted in this study. The importance of choosing a participant age of 15 years is due to the fact that, in most countries, participants are approaching the end of their compulsory education, and the literacies assessed thus would reflect at least partially their school learning outcomes and capability of applying these literacies in their future non-school environments (OECD, 2009).

Data source and participants

This current study used the PISA 2015 data from Australia and Taiwan. These two countries were intentionally chosen because their students’ performance on scientific literacy, mathematical literacy, and epistemological beliefs were each significantly above the OECD average (OECD, 2017a); however, the two countries’ students perform quite differently. Australia’s average science literacy score was 510, while Taiwan's average score was 532—about one-half years’ worth of schooling ahead of Australian students. According to the OECD (2017a) report, Australian science teachers report using “inquiry-based” pedagogy more frequently than their colleagues in Taiwan. Each of the two countries also exhibits distinct learning environments (Wallace & Chou, 2001), cultural traditions, and languages. In addition, there are distinctive differences in value perception between students of the two countries. Kee (2007) suggested that the values of Taiwanese students tend to be Confucianism-based, and focus more on self-discipline, while Australian students in a Western educational context place more value on self-benefit and personal interest. The population sizes of 15-year-old students in Australia and Taiwan in 2015 were very similar, at 282,547 and 287,783, respectively. The 2015 PISA data included 14,530 Australian students and 7708 Taiwanese students.

Variables

This study involved four primary constructs: inquiry-related learning activities, epistemological beliefs, scientific literacy, and mathematical literacy. The first two constructs were measured by the student questionnaire, while the literacies were assessed by various PISA test items. For scientific literacy, the major learning outcome domain assessed in PISA 2015 included three sub-competencies: explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically (OECD, 2017a). In addition, the assessed mathematical literacies in PISA 2015 were: formulating situations mathematically; employing mathematical concepts, facts, procedures, and reasoning; and interpreting, applying and evaluating mathematical outcomes (OECD, 2017a).

Inquiry-related learning activities

PISA 2015 required participating students to complete a background questionnaire that included questions relating specifically to their science classes; it used nine items to evaluate students’ self-report of inquiry-related learning activities (OECD, 2017a). Sample items included “students debate about investigations”, and “students explain their ideas”. Participants were asked to rate the frequency of each instructional practice by selecting one of four ordinal response options (from 1 = never or hardly ever to 4 = in all lessons).

The review of literature, along with the results of previous study (She et al., 2019), revealed that the items (not only the inquiry-related learning activities, but also the epistemological beliefs) could be further grouped and classified into sub-scales. For the purpose of classification, exploratory factor analysis (EFA) was conducted on the PISA 2015 Australia and Taiwan data to discern latent constructs among item responses. We computed the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy and conducted Bartlett’s test of sphericity (Tabachnick & Fidell, 2001). Results showed a high level of sampling adequacy (KMO = 0.880 for Australia and KMO = 0.911 for Taiwan) and significant sphericity [χ²(36) = 33,277.26, p < 0.001 for Australia and χ²(36) = 36,774.22, p < 0.001 for Taiwan], which showed that the data were suitable for factor analyses. Principal axis factoring with varimax rotation was used to assess the construct validity. The scree plot suggested that inquiry-related learning activities were represented by two latent constructs. Thus, constructs were labelled open inquiry (reflecting class activities that occur largely without explicit direction or overview of the teachers) with six items (i.e. When learning science there is a class debate about investigation; Students are allowed to design their own experiments; Students are required to argue about science questions; Students are asked to do an investigation to test ideas; Students spend time in the laboratory doing practical experiments; and Students are asked to draw conclusions from an experiment they have conducted), and structured inquiry (reflecting class activities that involve direction or overview of the teacher) with three items (i.e. When learning science the teacher explains science idea can be applied; The teacher clearly explains relevant science concepts to our lives; and Students are given opportunities to explain their ideas). After checking the content of the nine items assessing inquiry-related science activities and comparing the different classifications of inquiry by existing literature, we found that the two constructs resulting from the EFA aligned neatly with the classification of teacher-directed structured inquiry and student-directed open inquiry proposed. Factor loadings for the items associated with each of these two constructs in the Australia and Taiwan datasets ranged from 0.443 to 0.816 and from 0.466 to 0.781 (Table 1), respectively. The internal consistency reliability coefficients (Cronbach’s ɑ) for open inquiry were 0.821 (Australia) and 0.914 (Taiwan), while reliability coefficients for structured inquiry were 0.741 (Australia) and 0.761 (Taiwan). The total variance explained was 47.36% and 61.28% for the Australian and Taiwanese data, respectively.

Table 1 Factor loadings for inquiry-related learning activities based on exploratory factor analyses of Australian and Taiwanese samples

Epistemological beliefs

Over the last 50 years, research on epistemological cognition has experienced "three waves of scholarship”, which focus on different traditions, nomenclature and measurement methods. In the second wave, the focus is on students’ belief, it relies on the use of self-reports that are mainly Likert-type scales (Hofer, 2016; Lunn Brownlee et al., 2020). The survey design on epistemological beliefs of PISA 2015 also followed this direction.

The PISA 2015 student questionnaire used six items to evaluate students’ epistemological beliefs (OECD, 2017a). Participants were asked to indicate how much they agree with each statement using 4-point Likert response options (1 = strongly disagree to 4 = strongly agree), where higher values represent a higher level of epistemological beliefs.

EFA using principal axis factoring with varimax rotation was used to assess factor structure. The scree plot revealed that the six items reflected a single latent construct (Epistemology). Factor loadings for this latent construct in the Australia and Taiwan datasets ranged from 0.669 to 0.830 and from 0.677 to 0.881, respectively. The internal consistency coefficients (Cronbach’s ɑ) for the latent construct in the Australia and Taiwan data were 0.900 (Australia) and 0.934 (Taiwan). The total variance explained was 60.31% (Australia) and 71.09% (Taiwan) and internal consistency reliability coefficients (Cronbach’s ɑ) were 0.900 and 0.934, respectively. The above results reveal satisfactory reliability and construct validity evidence for scores from the epistemological belief instrument (DeVellis, 2011).

Scientific literacy

The forms of the PISA 2015 cognitive test of scientific literacy consisted of 184 items, and these 184 items were allocated into 12 clusters. Each participant was required to respond to two randomly selected clusters, with each cluster designed to occupy 30 min of testing time (OECD, 2017b). Because the content and number of test items attempted by each student were not the same, the test developers estimated ten plausible values (PVs) for each student that estimated the student’s science literacy. A set of PVs was developed based on item response theory and was randomly derived from the distribution of ability estimates, which represent the range of reasonable values for student literacy, and these plausible values are necessary to employ in analyses of PISA data involving scientific literacy (Mislevy et al., 1992; OECD, 2017b).

Mathematical literacy

PISA 2015 (OECD, 2017a) defined mathematical literacy as an individual’s capacity to formulate, employ, and interpret mathematics in a variety of contexts. It includes the competencies of reasoning mathematically and using mathematical concepts and procedures to explain and predict phenomena. The PISA 2015 cognitive test of mathematical literacy had 69 items. The representation of mathematical literacy also is based on PVs estimated by the test developers (OECD, 2017b).

Analysis

The first step in the analysis (previously described) used EFA on the PISA 2015 Australian (N = 14,530) and Taiwan (N = 7708) datasets to identify the dimensions in the Inquiry-related Learning Activities and Epistemological Beliefs scales. For the purpose of confirming the obtained dimensions and assessing construct validity of the questionnaire, this EFA then was coupled with Rasch analysis (Bond & Fox, 2007). Therefore, the Rasch Model anchored at the rating scale model was used to describe the status of students’ epistemological beliefs and their experiences with inquiry-related learning (Traub & Lam, 1985). The Rasch analysis also was used to validate the ranking within the Inquiry-related Learning Activities and Epistemological Beliefs scales according to their difficulty. In Rasch analysis, excluding missing data, the sample sizes for Australia and Taiwan were 10,663 and 6865, respectively. Then, we used structural equation modelling (SEM) to assess the posited causal relationships among variables. The model used three exogenous latent predictors—open inquiry, structured inquiry, and epistemology—and two endogenous observed outcomes—mathematical literacy and scientific literacy. The measurement model, including assessment of measurement invariance, first was assessed, followed by assessment of the structural model. Also, because each of the two outcomes (mathematical and scientific literacy) consisted of 10 sets of PVs, analyses were carried out separately with each set of PVs and the resulting test statistics averaged across these analyses, as recommended by OECD (2009). Additionally, as recommended by OECD and other authors (e.g., McNeish et al., 2017; Muthén & Satorra, 1995), standard errors of parameter estimates were adjusted for the school-level cluster effects as well as stratum effects using cluster-robust estimation, which in the present study was implemented using the TYPE = COMPLEX estimation procedure in Mplus. In all analyses, the supplied sampling weights were incorporated. Because the indicators of each factor were 4-category, ordinal measures, CAT-LS (i.e. WLSMV) estimation was used, as recommended by Rhemtulla et al. (2012).

Results

The Wright maps were used to present how Australian and Taiwanese students show their frequency of experience (inquiry-related learning activities) or attitude (epistemological beliefs). In addition, a multi-group structural model was fitted to assess relationships among Australian and Taiwanese students’ specific inquiry-related learning activities and epistemological beliefs and their mathematical and scientific literacies.

RQ1. How do the perceived inquiry-related learning activities and epistemological beliefs vary between Australian and Taiwanese students?

The Rasch rating scale model (Andrich, 1978) is appropriate for polytomous or ordered data resulting from the Likert response format employed in this study, moreover, Wright maps present the distribution of students’ frequency of experience inquiry-related learning activities and item difficulty on the same logit scale. In the case of non-achievement tests, item difficulty refers to how much of the latent trait the individual must possess before they positively endorse an item. It also means that easier items are common student behaviours/awareness that were easy to endorse, while more difficult items were less common behaviours/awareness that were difficult for students to endorse (Liu, 2010; Zile-Tamsen, 2017). Students are situated on the left side of the vertical dashed line, according to their frequency of experience (inquiry-related learning activities) or attitude (epistemological beliefs), from low to high or from disapproval to approval (bottom to top). The items are situated on the right, arranged by their difficulty. Items at the bottom indicate that they are easily approved by the participants. The number to the left of the decimal point represents the item number, and the number to the right of the decimal point represents the options for the item (1 = disagree, 2 = agree, 3 = strongly agree).

The Wright map of the inquiry-related learning activities (Fig. 2) reveals reasonably good targeting of the items on the sampled students as the cluster of items was mostly covered by the spread of persons. The results exhibited a normal distribution, with measures as wide as 5.813 logits (from − 3.171 to 2.642 logits) in the Australian dataset and 6.282 logits (from − 4.219 to 2.063 logits) in the Taiwanese dataset. Students closer to the top of the figure are those with higher frequency of experience in inquiry-related learning activities than those who are near the bottom. The results indicate that the Taiwanese students showed lower frequency of experiencing inquiry-related learning activity (M = − 1.268) than the Australian students (M = − 0.248).

Fig. 2

Wright map for inquiry-related learning activities

Figure 3 is the Wright map of the epistemological beliefs. The results exhibited a negatively skewed distribution, with measures as wide as 10.094 logits (from − 4.380 to 5.714 logits) in the Australian dataset and 9.913 logits (from − 4.714 to 5.199 logits) in the Taiwanese dataset. Students closer to the top of the figure represent those with greater approval of epistemological beliefs than those who are near the bottom. The results indicate that the Taiwanese students experienced greater agreement with the epistemological beliefs (M = 2.442; i.e. having more sophisticated views or better understanding about the nature of science) than the Australian students (M = 2.022).

Fig. 3

Wright map for epistemological beliefs

RQ2. How do correlational models of Australian and Taiwanese students’ specific inquiry-related learning activities and epistemological beliefs and their mathematical and scientific literacies compare?

Multi-group latent variable regression was used to answer research question 2. A two-step procedure was employed whereby the multi-group configural measurement model and measurement invariance first were assessed, followed by the structural model. The first step—assessment of the adequacy of the configural measurement model and measurement invariance of the three latent constructs across the two countries—was carried out by fitting and comparing successively constrained sets of models in which (1) a multi-group configural model (M0) first was fitted, followed by models with (2) factor loadings between groups constrained to be equal (i.e. metric invariance, model M1), and (3) factor loadings and intercepts constrained to be equal between countries (i.e. scalar invariance, model M2). Results (Table 2) showed good fit for each of the models, with evidence of scalar measurement invariance as evidenced ΔCFI ≤ 0.01 and ΔRMSEA ≤ 0.015 (Chen, 2007; Cheung & Rensvold, 2002). Each of the latent constructs—open inquiry, structured inquiry, and epistemology—additionally showed good evidence of factor reliability with, respectively, omega (McDonald, 1999) indices of 0.84, 0.76, and 0.90 (Australia) and 0.92, 0.79, and 0.95 (Taiwan). Average variance extracted (AVE) values ranged from 0.52 to 0.83, again reflecting good evidence of reliability.

Table 2 Results from invariance analysis of measurement model for latent constructs

Figure 4 and Table 3 demonstrate parameter estimates resulting from the multi-group structural model fitted to Taiwanese and Australian data. This model showed good fit to the data, based on Hu and Bentler (1999) guidelines (CFI ≥ 0.95, NNFI ≥ 0.95, RMSEA ≤ 0.06, SRMR ≤ 0.08), with the following observed fit statistics: ${\chi }_{avg}^{2}(261)$ = 8366.66, p < 0.001; CFI_avg = 0.973, NNFI_avg = 972, RMSEA_avg = 0.053, and SRMR_avg = 0.041, where statistics were averaged across analyses carried out with each set of PVs. Table 3 presents the estimated regression parameters for this model. As can be seen, in both Australia and Taiwan, each of the latent variables was a statistically significant predictor of each outcome (mathematical literacy and scientific literacy). Structured inquiry and epistemology were significant positive predictors, while open inquiry was a significant negative predictor. Additionally, statistically significant between-country differences in these relationship were observed. Specifically, when compared to the Australian sample, the Taiwanese sample showed 1) significantly stronger negative associations of open inquiry with mathematical and scientific literacy, and significantly stronger positive associations of structured inquiry with these outcomes. In contrast, significantly stronger associations of epistemology with these outcomes were observed in the Australian sample when compared to the Taiwanese sample. Additionally, when mean levels of the inquiry-related activities were compared for the two countries, Taiwanese students showed significantly lower levels (p < 0.001) of the latent open inquiry and structured inquiry constructs than their Australian counterparts, but showed significantly higher levels of epistemology (p < 0.001). These latter results are consistent with results observed in the Rasch analyses.

Fig. 4

Multi-group structural model for open inquiry, structured inquiry, and epistemological beliefs in mathematical and scientific literacy by country. ***p < 0.001

Table 3 Results of multi-group structural model analysis

Discussion

Using a large-scale, international dataset to facilitate a cross-cultural comparison, our findings make two unique contributions to advance the understanding of how inquiry-related activities are related to student scientific and mathematical literacies. First, the results of structural equation modelling revealed that Taiwan and Australia exhibited similar data patterns suggesting that structured inquiry learning and epistemological beliefs about science were significant and positive predictors of student mathematical and scientific literacy performance. Second, the current study further highlights and provides empirical evidence supporting the teacher’s role in the teaching of structured inquiry.

Structured inquiry activities are positively associated with learning outcome

One noteworthy finding of this study is the identification of factors pertaining to inquiry-related learning activities as measured in PISA 2015 having significant positive and negative relationships with student scientific and mathematical literacies. Previous studies have examined the effects of inquiry-related learning activities on student learning outcomes and found that the frequency of inquiry-based teaching practice has a negative association with student scientific literacy (Cairns & Areepattamannil, 2019; Jerrim et al., 2019). More recently, the disaggregation of PISA’s composite variable representing inquiry-related learning in PISA 2015 into single-item measures revealed that teacher-directive approaches in inquiry teaching have a positive association with student learning achievement (Cairns & Areepattamannil, 2021; Oliver et al., 2021). Thus, it is possible that some of the discrepancies between these findings and earlier findings may be explained by differences in how inquiry-related learning is operationalized—as a single construct, or as more fine-grained components. However, it should be noted that some researchers raise reliability concerns when using responses from single items as measures of constructs (e.g., Sarstedt & Wilczynski, 2009), and such single-item measures lack a measurement model or any defensible method of assessing measurement invariance between or among groups.

To examine the predictive effects of distinct types of learning activities, the current study classified the nine learning activity items in the PISA student questionnaire into two constructs; namely, open inquiry and structured inquiry. Multi-group latent regression analyses (Table 2) revealed that, in both Australia and Taiwan, structured inquiry and epistemology had significant positive associations with 15-year-old students’ mathematical and scientific literacy in both countries, while open inquiry had significant negative predictive effects on each outcome. Although previous meta-analysis of case studies has indicated the importance of teacher guidance for student engagement in inquiry-related activities (e.g., Furtak et al., 2012), we believe that large-scale and cross-cultural international study can provide additional evidence and further clarify what specific combinations of activities are beneficial to student learning outcomes. In contrast to Cairns and Areepattamannil’s (2019) PISA analysis using each of nine separate inquiry-related items as a predictor, the multi-group structural model of this study operationalized inquiry-based activity as two latent variables, where one of these latent variables (open inquiry) was measured by six inquiry teaching activities and was negatively associated with student scientific literacy, while the second latent variable (structured inquiry) was measured by three items and was positively predictive of student scientific literacy. These results serve to remind science educators and teachers that cautions should be taken when implementing the six open inquiry teaching activities for beginning or novice science learners. In addition, the positive relationship of structured inquiry activities with not only scientific literacy, but also mathematical literacy, is a finding that rarely has been explored previously. Specific activities related to structured inquiry (e.g., students being given opportunities to explain their ideas, or teachers explaining how ideas can be applied) are beneficial to student learning outcome performance. Such activities, which can be characterized as activities that are directed by the teacher and serve to consolidate and summarize open-inquiry activities, are an integral part of scientific inquiry (Yang et al., 2019).

Instructions incorporating plans for student/teacher explanation play a significant role

Hmelo-Silver et al. (2007) posited the importance of providing opportunities for students to elaborate their explanations. The PISA 2015 report (OECD, 2016a, 2016b) presented the individual relationships of the nine inquiry-related items with student performance. In contrast to item-level analyses, this study examined the nine items as two constructs—namely, student-directed open-inquiry and teacher-directed structured inquiry—using data from two high-achieving countries (Australia and Taiwan). The current study further highlights and provides empirical evidence that the teacher’s role in structured inquiry (especially pertaining to the relevance and applicability of these ideas) appears to be essential to the development of student literacy. As can be seen from the three structured inquiry activities (i.e. The teacher explains school science, how ideas can be applied; The teacher clearly explains relevance of concepts to our lives; and Students are given opportunities to explain their ideas), teachers play the significant role of structuring learning activities that provide opportunities for meaning-making through student–teacher interactions. Jayawardena et al. (2020) also confirmed this positive relationship with student science achievement and recommended that teachers encourage students to think about their previous knowledge and experience with regard to the new topic and ask them to conduct the experiment based on their prior knowledge.

In summary, before making a conclusion about the effects of inquiry-related learning activities, the quality of integrating inquiry-related learning, teacher/student explanation, and students’ level of cognitive engagement should be considered. Additionally, the reasons why the learning activities of open inquiry that have been recommended by the existing literature unexpectedly resulted in negative relationships with the student learning outcomes deserves further examination including, for example, the quality and depth of inquiry activities and teachers’ knowledge structure of scientific inquiry (Bartos & Lederman, 2014). In addition, readers are reminded that the negative predictive effect of open inquiry does not mean this kind of teaching strategy necessarily should be avoided. Given that the assessment of learning outcomes in this study was focused on the learning outcome of literacy defined by OECD’s framework, open inquiry may have potential benefits for specific groups of learners or on other kinds of learning outcomes (e.g., problem-solving skills or certain types of cognitive thinking skills).

As national curricular framework or standards (NRC, 1996) and international large-scale assessments (OECD, 2016a) consistently continue to emphasize the importance of inquiry-related learning activities, the identification and clarification of specific types of inquiry-related learning activities that are positively or negatively related with student scientific and mathematical literacies can provide empirical evidence to identify which learning activities are optimal, so as to provide reference for front line teachers. However, teachers are cautioned that this does not necessarily imply that those activities with negative relationships with science and/or mathematical literacies should be avoided. Open-inquiry activities should be regarded as a longer-term goal of science education that can be achieved gradually. Given that some inquiry activities (e.g., forming researchable questions and designing experimental procedures) are challenging even for undergraduate students (Cheng et al., 2021), our findings from the current study do suggest that for beginning learners of scientific inquiry, more structured explanations and guidance from teachers are strongly encouraged. It is clear that additional experimental-control studies (Wang, 2020) that more deeply examine factors resulting in negative effects on literacy are necessary. Furthermore, the negative correlational evidence between inquiry activities and student science achievement documented in the aforementioned literature should be interpreted with caution as correlations do not necessarily imply cause–effect relationships. It does suggest, however, that arguments against using inquiry activities in science classroom practices should not be disregarded entirely. It also suggests that inquiry-related activities that intend to promote conceptual understanding and application, higher-order thinking skills, and minds-on deep learning should not be equated with activities that only require students to manipulate objects instead of thinking about ideas (Hofstein & Lunetta, 2004). In addition, efficient use of time, appropriate and sufficient laboratory materials, and quality teacher preparation each are essential elements of effective instruction (OECD, 2016a).

Consistent predictive patterns but with different predictive values for different country-level learning contexts

The PISA 2015 report indicates that both Australia and Taiwan have high-performing education systems in science-related domains including above-average science performance and stronger-than-average epistemological beliefs. The results of multi-group SEM analyses revealed that structured inquiry and epistemological beliefs were positively and significantly correlated with scientific and mathematical literacies. Conversely, the SEM results indicated that open inquiry was negatively and significantly associated with scientific and mathematical literacies. These relationships were consistent for both Australia and Taiwan, but both the positive and negative associations of the inquiry activities with these outcomes were significantly and markedly stronger in Taiwan than in Australia. That is, when compared to Australian students, structured inquiry was more strongly positively associated with both mathematical and science literacy among Taiwanese students, while open inquiry was more strongly and negatively associated with each outcome. This suggests that, in countries such as Taiwan, extra care must be taken when planning and implementing inquiry-related learning activities, due to potential negative associations with student literacy. In particular, it is critical to ensure that sufficient levels of structured inquiry activities be included to facilitate inquiry. It is not sufficient, and indeed appears to be detrimental to student learning, to design learning activities that are too unstructured, and that do not incorporate plans for teacher explanation and opportunities for student explanation. Secondary school science teachers are encouraged to focus more on incorporating these directed teaching practices into inquiry-based learning activities discussed in this study.

The crux of this study is that it involves country-level distinctions in the relationships of inquiry-based activities to literacy outcomes. The findings that structured inquiry activities are supportive to student learning outcomes of scientific and mathematical literacy in Taiwan and Australia do not mean differences in culture and learning environment do not matter as much as the nature of the task with which students are asked to engage. We also provide empirical evidence demonstrating that the same type of inquiry-related activities applied in different cultural contexts has different results on learning outcomes. Reflecting on results of this study and our own literature review of studies relevant to student learning in the two countries (e.g., Kee, 2007; Wallace & Chou, 2001), we tentatively conjecture that in addition to the factors investigated in this study, cultural context and learning environment should also emerge as significant predictors of learning outcomes. Future studies are encouraged to explore how structured inquiry best can be implemented in different cultural contexts. In addition, given that many of the references regarding inquiry teaching are from the United States, an investigation how inquiry is implemented in US schools and comparison of its data with the findings of this study is strongly encouraged as a follow-up study.

Limitations and implications of the study

There are several limitations for this study. Firstly, readers are reminded that students participating in PISA are asked about the frequency with which they experience various pedagogical strategies. The variables being tested here reflect not only the type of inquiry-related activities, but very much the frequency with which they are applied in different jurisdictions. Given that the PISA 2015 student questionnaire primarily was focused on student science learning experiences, this study solely investigated the potential for science inquiry experiences (not mathematical inquiry experiences) as they relate to scientific and mathematical literacies. Readers are reminded that mathematical inquiry experiences could also predict students’ mathematical literacy. Secondly, readers are reminded that the PISA 2015 data used in this study did not assess the quality of teachers’ inquiry teaching practices or the level of students’ cognitive engagement in the nine learning activities; therefore, the potential effect of the quality teachers’ translation of inquiry-related teaching practices and cognitive level of students’ learning engagement on learning outcomes deserves elaborative investigation in future research studies. Such research can serve to more precisely target the specific features of inquiry-based learning that can enhance student development. Thirdly, despite the fact that PISA assessments have been carried out with standard procedures and quality assurance, students’ self-report of classroom learning activities still may not reflect actual classroom practices. Finally, the data informing this study are specific to 15-year-old students. In future studies, it would be illuminating to investigate more deeply how the age of the participants and the curricula within each country may contribute to the constructs and relationships examined in this study.

Conclusion

Overall, the results of this current international, cross-cultural comparison study identify and confirm that structured inquiry and epistemological beliefs were positively and significantly correlated with student scientific and mathematical literacies. The current study further highlights and provides empirical evidence that the teacher’s role in structured inquiry activities (e.g., teacher explains school science, how ideas can be applied) and promoting student epistemological beliefs appear to be essential to the development of student scientific and mathematical literacy. Our results suggest the importance of two strategies and their implications in science and mathematics education. First, as far as lesson planning is concerned, teachers are encouraged to consider moving beyond superficial inquiry activities requiring students’ physical engagement to activities that engage students in the appropriate level of cognitive practices (Hofstein & Lunetta, 2004). In view of the finding that the inquiry-related learning activities of structured inquiry and epistemology were significantly and positively associated with scientific and mathematical literacies, teachers are encouraged to design teaching materials and procedures in ways that match the inquiry level of student engagement and honour science epistemologies (Maeng et al., 2020); thereby guiding students toward deeper approaches of inquiry-related learning activities. Second, enhancing the complexity of students’ epistemological beliefs about science is a proactive strategy for improving science and mathematics academic achievement. In order to bring about improvement in student epistemological beliefs, it is not sufficient to merely tell students what they should believe. The use of contextualized historical cases or integrating the nature of science into teaching (Lin & Chen, 2002) can focus and enhance student understanding of how previous scientists explored ideas, planned investigations, inferred findings, discussed and debated with colleagues, and finally constructed consensus and understanding of specific phenomenon.