Do inquiry-based teaching and school climate influence science achievement and critical thinking? Evidence from PISA 2015

Abstract

Background

Inquiry-based teaching (IBT) and improving school climate are the two most commonly used strategies for fostering learning and critical thinking skills in students. IBT has positioned itself as one of the most recommended “active” methodologies for developing intellectual autonomy and complex thought processes, whereas school climate is considered to be a protective factor that helps mitigate adverse conditions and has a positive impact on teaching and learning. This study investigates the relationship between IBT and school climate with the academic performance of Colombian students in the PISA 2015 test.

Results

Using a two-level Hierarchical Linear Modeling (HLM), the results show a negative association between IBT and students’ scientific performance in the test. However, results show a positive association between IBT and students’ self-reported critical thinking skills. Results also show that school climate is a positive moderating factor on learning. Contextual factors such as the student’s grade level at the time of sitting for the PISA test, gender, and socioeconomic characteristics are strong predictors of science achievement in PISA.

Conclusions

These results reveal the need for additional research on the effect of the so-called active methodologies and school context on student learning. Regarding IBT, it seems that its influence is greater on student’s perception and motivation, than in acquisition of scientific knowledge. Positive school climate is a protective factor that can help to improve student learning. In Colombia, the quality of students’ educational outcomes has largely been attributed to their socioeconomic background. However, the results of this study show that, although students’ and schools’ socioeconomic characteristics continue to be strong predictors of academic performance, teaching practices can have an even larger impact.

Background

Inquiry-based teaching (IBT) is currently one of the most recommended and commonly used science teaching strategies (Duran 2016; Cueto 2015; Comisión Europea 2018). The main premise of IBT is that teachers must be facilitators of a student-centered learning (Freeman et al. 2014) and should play the role of colleagues and guides rather than mere transmitters of subject knowledge. Some literature mentions that IBT contributes to the development of intellectual autonomy in students (Abdal-Haqq 1998), and it has established itself as one of the most used teaching approaches due to its attributed potential to develop critical thinking skills, active learning, and information processing and synthesis skills (Hu et al. 2008; Minner et al. 2010).

There are reports in the literature that IBT-based educational interventions have resulted in gains in student achievement, scientific skills, comprehension of scientific concepts and jargon (Lloyd and Contreras 1987; Harlen 2013), critical thinking (Narode et al. 1987) performance on standardized tests (Gottfredson et al. 2005), and logical-mathematical thinking (Spronken-Smith and Walker 2010; Laursen et al. 2011). For instance, some authors report that IBT strategies focused on student-led experiments, laboratory practice, and interpretation of data and graphs yield better student outcomes compared with those exposed to traditional teaching methods such as lectures or direct instruction (Mattheis and Nakayama 1998). Another study mentions that teaching practices that incorporate research activities and evaluation of scientific concepts have a greater effect on students’ performance in chemistry rather than other factors such as teaching experience or the academic context (Roehrig and Garrow 2007).

Although those studies report a positive effect of IBT on learning (Kazempour 2013; Brew 2003), more robust studies have found contrary evidence. For instance, a study conducted in Qatar suggests that a student’s exposure to active methodologies that include practical or experimental work may be generally interesting and motivating, but less effective than expected in terms of academic achievement (Areepattamannil 2012). Another comparative study conducted in Canada, Australia, and New Zealand reports that students who were exposed to IBT activities in their classes had below-average levels of scientific literacy, even though they showed greater interest in science learning and greater scientific commitment (McConney et al. 2014). In this same line, a preliminary analysis of data from the 2015 version of the Program for International Student Assessment (PISA)—released soon after the administration of the test—already hinted at a negative association between IBT and scientific literacy (OECD 2016b). These contradictory scenarios reflect the need to carry out more research to determine the actual effect of the so-called active pedagogies or active methodologies, such as IBT, on student learning and achievement.

Another factor that occupies a prominent place in the literature of factors associated with student learning is school climate. Recent studies indicate that a favorable school climate can have a positive impact, among others, on students’ learning (Stewart 2007), health, and well-being (Thapa et al. 2013). Furthermore, a good school climate acts as a protective factor that can buffer against the effects of poverty or adverse socioeconomic conditions on academic achievement (Hopson and Lee 2011). In PISA (OECD 2016b), school climate comprises several dimensions that may influence student achievement: student absenteeism, disciplinary climate, teacher and student behaviors that hinder learning, and support of teachers to students (Fig. 1).

Fig. 1

Measures of school learning environment in PISA 2015 (OECD 2016b)

School climate has been defined as a measure of the frequency of personal conflicts and treatment among people (Coyne 2012), and reflects the standards, goals, values, interpersonal relationships, training, learning, leadership practices, and organizational structures that are part of an educational organization (Ramelow et al. 2015). A positive school climate encourages the development of students and the necessary learning for a more productive, satisfying, and participatory life in a democratic society (National School Climate Council 2010). Thus, interest in the effect of school climate on learning continues to grow, as it is considered a strong moderator of student learning. Despite its importance, we could not find empirical research for the Colombian context on the influence of school climate in the teaching and learning processes, specifically its effect on science teaching and learning.

However, the wealth of data that come along with the participation of Colombia in PISA offers us an opportunity to address questions and hypotheses related to educational practices and policies and factors that affect the academic performance of students. In addition to availability of data on students’ reading, math, and science skills (OECD 2016a), PISA provides robust data, among others, on indicators of IBT, critical thinking, and school climate.

Our research seeks to contribute to the literature on factors associated with student’s learning, particularly the effects of IBT and school climate on scientific performance. The questions that guide this study are as follows: What is the effect of IBT on scientific literacy (as measured by test scores in PISA 2015)? What is the effect of IBT on students’ critical thinking skills? To what extent does the school climate act as a moderator of the relationship between IBT, scientific performance, and the development of critical thinking skills?

Methods

Data

The primary sources of data for this study are the student, teacher, and school questionnaires of the PISA 2015 study. This database contains approximately 540,000 records of students from 72 participating countries. The Colombian sample includes 11,795 students. The data were obtained from the website of the program (OECD 2015).

Conceptual and analytical framework

In this study, Alexander Astin’s Input-Environment-Output (I-E-O) model (Astin and Lising 2012) is used to examine the relationship among students’ individual characteristics (input), school experiences (environment), and scientific performance and critical thinking skills (output). The main purpose of the I-E-O model is to measure the impact of the environment (or school-level variables) by controlling for individual characteristics such as gender, socioeconomic status, or family characteristics (Fig. 2).

Fig. 2

Astin’s I–E–O model (Astin and Lising 2012)

Following Astin’s model, the environmental variables selected to address the research questions are inquiry-based teaching (IBT), school climate, school’s socioeconomic status, nature of the school (public or private), and vocation (traditional academic school vs. technical or vocational school).

Astin identifies the outputs (O) as the student’s cognitive or affective gains after being exposed to the educational environment (Astin and Lising 2012). The output variables selected for this study are students’ scientific performance and critical thinking skills.

Data analysis

The study uses Hierarchical Linear Modeling (HLM) (Raudenbush and Bryk 2002) to examine the effect of individual and school characteristics on both the scientific performance and critical thinking skills of 15-year-old Colombian students. HLM is a variation of the ordinary least squares regression analysis and is used to investigate relationships between variables when the data are grouped into different hierarchical levels. This is the case of the PISA data, which are organized at the student, school, region, and country level. Unlike simple linear regression models, HLM is useful for calculating the effect resulting from the way students are assigned to different classes or schools, taking into account the characteristics of both the group and the individual (Myers and Myers 2015).

Colombia is a stratified country, and the socioeconomic characteristics of students and their families determine to a large extent the type of school they attend, so there may be little variation in the socioeconomic background of the students within each school. In other countries, schools accommodate students from different socioeconomic backgrounds. However, within the school, students’ socioeconomic characteristics determine the type of classes they are assigned to, and as a result, the variance within the school is affected. Therefore, a regression model that does not take into account the hierarchical structure of the data would not differentiate between these two education systems (OECD 2009). Equation 1 below details the analytical model for this study, displayed by level:

$$ \begin{aligned} Y_{ij} &= \beta{_{0}j}+\beta_{1j}{\text{Gender}}_{ij}+\beta_{2j}{\text{Grade}}_{ij}+\beta_{3j}{\text{ESCS}}_{ij}+\epsilon_{ij}\\ \beta_{0j} &= \gamma_{00}+\gamma_{01}{\text{Climate}}_{j}+\gamma_{02}{\text{IBT}}_{j}+\gamma_{03}{\mathrm{ESCS\, MU}}_{j}\\ &\quad+\gamma_{04}{\text{Vocation}}_{j}+ \gamma_{05}{\text{Nature}}_{j}+U_{0j}\\ \beta_{1j} &= \gamma_{10}\\ \beta_{2j} &= \gamma_{20}\\ \beta_{3j} &= \gamma_{30}. \end{aligned} $$

(1)

where, at level 1, Y_ij represents the outcome variables (scientific performance and critical thinking skills) measured for the ith level 1 unit nested within the jth level 2 unit; Gender_ij, Grade_ij, and ESCS_ij are level 1 predictors; β₀j is the intercept for the jth level 2 unit; β_1j, β_2j, and β_3j are the regression coefficients associated with X_ij for the jth level 2 unit; and ε_ij is the random error associated with ith level 1 unit nested within the jth level 2 unit.

At level 2, β_0j is the intercept for the jth level 2 unit; Climate_j, IBT_j, ESCS MU_j, Vocation_j, and Nature_j are level 2 predictors; γ₀₀ is the overall mean intercept adjusted for level 2 predictors; U_0j represents the random effects of the jth level 2 unit adjusted for level 2 predictors on the intercept; and γ₁₀, γ₂₀, and γ₃₀ refer to the coefficients for the predictors at level 1. The variables used in the model are explained in the following section.

Variables

As shown in Fig. 2, the variables for the analysis are organized around the three conceptual categories of Astin’s model: inputs, environment, and outputs.

First-level variables (input)

Gender

Gender reported by the student, recoded as 0 = male, 1 = female (ST004D01T)¹.

Grade

The grade level in which the student was enrolled at the time of sitting for the PISA 2015 test (ST001D01T). This categorical variable was transformed into five dummy variables, which denote the classification of each student in one of grades 7 through 11. Grade 11 is used as a reference category for the interpretation of regression coefficients.

ESCS

A composite index with a mean value of zero and a standard deviation of 1.0 representing the economic, social, and cultural status (ESCS) of students (OECD 2017). This index includes indicators for level of parental education, occupation of parents, and home possessions (OECD 2017).

Second-level variables (environment)

IBT

Inquiry-based teaching. A composite index with a mean value of zero and a standard deviation of 1.0, constructed using students’ responses to question ST098 (OECD 2016a) regarding the opportunities they have to explain ideas, conduct practical experiments in the laboratory, argue, draw conclusions about their experiments, discuss with teachers how scientific ideas can be applied to different phenomena, discuss their research, discuss the relevance of scientific concepts in their everyday lives, and conduct research to test ideas (see Additional file 1, Section 1).

School climate

A composite index with a mean value of zero and a standard deviation of 1.0, constructed with items included in both the student questionnaire (questions ST062, ST097, and ST100) and the school questionnaire (question SC061). Those items address the four dimensions of school climate as measured by PISA: student absenteeism, discipline, behaviors of students and teachers that obstruct learning, and teacher support to students (see Additional file 1, Section 3).

School ESCS

A measure of the socioeconomic and cultural status of the school. It is the school-level mean of the ESCS variable (ESCS MU).

Vocation

Modality of the school, recoded as 0 for academic schools and 1 for technical or vocational schools. This variable is called ISCEDO in the PISA database.

Type

Indicates whether the school is public (0) or private (1).

Dependent variables (output)

Scientific performance

It is a standardized variable with a mean of 500 points and a standard deviation of 100. It measures the scientific literacy of 15-year-old students in the use and application of scientific knowledge to identify questions, acquire new knowledge, explain natural phenomena, and draw conclusions based on scientific evidence (OECD 2016a). In PISA 2015, 10 plausible values were calculated for each student’s science score. Rather than using all 10 plausible values, in this study, we used only one plausible value as an outcome in the analysis. This is due to the fact that in analyses with more than 6000 cases, one plausible value has been shown to provide unbiased estimates of population parameters (OECD 2009, p. 43–46). The distribution of the 10 plausible values for the science performance of Colombian students is shown in Fig. 3. The figure shows that the means and standard deviations of the ten distributions across persons are almost similar to each other, and therefore, the analysis with one plausible value should not lead to loss of information.

Fig. 3

Distribution of plausible values for the science performance of Colombian students in PISA 2015

Critical thinking

This construct is measured by a composite index with a mean value of zero and standard deviation of 1.0. The index includes self-reported indicators of students’ ability to recognize scientific questions in the media, explain natural phenomena, recognize environmental changes that affect the survival of certain species, interpret data and scientific information, and asses the quality of arguments and explanations (questions ST129 and ST131; see Additional file 1, Section 2).

Construction of composite indices

As mentioned above, the concepts of inquiry-based teaching, school climate, and critical thinking are composite indices constructed with items included in the student and school questionnaires of the PISA study (OECD 2016a). Following the procedures established by PISA for the construction of these types of variables (OECD 2017), each index was built using the following model and principal component analysis (PCA):

$$ {\text{Index}}=\frac{\beta_{1}Q01'+\beta_{2}Q02' +\beta_{3}Q03' \cdots + \beta_{n}Qn'}{\varepsilon_{f}} $$

(2)

where β₁ through β_n are the factor loadings of the selected variables, Q01^′ through Qn^′ are the standardized values of the variables, and ε is the eigenvalue of the first principal component. The resulting indices have a mean value of zero and a standard deviation of 1.0, and the higher values represent, respectively, more exposure to IBT activities, a more positive school climate, and higher critical thinking skills.

The reliability analysis (Cronbach’s α) for each of these indices shows satisfactory results. The IBT index has a reliability of α=.84, and the critical thinking index and the school climate index both have a reliability of α=.81. Reliability is also satisfactory for each of the items that make up the indices (see Additional file 1, Section 4).

Results

Association between IBT and scientific literacy

The results of the unconditional model (Table 1) show that the total variance in students’ scientific performance that can be attributed to schools in Colombia is approximately 30% (F(1,363)=30,599,p<.05).

Table 1 Estimates of variance between and within schools and intraclass correlation (ρ)

The regression analysis including the predictor variables at the student and school level shows that the factors that best explain students’ performance in science are the grade level of the student, the frequency of inquiry-based teaching, gender, and the socioeconomic status of the school (Table 2). Each result is described below:

• (a) The grade level of the student when taking the PISA test is an important predictor of performance. Table 2 and Fig. 4 show that science scores are lower in grades 7 to 10 (grade 11 is used as a comparison category). Although all students taking the test were 15 years of age, the difference between 7th grade 11th grade students is 94.5 points (β=−94.5,p<.05).

  Fig. 4

  

  Science scores of Colombian students in PISA 2015 by grade

  Table 2 Hierarchical linear regression with science performance as the dependent variable

• (b) There is a significant difference in scientific performance between male and female students in Colombia, with males outperforming female students on an average of 25.3 points (β=−25.3,p<.05).

• (c) Scientific performance is also higher in technical or vocational schools than in academic schools (β=4.2,p<.05).

• (d) The socioeconomic index of the school (ESCS) is a strong predictor of students’ scientific performance. Two students at the same socioeconomic and cultural level who attend schools of different socioeconomic status show significant differences in performance; just a 1-point difference in a school’s socioeconomic index predicts differences in students’ scores of approximately 35 points (β=34.5,p<.05).

• (e) Students who report higher exposure to IBT have lower scientific performance than that of the students who are less exposed to IBT. As shown in Fig. 5, an increase of 1 standard deviation in the IBT index (i.e., higher exposure to IBT) predicts a decrease in test scores of 54.6 points (β=−54.6,p<.05).

  Fig. 5

  

  Association between IBT and the science scores of Colombian students in PISA 2015

• (f) An increase of 1 standard deviation in the school climate index (i.e., a more positive school climate) predicts an increase in science scores of 4.6 points (β=4.6,p<.05).

• (g) We did not find a significant interaction between IBT and school climate (β=1.1,p>.05).

Association between IBT and critical thinking

The results of the unconditional model show that the students’ critical thinking index does not significantly differ between schools in different regions of Colombia. The variance in the critical thinking index that can be attributed to schools is approximately 5% (Table 3). This is a negligible contribution (F(1,355)=.182,p>.05) and suggests that the effect of the school factor is null in the critical thinking index.

Table 3 Estimation of variance between and within schools and intraclass correlation (ρ)

The results of the HLM analysis are shown in Table 4. These indicate that the factors that best predict the self-reported critical thinking skills of the students are exposition to IBT, gender, and socioeconomic status of the school. These results are detailed below:

• (a) A 1-point increase (i.e., higher socioeconomic level) in the student’s socioeconomic index is associated with a slight increase in the critical thinking index (β=.055,p<.05).

  Table 4 Predictors associated with critical thinking

• (b) There is a positive association between IBT-based teaching practices and students’ critical thinking index (Fig. 6). An increase of 1 standard deviation in the IBT index predicts a positive change of 1.14 standard deviations in the students’ critical thinking index (β=1.138,p<.05).

  Fig. 6

  

  Association between IBT and students’ self-reported critical thinking skills

• (c) The critical thinking index did not vary significantly between grades, save for 8th grade students, who self-reported a critical thinking index score 0.23 standard deviations higher than the 11th grade used as a base of comparison (β=.232,p<05).

• (d) An increase of 1 standard deviation in the school climate index (i.e., the climate is more positive) predicts a positive change of 0.17 standard deviations in the critical thinking index (β=.17,p<.05).

• (e) There are no significant differences between male and female students in the critical thinking index (β=.022,p>.05).

• (f) There is a significant interaction between the IBT and school climate. This suggests that the better the school climate, the greater (more positive) the effect of IBT on critical thinking is (β=.372;p<05).

• (g) There are no significant differences between technical and academic schools in the critical thinking index (β=.064;p>05).

• (h) There are no significant differences between public and private schools in the critical thinking index (β=−.030;p>05).

Discussion

According to the results of this analysis, more evidence is needed to best understand the effect of inquiry-based teaching on science learning. Although several studies support the use of the so-called active pedagogies such as inquiry-based teaching as a way to improve students’ learning (Freeman et al. 2014; Gormally et al. 2009; Kvam 2000; Summerlee and Murray 2010), it seems that their effect is greater in helping students maintain their motivation and engagement and develop critical thinking skills, than in the development of scientific literacy. However, to have a real impact on learning, “active” strategies have to be more than just fun and exciting; they must be aimed at creating opportunities to develop connections to theories or complex ideas.

In Colombia, the quality of students’ educational outcomes has largely been attributed to their socioeconomic background. However, the results of this study show that although students’ and schools’ socioeconomic characteristics continue to be strong predictors of academic performance, teaching practices can have an even larger impact. As shown in Table 2, a 1-point increase in the school index of ESCS is associated with a 34.5-point increase in science scores, while a 1-point change in the IBT index predicts a decrease of 54.6 points.

This study shows that the effect of the students’ socioeconomic characteristics is not the same for all academic outcomes. As shown in Table 2, both school and student’s ESCS exert a strong effect when science performance is used as the dependent variable. However, when critical thinking is used as the dependent variable, their effect is barely tangible (Table 4). A possible explanation for this could be that the critical thinking index used in this study was estimated from self-reported items from the PISA student questionnaire, and therefore, it may be biased towards students reporting social desirable answers. Nevertheless, despite this potential bias and the need for further validation, critical thinking is widely acknowledged as an important outcome of science education, and although it cannot be used to generalize the results or make international comparisons, it can be useful as a starting point for further analysis and research (Marsden and Wright 2010).

Vertical stratification has an important effect on student learning. A large population of Colombian students are enrolled in grades that do not match to their chronological age, which contrasts with students from high-performing countries in PISA. Although the causes that lead to vertical stratification can be diverse, grade retention may be a big contributor, considering that more than 40% of Colombian students who sat for the PISA test report having repeated at least one grade. This leads us to suggest that it is necessary to delve deeper into this institutional practice and its effects on student learning.

This study shows that a positive school climate can be a protective factor for all students and a moderator of scientific performance and critical thinking skills. This suggests that student learning could be improved if actions are implemented to reduce absenteeism, improve discipline, change the habits and behavior of teachers and students that hinder learning, and establish support programs and guidance by the teachers.

Finally, we should mention that during the course of this study, Astin’s I-E-O model proved to be a useful analytical tool for helping us to conceptualize the connection between school-level variables (environment) and educational outcomes (outcomes), while controlling for the influence of students’ individual characteristics and socioeconomic background (inputs). As a conceptual framework, it naturally fits the nested structure of PISA data and the purpose of HLM analysis. However, it can be most useful as a friendly reminder to researchers and educational practitioners that socioeconomic or individual background factors cannot be easily changed by teachers or schools, and therefore, it is necessary to continue searching for in-school factors, within the control of practitioners and policymakers, that can make a difference to students’ learning. After all, each student has the right to an education that meets their unique needs, skills, and talents.

Conclusions

These results reveal the need for additional research on the effect of the so-called active methodologies and school context on student learning. Regarding IBT, it seems that its influence is greater on student’s perception and motivation, than in acquisition of scientific knowledge. Positive school climate is a protective factor that can help to improve student learning. In Colombia, the quality of students’ educational outcomes has largely been attributed to their socioeconomic background. However, the results of this study show that, although students’ and schools’ socioeconomic characteristics continue to be strong predictors of academic performance, teaching practices can have an even larger impact.