Bridging borders: assessing the impact of semester-long study abroad programs on intercultural competence development in undergraduate engineering students

Abstract

Background

Working and interacting with people from diverse backgrounds have become common in Engineering. Research has indicated that engineering graduates face challenges while working with a diverse workforce. Therefore, it is vital for higher education institutions to help engineering students develop intercultural competence skills by engaging them in intercultural learning activities. This study explores the impact of a semester-long study abroad program and its new curriculum on the intercultural learning gains of undergraduate engineering students. The study used a mixed methods design to assess the intercultural learning gains of the students enrolled in a study abroad program.

Results

The results of the study indicated that overall, as a group, students demonstrated significant gains in their Intercultural Development Inventory (IDI) scores from the pre- to post-test. Moreover, hierarchical clustering was conducted to group students into three clusters based on their IDI scores. The results of the clustering helped us to understand the differential growth of students from their pre- to post-test. Based on clustering students were grouped into high, moderate and low intercultural learning clusters. Further inductive thematic analysis was conducted to understand the depth of intercultural awareness of the students in general and discipline specific for each cluster. The results of the thematic analysis revealed that students in cluster 1 showed a deeper understanding of cultural differences, mindfulness, and openness while interacting with people from other cultures, whereas the students in clusters 2 and 3 showed moderate to superficial level of intercultural awareness.

Conclusions

The implication of the study extends to both practice and theory. From the practice perspective, the study discusses the need for creating intentionally structured study abroad programs that encourage students to reflect on their intercultural experiences and internalize takeaways. From the theory perspective, the study contributes to the Intercultural Development Continuum model, as it emphasizes the importance of meaningful gains in intercultural competence. The study also suggests strategies to improve the intercultural learning experiences of students that could help them move towards more intercultural mindsets.

Introduction

The need for intercultural competency has gained substantial attention in engineering as working with diverse populations has become a norm (Berka et al., 2021). With increasing technological advancements, companies focus on creating diverse work environments by recruiting talent across the globe (Jaiswal et al., 2021c). Studies have revealed that working within a diverse work environment can be challenging for fresh graduates (Bahrami et al., 2023; Du & Kolmos, 2009; Rock et al., 2016). It has also been observed that engineering students do not always get the opportunity to interact with a diverse student population (Blouin, 2022), and also the coursework is primarily technical (Lucietto & Russell, 2020). The technical nature of the coursework helps students to develop hard skills, but provides less opportunities to develop professional skills (Bahrami et al., 2023; Jaiswal et al., 2021a, 2023a; Magana et al., 2022). Therefore, introducing engineering graduates to the concept of intercultural competence can help them develop professional skills such as teamwork, communication skills, cultural self-awareness, openness, and interactive skills such as behaving appropriately when interacting with someone from a different cultural background (Berka et al., 2021; Jaiswal et al., 2023a, 2023b; Kulturel-Konak, 2020).

Helping students develop a global mindset is the top priority in today’s higher education (Starr et al., 2022). Intentionally structured study abroad programs have been considered beneficial in helping students develop intercultural competence and global competencies (Bittinger et al., 2022; Davis & Knight, 2018). Research has also shown that study abroad programs have resulted in improved twenty-first century skills, such as leadership and teamwork skills for students (Farrugia & Sanger, 2017). These programs help students develop cultural sensitivity, as analysis of student reflections after the study abroad trip has demonstrated a deeper understanding of different cultures, such as appreciating and recognizing biases they had earlier (Twyman & Knight, 2016). Chédru and Delhoume (2023) found that studying abroad for a semester is significant for students to understand and connect with different cultures. They suggest that educators and institution leaders can utilize the findings of this study to understand the impact of studying abroad and help prepare engineering students for collaborating and working with people from diverse backgrounds.

Literature review

To make engineers culturally aware and interculturally competent, higher education institutions are trying their best to develop mechanisms to integrate the concept of intercultural competence into the engineering curriculum. In the context of engineering, there is a lack of evidence that supports the effectiveness of approaches for embedding intercultural competence; therefore, scholars are still working to develop appropriate frameworks to assess and enhance intercultural gains (Davis & Knight, 2018, 2022; Yu, 2012). Prior studies in higher education have shown the importance of study abroad and structured curriculum to help students develop intercultural competence (Jin et al., 2024; Jones et al., 2019). For example, study by Krishnan et al. (2021) showed the impact of the structured curriculum and mentorship provided during the study abroad on the intercultural development of undergraduate students. The study used pre and post IDI and written reflection to assess intercultural competence. For this study, students were divided into experiment and control groups. Students in the experiment groups were provided 3–4 1-h sessions in the spring of 2017, during the summer of 2017 students were engaged in 3 h class for 2 weeks where instructors discussed about the culture and engaged them in various activities to develop cultural self-awareness and cross-cultural competence. The study abroad was conducted in the summer of 2017. Students went to India, and while on the trip, they were constantly mentored by interculturally trained faculty members and were engaged in reflection activities, discussions, and exercises related to intercultural learning. Experimental group students constantly worked with their fellow students in India to understand the healthcare system in India, which allowed them to reflect on their own culture and understand other cultures. Upon their return, instructors conducted a final debriefing that allowed students to reflect back on their intercultural experience. On the contrary, control group students were not provided such opportunities. The results of the IDI showed that students in the experimental group demonstrated both a statistically significant and a meaningful level of growth (+ 13.57 points), which was defined by Paras et al. (2019) as an increase of 7 or more points on the IDI’s Developmental Orientation score. Analysis of the reflection revealed that students developed cultural self-awareness, showed openness, and the ability to adapt to people from diverse cultures.

Similarly, a recent study by Bittinger et al. (2022) also used pre and post IDI and written reflection to assess the impact of the study abroad program and the structured curriculum on the intercultural learning gains of the students. The study emphasized the importance of an intentionally structured curriculum in helping first-year technology students to develop intercultural competence. In this study, first-year students participated in a Global Leadership Program developed by the study abroad office. The study showcases the results for 3 consecutive years 2017, 2018 and 2019. The curriculum for this program was grounded in the Intercultural Knowledge and Competence (IKC) VALUE (Valid Assessment of Learning in Undergraduate Education) rubric framework developed by the Association of American Colleges and Universities (AAC&U). This was a semester-long initiative. Students were required to complete reflections, assignments, and activities prior to their departure to Peru. Students stayed in Peru for 2 weeks: they stayed in various places for 2–3 nights with host families in Lima to get an understanding of the Peruvian culture, and then they stayed in an eco-lodge at Amazon and also in hotels. Students participated in class sessions that had a component of a site visit or guest lecture. They completed the 8–10 reflection journals during their stay in Peru; they were also constantly mentored and debriefs were conducted to help them internalize their experiences. Upon their arrival to the United States, the students met as a cohort for eight sessions and were engaged in various intercultural activities to gain a deeper understanding of their experiences. They were mentored throughout the program and completed IDI assessment pre, mid and post, as a part of the program. All the students showed a meaningful increase in their DO scores from pre- to post-test for all 3 years; the pre–post gains for 2017 were 21.76 points, for 2018 was 20.05, and for the year 2019, it was 22.17 points. The analysis of the written reflection revealed students’ openness and curiosity to learn about other cultures, an understanding of verbal and non-verbal communication, and confidence in interacting with people from diverse backgrounds after the study abroad trip. Overall, the program, curriculum, and constant mentoring helped the students to grow meaningfully in their intercultural journey.

It is also important to acknowledge when assessing the students’ intercultural competence using IDI that just an increase from pre to post does not showcase that students have grown interculturally. It is important to have an increase of 7 or more points from pre- to post-test, to conclude that students have had meaningfully developed intercultural skills and behavior (Mu et al., 2022). Recently, researchers have stressed the idea of meaningful growth and meaningful regression while discussing the IDI gains in the context of the study abroad program (Mu et al., 2022; Paras et al., 2019). Meaningful growth in the IDI score refers to the real-world gains in the intercultural competence of the students such as adaptation, cultural self-awareness, empathy, etc. (Paras et al., 2019). Meaningful regression refers to a decrease of 7 or more points on IDI, showcasing that students developed ethnocentric behaviors such as judging others, conducting cross-cultural comparisons, etc. (Mu et al., 2022).

Previous studies (Marx & Pray, 2011; Wang et al., 2022) also have emphasized the importance of providing students with opportunities to engage with different cultures that help them develop skills, such as empathy, cross cultural communication abilities, and global mindset. Studies by Tarchi and Surian (2022) and Berka et al. (2021) have demonstrated that both study abroad programs and intervention-based curricula contribute to the development of students’ verbal communication skills and empathy. Berka and colleagues conducted research and found that students enhanced their verbal communication abilities through interactions with team members. Furthermore, their study revealed that students who participated in both intervention-based curricula and study abroad programs demonstrated a strong understanding of cultural awareness by discussing the rules and biases of their own culture. These students exhibited a deep comprehension of cultural worldview frameworks, acknowledging and appreciating the cross-cultural complexity, including values, beliefs, and practices. Moreover, the students showed curiosity as they posed complex questions about diverse cultures and actively sought answers that incorporated multiple perspectives (Berka et al., 2021).

Therefore, through our study, we intend to examine the impact of a semester-long intentionally structured study abroad program in helping engineering graduates develop intercultural competence. This study contributes to the scholarship that focuses on intercultural competence in engineering education. By focusing on undergraduate students in engineering, this study provides insights into the factors related to fostering interculturally competent engineers toward facilitating intercultural skills, such as positive attitude, self-awareness, and effective communication. Therefore, our study intends to pursue the following research question: what is the impact of the intentionally structured study abroad program on the intercultural learning gains of engineering students?

Theoretical framework (Kolb’s learning cycle)

Learning is the result of knowledge gained by the learner through various experiences (Kolb, 1984). Therefore, allowing the learners to reflect, analyze and evaluate their experiences can lead to long-term learning gains. This study is grounded in Kolb’s Learning Cycle framework, as studying abroad is an experiential learning experience. The study abroad program allows the students to go through all four phases of Kolb’s learning cycle. The Kolb learning cycle identifies four phases that lead to effective learning gains. The four phases of Kolb’s learning cycle are concrete experience, reflection, abstract conceptualization, and active experimentation, as illustrated in Fig. 1.

Fig. 1

Kolb’s learning cycle

The concrete experience phase occurs when learners encounter a novel experience or reinterpret a prior experience from a new perspective. In our case, the study abroad program is a concrete experience for the students, as that allows students to indulge in a novel experience in a new country. In the reflection phase, learners learn to critically reflect on their experience by drawing on their prior experiences or new knowledge that they gained from their concrete experience. While studying abroad, mentors played a very important role in helping students to reflect on their experiences. Mentors met the students and engaged them in reflective discussions and exercises. The abstract conceptualization phase is activated when learners have learned to reflect, allowing them to think of new ideas and activate their critical thinking skills. In our case, the abstract conceptualization was activated by engaging students in guided written reflection activities. During these activities, students were scaffolded and given prompts to complete written reflections. The scaffolding helped foster students’ metacognitive thinking skills, allowed them to think critically about their past experiences, and connected them to new experiences. Mentors provided feedback on every reflection assignment completed by the students. The feedback from the mentors also served as an effective mechanism for the students to improve their reflective practices. Active experimentation is the fourth stage of Kolb’s learning cycle that allows the learners to apply their learned concepts in a real-world context. In our case, students were given various opportunities to interact with locals during their study abroad, allowing them to apply the intercultural skills they have learned. For example, in one of their assignments, students were required to interview someone from a different culture and learn about their intercultural experiences. Collectively, the four phases of the Kolb’s learning cycle helped us to take a deeper dive into the students’ experiences while studying abroad.

Methods

Learning design

The Semester Abroad Intercultural Learning (SAIL) Scholarship Program is an initiative by Purdue University that allows students to participate in a semester-long study abroad. During their time abroad, students are required to register for a one-credit-hour intercultural learning course. This study abroad program has three components: (a) pre-departure, (b) Growing, Learning and Understanding Everyone (GLUE) curriculum, and (c) mentoring.

Pre-departure

Pre-departure sets the tone for the study abroad programs. All the SAIL students attend a pre-departure orientation, during which they are given an overview of the course structure and procedures for engagement while abroad. Just prior to the start of their study abroad program, students are placed in cohorts based on program start dates and time zones. Once mentors are assigned to cohorts, students receive access to and instructions for completing the IDI. Mentors meet their cohort either in-person or virtually. During this meeting mentors introduce themselves to the cohort and introduce the students to the course syllabus and the major deliverables. This interactive 1-h session is crucial between the mentor and students as it helps them to know one another and students can ask clarification questions to the mentor. During this meeting the mentor also sets the date for their first synchronous live session. Students are then granted access to their learning management system (LMS), D2L Brightspace course page and are provided instructions to begin working through the first two units of asynchronous course content and assignments. Early in the curriculum, students view a pre-recorded group debrief video, which includes IDI results of all students taking part in the course that semester. This pre-recorded video helps students understand their IDI results and also the strategies that they can use to move ahead on the IDI continuum. As students work on their assignments, they are given periodic feedback on their deliverables. These feedback helps students to reflect on their study abroad experiences and make meaning of their experiences. At minimum, they receive feedback at least twice in each unit. The asynchronous component of the course is provided entirely via Brightspace and facilitated additionally through email announcements and communications.

GLUE curriculum

This course is a 1 credit-hour (course that requires students to spend 50 min per week) version of the Growing, Learning and Understanding Everyone (GLUE) curriculum. The GLUE curriculum is created by the Intercultural Curriculum Design Team, comprised of professional and graduate staff affiliated with the Center for Intercultural Learning and specially recruited instructional design experts and intercultural partners from across the university’s campus. The curriculum is grounded in three theoretical frameworks, they are, Intercultural Development Continuum, The IKC-VALUE Framework, and Sorrell’s Praxis model. In addition, the learning outcome of the course is focused on the higher order of Bloom’s Taxonomy, which allows students to analyze their learning, internalize the experience, and evaluate their learning gains. The curriculum as a whole comprises eight units of asynchronous instruction provided through the LMS, D2L Brightspace. Following every two units, students meet as a cohort along with their Mentor-instructor in a 90-min, live class session hosted on Zoom. Students also take post-IDI assessments at the end of their course. Those live sessions aimed to focus on challenging and advanced activities. Table 1 below presents the outline of the GLUE curriculum, and it also describes goals for each unit, assignments/activities, and the mentor-led live sessions. In addition, Appendix 1 contains the links to some of the activities mentioned in Table 1.

Table 1 Outline of the GLUE curriculum

Theoretical foundation of the GLUE curriculum

The GLUE curriculum emphasized the practical aspects of developing intercultural competencies within both local and global contexts, as well as addressing the needs of undergraduate education and future workspaces. The theoretical frameworks serve as the foundation for developing intercultural competence. For example, the Intercultural Praxis Model is a tool for understanding and navigating cross-cultural communication challenges. The model also played a crucial role in determining the learning objectives specific to the study abroad curriculum. The model consists of six stages: inquiry, framing, positioning, dialogue, reflection, and action. The inquiry component pertains to an individual’s curiosity, willingness, and adaptability to learn, comprehend, and question existing worldviews. The framing element emphasizes the capacity to shift perspectives according to different cultural frameworks. Positioning refers to understanding one’s own positionality in relation to others, such as recognizing privileged and marginalized voices. Dialogue is another crucial component that focuses on meaningful communication across cultures to establish connections and enhance understanding and empathy. The reflection framework involves introspection and observation of one’s own perspectives, viewing oneself as an agent for change. Finally, the action framework highlights a deeper understanding of intercultural elements and the ability to challenge existing biases through proactive engagement (Sorrells, 2022). Each stage of the model serves as an opportunity to understand intercultural communication and initiate interaction. Since helping students develop an understanding of other cultures and interacting appropriately is an objective of the program, the model provided students with the opportunity to navigate complex and challenging intercultural situations (Sorrells, 2022).

Another objective of the program is to help students understand and reflect on their own cultural values and also develop appropriate skills, behavior and attitudes to interact with other cultures. Therefore, the six elements of the American Association of Colleges and Universities (AAC&U) Intercultural Knowledge and Competence (IKC) rubric were instrumental in setting the learning objectives for each assignment, encompassing activities and reflective exercises. The IKC VALUE (Valid Assessment of Learning in Undergraduate Education) Framework, as proposed by Bennett (2008), elucidates the cognitive, affective, and behavioral competencies required to effectively engage in diverse cultural environments. This framework includes following six dimensions:

• 1. cultural self-awareness (emphasizes the recognition of personal cultural biases)

• 2. knowledge of cultural worldview (understanding values, politics, history, economy, and communication styles associated with specific cultures)

• 3. empathy (capacity to comprehend emotions by adopting different perspectives)

• 4. verbal and nonverbal communication (ability to comprehend diverse communication styles and the values associated with them across different cultures)

• 5. curiosity (reflecting upon diverse cultural perspectives and asking probing questions about different cultures)

• 6. openness (ability to eliminate judgmental attitudes towards cultures that differ from one’s own), (AAC&U, 2009).

The Intercultural Development Continuum (IDC) is the third theoretical framework. It was used to establish the desired learning outcomes for the GLUE course and guide the design of formative instructional content. The IDC comprises five stages, including denial, polarization, minimization, acceptance, and adaptation (Hammer, 2012). The denial mindset reflects an inability to comprehend and accept cultural differences. Polarization refers to a mindset that categorizes cultural differences as “us” versus “them.” Minimization highlights the recognition of commonalities across cultures while failing to deeply understand the nuances of cultural differences. On the other hand, acceptance and adaptation represent global intercultural mindsets that appreciate and embrace differences while also acknowledging them on a deeper level (Hammer, 2012). Using pre-test scores from an assessment for this program helps the mentors to evaluate the position of the students on IDC and develop strategies accordingly.

Virtual mentoring

Mentors play a very important role in the study abroad program. All the mentors are trained professionals with years of experience in using the Intercultural Development Inventory (IDI) to assess intercultural learning gains. Being IDI-trained allows the mentors to provide stage-based feedback to students throughout the course. The stage-based feedback allows the students to move ahead on the IDC continuum and develop interculturally. The mentors are also responsible for providing feedback on student reflections. Throughout the program, mentors meet the student cohorts virtually and engage them in virtual 90-min live sessions (see Table 1), using the Zoom platform. During the session, the mentor engages students in reflective practices by asking them to think through their intercultural experiences while studying abroad. This thinking exercise helps students to learn to reflect on their experiences. The activities and discussions in live sessions aim to help students move from the knowledge, comprehension, and application of intercultural competence concepts to analysis, synthesis, and evaluation of their learning experiences.

Research setting and participants

This study was conducted at a large Midwestern university. The participants for this study were undergraduate students enrolled in the SAIL (Semester Abroad Intercultural Learning) scholarship program for Spring 2022. The participants of this study participated in a semester-long study abroad program and had voluntarily self-selected into a one-credit online mentored support course for their intercultural learning while abroad, incentivized by a tuition–credit scholarship upon completion of the course. A total of 74 undergraduate engineering students enrolled in the SAIL program for Spring 2022. Permission from the Institutional Review Board was obtained prior to the start of the study.

Data collection

We used a mixed-method approach to understand the student experiences in this semester-long study abroad program (Creswell, 2018). Both quantitative data and qualitative data were collected as the part of the program. The quantitative survey data were collected using the pre and post-IDI assessments. The pre-IDI was conducted at the start of the program, and the post-IDI was conducted at the end of the study abroad program. The students were also engaged in reflections throughout the course, the qualitative data was collected in the form of student reflections. The reflection that is analyzed in this study was conducted in the last week of the program. Since the intent was to understand the impact of the program on the intercultural learning gains of students, therefore, we found it appropriate to analyze the reflections completed during the last week of the program. Since the course is designed on the principles of experiential learning, in this final assignment, students were required to interview someone who is from a different culture but has been in the engineering profession for at least 2 years. During the interview, students were supposed to ask the participants about their experience of working with people from other cultures, the challenges and benefits of working in a multicultural environment, and their perspectives on intercultural competence in engineering. For the reflection piece of the assignment, students were asked to transcribe the interview, and they were asked to write a narrative highlighting the key away from the interview. They were also supposed to write their perceptions about working and interacting with people from various cultures and the importance of intercultural competence in their discipline. This assignment was due for submission in the last week of the course so that students could draw on their experiences of studying abroad and reflect on their interactions with students of different cultures. The goal of the assignment was to help students understand the perspective of people from similar professions on intercultural competence.

We chose to use both the IDI data and qualitative data to understand the intercultural development of the students is grounded in the previous literature. A recent study by Tamer et al. (2023) indicated that just using the IDI as a measure for intercultural competence is not sufficient as it is a self-reported assessment, and students tend to overestimate themselves in the pre-test, which might provide an inaccurate measure of intercultural competence when pre- and post-test results are compared. In addition, other factors that might impact the IDI data are individuals’ personal experiences, biases, background, etc. (Terzuolo, 2018). Therefore, Wong (2015) suggested to use qualitative data along with IDI data to assess the intercultural competence development of the students.

Intercultural Development Inventory (IDI) instrument

The Intercultural Development Inventory (IDI) is a widely used and psychometrically valid instrument to measure the intercultural learning of students (Hammer, 2011). The initial version of the IDI consisted of 145 items; these items were generated by a panel of experts based on content analysis. To validate the instrument, 226 participants were selected, and they completed the 145-item questionnaire; exploratory factor analysis was conducted. The results of the factor analysis helped to reduce the number of items from 145 to 122. In the second phase, a confirmatory factor analysis was conducted, 591 participants completed the survey, and the results of the confirmatory factor analysis helped to reduce the number of questions to 52. Later on, two questions were dropped by the panel of experts as they lacked clarity, and a new version of the IDI consisted of 50 multiple choice questions measured on 5-point Likert scale (Hammer et al., 2003). Furthermore, the 50-question IDI was validated using another round of confirmatory factor analysis, and that led to the creation of the version 3 IDI instrument that is used for this study (Hammer, 2011). The IDI is a widely used instrument and has been used in multiple studies to assess the intercultural learning gains of the students (Krishnan & Jin, 2022; Starr et al., 2022). The IDI assess the participants intercultural competence based on five stages, see Fig. 2, they are denial, polarization, minimization, acceptance, and adaptation. It also categorizes participants into monocultural and intercultural mindsets based on their developmental orientation scores. According to the instrument, the participants whose scores fall in denial and polarization categories tend to have a monocultural mindset, whereas the participants in the acceptance and adaptation phases have an intercultural mindset. The minimization is a transition stage from a monocultural to an intercultural mindset (Hammer, 2011; Haskollar & Kohli Bagwe, 2023). The IDI also provides developmental orientation and perceived orientation scores. The developmental orientation (DO) score shows the actual level of intercultural competence of the participant. The perceived orientation (PO) scores represent the perception of the participants regarding their intercultural competence. Therefore, studies use DO scores as a measure to assess the intercultural competence of the participants. The IDI provides an intercultural development plan (IDP) for each participant, the plan delineates the strategy that the participant could follow to move ahead on the intercultural development continuum (IDC).

Fig. 2

Intercultural development continuum

Data analysis

Quantitative data analysis

The study used quantitative analysis to analyze the IDI data, specifically the DO scores, obtained as the pre- and post-test scores. Prior to conducting the quantitative analysis, the normality of the data was assessed using the Shapiro–Wilk test, the analysis confirmed that the data was normally distributed as the p value was not significant for the pre-test DO scores [W(74) = 0.99, p = 0.601] and for post-test DO scores [W(74) = 0.98, p = 0.466]. Furthermore, the pre-test and post-test data were also tested for outliers, and no outliers were found, please see the graphs in Appendix 2. The quantitative data were analyzed using descriptive statistics to identify the mean and spread of the DO scores for the students. Furthermore, an unsupervised machine learning algorithm known as hierarchical clustering was used. An unsupervised machine learning algorithm helps us to detect unknown patterns in a dataset that is difficult to observe or find manually (Le Quy et al., 2023). Hierarchical clustering has been widely used in the context of the education data mining (Jaiswal et al., 2021b, 2022; Psaromiligkos et al., 2009), as it allows to organize the data with similar scores together. Specifically for this study, since the data set was not very large (N = 74 students), Ward hierarchical clustering was used as it has been identified as an effective approach to cluster data from small data sets (Antonenko et al., 2012; Jaiswal, 2022; Jaiswal & Karabiyik, 2022; Jaiswal et al., 2021a; Medová & Bakusová, 2019). Before conducting the clustering, it is important to identify the optimal number of clusters. We used the elbow method to identify the optimal number of clusters that could be generated by a data set (Yuan & Yang, 2019). Moreover, to examine if there was a significant difference between the pre- and post-test scores for each cluster t test was performed.

Qualitative data analysis

Since the quantitative results helped to group the students into three clusters, we wanted to triangulate our findings by conducting a qualitative analysis of the student reflections. Therefore, to gain a deeper understanding of intercultural learning gains for students in each cluster, we conducted the qualitative analysis using inducted thematic analysis. Details about the analysis are presented in the subsequent paragraphs.

Positionality of the researchers

The qualitative data were analyzed by the first and second authors as they were at the outsider end of the positionality continuum. Both authors joined the research project in the summer of 2022 after the completion of the semester long study abroad. Therefore, the authors did not have any direct interaction with any participants nor any intrinsic motivation to find evidence of learning in the course. Both authors are experts in the thematic analysis of qualitative data and also possess expertise in intercultural learning frameworks.

Data coding

The qualitative analysis was conducted by the first and second authors, who worked together to perform an inductive thematic analysis based on the steps delineated by Braun and Clarke (2012): getting familiar with data, creating the initial codes, identifying the potential themes, reviewing the themes, defining the themes, and producing the final report.

To ensure the trustworthiness of the analysis, the first and second author coded the data separately. First, descriptive coding was conducted, followed by a round of axial coding to generate the final themes. In this process, we independently read ten reflections from each cluster and developed descriptive codes to describe the student experiences. Upon completion of the first round of descriptive coding, we met and discussed our codes, resolving any inter-rater differences in initial coding through consensus. The discussion helped us to merge similar codes and develop categories and sub-categories. The final codebook was created, and we used the codebook to code the data. Since the data set was small, we used the final codebook to recode the data for all the students in each cluster. Upon completion of the second round of coding using the codebook, we calculated the inter-rater reliability, which was 81%. The results of the thematic analysis are discussed in the qualitative results section.

Results

Quantitative results

Descriptive statistics and t test comparison

The results of the descriptive statistics revealed that overall, as a group, students showed a meaningful increase in their DO scores from pre-test to post-test, see Table 2. The post-test mean scores indicate that students as a group moved to the center of minimization on the intercultural development continuum. In addition, the student group demonstrated a meaningful gain (gain of 7 or more points on IDI) in the intercultural competence (Jones et al., 2019).

Table 2 Mean and SD of students enrolled in the study abroad program

Further paired sample t test was conducted, and it was found that as a group, students demonstrated a significant increase in their DO scores from the pre-test (M = 92.16, SD = 15.73) to the post-test (M = 101.66, SD = 18.01, t = 5.93, p < 0.00001). The line graph in Fig. 3 represents the DO scores of the student group for pre and post-IDI assessment.

Fig. 3

Line graph showing the pre and post DO scores of all students

Results of cluster analysis

The Ward Hierarchical clustering was conducted using R programming. The pre and post-DO scores of the students were used as the input for the algorithm. The first step was to determine the optimal number of clusters, and using the elbow method, we identified three clusters that were optimum for the current data set. The clusters were named as follows:

Cluster 1: high intercultural learning cluster (C1) Students (N = 11) showed a large increase in their mean DO score from pre- to post-test. Students moved from the lower end of minimization to acceptance on the IDC. Figure 4 shows the change in the DO score values for the C1 students. From Fig. 4 we note that all of the students (S1….S11) showed meaningful increase in the DO score for C1.

Fig. 4

Change in DO-scores for each cluster

Cluster 2: moderate intercultural learning cluster (C2) Students (N = 39) showed a moderate increase in their mean DO score from pre- to post-test. Students moved from the lower end of minimization towards the upper end of minimization on the IDC. Figure 4 shows the change in score for the C2 students. All the students (S12….S50) in this cluster showed an increase in their DO scores, and most of the students showed meaningful gains in their DO score, except for S25 = 6.02, S31 = 6.29, and S47 = 6.47.

Cluster 3: low intercultural learning cluster (C3) Overall, students (N = 24) in C3 showed a decline in the mean DO score from pre- to post-test. Students were towards the center of minimization in their pre-test mean but shifted backwards by 5 points towards the lower end of minimization. Moreover, to be specific, out of 24 students, 18 showed a decline in their DO scores. However, only eight students showed a meaningful regression in their DO scores from pre- to post-test: S53 = − 16.19; S56 = − 11.74; S57 = − 8.68; S58 = − 8.34, S60 = − 25.26, S61 = − 13.48, S65 = − 8.16, S68 = − 19.60.

Comparison of pre- and post-test DO scores clusterwise

Table 3 represents the mean, standard deviation and t test scores of each cluster. To analyze the pre–post DO scores paired sample t test was conducted between the pre- and post-DO scores for each cluster. From Table 2 we can interpret that students in C1 and C2 demonstrated a statistically significant and meaningful gain in the DO scores from pre to post. In contrast, C3 students demonstrated a statistically significant decrease in the DO scores from pre to post. It is also important to note that as a group C3 students did not show a meaningful regression (decrease of 7 or more points) on the IDI. That means that there are few students (N = 8) who have shown a meaningful decline.

Table 3 Clusterwise mean and SD of students enrolled in the study abroad program

Qualitative results

Inductive thematic analysis was conducted to identify the emerging themes for students in each cluster. The objective of the analysis was to find the commonality and differences in study abroad experiences among the participants in each cluster. In the subsequent paragraphs, we have discussed the themes that emerged under each category.

Themes for intercultural competence

Sub-theme for Cluster 1: developed an understanding of intercultural communication and showed empathy toward other cultures

Students in C1 demonstrated a deeper understanding of verbal and non-verbal communication, showed empathy, and exhibited a positive attitude towards cultural differences. For instance, Morgan mentioned that she believes that knowledge of intercultural competence is crucial in this era as the world is becoming global. “I learned that intercultural competence is very important especially, in the era of globalization and the new technology.” Morgan also acknowledged that after studying abroad, she had developed the ability to communicate with people from diverse backgrounds. In addition, she admitted the fact that communicating with someone with a different background is not easy. There are chances of miscommunication, but the study abroad experience has taught her to become emotionally resilient and communicate calmly during such interactions. “I can interact with people from different cultures, which can be very difficult. However, when there is miscommunication, I have learned how to manage my emotions so that we can continue to communicate calmly.”

Similarly, another student Jake mentioned that he also believes that the intercultural competence that he has developed through this study abroad experience has helped him to respond well to ambiguous situations. “I believe an important aspect of intercultural development is learning to respond well to ambiguity.” He also acknowledges that living in a country that speaks a different language than his can lead to situations where two people from different backgrounds find it difficult to understand one another. In such cases, Jake has learned to remain calm and ask more clarification questions. “By living in a country with a different language than my own, I have had many instances where I do not fully understand the situation. I have learned to ask more questions if I do not understand, and I do not get as frustrated with lack of understanding within the situation.”

In both these cases, we see how students suspended their ability to judge; rather, they were focused on deeply comprehending the differences and responding calmly to the situation.

Sub-theme for Cluster 2 and Cluster 3: showed willingness to understand and adapt when interacting with other cultures

Students in C2 and C3 showed their willingness to change their behavior when interacting with people from different cultures. For example, John mentioned that as a future engineer, he needs to develop flexibility and adapt his communication style when interacting with someone from a different culture. He also mentioned that he needs to be articulate enough so that others can understand him. John says, “As a future engineer, I need to become more adaptable to different cultures. I should learn to articulate my thoughts clearly and must not assume that a person from another culture will understand something I say.”

Similarly, Jenny from C3 mentioned she started putting effort into understanding people from other cultures as she feels showing an openness towards understanding others will lead to friendly and easy communication. Jenny says, “I learned to put in the effort of understanding people from different backgrounds and how it opened the doors to better friendships and easier communication.”

In both instances, we saw that John (C2) and Jenny (C3) showed openness and mindfulness toward other cultures, but it is important to note that they lacked the ability to empathize and adapt, as shown by C1 students.

Theme for engineering

Sub-theme for Cluster 1: knowledge of Intercultural competence will help to understand others and create inclusive products

Students from C1 felt that being cognizant of intercultural competence is important for engineers as it will allow us to understand others. Students also felt that understanding others’ perspectives is crucial in engineering as it will help to create inclusive products. For example, Kent felt that having intercultural competence is important for engineers as being interculturally trained allows them to understand other perspectives, as individuals from different cultural backgrounds may have different perspectives on solving the same problem. Kent says, “Intercultural skills are important in engineering as people from other cultures may have seen the problem differently than me, as they are coming from a different culture with an entirely different upbringing.” Furthermore, Kent stresses that understanding others’ perspectives will help engineers to create safe, efficient, and helpful products. She says, “In the engineering field, we want to come up with the safest, most efficient, and helpful technology and this comes from considering a multitude of solutions and perspectives.”

Sub-theme for Cluster 2: intercultural competence is important to secure an engineering job

Students in C2 agreed that intercultural competence is crucial for them, they also mentioned how developing intercultural competence can help them to secure a job in the engineering field or make them an ideal candidate for a job role. For example, Ethan wants to join the aviation industry, and he feels that employers will value the intercultural skills that he has gained through the program. He says, “I think in the future, especially in the aviation industry, having intercultural competency is something valued by employers, because it shows a strong passion, curiosity, and desire to learn about other cultures.”

Similarly, Jessica also mentioned that her experience of studying abroad would be very useful when she interviews for her job, as she can mention her experiences and intercultural skills that she developed through the program. Jessica says, “I think I would communicate what I have learned to prospective employers by focusing on my ability to work in an intercultural team during an interview process”. She believes that the employer will focus more on technical questions, but in the behavioral round, she can talk about her intercultural skills. Jessica says, “Obviously, employers will want to know that I have the technical skills needed for the job, but when asked more behavioral questions, I will be able to talk about my experience abroad and in this course to demonstrate some of the intercultural skills I have gained.”

Sub-theme for Cluster 3: intercultural skills are crucial for developing good work relationships and communicating effectively

Students in C3 felt that developing intercultural skills is valuable as an engineer. Engineering as a field requires engineers to work and interact with diverse people, so having these skills will help them to develop healthy work relationships and communicate effectively at work. For example, Juno mentions, “In the workplace, especially in engineering it will translate to making good communication with co-workers a lot faster and versatility for different work locations.”

Students also acknowledged that intercultural skills are rarely seen in engineering graduates, so having those skills can set them apart as they will be able to communicate effectively with people from diverse backgrounds. For example, Elan also mentioned that intercultural competence is important for material engineering, and there are very few people with such skills. Therefore, having developed intercultural communication skills, she will be able to convince her employers about her expertise in communicating across cultures. Elan says, “Materials engineering is such a broad industry but there are very few people who have competencies in the area. Going forward I’ll be able to convince future employers of my agility in the workplace. I will be able to discuss how learning how to navigate different cultures can be applied to the working styles of different teams or coworkers and how I can adapt to their preferred style of communication”.

The themes for engineering illustrate that students for all three clusters valued intercultural competence. Students in C1 viewed intercultural competence as a lens to create inclusive products. Students in C2 and C3 perceived it as an important skill that would be valued in the workplace.

Discussion

The results of the study are important for engineering educators as the study demonstrates the importance of mentoring and reflection for the development of intercultural competence in Engineering education abroad. The results of the mean pre–post DO scores for all 74 students indicated that as a group students moved meaningfully on the IDC and developed intercultural skills. The results of the current study are consistent with other studies that focused on assessing the effectiveness of intentionally structured study abroad initiatives. For example, the study by Bittinger et al. (2022) and Davis and Knight (2022) has emphasized how intentionally structured study abroad programs can bring about changes in the intercultural competence of students. The results of Krishnan et al. (2017) and Bittinger et al. (2022) are consistent with the overall results of our study, where we also witnessed that, as whole, engineering students demonstrated a significant and meaningful gain in intercultural learning. However, a recent study by Mu et al. (2022) has argued that achieving meaningful gain on IDI after participating in a study abroad program is impacted by several programmatic and personal factors. So many aspects of a student’s experience (from where they study abroad to the language of instruction, their living situation, their social and embodied identities relative to the host culture, their preparedness for study abroad, motivations for the experience, emotional resilience, and many more) will influence their capacity to learn and grow while on study abroad. Therefore, even after an effective intervention or study abroad program has been designed, some students can still regress on the Intercultural Development Continuum. Niehaus and Nyunt (2020) have suggested that it is important to understand the study abroad experience of the students beyond the group level. Therefore, in this study, we wanted to take a deeper dive into the data and use an unsupervised clustering method to explore the data at the sub-group level. Specifically for this study, we used Ward hierarchical clustering that helped to divide the 74 students into three small groups. Further statistical analysis was conducted to understand the intercultural learning gains of the three groups. The results of the analysis revealed that students in two groups C1 and C2 demonstrated a statistically significant and meaningful increase in their DO scores. A meaningful increase in IDI scores indicates real-world intercultural learning gains for students. The last group (C3) showed a statistically significant decrease, but the regression was small enough as a whole to not be meaningful (Mu et al., 2022)—that is, while students did show some small average decrease in the DO score in this cluster, that decrease likely did not make a real-world impact on their intercultural competence except for the few students in the group (N = 8) whose IDI regression was greater than 7 points.

It is also important to note that students in all three groups started at minimization. It was interesting to observe that in the post-test, the students in C1 moved to acceptance. One C1 student in particular achieved exceptionally high gains in the post-test IDI scores, but such documented growth with the IDI in mentored semester-long study abroad is not unusual, as in the study by Jones et al. (2019) growth of more than 60 points has been observed for 0.5% of the students. C2 students moved from the lower end of minimization towards the upper end of minimization as a group, and C3 students began close to the center of minimization but regressed to the lower end of minimization as a whole. Based on the prior literature (Vande Berg et al., 2009), the increase in the DO scores for C1 and C2 students may have been shaped by multiple programmatic factors such as the duration of the study and the mentored support course they engaged in while abroad; in our case, the study abroad program was semester-long, which provided opportunity for students to interact and understand people from other cultures; consistent mentoring could be another reason, and intentionally structured curriculum also helped the students in both clusters to show gains in their DO scores after the study abroad program.

In the C3, though, we saw that students showed some decline in the DO. Prior studies (Ching, 2021) have argued that some of the potential reasons for the decrease could be because students experience culture shock while studying abroad, or they may not feel comfortable interacting with people from other cultures. Studies (Ching, 2021; Kim & Okazaki, 2014) have also indicated that students who are younger in age may find studying abroad a difficult experience. Moreover, Terzuolo (2018) has pointed out that female students have higher level of intercultural competence than male students. For C3 students we know that all the students are undergraduate students. From the demographic data, we note that for the C3 the percentage of domestic students, and male students was larger as compared to other clusters, therefore, demographic and personal factors highlighted in prior literature could be the reason for the decline in intercultural competence shown by the IDI for C3 students.

Streitwieser and Light (2018) have argued that study abroad experiences are messy, therefore capturing the student experiences and intercultural development only through a IDI survey is not enough, therefore we need to take a deeper plunge into the experiences by introducing some qualitative techniques (Davis & Knight, 2022). Moreover, to triangulate our findings and to ground our choice of methods as proposed in recent literature (see, Tamer et al., 2023), we conducted qualitative analysis of the final reflections along with assessing the intercultural learning gains using the IDI. We used inductive thematic analysis to analyze the reflection data. We also chose to perform the qualitative analysis clusterwise over the whole data set, as we intended to understand the student experiences and intercultural journey for students in each cluster and we also wanted to interpret the qualitative themes that emerge from the inductive analysis with the change in DO scores for students in each cluster. The results of the qualitative analysis were categorized into two broad categories: (i) theme for intercultural competence, (ii) theme for engineering. The theme for intercultural competence for C1 students indicated that studying abroad helped C1 students develop a deeper understanding of verbal and non-verbal communication and empathy. They were able to suspend their judgment and showed a positive attitude toward other cultures. We saw that the students did a good job of pulling themselves out of their comfort zone and adapting to new behaviors, similar findings were noticed in Mu et al. (2022) for some students who demonstrated large meaningful gains in their post-DO scores. The qualitative results for C1 align with the IDI results as students in C1 showed exponential growth in intercultural competence; they grew from lower minimization to acceptance stage on the IDC; therefore, studying abroad helped them to comprehend the cultural differences and appreciate them. At the same time, C2 and C3 students showed flexibility and openness to change their behaviors when interacting with other cultures. It is also important to note that both C2 and C3 students are in minimization based on their IDI score and minimization is the transition zone from a monocultural mindset to an intercultural mindset (Hammer, 2011; Haskollar & Kohli Bagwe, 2023). Therefore, through these themes, we are observing that students in C2 and C3 have developed skills and attitudes to navigate cultural differences and develop interculturally. Whereas the C1 students are at acceptance (intercultural mindset) and showed a deeper understanding of intercultural values, over C2 and C3 students.

Moreover, considering the dimensions of the AAC&U IKC rubrics, we also observe that C1 students focused on the behavioral dimension of the rubrics they are verbal and non-verbal communication and empathy. Whereas students in C2 and C3 focused on the affective dimension of the rubric, that is, openness. Similar results were observed in the study by Maharaja (2018), where students demonstrated openness and flexibility to adapt to other cultures after their study abroad program. C2 and C3 students did show the urge to change their behaviors and adapt to other cultures, but at the same time they lacked the full awareness of behaviors expected in other cultures. The students in both clusters focused on identifying a common ground by avoiding any biased behavior or stereotype. The qualitative analysis aligns with their stage on the IDC, both the clusters focused on the commonality, and hence they are in minimization stage as per their IDC.

The second broad category for thematic analysis was the themes for engineering. We found that students in all three clusters identified intercultural competence as a valuable skill for engineers. For instance, students in C1 focused on how being interculturally competent can help engineers understand the perspective of other engineers from various backgrounds and will help them to design inclusive and efficient products. The C2 students found intercultural competence an important skill for engineers as being interculturally competent will differentiate them from other candidates when they are interviewing for the job. The C3 students identified intercultural competence as a rare skill for engineers and mentioned that having intercultural competence can help them develop good work relationships and communicate effectively. Importantly, all of the clusters identified extrinsic, instrumental motivations for developing intercultural competence as engineers—that is, the value for this development seems mostly professional in nature and is often tied to success in the workplace. Only the C1 cluster additionally noted more prosocial or altruistic values associated with the development of intercultural competence, for example, implying that inclusive innovation is ethically valuable in and of itself, for its impact on users and communities, not only as an expected skill in their career.

The qualitative findings of the study concur with the prior studies conducted at the intersection of study abroad and intercultural competence and have used qualitative approaches to assess intercultural gains. For example, the themes that emerged in the study by Krishnan et al. (2021) are similar to our themes for intercultural competence. The study by Krishnan and colleagues showed students developed openness, showed the ability to see other’s perspectives, demonstrated tolerance toward cultural differences, and became emotionally resilient after their study abroad trip. These themes are consistent with the findings of our study, where we saw similar behavior in the students, that students in C1 were more tolerant and emotionally resilient, whereas students in C2 and C3 showed openness and ability to see other’s perspectives. Similarly, for the themes for engineering align with the study by Twyman and Knight (2016). The study used qualitative analysis to understand the experiences of undergraduate-level engineering students after their study abroad trip. The findings of the study align with our themes as engineering students in both studies learned to appreciate other cultures and were able to understand the interdisciplinary nature of engineering and the value of cultural understanding when developing complex engineering solutions. Furthermore, the principal advantage of data triangulation, specifically incorporating qualitative analysis alongside quantitative scores, is its potential to reveal unexpected patterns in the data. This approach can help to uncover insights contrary to what the quantitative assessment scores alone might suggest. For instance, for C3 students, the quantitative scores indicated a decrease in the overall post-intercultural competence, but conducting the qualitative analysis revealed that students did develop mindfulness, an attitude to adapt to other cultures, and understood the benefits of communication skills within engineering. Therefore, triangulating the findings helps us uncover intercultural growth of respondents not captured by the quantitative measures.

Implications for teaching and learning

The study was conducted in the engineering discipline but the described curriculum and the activities can be replicated for STEM students in general to help them develop interculturally. The practical implication of this study is to provide STEM faculty members with guidelines for developing a semester-long study-abroad program. The study also provides the curriculum design for creating a theoretically grounded curriculum to help support students while studying abroad. Using Kolb’s learning cycle to design the study abroad program provided a strong foundation to incorporate experiential learning and reflection components into the program. The theoretical frameworks used to design the curriculum served as a basis for developing the learning design for the students. Overall, the results of the study are interesting and important to the STEM discipline as there are very few studies in STEM that discuss the intercultural development of students through intentionally designed curricula and mentored semester-long study abroad programs.

Conclusion, limitations, and future work

The study demonstrated the effectiveness of intentionally designed study abroad programs for engineering graduates. The results of the study confirm that engaging students in a long-term study abroad initiative, constantly mentoring them, and allowing them to participate in critical reflection activities can help students develop interculturally. The study also provides a blueprint for creating effective study-abroad initiatives for STEM graduates. Finally, the cluster analysis offered unique insights. Because unsupervised statistical clustering was the primary method to explore the quantitative data, with the clusters later applied as an organizing structure for the qualitative dataset, the research team was able to compare learning artifacts of statistically clustered students to other clusters instead of simply considering all learning artifacts as single homogenous dataset. This approach yielded not only comparisons between clusters but also triangulation between quantitative and qualitative results for each cluster.

However, the study does have some limitations. The study solely focused on the reflection reports of the students to assess the intercultural gains of the students. As noted in the literature review, self-report quantitative measures are often critiqued as insufficient evidence of intercultural growth. While the mixed methods approach in this case did provide additional evidence of learning and insight into student motivations, the nature of the qualitative data included was limited by the fact that it was existing data in the form of reflective learner artifacts from the course—that is, the qualitative data also could be considered self-reported. Other approaches that could yield more direct assessment of growth would be observation of behavior or analysis of authentic dialogue; these could address the critiques of self-report data that equally apply to the quantitative and qualitative sources in this and most other similar studies.

Despite the patterns of sub-group variation revealed by the cluster analysis in this study, the results still fall short of fully explaining why some engineering students are so much more successful in developing intercultural competence than others. This is primarily a limitation of the existing data approach. With learner artifacts, for instance, analysis can only address what was collected or what the students were asked to reflect about, and the nuances of their individual experiences while on study abroad are simply not captured there. To better understand the variation among students and the relationship between who students are and how they develop, future studies should collect detailed post-study abroad data, both quantitatively, in the form of a culminating experience survey, and qualitatively, with focus groups or in-depth interviews. However, future work would benefit from comparing study abroad students who have mentored support to those who do not, or comparing engineering students who study abroad to those who participate in some other globalized learning experience.

Appendices

Appendix 1

Activity	Learning objective	Link to the activity
The name game	As a result of this activity, participants will be able to:  • Recognize the significance of names in shaping personal identity  • Explore various cultural naming conventions  • Appreciate the value of addressing individuals by their preferred names and ensuring correct pronunciation	https://hubicl.org/toolbox/tools/51/objectives
Core cultural value and cultural mapping worksheet	As a result of this reflection, students will be able to:  • Reflect on their personal and cultural values  • Identify, define, and examine value differences within and between cultures  • Start forming hypotheses about the values of the host culture	https://hubicl.org/toolbox/tools/363
Human values continuum	As a result of this activity, participants will be able to:  • Develop an understanding of the cultural values and frameworks  • Visually observe differences, even among individuals from the same cultural group	https://hubicl.org/toolbox/tools/37/objectives
Changing stereotypes to generalizations	As a result of this reflection, participants will be able to:  • Enact a bridging technique (changing a stereotype to an actionable hypothesis)	https://hubicl.org/toolbox/tools/413
Describe–interpret–evaluate (D–I–E) activity	As a result of this activity, participants will be able to:  • Distinguish between description (D), interpretation (I), and evaluation (E)  • Show that people often start with interpretation and evaluation, rather than description  • Illustrate how personal experiences and cultural values shape our interpretations and evaluations  • Practice describing, emphasizing the importance of observations	https://hubicl.org/toolbox/tools/25
Scenery, machinery, people	As a result of this activity, participants will be able to:  • Understand and articulate how we place people into categories  • Understand how empathy impacts how we form relationships  • Explain how they put people into categories in their lives	https://hubicl.org/toolbox/tools/109
Flowers’ point of view	As a result of this activity, participants will be able to:  • Exercise imaginative empathy  • Identify both possibilities and limits of empathy	https://hubicl.org/toolbox/tools/270

Appendix 2

See Fig. 5.

Fig. 5

QQ plot for pre–post data