Motivational climate predicts effort and achievement in a large computer science course: examining differences across sexes, races/ethnicities, and academic majors

Abstract

Background

The motivational climate within a course has been shown to be an important predictor of students’ engagement and course ratings. Because little is known about how students’ perceptions of the motivational climate in a computer science (CS) course vary by sex, race/ethnicity, and academic major, we investigated these questions: (1) To what extent do students’ achievement and perceptions of motivational climate, cost, ease, and effort vary by sex, race/ethnicity, or major? and (2) To what extent do the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major? Participants were enrolled in a large CS course at a large public university in the southeastern U.S. A survey was administered to 981 students in the course over three years. Path analyses and one-way MANOVAs and ANOVAs were conducted to examine differences between groups.

Results

Students’ perceptions of empowerment, usefulness, interest, and caring were similar across sexes and races/ethnicities. However, women and Asian students reported lower success expectancies. Students in the same academic major as the course topic (i.e., CS) generally reported higher perceptions of the motivational climate than students who did not major or minor in the course topic. Final grades in the course did not vary by sex or race/ethnicity, except that the White and Asian students obtained higher grades than the Black students. Across sex, race/ethnicity, and major, students’ perceptions of the motivational climate were positively related to effort, which was positively related to achievement.

Conclusions

One implication is that females, Asian students, and non-CS students may need more support, or different types of support, to help them believe that they can succeed in computer science courses. On average, these students were less confident in their abilities to succeed in the course and were more likely to report that they did not have the time needed to do well in the course. A second implication for instructors is that it may be possible to increase students’ effort and achievement by increasing students’ perceptions of the five key constructs in the MUSIC Model of Motivation: eMpowerment, Usefulness, Success, Interest, and Caring.

Introduction

The motivational climate in a course has been defined as “the aspects of the psychological environment that affect students’ motivation and engagement within a course” (Jones et al., 2022b, p. 1). Understanding students’ perceptions of the motivational climate is important because positive motivational climate perceptions have been associated with many positive outcomes, such as increased student engagement (Christenson et al., 2012; Jones et al., 2021, 2023a), and higher instructor and course ratings in undergraduate courses (Jones & Skaggs, 2016; Jones & Wilkins, 2023; Wilkins et al., 2021). The motivational climate in a course is perceived by each student individually, and therefore, students’ perceptions of the motivational climate can differ significantly in some cases (Jones et al., 2022; Lam et al., 2015; Schweig & Martínez, 2021). Furthermore, these differences in perceptions can vary by sex/gender and race/ethnicity (Harper et al., 2019; Schenke et al., 2017). Understanding differences in perceived motivational climates within courses by sex/gender and race/ethnicity could provide insights into how teaching practices can have differential effects on students’ engagement, persistence, and achievement in courses.

The purposes of this study were to examine students’ achievement and perceptions of motivational climate in a computer science (CS) course to determine the extent to which (1) students’ achievement and perceptions of motivational climate, cost, ease, and effort vary by sex, race/ethnicity, or major, and (2) the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major. Having a better understanding of how these motivational climate constructs vary by sex, race/ethnicity, and major could be useful because these constructs can be manipulated by the instructor (Hulleman et al., 2017; McGinley & Jones, 2014). Consequently, it may be possible to develop interventions that focus on certain aspects of the motivational climate to increase the motivation and engagement of the targeted groups. Targeting students in lower-level CS courses can be especially important because students’ perceptions of lower-level courses can contribute to their decision to remain in their major (Jones et al., 2014, 2016; Nelson et al., 2015). Because engineering students are often required to take these lower-level CS courses, and CS courses are sometimes located within engineering departments, the findings of this study may be relevant beyond CS to the field of engineering more broadly. Women and racially minoritized students are also a key population to target for recruitment and retention in CS and engineering because they are underrepresented in these professions and earn fewer degrees compared to represented White and Asian students (Meiksins & Layne, 2022; National Center for Science & Engineering Statistics, 2021). Ultimately, instructional strategies could be employed to engage all engineering students more deeply in the course activities to create an equitable and inclusive climate that helps to recruit and retain them within CS and engineering majors, which could lead to more employee diversity in these disciplines.

Literature review

The MUSIC Model of Motivation Theory

We used the MUSIC Model of Motivation Theory (Jones, 2009, 2018, 2020) as our theoretical framework because it has a strong theoretical and empirical basis, and it can be applied within CS and engineering courses to better understand students’ motivations and course engagement (e.g., Jones et al., 2013, 2014, 2016; Hall et al., 2013; Mora et al., 2017; Tendhar et al., 2017). The MUSIC model includes five course perceptions that comprise the motivational climate within a course: eMpowerment, Usefulness, Success, Interest, and Caring (the initial sounds of the words form the acronym MUSIC). Empowerment refers to students’ perceptions of control (e.g., choice, freedom) within their learning environment. Usefulness refers to students’ perceptions of the usefulness of the course content or activities in relation to their goals, either in the short- or long-term. The Success component focuses on the extent to which students believe that they can succeed in the course activities. Interest is the extent to which students perceive the course to be interesting and enjoyable. Lastly, Caring refers to students’ perceptions that other people in their learning environment (i.e., the instructor, other students) care about their learning and well-being. The MUSIC model was developed based on decades of motivation research in the social sciences. For example, Jones (2018, 2020) lists 28 theories that were used as evidence for different aspects of the MUSIC model, including, but not limited to, arousal theories (Berlyne, 1960; Duffy, 1957), attribution theory (Weiner, 2000), competence theories (Elliot & Dweck, 2005; White, 1959), expectancy-value theory (Wigfield & Eccles, 2000), flow theory (Csikszentmihalyi, 1990), goal theories (Ford, 1992; Locke & Latham, 2002), interest theories (Renninger & Hidi, 2015; Schraw & Lehman, 2001), self-determination theory (Ryan & Deci, 2020), self-theories of intelligence (Dweck, 1999), self-worth theories (Covington, 1992), and social cognitive theory (Bandura, 1986).

An overview of the MUSIC model is presented in Fig. 1, and it includes several elements that are important to the present study. The left side of the figure shows how variables that are external to students (e.g., instructor’s teaching strategies, ease of course, and environmental conditions) interact with those that are internal to students (e.g., cognition, affect, needs/desires, identity beliefs, personality characteristics, abilities). These external and internal variables then affect students’ perceptions of the motivational climate (i.e., MUSIC perceptions) within a course. For example, a Black female student may perceive that she can succeed in a CS course and find the course activities interesting because of a combination of external factors (e.g., she believes that the course will be relatively easy because the teacher incorporates cooperative learning and she usually succeeds in classes that use this approach) and internal factors (e.g., she generally enjoys CS activities; she believes that the knowledge she gained in prior classes will help her in this class). As a result (moving from left to right in Fig. 1), she has high perceptions of Success and Interest (and possibly eMpowerment, Usefulness, and Caring), which can lead to increased motivation and engagement/effort in the course, which may result in a high course grade. This outcome can cycle back (as depicted by the arrow from right to left at the bottom of Fig. 1) to further strengthen her ability beliefs and interest in CS. Thus, the MUSIC model is a dynamic cycle that explains how students’ motivations in a course can change over time, which can be measured in minutes or in months or years depending on the purpose of the measurement.

Fig. 1

Copyright 2018 by Brett D. Jones. Reprinted with permission

The MUSIC Model of Motivation Theory as it relates to students in a course. Adapted from “Motivating Students by Design” by B. D. Jones, 2018, p. 13.

Motivation is defined as “the extent to which one intends to engage in an activity” (Jones, 2018, p. 5); and therefore, students can be motivated to do many things, but may not engage in all of them. This possibility is depicted by the “cost/benefit decisions” rectangle in Fig. 1. For example, students may not engage in courses even when they are motivated to do so because they are more motivated by other activities (e.g., other courses, extracurricular activities).

Although it is possible for students to be motivated based on only one of the MUSIC model perceptions (such as Success), students’ perceptions of the other MUSIC model components can also affect their motivation and the motivational climate in the course. For instance, despite high success expectations, a female student can perceive that the instructor is uncaring towards female students in the class and/or that cooperative learning activities are uninteresting or useless in helping her to learn the concepts. These low perceptions of caring, interest, and usefulness interact with her high success perceptions to affect her motivation and engagement.

Relationships between MUSIC perceptions and other variables

It is impossible to statistically test the entire MUSIC model as one whole model because of the large number of variables involved and the limits of statistical analyses. Therefore, researchers have tested various parts of the model by investigating relationships among some of the variables. In CS courses, researchers have documented relationships between students’ MUSIC perceptions and their broader CS beliefs and goals, but this research is only beginning (Jones et al., 2023b). More research has been conducted within engineering courses where researchers have documented relationships among students’ MUSIC perceptions and a variety of outcomes, such as course effort and some of the internal variables shown in Fig. 1 (e.g., engineering identification). For example, when students’ MUSIC perceptions of an engineering course were included in a structural equation model with course effort, engineering identification, and other variables, all of the MUSIC perceptions predicted engineering identification (Jones et al., 2014). However, only Empowerment, Usefulness, and Interest were significantly related to course effort, and only Success was associated with estimated course grade when all of the MUSIC variables were included in the model. Subsequent studies of students’ engineering courses also documented significant associations between at least some MUSIC perceptions and students’ engineering identification (in Jones et al., 2016, all MUSIC perceptions except Interest were significantly associated with engineering identification; in Tendhar et al., 2017, Success, Interest, and Caring were significantly associated with engineering identification). These findings demonstrate that while at least some MUSIC perceptions are related to course effort and engineering identification, the significant relationships can vary as the context changes and when different variables are included in the model. For these reasons, it is important for researchers to include all five MUSIC variables in studies to ensure that the study controls for a range of students’ course perceptions.

Differences across MUSIC perceptions by sex and race/ethnicity

In this section, we highlight some of the research that has documented differences between sexes and races/ethnicities for the variables included in this study. We begin with the Success variable because students’ success expectancies (a.k.a., self-efficacy; Bandura, 1986) in a course (i.e., students’ confidence in their ability to succeed in the course) has been shown to predict their persistence, achievement, and major/career choices in CS (Beyer et al., 2003) and engineering (Jones et al., 2016; Lent et al., 1986). Some studies documented that females’ self-efficacy in CS (Pirttinen et al., 2020; Tellhed et al., 2023), engineering (Besterfield-Sacre et al., 2001; Jones et al., 2010), and STEM domains (Chan, 2022; Young et al., 2018) are lower than those of males. Yet, other studies found no differences in engineering students’ self-efficacy beliefs by sex/gender (Concannon & Barrow, 2012; Lent et al., 2005; Mamaril et al., 2016). A more consistent finding related to Success is that Asian students tend to report lower self-efficacy in courses than students of other races/ethnicities because they have a higher fear of failure and anxiety, often due to parental pressure (D’Lima et al., 2014; Ma et al., 2018; Wilson et al., 2015; Zusho et al., 2005). Finally, in one study that investigated the differences in self-efficacy among minorities in STEM, researchers found that Hispanic students reported higher levels of self-efficacy compared to White and Asian students (Wilson et al., 2015).

Related to the Caring variable, a substantial number of studies have demonstrated that female engineering students perceive a lack of inclusion or acceptance, and greater discrimination or disrespect from faculty and fellow students, which is often reflected in their low caring ratings (Brainard & Carlin, 1998; Marra et al., 2009; Vogt et al., 2007). Women of color can face a “double bind” (Malcom et al., 1976) as they experience hostility and microaggressions within the environment that are associated with their identities as a woman and as a person of color (Blosser, 2020; Cross et al., 2021). Regardless of their sex, students of color in higher education can endure emotional injury as a result of racial stereotypes and microaggressions (McGee, 2016).

Related to the Effort variable, females have reported putting forth more effort than males in engineering courses and courses in similar fields (Hawks & Spade, 1998; Vogt et al., 2007). Similarly, Asian American students tend to spend more time on homework than White American students (Mau, 1997). Other researchers studying first-generation Latina women have found that when these women attend a predominantly White institution, they attribute their academic achievement and outcomes to the effort they put forth (Gutierrez-Serrano et al., 2022). They also reported that the high personal value those students placed on being in college, and the appreciation they received from their parents were two major contributing factors for their high motivation to improve performance, achieve, and persist.

Most studies have documented that male and female students’ engineering course grades are not significantly different (Hawks & Spade, 1998; Jones, 2010). However, Asian students tend to receive higher grades than non-Asian students (Chen & Stevenson, 1995), and racially minoritized students tend to earn lower grades than Asian and White students as undergraduates (Whitcomb & Singh, 2021). Although some studies showed that Black students were more likely to report higher self-perceived abilities and confidence in learning CS than White and Hispanic students (Wang et al., 2017), other studies documented that Black students had the lowest STEM persistence rates (Pedraza & Chen, 2022), and had lower achievement in certain STEM courses compared to their White peers (Tyson, 2011).

Present study

We chose to study a lower-level CS course (CS2) that is the second course in the software engineering track of courses required for all CS majors and minors. It is the minimum requirement for acceptance to the graduate programs in CS at the participating university. Approximately 70% of the students in this course tend to be engineering majors (Hooshangi et al., 2022), many of whom intend to major or minor in CS. Some students are required to take the course for a non-CS or non-engineering major such as Discrete Mathematics, Computational Neuroscience, or Computational Modeling and Data Analytics, while others may be simply exploring the CS discipline. Consequently, this course can help to not only retain CS and engineering students in their major, but also to attract non-CS and non-engineering students to consider a minor or master’s degree in CS.

Students can enroll in this CS2 course as early as their first semester at the university if they earn computer science (CS1) credit in high school. First-year CS and engineering students in the course have a high pass rate and a low DFW rate relative to other students (Hooshangi et al., 2022). To improve the overall course climate and to recruit and retain students in CS majors and minors, we studied the motivational climate for both the students who plan to continue in the discipline as a major or minor (CS students) and those who do not plan to continue (non-CS students). We were also interested in how student demographics were related to student motivation, which would assist us in targeting course designs that can broaden participation within CS.

Within the context of this lower-level, undergraduate CS2 course, our two research questions were the following: (1) To what extent do students’ achievement and perceptions of motivational climate, cost, ease, and effort vary by sex, race/ethnicity, or major? and (2) To what extent do the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major?

Method

Author positionality

We provide this author positionality statement to increase the transparency and trustworthiness of the study (Darwin Holmes, 2020). We conducted the present study with the hopes that the findings could be used to improve the motivational climate within a CS2 course in ways that would motivate and engage all students in the course, and potentially retain and attract students to the CS discipline. Although it has been disappointing that we were unable to include all students in the present study (as explained in the following section), we believe that this study is a beginning step that can lead to other studies that do include all students. The second author was an “insider” (Herr & Anderson, 2005) because she was the instructor for the CS2 course included in the present study. The other authors were “outsiders” in that they were not associated with the CS2 course or department; however, they were employed by the same university. The first author identifies as a White male educational psychology professor, the second author as a White female CS professor, the third author as an Asian male research methodologist, and the fourth author as a White female educational psychology doctoral student. Although our perspectives are limited by our experiences and identities, we believe our diverse perspectives have strengthened this study in meaningful ways.

Participants and procedures

Participants were enrolled in a CS2 course within an engineering college at a large public university in the southeastern United States. A total of 981 students completed a survey assignment for the course and agreed to participate in the study. The survey assignment was administered in three different sections of a course that was offered in three consecutive years. Students completed the survey near the end of the course as part of their regular coursework and received full credit if they completed it, regardless of whether they agreed to participate in the study or not. This study was approved by the Institutional Review Board at the university (IRB #17-057) and students completed a consent form to participate. The number of students that consented to participate were as follows: 229 of 368 students (62.2%) consented to participate in Year 1 (Fall 2019), 395 of 550 students (71.8%) consented to participate in Year 2 (Fall 2020), and 357 of 521 students (68.5%) consented to participate in Year 3 (Spring 2021).

Students self-reported their sex as either male (n = 750, 76.5%), female (n = 225, 22.9%), or other (n = 6, 0.6%), and their race/ethnicity as White or Caucasian (not Hispanic; n = 454, 46.3%), Asian or Pacific Islander (n = 393, 40.1%), Black or African American (n = 40, 4.1%), Hispanic (n = 39, 4.0%), Native American (n = 1, 0.1%), more than one of the options provided (n = 45, 4.6%), or another race/ethnicity not provided as an option (n = 9, 0.9%). Most of the students were undergraduates, with 249 (25.4%) first year students, 506 (51.6%) sophomores, 193 (19.7%) juniors, 26 (2.7%) seniors, six master’s students (0.6%), and one doctoral student (0.1%). Students selected the major or minor that best described them using the following categories: CS major (n = 328, 33.4%), intending to be a CS major (n = 271, 27.6%), CS minor (n = 163, 16.6%), computational modeling and data analytics (CMDA) major (n = 158, 16.1%), mathematics major (n = 13, 1.3%), or another major (n = 48, 4.9%). Throughout this article, we sometimes combine some of these students to form two groups: “CS students” for those committed to the discipline (CS majors, intending to be a CS major, and CS minor) and “non-CS students” (CMDA majors, math majors, and other majors).

The six students who self-reported their sex as non-binary were not included in the analyses that involved comparing sexes because there were not enough students in this group to provide the statistical power needed for those analyses. Future studies are needed to better understand possible sex and gender identity differences within non-binary groups. We use the term “sex” in this study because we are referring to students’ sex assigned at birth. We did not collect data that would allow us to categorize students by their gender, which is a socially constructed psychological construct that refers to the roles, behaviors, activities, and attributes that are considered appropriate for women and men in society (American Psychological Association, 2014).

With respect to race/ethnicity, the one Native American student and the nine students who indicated that their race/ethnicity was not listed as an option were not included in the analyses that involved comparing races/ethnicities because there were not enough participants to provide adequate statistical power. In addition, we did not include the 45 students who reported their race/ethnicity as more than one of the options provided. It was inappropriate to place these 45 students in a group together because these students may not share similar racial/ethnic identities. Therefore, our analyses that involved comparisons for race/ethnicity included four groups of students that we refer to as: White (or Caucasian, not Hispanic), Asian (or Pacific Islander), Black (or African American), and Hispanic. We acknowledge that Asia and the Pacific Islands are composed of many different races/ethnicities and that there may be differences among students who we categorized in the Asian subgroup. However, we did not examine these differences because we did not provide students with options on the survey for different racial/ethnic subgroups within the Asian/Pacific Islander category. Future studies could examine whether it is possible to separate Asian/Pacific Islander students into meaningful subgroups based on the analyses conducted.

Course and instructor

The course was offered through the CS department and was a required course for all CS majors, all CS minors, all computational modeling and data analytics (CMDA) majors, and some math and neuroscience majors. The CS department was part of the College of Engineering, and many of the CS minors tended to major in one of the other 12 engineering disciplines offered at the university. The course was an intensive computer programming course that focused on data structures and software design. The course format varied over the years (e.g., face-to-face, online, a hybrid version that was partially face-to-face and partially online), but the categories of assignments and their percentage of the final course grade remained consistent across years. The structure of the course included content delivery, formative quizzes, short-form coding practice, weekly lab programming assignments, and five larger programming projects. The final project was designed and completed by students who worked together in teams.

The same instructor taught all three sections of this CS course. The instructor self-identified as a White female and had taught the course for four years prior to the semester in which the data were collected for the present study.

Measures

As part of the course survey, students rated items on a 6-point Likert-format scale with descriptors at each point (1 = Strongly disagree, 2 = Disagree, 3 = Somewhat disagree, 4 = Somewhat agree, 5 = Agree, 6 = Strongly agree). All of the measures described in this section (except for achievement) used the same Likert-format scale and the items on the scales were combined into one section and presented to each student in a different, random order to reduce order bias (Schwartz, 1999).

Motivational climate. Students’ perceptions of the motivational climate were assessed using the 26-item, College Student version of the MUSIC Model of Academic Motivation Inventory (Jones, 2012/2022). The MUSIC Inventory is composed of five scales that measure the extent to which students perceive that: they have control of their learning environment in the course (Empowerment scale, 5 items), the coursework is useful to their future (Usefulness scale, 5 items), they can succeed at the coursework (Success scale, 4 items), the instructional methods and coursework are interesting (Interest scale, 6 items), and the instructor cares about whether the student succeeds in the coursework and cares about the student’s well-being (Caring scale, 6 items). Example items include the following: “I have the freedom to complete the coursework my own way” (Empowerment), “In general, the coursework is useful to me” (Usefulness), “I am confident that I can succeed in the coursework” (Success), “The coursework is interesting to me” (Interest), and “The instructor cares about how well I do in this course” (Caring). The User Guide (Jones, 2012/2022) includes the complete inventory, administration instructions, and validity information. The internal consistency reliability of the scale scores was very good in other studies of undergraduate students (α ranged from 0.82 to 0.87 in Chittum et al., 2019, and from 0.91 to 0.96 in Jones & Skaggs, 2016).

Ease of the course. The three-item Ease of Course scale (Jones et al., 2021) was used to assess the extent to which students perceived the course to be easy. The items are: (1) “This course is very easy for me”, (2) “I don’t need to work my hardest to get a high grade in this course”, and (3) “In this course, I can get the grade I want with very little effort.” This scale has demonstrated acceptable internal consistency reliability in other studies with undergraduate students (α = 0.73, Jones et al., 2021; α = 0.82, Jones et al., 2022b).

Effort in the course. The four-item Course Effort scale (Jones, 2019) was used to measure the amount of effort that students believe that they are putting forth in a course. An example item is, “In this course, I put forth my maximum effort” (the complete scale is available at Jones, 2012/2022). The internal consistency reliability for the scores was good in other undergraduate courses (α = 0.93, 0.87, 0.94, 0.83, and 0.79 in Jones, 2019; α = 0.87 in Jones et al., 2021).

Cost of engaging in the course. A three-item Time Cost scale was used to measure the extent to which students did not have the time to put into the course. This scale was presented in Jones et al., (2021, α = 0.86) and was based on a scale developed by Kosovich et al. (2015). The items in the scale were (1) “This course requires too much time”, (2) “Because of other things that I do, I don’t have time to put into this course”, and (3) “I’m unable to put in the time needed to do well in this course.”

Achievement in the course. Students’ achievement was measured with their final end-of-course grade. The grade was calculated similarly for all three courses with approximately one-third of the grade based on exams, one-third based on projects, and one-third based on labs, homework, and participation. The grades are reported in this study as percentages that could range from 0 to 100%.

Analysis

We used SPSS (version 27) to compute the descriptive statistics, correlations, and Cronbach’s alpha values for all of the study variables. We used SAS (Version 9.4) for all of the other statistical analyses (i.e., MANOVAs, ANOVAs, path analyses). We set the significance level at 0.05 for all of the analyses.

To answer our first research question (To what extent do students’ achievement and perceptions of motivational climate, cost, ease, and effort vary by sex, race/ethnicity, or major?), we conducted three one-way MANOVAs to compare the means between groups. The null hypothesis in a one-way MANOVA states that there is no difference in the means of all dependent variables across groups, which is evaluated by Wilk’s Λ. When this null hypothesis was rejected (as indicated by Wilk’s Λ), we identified the source of the difference(s) by conducting a one-way ANOVA for each dependent variable. The null hypothesis in a one-way ANOVA states that there is no difference in the means of the dependent variable across groups, which is evaluated by the F test. When this null hypothesis was rejected, we identified the source of difference(s) by conducting multiple comparisons in which we used the Tukey–Kramer adjustment to account for the unbalanced cell sizes and to control the overall Type I error rate (Kramer, 1956; Tukey, 1953).

To answer our second research question (To what extent do the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major?), we conducted a path analysis using the model shown in Fig. 2. We included a path from Success to Grade in the model because although students who believe they can succeed may put forth more effort (as depicted by the arrow in Fig. 2 from Success to Effort), students’ success expectancies may also directly affect their grades without increasing or decreasing their effort. For example, this may occur when students have high success expectancies but do not need to put forth high effort because they believe the coursework is easy, perhaps because they already have many of the abilities needed to attain a high grade. The covariances between the five MUSIC variables and the cost variable were all freely estimated because they are exogenous variables. We fit the model using students who did not believe that the course was easy because when students believe that the course is easy, they do not need to put forth effort (Jones et al., 2021). Students who rated course ease less than 4.0 were identified as those who did not believe that the course was easy (n = 801; 82% of the participants). For parameter estimation, we used the MLSB/MLM estimation method (i.e., maximum likelihood parameter estimates with standard errors and a mean-adjusted Chi-square test statistic) because it can accommodate data from nonnormal distributions and generate scaled fit indices (Satorra & Bentler, 1994).

Fig. 2

Part of the MUSIC Model tested in this study

To compare the results across sex (females and males), race/ethnicity (White, Asian, Black, and Hispanic), and major (CS majors, CS minors, CMDA major, and Other majors), we conducted multi-group analyses. Within each comparison (e.g., comparison between sexes), we first fit an unrestricted model in which the path model in Fig. 2 was fitted to all the groups (e.g., the female group and the male group) simultaneously and all the parameters were freely estimated. Then, we fit a restricted model, in which eight path parameters (i.e., the five MUSIC variables to Effort, Cost to Effort, Success to Grade, and Effort to Grade) were equated between the groups (e.g., the female group and the male group) and all other parameters were freely estimated. Finally, we computed the scaled Chi-square difference (Satorra & Bentler, 2010) to compare the unrestricted and restricted models.

Results

Descriptive statistics, correlations, and reliabilities

We provide the descriptive statistics for all of the study variables in Table 1. The means for students’ MUSIC perceptions were fairly high and ranged from 4.3 to 5.2. The mean for Cost was about average (M = 3.4), the mean was below average for Ease (M = 2.8), and the mean was fairly high for Effort (M = 4.8). The grades ranged from failing (54.4%) to almost perfect (99.3%) with a mean value of 87.9% (SD = 6.8). The skewness and kurtosis values were acceptable for all of the variables (|< 2|) except for Usefulness and Caring, which had slightly higher kurtosis values of 3.02 and 2.23, respectively, because some students rated both of these constructs with the highest scale value.

Table 1 Correlations, Cronbach’s alpha values, means, and measures of distribution for the variables

We determined that the internal consistency reliability, as assessed by Cronbach’s alpha (α), was good for all of the scales (George & Mallery, 2019) and ranged from 0.85 to 0.90 for the MUSIC Inventory scales. The MUSIC variables were correlated moderately (r ranged from 0.45 to 0.67), which is typical for undergraduate samples of students (e.g., Jones & Skaggs, 2016; Jones et al., 2021).

Results for Research Question 1

We conducted three one-way MANOVAs to answer Research Question 1 to determine the extent to which students’ achievement and perceptions of motivational climate, cost, ease, and effort varied by sex, race/ethnicity, or major. All three of the MANOVA analyses were statistically significant (for sex, Wilk’s Λ = 0.910, p < 0.001; for race/ethnicity, Wilk’s Λ = 0.888, p < 0.001; for major, Wilk’s Λ = 0.830, p < 0.001). Given the significant results from all three MANOVAs, we examined the one-way ANOVA results for each dependent variable and presented the results in Table 2. Compared to males, females reported lower values for Empowerment, Success, and Ease, and higher values for Cost and Effort. These differences (and those for race/ethnicity and major) are explained in detail in the “Discussion” section.

Table 2 Means, standard deviations, and one-way ANOVAs for the study variables by sex

Table 3 includes the results of the one-way ANOVAs by race/ethnicity and shows the comparison by variable for White and Caucasian (not Hispanic), Asian or Pacific Islander, Black or African American, and Hispanic students. We documented differences between the four groups for Empowerment, Success, Interest, Cost, Ease, and Grade. Hispanic students reported higher Empowerment than White and Black students. White students had higher Success expectancies than Asian students. Asian and Hispanic students reported higher Interest than White and Black students. Asian students reported higher values for Cost than White students, and lower values for Ease than White students. Black students obtained lower grades than White and Asian students.

Table 3 Means, standard deviations, and one-way ANOVAs for the study variables by race/ethnicity

To check for a possible interaction between sex and race/ethnicity, we also conducted a two-way MANOVA with sex, race/ethnicity, and their interaction. The MANOVA test indicated that the interaction effect between sex and race/ethnicity was not significant (Wilk’s Λ = 0.964, p > 0.05).

The results of the one-way ANOVAs by academic major are provided in Table 4. We combined current CS majors with students intending to become CS majors because we assumed that they would have similar motivations and goals. We also created an “Other” group that consisted of all students who were neither CS majors, CS minors, nor CMDA majors. We identified differences between the groups for all of the variables except Effort, which was fairly high for all groups (M ranged from 4.6 to 4.8). The grades were also similar across groups except that the CMDA majors earned lower grades than the other three groups. As a trend, CS majors reported higher MUSIC perceptions than the other groups, although they were similar to CS minors for Success, Interest, and Caring. The CS minors reported values similar to the non-CS students (i.e., the CMDA majors and the “Other” group) for many of the variables. The two groups that comprised the non-CS students (i.e., the CMDA majors and the “Other” group) were not statistically significantly different on any of the variables except that the CMDA majors earned lower grades. Table 5 presents a summary of all the findings related to Research Question 1.

Table 4 Means, standard deviations, and one-way ANOVAs for the study variables by major

Table 5 Summary of findings for differences in the variable means by sex, race/ethnicity, and major

Results for Research Question 2

We began answering our second research question (To what extent do the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major?) by fitting the model in Fig. 2 to data from the 801 students who did not perceive the course to be easy. The standardized path estimates are provided in Fig. 3.

Fig. 3

Standardized path estimates between the study variables for 801 students

The commonly reported fit indices are presented in Table 6 in the “All” column and indicated good to excellent fit of the model because the SRMR and RMSEA were less than 0.08, and the CFI and NNFI/TLI were greater than or equal to 0.95 (Hu & Bentler, 1999; Kline, 2013). The MUSIC variables and cost explained 16% of the variance in Effort, whereas Effort and Success explained 19% of the variance in Grades.

Table 6 Fit indices for the path analyses

For all three multi-group analyses (i.e., sex, race/ethnicity, and major), the fit indices indicated adequate fit of all the unrestricted models, and the scaled Chi-square difference tests indicated no significant difference between the unrestricted and restricted models. These findings indicate that the model in Fig. 2 does not vary by sex, race/ethnicity, or major.

Table 7 provides the unstandardized and standardized estimates for the path parameters in the model shown in Fig. 2 and for the restricted models for sex, race/ethnicity, and major. The significant path estimates in all three restricted models (i.e., for sex, race/ethnicity, and major) were the same paths as those that were significant in the model that included all students: Empowerment to Effort (except for the race/ethnicity model in which case it is close to significance at p = 0.066), Usefulness to Effort, Interest to Effort, Effort to Grade, and Success to Grade. All of these significant path estimates were positive as anticipated, except that the path between Empowerment and Effort was negative. The path estimates from Success to Effort, Caring to Effort, and Cost to Effort were not statistically significant.

Table 7 Unstandardized and standardized estimates for the paths between the variables

Discussion

RQ1: variations by sex, race/ethnicity, and major

Our first research question asked: In an undergraduate CS course, to what extent do students’ achievement and perceptions of motivational climate, cost, ease, and effort vary by sex, race/ethnicity, or major? The answer to this question is important because if differences between groups exist, they may create a motivational climate that is unmotivating or unengaging to some students, which could lead to course conditions that are inequitable or not inclusive of all students. It is important to note that some of the differences we documented are relatively small; therefore, we do not want to exaggerate the possible importance of those findings. For example, although Asian students reported statistically significantly higher scores for Interest than White students, the practical significance may not be overly meaningful given that the means only differed by 0.2 on the six-point scale (the mean was 4.5 for Asian students and 4.3 for White students). Therefore, we caution readers not to make hasty generalizations from all of these findings, and rather, to consider whether the differences are practically meaningful. As an example of findings that are fairly substantial and may have practical significance, consider the differences in Grades. The Black students obtained significantly lower grades (M = 83.6%) than the White (M = 88.1%) and Asian (M = 88.0%) students. Understanding the sources of these differences could lead to solutions that could help Black students to obtain higher grades.

Sex

We first examined the extent to which females and males perceived the motivational climate (i.e., the MUSIC constructs) in the course differently. We found that both sexes had similar perceptions of Usefulness, Interest, and Caring. This finding indicates that there are many similarities in how the overall motivational climate in this course is perceived by females and males. The differences in MUSIC perceptions were that females reported lower values for Empowerment and Success. Other differences were that females reported the course was harder, they put forth more effort, and the time cost was higher than males. The final grade earned by females was not statistically significantly different from males, which is consistent with the findings of previous studies (Huang & Brainard, 2001; Takahira et al., 1998).

The fact that females reported lower Success expectancies than males indicates that females were less likely to perceive that they could succeed in the course. These findings are consistent with the results of other studies that have documented lower Success expectancies (self-efficacy) for females (e.g., Besterfield-Sacre et al., 2001; Jones et al., 2010; Marshman et al., 2018; Pirttinen et al., 2020; Young et al., 2018). Combined with the results for Ease, Effort, and Cost, these findings provide a picture that females do not expect to do as well as males because they find the course more difficult, which requires them to put forth more effort and takes more time. These struggles could lead to many undesirable outcomes, such as having less time available for other coursework and non-academic activities, and could cause females to question whether they are “good enough” to succeed in CS and pursue a CS career (Margolis et al., 2000). These findings could also shed some light on the types of psychological struggles that female students are more likely to endure, such as anxiety and depression, which can drastically impact their persistence (Seymour, 1995).

The fact that the lead instructor of the courses was a woman may have had a positive effect on women’s perceptions because role models can influence female students’ attitudes towards engineering (Casad et al., 2018; Wallace & Haines, 2004) and other STEM fields (Stout et al., 2011). Future studies could compare how students’ perceptions of the motivational climate differ in courses taught by men.

Race/ethnicity

Differences by race/ethnicity varied somewhat across the five MUSIC perceptions, but we identified a few patterns that we present in this section in the order of the size of the participant groups in this study from largest to smallest (i.e., White, Asian, Black, Hispanic). White students reported higher values than Asian students for Success and Ease, and lower values than Asian students for cost. White students also obtained higher grades than the Black students. Finally, the White students reported lower values on Empowerment than Hispanic students and lower values on Interest than the Asian and Hispanic students. We discuss all of these findings within the context of the other race/ethnic groups in the following paragraphs.

The findings that Asian students reported lower Success expectances and higher time Cost than White students are consistent with other studies in which Asian students have reported lower Success expectations than other students (D’Lima et al., 2014; Wilson et al., 2015). For instance, in studies of college students, Asian American students displayed significantly higher levels of anxiety than their Anglo-American counterparts because they feared that their failure could result in severe consequences (Steinberg et al., 1992; Zusho et al., 2005). Asian American students can feel pressure from their parents’ high academic standards (D’Lima et al., 2014; Steinberg et al., 1992). In some studies, this increased pressure and fear of failure has led Asian American students to put forth increased effort and achieve higher than other students (Ma et al., 2018). However, that was not the case in the present study. Asian students put forth the same amount of effort as the White, Black, and Hispanic students, and they earned the same final grades as the White and Hispanic students.

One possible reason that Asian students may report higher time Cost than White students is that English is the second language for some of these students; and therefore, it may take them more time to comprehend and produce materials in English. Campbell and Li (2007) noted that in New Zealand, Asian students faced language barriers that caused them to spend a lot of time preparing for class, looking up new words in the dictionary, and improving their English language skills. We did not ask students if English was their second language, but future studies could include this variable to better understand how language may be a barrier to students’ success. It is also possible that there are other differences between the Asian and White students that led the Asian students to report higher time Cost, such as they enrolled in more courses or were engaged in more extracurricular activities. Future students could also examine these possibilities.

Black students tended to respond similarly to students in one or more of the other racial/ethnic groups. The exceptions were that Black students reported lower Empowerment than Hispanic students, and lower Interest than Asian and Hispanic students; and Black students obtained lower grades than White and Asian students. Black students were not statistically different from White students on any of the motivation-related variables, they only differed in that their grades were lower. This finding is consistent with previous studies which reported that Black students as a racially minoritized group tended to have lower grades (Whitcomb & Singh, 2021) and lower achievement than their White and Asian peers (Tyson, 2011). One reason for this trend could be that, on average, Black students did not have the prerequisite skills to do as well in the course because K-12 schools that serve high populations of Black students lack accessibility to quality and advanced learning experiences in and exposure to STEM (London et al., 2021).

One consistent finding was that the perceptions of the Hispanic students were as high or higher than any of the other three races/ethnicities for all five MUSIC variables. Hispanic students reported higher Empowerment than White and Black students and higher Interest than White and Black students. Hispanic students reported values for Cost, Ease, and Effort that were similar to all three of the other racial/ethnic groups, and their grades were not statistically different from the other three groups. These findings may be attributed to a variety of factors. Hispanic students may have reported higher values for the five MUSIC variables because, compared to White and Asian students, Hispanic students in other studies have reported more positive perceptions of CS, have been more likely to report that they would have a job in which they would need to know CS (i.e., CS is useful to them), and have been more likely than White and Black students to report that they were very interested in learning CS (Wang et al., 2017).

Major

The CS majors reported higher values than the non-CS students on all of the MUSIC variables except Empowerment (Empowerment was the same for CS majors and the “Other” group), whereas the mean values for the CS minors tended to be between the CS majors and the non-CS students. The CS majors (and the CS minors to a lesser extent) had chosen the CS field as a focus of their studies and likely have an affinity for programming and an interest in software development. All CMDA students were required to enroll in this course and some of them may be interested in some aspects of the course (e.g., modeling and analysis) but not other aspects (e.g., software development). Compared to CS students, the CMDA students may have less experience and confidence in their programming skills. Because this course is focused on building programs, it is likely to be more appealing to CS students who tend to be interested in designing, building, and creating.

Typically, the CMDA majors reported lower values for the MUSIC variables than the other students and they received lower grades than the other students. Because they reported putting forth a similar amount of effort as the other groups, it is possible that they had less prior knowledge and skills related to the topics in this course. Because the CMDA students are not in the College of Engineering and the requirements for admittance into the CMDA major is less rigorous, these students may have a history of lower academic achievement overall compared to the engineering students. In addition, the CMDA students may have had less programming experience and been less prepared to manage a CS course with a heavy workload and firm deadlines. Furthermore, although some CMDA majors had completed a previous course that taught the Java knowledge needed in the current course, many CMDA majors enrolled in a 2-course sequence in Python that was designed to support their academic path, but may not have prepared them adequately for the present course. Consequently, some CMDA students may need a stronger support system and preparation than was available.

RQ2: relationships between MUSIC perceptions, cost, effort, and achievement

Our second primary research question was the following: In an undergraduate CS course, to what extent do the relationships between students’ achievement and perceptions of motivational climate, cost, and effort vary by sex, race/ethnicity, and major? We tested the model in Fig. 2 and the results provided evidence that the model was valid for use in this CS course. That is, students’ MUSIC perceptions (with the exception of Empowerment, as discussed in the following paragraph) and cost were positively related to their effort and their effort was positively related to their achievement. These findings add to the evidence produced in other studies of engineering courses that demonstrate that students’ MUSIC perceptions are related to their effort and achievement (e.g., Jones et al., 2014).

Although all of the MUSIC components were significantly positively correlated with Effort (see Table 1), when all five MUSIC components were included in the model shown in Fig. 2, only Usefulness and Interest were positively significant predictors of Effort, whereas Empowerment was negatively predictive of Effort. These findings occurred because the MUSIC components are correlated, and only when they are all included in the model at the same time is it possible to determine which variables are the best predictors of Effort. Usefulness and Interest were also found to be significant predictors of course effort in a study of students in an engineering course that included all of the MUSIC variables in the model at once (Jones et al., 2014). These findings in engineering courses are somewhat different from those in studies of undergraduate students in non-engineering courses. As examples of studies that included all of the MUSIC variables in the model, Interest and Caring were significant predictors of Effort in an undergraduate geography course (Jones et al., 2021), and Success and Interest (and to a lesser extent, Empowerment and Caring) were significant predictors in undergraduate language courses (Jones et al., 2023; Li et al., 2022). Future research could examine whether Usefulness and Interest are the most significant predictors of Effort in all types of engineering courses or if the predictors vary by engineering course.

We are not sure why Empowerment negatively predicted Effort in the path analyses (except in the race/ethnicity model). One plausible explanation is that the part of the variance for the Empowerment construct that predicts Effort overlaps with the Usefulness and Interest constructs, and the remaining part of the variance for the Empowerment construct is negatively related to Effort. Empowerment refers to the amount of autonomy and control that students have within the course. Having some autonomy may increase students’ perceptions of interest in and usefulness of the course topics and activities. However, this autonomy may also allow students to use strategies that require less Effort, in which case increased Empowerment could lead to decreased Effort. These speculations need to be examined more thoroughly in future research to understand the role of Empowerment in relation to Effort and the other MUSIC variables.

The relationships between the variables depicted in Fig. 2 were consistent across sex, race/ethnicity, and major, which provides evidence that the part of the MUSIC model shown in Fig. 2 can be used with a variety of CS and non-CS students. This adds to a growing body of research supporting the relationships between variables in the MUSIC Model of Motivation Theory (e.g., Jones, 2010, 2019; Jones et al., 2021). The fact that time Cost was not a significant predictor of Effort is inconsistent with the MUSIC model theory because Cost is considered to be one of the variables that can affect students’ motivation and engagement in a course (see Fig. 1). For the students in this course, other factors are more important than time Cost in students’ decisions to put forth effort. It may be that students view this course as one of their first important courses in CS; and therefore, they want to put forth the effort needed to get a high grade regardless of the time Cost involved.

Limitations

The findings of this study must be interpreted within the study limitations. First, the course included in this study may not be representative of all CS courses. However, courses similar to this course are offered at most public universities that offer engineering and/or CS degrees. Second, we were unable to include students who self-identified as more than one race/ethnicity option provided (because we could not assume that these students would form a homogenous group) or as another race/ethnicity not provided (because they did not report their race/ethnicity). Furthermore, we were unable to include any students who were either not White or Asian, or non-binary in the path analyses because there were not enough students in these groups to provide the statistical power needed to conduct robust analyses. Future studies could include enough students from a variety of races/ethnicities and sexes/genders to permit statistical comparisons across these groups. Third, because students were surveyed near the end of the course, students who dropped out of the course earlier in the semester were not included in the study. It is possible that students who dropped out of the course had different course perceptions and grades than those who completed the course. Finally, students in Year 2 and Year 3 completed the surveys during the COVID-19 pandemic, which may have caused undue stress on the instructor and students, thus limiting the generalizability of these findings to courses during non-pandemic years.

Implications and conclusion

One implication of this study for instructors is that females, Asian students, and non-CS students may need more support, or different types of support, to help them believe that they can succeed in CS courses. On average, these students were less confident in their abilities to succeed in the course and were more likely to report that they did not have the time needed to do well in the course. Because the female and Asian students earned grades that were statistically similar to the other students, there does not seem to be a problem with respect to how much students are learning and achieving, except for the CMDA students. However, having lower success expectancies could lead these students to question whether they want to persist in CS-related disciplines (Bandura, 1986). Asian students may have external pressure from parents to motivate them to continue in CS-related disciplines (Ma et al., 2018), and non-CS students may not need to have high CS success expectancies to succeed in their major. However, female CS and engineering students may question their CS ability and begin to think that CS is not a good career fit for them, which could lead to them losing interest or disidentifying with CS (Jones et al., 2015; Margolis et al., 2000). Thus, having a lower course self-efficacy could be especially detrimental to females. In addition, if non-CS majors (e.g., other engineering students) have lower success expectancies, they may be less likely to consider pursuing a minor or graduate degree in CS. And, even with parental or other external pressures to continue in CS, Asian students may benefit from higher success expectancies. By being aware of students’ lower success expectancies, instructors may be more likely to identify and remediate situations or issues in ways that can help these students believe they can succeed, resulting in a more equitable and inclusive learning environment. Further research could examine whether increasing students’ success expectancies can lead to positive outcomes besides increased achievement.

A second implication for instructors is that it may be possible to increase students’ effort and achievement if they increase students’ MUSIC perceptions. Studies indicate that students (regardless of their sex/gender, race/ethnicity, or background) are likely to take more CS courses when they have excellent instructors who use pedagogically sound practices (Beyer, 2014). In the present study, students’ perceptions of Usefulness and Interest were the most significant predictors of their Effort, which predicted their grades. In addition, Success predicted course grades directly. Although correlation does not imply causation, it is reasonable to suspect (until experimental studies can be conducted) that increasing students’ Usefulness perceptions and Interest in the course content and activities could lead to increased effort. Or, considered from a different vantage point, poor course designs can decrease interest in science and mathematics courses, especially for women (Seymour, 1995). Many different strategies are available to instructors to increase usefulness perceptions (e.g., connecting the course activities to the “real-world” and students’ future goals [Spence et al., 2022]) and situational interest (e.g., incorporating novelty, piquing students’ curiosity, stimulating emotional arousal, varying activities [see Jones, 2018]). A variety of strategies can be used by instructors to increase students’ success expectancies, including those that help them to succeed (e.g., divide long assignments into shorter, more manageable assignments) and those that provide supportive verbal feedback (Bandura, 1986). Another benefit to increasing students’ MUSIC perceptions is that these perceptions are highly predictive of the instructor and course ratings provided by students (Jones et al., 2022b; Wilkins et al., 2021), and these ratings are often important to faculty because they can affect personnel decisions, such as annual reviews, merit raises, and promotion decisions (Stroebe, 2020).

Compared to CS majors, non-CS students reported that the course was less useful and interesting to them, which could lead to less effort in the course given that Usefulness and Interest were significant predictors of Effort. It may be possible to increase students’ effort and achievement (especially for the CMDA majors) by identifying teaching strategies that could be used to improve their perceptions of Usefulness and Interest. Doing so may also increase the likelihood that they would choose CS as a minor, as students studying other disciplines who perceive CS to be useful to their disciplines are more likely to be attracted to CS (Carter, 2006).

A final implication is that CS instructors can assess students’ MUSIC perceptions of the motivational climate using the MUSIC Inventory. In the present study, we used the MUSIC Inventory results to examine the motivational climate within a class and within subgroups (e.g., females vs. males). However, recent studies have demonstrated that these data can also be used to assess individual students’ perceptions, which could provide further insight into the variety of perceptions within the class (Jones et al., 2022a). Analyzing data at the individual level could help instructors to provide more equitable and inclusive learning experiences to all students. Data can also be collected over time within the same course to provide instructors with feedback about how different course activities affect the motivational climate.

In sum, the results of this study are potentially useful to instructors, course designers, and administrators interested in creating equitable motivational climates within courses. In this study, a positive motivational climate was associated with increased student effort and achievement across sex, race/ethnicity, and academic major. By attending to the various aspects of the motivational climate (i.e., students’ perceptions of empowerment, usefulness, success, interest, and caring), instructors can also increase the likelihood that their course is equitable and inclusive of all students. We hope that this study will inform future studies and that CS and engineering departments will take these findings into account in their efforts to expand and diversify their fields and attract and retain women and racially minoritized students in STEM fields.