This section is subdivided based on the outcomes of the development of the CICS and the EBIP Adoption Scale, including subsections on descriptive outcomes, CICS factor analysis results, climate and adoption scale results considered together, factor analysis results, and analyses based on select demographic variables. A discussion of each of the outcomes is included here for clarity, followed by a Conclusions section.
Descriptive outcomes for the CICS and EBIP adoption scale
For the overall means and standard deviations for all of the CICS survey items, see Tables 1, 2, and 3. Note that for the first two sections of the CICS, each item was answered on a seven-point semantic differential scale, with the left-most response coded as 1 and the right-most response coded as 7. For example, for the item “I believe that the campus culture (‘does not value teaching’ to ‘values teaching’),” a lower score means that faculty responses were closer to the left-most “does not value teaching” anchor (1), and a higher score means that faculty responses were closer to the right-most “values teaching” anchor (7), with an exact midpoint at 4.0. For this particular item, the mean response value was 5.22 (SD = 1.5), meaning that across all faculty respondents, on average, they tend to believe that the campus culture values teaching. With regular and meaningful measurement, answers to particular items can be helpful. For instance, observing relatively high values on the initial measurement can inform researchers that the current campus climate on a particular issue is highly positive; given this observation, efforts to significantly increase perceptions may be difficult due to ceiling effects.
The descriptive outcomes for the EBIP adoption scale responses consist of scale scores and how they map onto Dormant’s (2011) CACAO change model adoption stages. For these results, see Additional file 1: Table S1. This type of measure could be particularly valuable over time, as shifts in departmental culture can be tracked based on the distribution of faculty across different stages of EBIP adoption.
Factor analytic outcomes for the CICS
All responses to the 28-item CICS, items were subjected to exploratory factor analysis using a varimax rotation, eigenvalues > 1.25, and factor loadings > 50. A five-factor solution emerges explaining 54.1% of the variance.
The theme that emerges for factor 1 (items 14, 2, 8, and 15; see Tables 1, 2, and 3) is the free choice of teaching methods, which involves the encouragement of using new teaching methods as well as collaborative discussions; inter-item reliability using Cronbach’s α = .797. The higher the factor 1 score, the greater the belief that the free choice of teaching methods exists. The theme for factor 2 (items 27, 28, 26, and 25) is institutional support, meaning that there is adequate support for teaching, assessment, evaluation, and the availability of physical, flexible spaces for teaching; inter-item reliability using Cronbach’s α = .805. The higher the factor 2 score, the greater agreement that there is institutional support for teaching. The theme for factor 3 (items 10, 4, and 23, with reverse coding for item 10) is teaching-research balance, including the relative valuing of teaching and research in hiring as well as promotion and tenure decisions; inter-item reliability using Cronbach’s α = .759. The higher the score for factor 3, the more that teaching is valued over research, including hiring and promotion and tenure decisions. The theme for factor 4 (items 9 and 12, with both items reverse-coded) is the encouragement to use evidence-based instructional practices, especially as related to assessing student learning outcomes; inter-item reliability using Cronbach’s α = .619. The higher the score for factor 4, the greater the belief that the campus climate encourages the use of evidence-based instructional practices. Lastly, the theme for factor 5 (items 22 and 6, with item 6 reverse-coded) is teacher connectedness, involving the connections and conversations with teaching colleagues; inter-item reliability using Cronbach’s α = .615. The higher the score for factor 5, the greater connectedness with teaching colleagues, especially as related to teaching discussions. Even though the inter-item reliabilities are low for factor 4 and factor 5, they were retained here for explanatory purposes.
Combination of climate and adoption stage: CICS factor scores and EBIP adoption scale outcomes
Scores from the five CICS factor scores were correlated with EBIP adoption scale scores. Due to five correlation coefficients being generated, a Bonferroni correction was employed to minimize family-wise error. The resulting p critical value (p
crit) is .01. EBIP adoption scale scores are significantly correlated with (a) factor 1 (the free choice of teaching methods), r(531) = .13, p = .004; (b) factor 3 (teaching-research balance), r(531) = −.18, p < .001, (c) factor 4 (encouragement to use evidence-based instructional practices), r(528) = .14, p = .002, and (d) factor 5 (teacher connectedness), r(531) = .22, p < .001. What does this mean? The higher the self-reported stage on the EBIP adoption scale (a) the greater the perception of free choice in teaching, (b) the greater the weighting of teaching in considering teaching-research balance, (c) the greater the perceived encouragement on campus to use evidence-based instructional practices, and (d) the more connected the faculty member feels to other teachers on campus.
Select demographic variables as related to CICS scores
For all of the CICS-related analyses in this section, the Bonferroni correction was used for the five comparisons, resulting in p
crit = .01.
Age
Answers to the items which comprise factor 1 (the free choice of teaching methods) were significantly correlated with age, r(493) = .15, p = .001. Younger faculty reports greater freedom to select the teaching method of their choice. Answers to the items which comprise factor 3 (teaching-research balance) were significantly correlated with age, r(493) = −.12, p = .008. With the negative correlation, younger faculty members report their belief that research is valued over teaching in the teaching-research balance.
Teaching workload
Respondents were asked to report the approximate percentage of their workload that involves teaching. There is a significant correlation between responses to factor 3 (teaching-research balance) and responses to the teaching workload item, r(526) = .13, p = .002. Faculty members reporting higher workload percentages for teaching perceive teaching is more valued in hiring decisions and promotion and tenure decisions.
Tenure/tenure track vs. non-tenure track
When the responses are compared between tenure/tenure-track faculty and non-tenure-track faculty, significant differences emerge for two CICS factors: (1) tenure/track faculty (mean = 3.18, SD = 0.9) score significantly lower than non-tenure-track faculty (mean = 3.53, SD = 0.8) on factor 2 (institutional support), t(526) = −4.76, p < .001 and (2) tenure/tenure-track faculty (mean = 2.58, SD = 1.3) score significantly lower than non-tenure-track faculty (mean = 3.45, SD = 1.2) on factor 3 (teaching-research balance), t(526) = −7.89, p < .001. Tenured/tenure-track faculty believe there is less institutional support for teaching compared to non-tenure-track faculty, and tenured/tenure-track faculty believe that research is more valued over teaching as compared to the balance perceived by non-tenure-track faculty.
Office on campus
For the CICS factor scores, there were three significant differences in answers between those individuals with an office on campus and not having an office on campus: (1) individuals with an office (mean = 3.33, SD = 0.8) scored significantly lower than individuals with no office (mean = 3.68, SD = 0.8) on factor 2 (institutional) support, t(530) = −4.07, p < .001; (2) individuals with an office (mean = 2.96, SD = 1.3) scored significantly lower than individuals with no office (mean = 3.73, SD = 1.1) on factor 3 (teaching-research balance), t(530) = −5.91, p < .001; and (3) individuals with an office (mean = 5.00, SD = 1.3) score significantly higher than individuals with no office (mean = 4.52, SD = 1.4) on factor 5 (teacher connectedness), t(530) = 3.64, p < .001. When answers to the office on campus item are compared to academic status (non-tenure track vs. tenure/tenure track), there is a significant association in the pattern of answering these two items, X
2(1) = 75.99, p < .001; 98.5% of tenure/tenure-track faculty members have an office on campus compared to 66.1% of non-tenure-track faculty members.
Faculty members with an office on campus actually believe that there are fewer institutional resources for teaching compared to those faculty without offices on campus. Faculty members without an office believe that the institution values teaching over research more than faculty members with an office. Lastly, faculty members with an office report greater connectedness to other teachers on campus compared to those faculty without offices on campus.
Gender
There were no significant differences between male and female responses on each of the five CICS factors.
Demographic variable relationships with EBIP adoption scale scores
EBIP adoption scores were significantly correlated with answers to the item about the percentage of workload involving research, r(367) = −.12, p = .027; with the negative correlation coefficient, the less workload involving research, the higher the EBIP adoption score. There is a significant difference between tenure/tenure-track faculty (mean = 3.82, SD = 2.0) and non-tenure-track faculty (mean = 3.42, SD = 2.2) on their EBIP adoption scores, t(526) = 2.08, p = .038; tenure-track faculty report significantly higher EBIP adoption scores. There is a significant difference in answers for those individuals with an office (mean = 3.75, SD = 2.1) and those individuals who do not have an office (mean = 2.91, SD = 2.3) on EBIP adoption scores, t(530) = 3.84, p < .001; those with an office report higher EBIP adoption score. Also, there is a significant difference between females (mean = 3.86, SD = 2.1) and males (mean = 3.18, SD = 2.1) on EBIP adoption scores, t(498) = −3.51, p < .001; females report higher EBIP adoption scores than males.
EBIP departmental profiles
With the existence of individually based EBIP adoption scores, departmental profiles can be created to depict the climate or culture within a department concerning the adoption of evidence-based instructional practices. There are strong advocates for changes in STEM education (Freeman et al., 2014; Wieman 2014), and utilizing EBIP departmental profiles for STEM departments could provide a new measure of assessing the landscape. Following the calculation of EBIP scores in the current study, departmental profiles were created for each of the STEM departments under study, see Fig. 1 for examples of STEM department profiles. By reviewing the departmental profiles such as in chemistry or computer science, campus leaders interested in the transformation of both faculty practice and institutional climate may realize that a one-size-fits-all approach in encouraging faculty members to adopt evidence-based instructional practices will likely not work. For instance, multiple strategies for EBIP adoption are needed in Chemistry due to the diversity of scores on the EBIP adoption scale (Fig. 2). However, campus leaders might decide to provide more resource-intensive support to Computer Science since the bulk of respondents are already EBIP adopters.
It is clear to see that different faculty members are aligned at different points on the EBIP Adoption Scale; thus, strategies for those individuals at the awareness stage should be different than the strategies needed for those in the mental tryout or adoption stages. Department profiles could be a powerful source of information for campus leaders in determining tipping points for localized, grassroots efforts to affect teaching practices.