Developing and evaluating a pollination systems knowledge assessment in a multidisciplinary course

Abstract

Background

Although pollinators play an integral role in human well-being, their continued global decline reflects the need to provide and evaluate general pollinator knowledge to promote their conservation. Enhancing learners’ understanding of the complexity inherent in pollination systems within the science classroom may help them make more informed decisions regarding pollinator conservation actions. By measuring conceptual understanding of pollination systems, science educators can identify learners’ knowledge needs and inform their teaching in science classrooms. Based on previously developed theoretical frameworks describing pollination systems knowledge, we created and evaluated a new instrument to assess pollination systems and conservation actions knowledge. The Pollination Systems Knowledge Assessment (PSKA) is a multiple-true–false instrument containing 18 question stems and 70 accompanying T–F items encompassing three organizational components of pollination knowledge regarding (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and pollinator conservation.

Results

We refined the PSKA based on expert discussions, think-aloud interviews, and pilot testing before and after presenting a wild pollinator conservation unit within a postsecondary science literacy course. The PSKA elucidated learners’ misconceptions and revealed discriminating items from the three organizational components of pollination systems knowledge.

Conclusions

The PSKA may aid educators in exploring learners’ conceptual understanding, identifying areas of misconceptions, and refining educational programming aimed at improving learners’ pollination systems knowledge.

Introduction

Although pollinators play an integral role in human well-being, their continued global decline reflects the need to provide and evaluate general pollinator knowledge to promote their conservation (Golick et al., 2018; Potts et al., 2016; Schönfelder & Bogner, 2018; Westerhold et al., 2018). Many studies and organizations have implemented interventions, such as citizen science projects (BumbleBeeWatch.org; Domroese & Johnson, 2017; Saunders et al., 2018) or hands-on activities regarding pollinators (Cho & Lee, 2018; Schönfelder & Bogner, 2018), to encourage pro-pollinator conservation attitudes and behaviors. Success of these environmental education programs is often determined by increased positive perception or conservation motivations towards pollinators (Cho & Lee, 2018; Schönfelder & Bogner, 2018; Stanisavljević & Stanisavljević, 2017) or increased participant engagement in science learning (Domroese & Johnson, 2017; Saunders et al., 2018). However, there is no systematic method to assess the success of implemented interventions in terms of pollination systems knowledge gains, nor to identify learners’ misconceptions regarding pollinator knowledge that may hinder pollinator conservation behaviors. We aimed to meet this need by developing and evaluating a novel instrument that assesses learners’ understanding and misconceptions about pollination systems and conservation actions. Without a purposeful assessment, science educators and organizations will not be able to identify instructional strategies that achieve program outcomes and address misconceptions of pollination systems to encourage informed decision-making regarding pollinator conservation strategies.

Identifying pollination and pollinator misconceptions through a structure–behavior–function approach

Misconceptions can form a novice’s conceptual understanding of a concept or a system (diSessa, 2014; Sands et al., 2018; Smith et al., 1994) and affect subsequent knowledge construction. Applying a structure–behavior–function (SBF) theory approach to instructional interventions has been suggested to help novice learners understand complex systems (Hmelo-Silver & Azevedo, 2006; Hmelo-Silver & Pfeffer, 2004). The SBF theory helps parse out the various interrelated levels within dynamic systems (Hmelo-Silver & Pfeffer, 2004; Vattam et al., 2011), which can allow educators and researchers to identify what components of the system students’ have difficulty understanding. The SBF framework comprised structures regarding components of the system, behaviors that deal with mechanisms resulting in an outcome, and functions related to the purpose of the components in the system (Hmelo-Silver & Pfeffer, 2004). Studies utilizing SBF theory show that experts can reason about abstract concepts within a complex system while novices focus on superficial structures of the system and have little understanding about the interrelated abstract relationships (Hmelo-Silver et al., 2007; Vattam et al., 2011). The SBF approach is ideal to study student understanding of pollination systems as it reveals novice learners’ persisting misconceptions at multiple interactive scales from the pollination process to the plant–pollinator interactions and system functioning. In our work, we considered misconceptions to be ideas inconsistent with currently accepted scientific evidence (Smith et al., 1994; Wynn et al., 2017).

Only a handful of studies have explored what people know about pollination, plant–pollinator interactions, and pollinator conservation actions (Golick et al., 2018; see also Silva & Minor, 2017; Westerhold et al., 2018). Though the public is aware of the declining populations of insect pollinators, majority of the focus has been on the popular honeybee (Apis mellifera) rather than wild insect pollinators (Hall & Martins, 2020). Studies that report on the perceptions of pollinators tend to focus on insects in general (e.g., Cinici, 2013; Lemelin et al., 2017; Shepardson, 2002; Snaddon & Turner, 2007) or solely on bees (O’Connell et al., 2018; Silva & Minor, 2017; Wilson et al., 2017). For example, when asking adolescents about their experience, knowledge, and attitudes toward bees, Silva and Minor (2017) found that greater understanding of bees was associated with more positive attitudes towards bees, though students did not often understand the interrelated abstract relationship between bees and plants (e.g., plant reproduction is possible because bees carry and transfer pollen among flowers). They also found adolescents had misconceptions regarding bee products and nest building, along with difficulty recognizing the plant–bee relationship and the role bees play in their environment (Silva & Minor, 2017). Few studies explore perceptions of multiple insect pollinators or non-insect pollinators like birds and bats (e.g., Golick et al., 2018; Hevia et al., 2021; Munyuli, 2011). Golick et al. (2018) described college students’ conceptual understanding of pollination and pollinator conservation in detail through qualitative interviews. They found that students often struggled describing pollinating structures on pollinators, identifying different types of pollinators, and associating pollination as a separate process from fertilization and seed dispersal within a plants’ reproductive process. Additionally, students held misconceptions related to the characteristics of pollinating organisms in ecosystems. Likewise, they also found that students have difficulty recognizing actions that could be taken to conserve pollinators, calling for a need of more action-oriented pollinator conservation education.

Although many are interested in conserving pollinators, this interest may not always result in effective conservation actions, as few understand how to efficiently promote pollinator conservation (Hall & Martins, 2020; Wilson et al., 2017). For example, Westerhold et al. (2018) found that horticultural employees, who have direct interactions with homeowners, had weak knowledge of pollinator-friendly plants, which may be detrimental for pollinator conservation in local gardens. The lack of focus on plant awareness regarding pollinator–plant interactions further adds to misconceptions of pollination systems. This is often characterized as plant blindness (Balding & Williams, 2016), meaning that people do not focus on plants in their surroundings (or educational programming), and thus are not aware of plants’ importance. Most recently, Parsley (2020) proposed the term plant awareness disparity to discourage the ableist connotation of plant blindness while keeping at its core the lack of attention, attitude, knowledge, and relative interest in plants that lead to misconceptions and lack of conservation actions towards plants. Hershey (2004) developed a list of common plant misconceptions to focus on while teaching topics like pollination or plant reproduction; for example, teachers may have to address the misconception that pollination is different from fertilization and pollination does not ensure fertilization. Adding to Hershey’s (2004) list of misconceptions, Wynn et al. (2017) combed through the literature for common plant misconceptions and found that most may be related to two types of thinking errors: insufficient knowledge and early childhood ideas. They listed several plant misconceptions related to pollination involving plant structures, growth, and development as well as a plant’s reproduction process (Wynn et al., 2017). Some of these misconceptions persisted past the high school level, supporting their assertion that misconceptions of plants can persist over time regardless of whether individuals have been exposed to scientifically accepted information. Additionally, students have difficulty understanding abstract mechanisms involved in evolution (i.e., natural selection; Gregory, 2009; Nehm et al., 2012), and may be less likely to recognize implications of coevolution on plant–pollinator structures, behaviors, and functions associated with pollination within the classroom. For example, students may not have a strong understanding of the specialized evolutionary relationships that develop between animal pollinators and plant species, and instead may maintain a view that pollination could occur by chance encounters by any animal or plant species (Golick et al., 2018). These persistent misconceptions may affect what pollinator conservation actions people support or enact in the real-world. The difficulty with understanding the process of pollination and pollinator–plant interactions may rest in recognizing the complexity of the various components involved.

A pollination systems-thinking and conservation actions framework

Golick et al.’s (2018) work aimed at explaining learners’ understanding of pollination systems by building theoretical frameworks, developed deductively from an SBF approach, which described college students’ levels of knowledge sophistication. They described three organizational components regarding pollination systems knowledge: (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation. They suggested that an SBF teaching approach could help learners engage in systems-thinking by understanding the relationships (or behaviors) among the structures and functions of the pollination system.

By creating an instrument that elucidates student thinking of the structures, behaviors, and functions involved in pollination systems, educators can identify students’ misconceptions across multiple components of the system and modify their educational programming accordingly. Likewise, they can utilize the instrument to assess the effectiveness of their educational programming at achieving specified learning objectives (Golick et al., 2018; Smith & Tanner, 2010). To our knowledge, there is no published instrument that evaluates pollination systems knowledge, misconceptions, or the success of educational programming related to pollination systems knowledge. We set out to address this gap by building the Pollination Systems Knowledge Assessment (PSKA) instrument based on Golick et al.’s (2018) three frameworks that described student understanding of pollination systems and discussing the properties of the instrument through psychometric evaluation (item statistical analyses).

Methods

We present the development and evaluation of the Pollination Systems Knowledge Assessment (PSKA) instrument. We designed the instrument using the three organizational components of pollination systems knowledge proposed by Golick et al. (2018), as well as findings from their interviews, and then tested the instrument in a large-enrollment college course. Our research was done with the approval of an IRB #20140813907 EP.

Our intention was to develop a comprehensive pollination systems knowledge instrument that encompassed structures, behaviors, and functions across multiple components of the system. As such, we designed the questions (items) of the instrument around the three organizational components of Golick et al. (2018): (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation. We also included concepts that students struggled in understanding based on Golick et al.’s (2018) interviews to enhance content validity of pollination systems knowledge. The first component includes pollination knowledge for the types of plants, plant structures, and the purpose of pollination for plants. The second component specifies pollination knowledge related to types of pollinators, pollinator structures, purpose of pollination for the pollinator, and the survival needs for pollinators. Lastly, the third component categorizes pollination knowledge about the human–pollinator relationships, role of pollination in the environment, and conservation actions.

Instrument—item development and implementation

Research-based assessments, hereafter instruments, aim to measure students’ conceptual understanding of a concept(s) (D'Avanzo, 2008; Garvin-Doxas & Klymkowsky, 2008; Sands et al., 2018) as well as evaluate the effectiveness of instruction at facilitating that understanding (Furrow & Hsu, 2019; Madsen et al., 2017; Porter et al., 2014). These instruments are composed of various questions or items (Sands et al., 2018), that focus on key concepts and capture correct and incorrect responses. Many instrument studies use a multiple-choice question (MCQ) format to assess understanding, posing a question along with multiple answer choices, and allowing the participant to choose only one (Sands et al., 2018). Although the MCQ format is useful in providing insight on conceptual understanding if learners pick answer choices that reflect common misconceptions, the MCQ format is limited as only information for one answer choice is given. Some have sought to overcome this limitation by developing instruments in a multiple-true–false (MTF) format (Brassil & Couch, 2019; Couch et al., 2015). Unlike the MCQ format, the MTF format consists of a question stem followed by multiple true/false answer statements (i.e., items) that allow educators to identify what learners understand in each item. The MTF format has shown to better distinguish individuals’ thinking than a MCQ format by identifying students with full, partial, or minimal understanding of the various answer statements (Brassil & Couch, 2019; Couch et al., 2015; Sands et al., 2018).

We began developing our instrument in fall 2018 by creating questions in a multiple-true–false (MTF) format, informed by the three organizational components of pollination systems knowledge proposed by Golick et al. (2018): (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation, as well as concepts students had difficulty with based on their findings. We engaged in an iterative item-development process, creating a question stem followed by several items (Fig. 1; Brassil & Couch, 2019; Couch et al., 2015; Hubbard et al., 2017). We followed accepted item-writing standards, like avoiding jargon (Frey et al., 2005). We then administered the initial version of the PSKA (18 question stems and 67 items) at the beginning of a required multidisciplinary science literacy course in fall 2018 to 256 undergraduate students. The science literacy course consists of undergraduates using a structured decision-making tool that incorporates their personal values and scientific information to reason about complex socioscientific issues (Alred & Dauer, 2020; Dauer et al., 2021, 2022). Instruction of the wild pollinator unit included discussing the ecology of pollination systems and the multiple alternative solutions for conserving wild pollinators.

Fig. 1

Example of a multiple true–false item included in the Pollination Systems Knowledge Assessment (PSKA)

We subsequently conducted group discussions about the items with entomology experts, faculty, and postdocs (n = 3) in a natural resources department as well as semi-structured think-aloud interviews with both undergraduate and graduate students (n = 5), to ensure clarity of each item, improve face validity, and response-process validity of the instrument. During the interviews, participants first completed the PSKA and then were asked to talk about the reasoning for their answers. We modified most items more than once based on these discussions and interviews, as well as preliminary item analyses. We then administered the second version of the PSKA (18 question stems and 71 items) to 280 undergraduates at the end of the science literacy course in fall 2018. Based on additional expert (n = 3) and student (n = 4) feedback, we refined the instrument and implemented the final version of the PSKA (18 question stems and 70 items) instrument with 99 undergraduates (Table 1) on the last day of the science literacy course in the Spring 2019 semester as an online survey. Subsequent statistical analyses are based on student responses for the final version of the PSKA.

Table 1 Self-reported student demographics (n = 99) for the PSKA

Item statistical analyses—difficulty and discrimination

We describe item statistics to determine how well undergraduates performed on individual items that encompass different components of the pollination system. For each T/F item, we scored responses as 0 = incorrect or 1 = correct. We computed item statistics by determining item difficulty (P), and item discrimination (D). Item difficulty values represent the proportion of correct responses on a particular item; lower item difficulty values indicate that a lower proportion of students answered that item correctly. We calculated item difficulty by counting the total number of correct responses (n₁) divided by the total number of responses (n) (P = n₁ / n). We also calculated item discrimination values as they indicate the degree to which the item can differentiate among students’ abilities in terms of their overall test score (Boopathiraj & Chellamani, 2013). To calculate item discrimination, we divided our sample (n_students = 99) into three groups (bottom, middle, and top) based on their overall scores. For each item, we subtracted the proportion of correct responses of the bottom third group of students (n_L) from the proportion of correct responses for the top third group of students (n_H) (D = n_H – n_L) (Couch et al., 2015; Doran, 1980). High item discrimination values (greater than 0.2) suggest that individuals who scored high on the instrument answered the item correctly whereas those who had a lower test score did not (Couch et al., 2015).

Instrument statistical analysis

We performed a confirmatory factor analysis (CFA) to determine whether item response patterns aligned with the three organizational components that guided Golick et al.’s (2018) pollination systems frameworks: (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation. Our intention was to build a comprehensive pollination systems knowledge instrument that encompassed structures, behaviors, and functions across multiple components of the system, so the PSKA had items that pertained to structures, behaviors, and functions within each organizational component. We did not necessarily presuppose that students’ pollination systems knowledge would differentiate into the three organizational components as the structure of students’ knowledge may or may not be aligned with the way researchers organized the concepts. In particular, Golick et al. (2018) developed the three organizational components to capture the various domains underlying a complex system but do not necessarily imply that students at a given stage would have distinct levels of understanding across the components. Additionally, we used Golick et al.’s (2018) frameworks as guides to develop question stems and items and relied on entomologists’ expertise to further expand and refine the instrument. As such, items varied somewhat from concepts within Golick et al.’s (2018) pollination systems frameworks.

We used the lavaan package in R (Rosseel, 2012) to evaluate the CFA models for goodness of fit by comparing their CFI (comparative fit index) and TLI (Tucker–Lee index) (Brown, 2015). We used the DWLS estimator to account for our binary data. Because these indices are sensitive to sample size, we also looked at how well the models fit the data by comparing the RMSEA (root-mean-square error of approximation) between the two models (Brown, 2015). Subsequently, we calculated Cronbach’s alpha (α) in R as one way to determine internal consistency, or reliability, of the instrument. This calculation measures the interrelatedness of items, so that high α values, ranging from 0–1, represent higher consistency among the items (Connelly, 2011). Finally, we also determined aggregate scores for the three organizational components.

Results

The final version of the PSKA comprised 18 question stems, each containing 3 to 5 true–false items, for a total of 70 items (Additional file 1: Appendix A). We present 99 undergraduates’ overall performance distribution in Fig. 2. The average percent correct was 75.3% ± 7.6 SD. There were several items that students consistently missed, which indicated students may hold misconceptions about concepts within three organizational components of pollination systems knowledge: (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation (described more fully in the discussion).

Fig. 2

Frequency distribution of students’ overall percent of correct items for the final version of the PSKA instrument in Spring 2019. Each bar represents the percentage of students whose test scores fall within the given percent correct bins. n = 99

Item statistics—difficulty and discrimination

We noted item difficulty values to determine if there were certain concepts students struggled in understanding. Lower difficulty values indicate a lower proportion of correct responses, and our calculated item difficulty values ranged from 0.10 (a very difficult item) to 1.0 (very easy item; Fig. 3; Additional file 1: Appendix B). The multiple T/F format of the PSKA allowed us to distinguish which items students struggled with. Lower item difficulty values (< 0.4) appeared within each of the three organizational components, indicating that students had difficulty with multiple concepts across structures, behaviors, and functions related to pollination systems. Table 2 lists the most challenging items across each organizational component that reported low item difficulty values, (i.e., a lower proportion of students responded correctly to these items). Our findings suggest that the PSKA can assess if students struggle with understanding concepts related to pollination systems knowledge.

Fig. 3

Item difficulty and discrimination values for the PSKA, in Spring 2019. Each triangle represents an item; these are ordered based on their difficulty and discrimination values. For item difficulty, lower values indicate a lower proportion of correct answers (i.e., a difficult question). For item discrimination, higher values indicate that the top-third of students outperformed the bottom-third of students

Table 2 List of items from the PSKA that 40% or fewer students answered correctly in Spring 2019 (n = 99) within each organizational component from Golick et al.’s (2018) framework

We also identified item discrimination values to note how well each item distinguishes between low- and high-performing students. Most item discrimination values ranged from 0.10 to 0.63, with 28 item discrimination values being below 0.10 (Fig. 3; Additional file 1: Appendix B). This indicated that the top third of students outperformed the bottom third for most items by 10–63%.

Table 3 lists the items that had had high discrimination values (≥ 0.2) for each organizational component of the PSKA, suggesting that there may be multiple items instructors can use to quickly differentiate how well students may perform on the instrument overall. These findings suggest that the PSKA can discriminate among most to least knowledgeable students.

Table 3 List of items from the PSKA with high discrimination values (greater than 0.2) during Spring 2019 within each component from Golick et al.’s (2018) framework

Instrument statistics

We performed CFA analyses in R, which recommended we remove three items that students answered 100% correctly. These items consisted of students correctly identifying (i) bees as common pollinators, (ii) that some pollinators have special structures pollen sticks to, and (iii) that some pollinator-friendly practices to enact include planting a variety of plants that can bloom in different seasons. There were many easy items on the PSKA, potentially indicating that students generally had a high ability to answer these items (see Additional file 1: Appendix B). Our CFA analyses revealed that the baseline model (one-factor, CFI = 0.71, TLI = 0.70, RMSEA = 0.039, SRMR = 0.205) and our proposed model (three-factor, CFI = 0.71, TLI = 0.70, RMSEA = 0.039, SRMR = 0.205) based on Golick et al.’s (2018) three organizational components of pollination systems knowledge, had similar fit characteristics. This meant that items did not group around the three organizational components; rather, our findings indicated that there was only one general knowledge component that explained the item response patterns. Our calculated Cronbach’s alpha for the full instrument was 0.698 with a 95% confidence interval of 0.61 – 0.75. The aggregate scores (mean ± SD) for the three organizational components of the instrument showed similar student response patterns across them (plant structures: 0.72 ± 0.24; pollinator structures and behaviors: 0.78 ± 0.22; pollination systems function and conservation actions: 0.75 ± 0.31).

Discussion

To our knowledge, there is no published instrument that evaluates pollination systems knowledge, misconceptions, or the success of educational programming related to pollination systems knowledge. Our study addressed this gap by building the Pollination Systems Knowledge Assessment (PSKA) instrument based on Golick et al.’s (2018) frameworks that encompassed three organizational components of pollination systems knowledge and discussing the properties of the instrument through psychometric evaluation. Throughout the instrument development process, we created items that were scientifically accurate and reflected learners’ thinking by integrating feedback from experts and interviewed students. We administered the final version of the PSKA, designed in a multiple-true/false (MTF) format, in-class to 99 undergraduates in a multidisciplinary science literacy course as an online survey (Additional file 1: Appendix A).

Item statistics—difficulty and discrimination

Our item statistics revealed difficult items from each of the organizational components of Golick et al.’s (2018) pollination systems knowledge frameworks (Table 2). Items with lower difficulty values may reveal misconceptions that students hold about a particular concept. For example, many students in our sample incorrectly thought that pollination was required by all plants to reproduce. These difficult items can help educators plan and adjust their educational programming to address those misconceptions. With the PSKA, the misconceptions involved recognizing basic components of the pollination process, the necessity of coevolution directing plant–pollinator interactions, and the consequences of conservation actions, which we next describe in more detail.

Students struggled in disassociating the pollination process from fertilization and seed dispersal. For example, only 21% of students (n_students = 99) correctly identified that the pollination process does not “include the process when sperm fertilizes the egg” (Table 2). The pollination process only involves the transfer of pollen (Greber et al., 2013), after which fertilization can potentially occur. Seed dispersal also occurs after pollination and follows fertilization (Greber et al., 2013). However, this was unclear to many, as only 35% of students correctly indicated that the pollination process does not “include the dispersal of seeds by elements (wind or water)” (Table 2). Similarly, pollination is only a requirement for plants’ sexual reproduction as many plants reproduce asexually (Greber et al., 2013). However, only 26% of students correctly indicated that pollination was not a requirement for plants’ reproductive process. This is consistent with findings from previous studies (Golick et al., 2018; Hershey, 2004; Schussler, 2008), indicating that students may be overgeneralizing due to naïve ideas of plants’ reproductive process.

Students also held misconceptions regarding an organism’s role as a pollinator in an ecosystem, as only 22% of students correctly answered that not “any animal that is attracted to a flower can play a role as pollinator in ecosystems.” Similarly, 41% of students incorrectly indicated that “pollen is regularly transferred between flowers by cows’ noses as they are eating flowers.” Students may not have a full grasp on the way in which plant–pollinator relationships developed very specific structures and mechanisms over evolutionary time that dictate pollination success. Our findings support what Golick et al. (2018) concluded, that students may not fully understand the coevolutionary mechanisms (i.e., adaptation over time) that transpire between pollinators and plants leading to pollination between coevolved mutualistic species. Rather, many students think about pollination as a chance encounter that happens with many types of organisms potentially acting as a pollinator. This may be due to the difficulty students have when reasoning about time; for example, Ageitos et al. (2019) found that students’ reasoning about genetics and evolution tended to focus on short time frames. As novices, students have difficulty understanding abstract mechanisms involved in evolution (e.g., natural selection; Gregory, 2009; Nehm et al., 2012) and may be even less likely to recognize those associated with coevolution. Student understanding of coevolution has received much less attention than evolution (Ziadie & Andrews, 2018), indicating a new area of exploration for which pollination may be a useful context to probe students’ understanding of the underlying abstract mechanisms. Importantly, if novices do not recognize coevolved plant–pollinator relationships, they may be less likely to support science-based conservation efforts towards specific pollinators if they believe that all organisms can pollinate any plant, or that pollination happens coincidentally.

Much like Golick et al. (2018) reported, students also did not perform well on items concerning pollinator conservation actions. For instance, 90% of students indicated that avoiding the use of all types of pesticides on their lawn is a pollinator-friendly practice (Additional file 1: Appendix A). Not all pesticides are risky to pollinators if properly applied or combined with other management techniques (Biddinger & Rajotte, 2015). For example, a study on apple orchards found that applying certain insecticides and fungicides in the fall could be safe enough for honeybees when they emerge in spring (Heller et al., 2020). Though indiscriminate use of pesticides may lead to pollinator decline, some pesticides (like an herbicide) may be help make a landscape better for pollinators by removing non-flowering plants. Similarly, 82% of students specified that decreasing urban land development helps conserve pollinators. Students may not be considering the beneficial effect urban landscape projects can have in conserving wild pollinators. Several studies report that increasing floral resources in urban areas could potentially conserve pollinators by increasing their food supply (Davis et al., 2017; Hicks et al., 2016; Simao et al., 2018). Students may not recognize the nuances involved in the consequences of each conservation action, which can affect subsequent decisions they make involving pollinator conservation.

Our item statistics also showed high discriminating items that encompassed concepts across multiple components of pollination systems (Table 3). One of the purposes of having discriminating items is to quickly assess whether students who performed high on the PSKA have a higher ability (i.e., knowledge) on an item than low-performing students. For example, top-performing students in our sample were able to correctly identify that pollinators do not intentionally pollinate but instead harvest nectar or pollen for their own resources (i.e., pollination is a byproduct). However, the instrument also had low discriminating items that may have resulted from very easy items (i.e., items a high proportion of students answered correctly). For example, all students (n = 99) correctly identified that some pollinators have special structures that pollen sticks to. Similarly, all students correctly identified that planting a variety of plants that can bloom from spring to fall is a pollinator-friendly practice they could engage in. Since many concepts within the PSKA were discussed during the wild pollinator unit in a science literacy course, these items may be easy for students as they recalled the content. Another explanation may be that students recalled information from their high school science education, an often-cited source of information from participants in our focus groups during the development of the instrument. Low discriminating items may also result from concepts students’ struggle to understand. For example, just 22% of students were able to correctly identify that only certain animals attracted to flowers play a role as pollinators in ecosystems. As the PSKA had many high discriminating items, educators can implement these to quickly assess whether there are differences in pollination systems knowledge among their audience or rank students from most to least knowledgeable on a concept.

Instrument statistics

Our CFA analysis did not statistically support the three-factor model representing the three organizational components from Golick et al.’s (2018) pollination systems frameworks: (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation. Our intention was to build a comprehensive pollination systems knowledge instrument that encompassed structures, behaviors, and functions across multiple components of the system. Our finding that the items did not separate into the three organization components suggests that a given student tended to have similar knowledge across the components (rather than distinct knowledge of one component over another). It may be that the organizational components of the PSKA assessment reflect researchers’ conceptual organization, but do not reflect independent facets of students’ knowledge of pollination systems. Furthermore, students learned about the different components at the same time throughout the semester, which would further lead to them having similar knowledge levels across components. Our small sample size may have also affected our analysis, indicating a need for more data. The Cronbach α was 0.698, indicating that the internal consistency of the instrument was moderate, but only slightly below the range of the acceptable value of 0.7 (Kline, 2000). Within the one-factor CFA model, many low factor loadings as well as unsuitable fit statistics suggest that there may be a more complex pattern or structure underlying student thinking. These findings indicate the need for further instrument evaluation (e.g., employ instrument to a larger audience, review item language through interviews, explore alternative factor structures, and implement the instrument in multiple contexts). Further, the aggregate scores of the three organizational components showed that students have similar response patterns across them, so elicited pollination knowledge should be reported as a single score and reviewed item by item rather than reporting three separate scores. But we suggest that the PSKA remains functionally useful as the elicited knowledge gives educators a comprehensive overview of what their audience knows and struggles to understand about pollination systems and pollinator conservation actions. Our findings demonstrate that the instrument elucidates misconceptions, which is essential information educators can use to tailor their educational programming and target their audiences’ misconceptions as well as assess if their desired pollination systems knowledge base has been achieved.

Overall, our instrument was designed to measure general knowledge about pollination systems and conservation actions. This may be helpful for educators as they can tailor the instrument to suit their programming needs, which can vary by age and conceptual areas. For example, students in our study had a high knowledge ability in identifying plant structures associated with pollination, an expected result for undergraduates who are mostly STEM majors. But this may not be the case for elementary school students who are beginning to learn about the pollination process. Educators can thoughtfully tailor the PSKA by implementing items from each organizational component of the instrument that are appropriate for their desired audience and elicit participants’ understandings and misconceptions from each of these conceptual areas of the pollination system.

Limitations

Our study had several limitations. First, the PSKA was only evaluated during two semesters of one course at one institution with small sample sizes, limiting its generalizability to other populations. Though we worked to enhance content validity when building the PSKA, we did not evaluate additional areas of validity (e.g., predictive validity). Another limitation of our instrument may be the MTF format we chose as this design could induce student thinking towards an all-or-nothing approach toward conservation action, and thus students may not consider the inherent nuance of the pollinator conservation action. We need further studies to examine student thinking around conservation actions towards pollinators, including designing instruments in open-ended format to address the MTF format limitation.

Conclusions

In summary, the PSKA is a valuable tool that was able to distinguish differing knowledge abilities and elucidate misconceptions about pollination systems and pollinator conservation actions. Developing the PSKA based on Golick et al.’s (2018) pollination systems frameworks, designed from an SBF approach, allowed us to reveal foundational misconceptions on multiple components of the pollination system (i.e., (1) plant structures, (2) pollinator structures and behaviors, and (3) pollination systems function and conservation action). When science educators implement the instrument, they can assess baseline levels of pollination knowledge as well as identify what components of the pollination systems learners’ struggle understanding. We recommend that formal and informal educators (K-16) review and implement items from each organizational component of the PSKA that fit their audience and programming needs. In this way, educators maintain adequate breadth of pollination systems knowledge. For example, elementary school educators may choose items that are the easiest from each component of the PSKA while postsecondary educators may choose more difficult items. Similarly, informal educators may choose to ask items from each organizational component of the PSKA that pertain to their content. Educators may also use high discriminating items to quickly rank students from most to least knowledgeable for certain concepts (e.g., pollinators do not intentionally pollinate). Likewise, educators can employ difficult items to see if learners’ misconceptions persist after presenting their educational programming.

We suggest further studies implementing and evaluating the PSKA with wide-ranging audiences (e.g., variety of ages and backgrounds) in multiple educational settings (e.g., formal, and informal). We propose the use of this instrument as a first step in assessing learners’ knowledge about pollination systems and pollinator conservation actions. Assessment tools like PSKA may aid science educators in exploring learners’ conceptual understanding, identifying areas of misconceptions, and thus refining educational programming aimed at improving learners’ knowledge of pollination systems. Developing an assessment using an MTF format can also help science educators identify specific concepts and processes students struggle in understanding within a complex system. Similarly, an assessment guided by an SBF approach is beneficial as educators may be able to comprehensively evaluate student thinking of the interconnected levels within any system.