Table 10 Inter-rater reliability for the Modification Identification Framework (MIF): Gwet’s AC1 values (per code)
Code | Gwet’s AC1 | ||
|---|---|---|---|
Instructional practice | Level of delivery | Individual | 1 |
Group of students | 1 | ||
Demographic group of students | 0.97 | ||
Individual instructors | 0.72 | ||
Instructors teaching the same course | 0.97 | ||
Entire department | 0.95 | ||
Entire university or system/consortia of universities | 0.98 | ||
Nature of modification | Tailoring/tweaking | 0.92 | |
Adding elements | 0.96 | ||
Removing/skipping elements | 0.92 | ||
Shortening/condensing | 0.98 | ||
Lengthening/extending | 0.96 | ||
Substituting elements | 0.97 | ||
Reordering RBIS modules | 0.99 | ||
Integrating another RBIS | 0.99 | ||
Integrating strategy into other RBIS | 0.98 | ||
Repeating elements | 0.99 | ||
Loosening structure | 0.85 | ||
Departing from RBIS | 0.99 | ||
Context | Format | 1 | |
Personnel | 1 | ||
Population | 0.98 | ||
Setting | 0.99 | ||