ADDENDUM: Two diagrams follow that represent data not available when we wrote the preceding article for "To Improve the Academy." These illustrate use of knowledge surveys for unit level development. Figure 1 shows a stacked area graph where values plotted are averages of students' confidence of content mastery. Complete mastery of all students on an item would register as a "3;" complete ignorance would register as a "1." Sequence of coverage is from left to right, with four surveys from four class levels compiled into one graph. Lower black shading reveals average general knowledge at start of course; lighter gray shading reveals average general knowledge at end. The freshman course shows some prior familiarity with initial material covered but a gradual decrease in mastery as the course proceeds. The sophomore course appears to be highly successful; students enter without much mastery and leave with a high degree of it, with results consistent across the entire course. The large amounts of black that appear in junior and senior courses reveal that much material covered here duplicates that from prior courses.

Figure 1. Combined results of knowledge surveys from four undergraduate courses within a single program.

Figure 2 explains much of the results seen in Figure 1. Sequence and scale of items is the same, but items here show the Bloom rating of each survey item plotted in a bar graph. There are more higher Bloom-level challenges in the freshman class than in all the other courses combined, and it is designed at too high a level to promote optimal mastery by beginners. The sophomore course matches the students' needs well. It has good challenges but still is strong on building basics. The junior and senior courses should be heavier on higher-level challenges. Removal of material duplicated in earlier courses would allow this.

Figure 2. Record of ratings of Bloom levels of items presented in knowledge surveys depicted in Figure 1.

Together, colleagues in the department can see precisely what topics should be covered and where. Such knowledge helps a unit to optimize design of its curriculum by assessing content directly in detail. This department learned to emphasize more basics in the introductory class and to provide challenging opportunities in upper division courses. Such changes ultimately will produce much higher quality education for students in the program.

There are few clearer examples of the need for assessment and the limitations of evaluation. The student evaluation ratings were not bad in any of these courses, and no student ratings could ever show the information revealed here that helped generate such improvements.