Table 1. Excerpt of six items from a 200-item knowledge survey. Students respond in accord with instructions for either a bubble sheet (as above) or a web-based format.

Figure 1 shows the pre- and post-course results from the 200 items in the same semester-long course. The values plotted are the class average of responses to each item, and only the students who completed both pre and post surveys are included. When needed, the instructor can call up the data at the level of each individual student. The figure provides insights that global summative student evaluations would never reveal, such as better pacing (see caption) as a key to improve learning when the instructor again teaches the course.

Figure 1. Pre- & post-course results of a knowledge survey (From Nuhfer, 1995). Ordinate scales are from 1 (low confidence) to 3 (high confidence). The survey elicited confidence ratings to 200 items (abscissa) in the order in which students encountered items in the course. The lower darker area (on this and all similar figures in this paper) reveals the class averages on confidence to address each item at the start of class; the upper shaded area displays the ratings to the same items at the end of class. The course, instructor and learning experience all received "A" ratings from students on summative evaluations, but lower learning did occur in the final two weeks of classes when the final topic on coastal hazards was hurriedly covered. The right side of the graph (approximately items 170-200) reveals this learning gap. Better pacing eliminated this gap when the instructor next taught the course.

The need for a more direct assessment of learning

Knowledge surveys address learning in specific detail, as opposed to the global summative item: "Evaluate this course as a learning experience." Summative student ratings are frequently used to evaluate professors for rank, salary and tenure. Summative ratings are measures of students' general satisfaction, which results from a mix of content learning, cognitive development and affective factors. The most thorough research on the relationship between content learning and summative ratings produced a correlation coefficient derived by meta-analysis of r = 0.43 (Cohen, 1981). That relationship is strong enough to prove that, in general, students learn more from professors whom they rate highly, but it is too weak to allow learning in any single class to be assessed by inference based on student ratings of the class, the individual professors who taught it, or the "overall learning experience." Researchers such as Feldman (1998) have shown that educational practices that produce the most learning are not exactly the same as those that produce the highest student ratings. Because student ratings alone cannot reliably reveal learning outcomes produced by individual instructors or courses, good assessment requires a more direct assessment of learning. We believe knowledge surveys, properly used, serve this purpose. Knowledge surveys were created to allow instructors to prove that their courses produced specific changes in students' learning and to disclose the detailed content of a course to students (Nuhfer, 1993, 1995). Tobias and Raphael (1997) grasped the usefulness of the tool and included it as a best practice in their book. Knowledge surveys do achieve their original purposes, but employment of this tool encourages further instructional improvements that are discussed in this paper.

DISCUSSION

Knowledge surveys that enhance high-level thinking

When instructors develop sophistication with this tool, they can code each item according to levels of reasoning, such as those of Bloom (1956- see Tables 1 & 2). This coding allows an instructor to see the course as a profile of levels of inquiry addressed (Figure 2) and verify that the course plan accords with the original intentions of the instructor and the purposes of the course. As noted by Gardiner (1994), most testing pinpoints low-level content knowledge, even though most instructors really aspire to teach some higher-level critical thinking. If an instructor includes "critical thinking" as her/his course objective, and the detailed knowledge survey reveals overt emphasis on mere memorization, then recognizing this discrepancy is empowering. Such revelations allow course redesign that will ensure the desired critical thinking experiences.

Table 2. The six levels of Bloom's (1956) classic taxonomy of educational objectives together with question prompts that commonly elicit a response at a particular reasoning level. Note that Bloom's items usually reflect instructors' questions, while the levels of thinking presented in other more sophisticated schema (Perry, 1999; King and Kitchener, 1994) reflect students' answers.

Interpretations of graphs like Figure 2 require care. Although 54% of knowledge survey items addressed the lowest Bloom levels, this does not imply that 54% of course time was spent addressing lower levels. In fact, high-level open-ended questions require disproportionately more time to confront than do simple recall knowledge items. Figure 2 verifies that students encountered no less than (they actually encountered more) 28 analytical, 16 synthesis and 16 evaluation challenges in this course. Yet, students can do synthesis and evaluation poorly, thus operating at the lower levels of the Perry (1999) model or the Reflective Judgment model (King and Kitchener, 1994). If one wants to use such a presentation like Figure 2 to prove that students truly mastered some critical thinking in this course, then one must disclose the rubrics used to evaluate students' answers to representative high-level items. (Rubrics are the disclosed criteria that instructors use for evaluation of a test question or project.) When instructors present rubrics along with evidence like Figure 2, the assertion that students addressed high-level challenges with high levels of sophistication is hard to refute. Table 4 near the end of this paper gives rubrics for a few such questions.

Figure 2. Levels of thinking represented in a knowledge survey. Data are from the same class and knowledge survey shown in Figure 1, but here have been rearranged to present the course outcomes as a profile of levels of educational objectives (Bloom, 1956) encountered in the course. The graph reveals that reduced learning in the final two weeks (Figure 1) occurred in material typified by the lower Bloom's levels.

Does increased confidence reveal increased learning?

The video, Thinking Together: Collaborative Learning in Science, produced by the Derek Bok Center (1993) at Harvard shows a brief paired-learning exercise in Dr. Eric Mazur's introductory physics class. The students confront a problem, answer a multiple-choice question about the problem and rate the confidence that their own answer is correct. The students then engage in paired discussion to convince the partner that one's own answer is correct. Thereafter, the entire class debriefs and summarizes results. In that video, a bar graphic displays a positive relationship between confidence and correctness. Mazur (personal e-mail communication, November 29, 2001) revealed that there is no strong correlation initially between correctness and confidence to answer correctly, but overall class confidence rises significantly as discussion of the problem proceeds and greater understanding results.

We discovered that a few students will occasionally exhibit overconfidence of their abilities at the start of a class, but every class average we have examined has invariably been a very good representation of that class's knowledge and abilities. The confidence rating to address content does indeed parallel ability to address it in an exam situation (Figure 3), especially when a teacher designs effective teaching/learning experiences for the topic. Learning gaps revealed in post-course surveys reflect strong concurrence among nearly all students of little confidence to address an item. Interviews with professors who use knowledge surveys show that they nearly always know what produced a so-designated gap. Often a gap reveals a topic not covered, or addressed inadequately (see caption, Figure 1).

 The best results occur when survey items clearly frame specific content, and students take the survey home to complete it with plenty of time for self-reflection. Instructors should refer frequently to the survey throughout the course and remind all students to monitor their progress. Like any tool, a user's ability to be effective with it increases with practice and experience.

Figure 3. Comparison of normalized reported knowledge and final examinations results from astronomy class. Students were slightly overconfident about their knowledge on several of the questions from the first part of the course and less confident about their knowledge level of material toward the latter part of the course. The latter portion of the course covered material that was conceptually and mathematically new to the students. (from Knipp, 2001)

 

Knowledge surveys promote preparation and organization.

Feldman (1998) used meta-analyses to tease apart the instructional practices that produce learning as opposed to those that produce high ratings of student satisfaction. He discovered that the most important instructional contribution to learning was the instructor's preparation and organization of the course. However, this practice ranked only sixth in importance in gaining high ratings of satisfaction. Satisfaction depended much more strongly on the professor's enthusiasm and stimulation of interest. The National Survey for Student Learning (Pascarella, 2001) thoroughly supports Feldman's findings about student learning. The use of knowledge surveys as a best practice can be justified on the basis of the dominant evidence: nothing a teacher can do to produce learning matters more than the efforts put into course preparation and organization.

The process of making a knowledge survey-laying out the course in its entirety- considering the concepts, the content, levels of thinking, and questions suitable for testing learning of chosen outcomes is extremely conducive to organization and preparation. Further, the act of disclosing the course in its entirety is akin to providing students with a detailed road map to the course. Students know the content, the sequence in which it will come, and the levels of challenge demanded. Erdle and Murray (1986) confirmed that students perceive disclosure as moderately important in physical science and humanities courses, and more important in social science courses. Simply put, the research shows that if one carefully decides what one is going to teach and conveys this clearly to students, the more likely the students will achieve the learning outcomes desired.

Knowledge surveys boost practice of the Seven Principles.

 Chickering and. Gamson (1987) summarized the outcomes of a Wingspread Conference in which attendees expressed consensus by drafting "Seven Principles of Good Practice" for succeeding with undergraduates. Developers know well that getting some faculty to adopt progressive pedagogical practices can be difficult. Most faculty relate better to content learning than to the practices that might better produce learning. The fact that knowledge surveys reach faculty where they are-with an obvious relationship to the content they value-makes these a good tool developers can use to introduce faculty to thinking toward improved practice. Knowledge surveys pre-dispose a class to making use of all seven principles.

1.    Good practice encourages student - faculty contact. One reason that few students ever come to a professor's office for help is because they are often unaware of what they do not know or understand. Once students confront a knowledge survey and understand its use, students can more clearly see their need to seek help. A pre-course survey can also reveal which students have the most confidence with the material and which do not (Figure 4). Such insights permit faculty to know something about each student and make them aware of each student's possible needs for extra assistance before the class is even underway. Knowledge surveys also indicate which individual students really have the prerequisites needed to engage the challenges forthcoming in the course.

Figure 4. A pre-course knowledge survey used to identify students by confidence level at start of a course. Data here are from a fifty-item music theory course with students' names changed. The data sorted and graphed with students' names results from averages computed on each student's responses to all items plus a separately calculated class average (lighter bar).

2.    Good practice encourages cooperation among students. Knowledge surveys help to impart several of the five basic elements of cooperative learning (Johnson, Johnson and Smith, 1991; Millis and Cottell, 1998). Individual accountability includes the critical ability of individuals to be able to accurately assess their own level of preparedness or lack thereof. When a course has detailed disclosure, students more readily know when they have deficiencies, making them more receptive to engaging in positive interdependence, promotive interaction and group processing to overcome deficiencies. Pre-course knowledge survey results such as the kind shown in Figure 4 provide the information needed to form heterogeneous cooperative groups composed of members with known, varied abilities.

3.    Good practice encourages active learning. Knowledge surveys can be a powerful prompt for addressing high-level thinking. When students receive both good example items and a copy of Bloom's taxonomy, they can make up their own new test questions for a unit. Such questions will address the material and appropriately high Bloom levels. A simple assignment could be: "You already have seven questions on this unit in your knowledge survey. Address the material in the unit, use Bloom's taxonomy, and see if you can produce seven good questions that are even more challenging. If yours are better, I may use them on the knowledge survey for this course next term." When students know the important concepts and outcomes desired in a unit lesson, groups can become resources, thus structuring peer teaching into a course while assuring quality outcomes.

4.    Good practice gives prompt feedback. When a detailed knowledge survey is furnished, it allows students to monitor their progress through the course. One of the first signs that an instructor has produced a survey of good quality is a query from a student: "Will I really be able to learn all these new things?" Prompt feedback delivered by a survey is the students' own continuous tracking of knowledge gains as the course unfolds. When students can create their original test questions that address the material at a respectable level of thinking, they have reached what probably constitutes adequate preparation and understanding. If any student must be absent, the survey immediately discloses the material missed and reveals to an extent the work required to master it.

5.    Good practice emphasizes time on task. Full disclosure at the start of a course allows timely planning and study. A review sheet given out before an exam will not reveal to students what they do not know in a timely manner, and it will promote mere cramming rather than planned learning. Faculty who plan courses well and disclose them at the detail of a knowledge survey, quickly discover that the survey keeps them honest. Some student who is following his/her progress will invariably say: "Excuse me, Dr. ----, but it seems like we didn't address that item #-- about----." Perhaps that class launched into a discussion that carried more value than the item skipped and an expressed item of more value now replaces the original item.  On the other hand, perhaps the class launched into a digression and the instructor forgot to address a particularly important point that he/she must yet address. When a class inadvertently strays off track, knowledge surveys reveal whether straying resulted in any important omissions. Surveys also require students to engage material repeatedly. Some of the earliest research on cognition deduced the benefits of time spent in repetition to learning. The use of knowledge surveys insures at least two additional structured engagements with the entire course material.

6.    Good practice communicates high expectations. Students sometimes complain that instructors teach at one level and test at a more challenging level. Knowledge surveys offer the opportunity to detail, in a timely manner, the level of challenge that students should expect. The materials needed to build a rudimentary knowledge survey already reside in most professors' computers-in the past examinations, quizzes, and review sheets they provided the last time they taught the course. The only work required is to arrange items from these in the order of course presentation. Students appreciate the focus that a knowledge survey brings to the study process, and they will rise to expectations conveyed in a survey, particularly if instructors assert that some of the higher-level items will likely appear on a final exam.

7.    Good practice respects diverse talents and ways of learning. One of the best ways to address diverse learners is to be certain to present and engage materials in a variety of ways, in particular ways that make sense in terms of how the brain operates in the learning process (Leamnson, 1999). When one actually has a blueprint of content and levels of thinking that one wants to present, it quickly allows one to ask "What is the best way to present this item, then the following item?" Without such a plan, one can too easily end up lecturing through the entire course, even when the desired outcomes literally scream for alternative methods. A detailed plan of outcomes will obviate a correlative plan for reaching these (see "Pedagogies" Table 4).

Applications at the Unit Level

It is easy to conceive of tools applied to a single course (Figures 1 and 2) as applied to a four-year curriculum, with the content outcomes and levels of thinking desired plotted on the abscissa from freshman through senior year. We authors have just begun to scratch the surface of using knowledge surveys in unit-level development, but it is exciting and we hope readers will extend this kind of thinking to units at their own institutions.

When professors have detailed blueprints of their own courses, this permits a larger unit to sit down and have the necessary conversations about what content should be taught and when, what levels of thinking should be stressed and when, and what pedagogies and experiences should occur and when. One department at the senior author's institution has now committed to post syllabi and knowledge surveys for all courses on the web. A senior comprehensive exit exam will draw its questions from the knowledge surveys of courses taken by students. A student can thus see in detail every course and will have relevant study guides to any exit examination. A college at the same institution is now using knowledge surveys to plan and assess the curricula of various departments in that college. Both authors' institutions have discovered that when adjunct faculty are recruited or an institution has high turnover of instructors, knowledge surveys convey expectations for outcomes to new or part-time professors and help maintain course continuity and course standards.

Table 3 shows eleven learning outcomes for a university-level general education requirement. These resulted after eight science professors gathered to answer the question: "Why are we requiring students to take a science course, and what outcomes justify their costs in time and tuition dollars?" Together, these items constitute a model for science literacy-understanding what science is, how it works and how it fits into a broader plan for education. The institution regularly assesses these items by knowledge survey in each required core science course.

Table 3. Global outcomes for a core science course. These were devised when all science departments at an institution engaged in the necessary conversation to formulate unit level outcomes. Knowledge survey items can address high-level concepts such as these and allow assessment that is informative and effective.

Figure 5 shows a less positive outcome in a course given in a department that had not held the necessary conversations. The process shows that the course tied poorly into preceding courses. The students initially engaged new material and then spent a substantial amount of the course on material already previously covered elsewhere. By recognizing the duplication, the instructor was able to revise the course, and she achieved much better results the following year.

Figure 5. Knowledge survey revealing curriculum design deficiency. This 48-item survey reveals significant material taught to students that the students had already learned from some previous course(s). Minimal duplication occurs when a department uses knowledge surveys to inform curriculum design and prevent unplanned overlap.

Table 4. A knowledge survey utilized as a basis to select pedagogies, assign homework, author rubrics and design an exercise for a self-assessment journal (see Alverno College, 2000). By stating the content learning outcomes in detail, clear choices emerge that engage learners in diverse experiences. The rubrics and self-assessments relate to both the course content and the global core learning outcomes (Table 3). Such an approach epitomizes the "instructional alignment" concept of Cohen (1987), wherein alignment of intended outcomes with instructional processes and instructional assessments produces learning outcomes about two standard deviations beyond what is possible without such organization.

Summary

Knowledge surveys offer a powerful way to achieve superb course organization and to enact instructional alignment. Surveys serve as powerful assessment tools that provide a direct, detailed assessment of content, learning and levels of thinking. The time invested to produce a knowledge survey returns worthy benefit in enhanced learning that results from their use. Faculty appreciate that knowledge surveys require no in-class time to administer, that they improve planning and preparation, that they validate student learning much better than summative student ratings, and that they can construct knowledge surveys mostly from what is already available in their office computers.

  ADDENDUM click here

References

Alverno College Faculty, (2000). Self Assessment at Alverno College. G. Loacker, (Ed.) Milwaukee, WI: Alverno College

 

Bloom, B. S. (1956). Taxonomy of Educational Objectives- The Classification of Educational Goals. Handbook I. - Cognitive Domain, New York, NY: David McKay.

 

Chickering, A. W. and Gamson, Z. (1987). Seven principles for good practice in undergraduate education. American Association of Higher Education  Bulletin, 39 3-7. Also Chickering, A. W. and Gamson, Z. (1987). Seven principles for good practice in undergraduate education. Racine, WI: The Johnson Foundation, Inc./Wingspread.

 

Cohen, P. A. (1981). Student ratings of instruction and student achievement: a meta-analysis of multisection validity studies. Review of Educational Research, 51 (3) 281-309.

 

Cohen, S. A. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16 (8) 16-20.

 

Erdle, S. and Murray, H. G. (1986). Interfaculty differences in classroom teaching behaviors and their relationship to student instructional ratings. Research in Higher Education, 24 (2) 115-127.

 

Feldman, K. A. (1998). Identifying exemplary teachers and teaching: evidence from student ratings. in Teaching and Learning in the College Classroom 2^nd edition, K. A. Feldman and M. B. Paulsen, (Eds.) Needham Heights, MA: Simon & Schuster, 391-414.

 

Gardiner, L. F. (1994). Redesigning Higher Education: Producing Dramatic Gains in Student Learning. Washington, DC: ASHE-ERIC Higher Education Report No. 7

 

Johnson, D. W. Johnson, R. T. and Smith, K. A. (1991). Active Learning: Cooperation in the College Classroom. Edina, MN: Interaction Book Co.

 

King, P. M. and Kitchener, K. S. (1994). Developing Reflective Judgment. San Francisco, CA: Jossey-Bass.

 

Knipp, D. (2001). Knowledge Surveys: What do students bring to and take from a class? United States Air Force Academy Educator, Spring, 2001. Available since August 2006 at http://www.isu.edu/ctl/facultydev/KnowS_files/KnippUSAFA/KSKNIPPUSAFA.html.

 

Leamnson, R. (1999). Thinking About Teaching and Learning: Developing Habits of Learning with First Year College and University Students. Sterling, VA: Stylus.

 

Millis, B. J. and Cottell, P. G. (1998) Cooperative Learning for Higher Education Faculty. Phoenix, AZ: Oryx Press (now available through Greenwood Press).

 

Nuhfer, E. B. (1993). Bottom-line disclosure and assessment. Teaching Professor, 7 (7) 8.

 

Nuhfer, E. B. (1996). The place of formative evaluations in assessment and ways to reap their benefits. Journal of Geoscience Education, 44 (4) 385-394.

 

Pascarella, E. T. (2001). Cognitive growth in college. Change, 33 (1) 21-27.

 

Perry, W. G. Jr. (1999). Forms of Ethical and Intellectual Development in the College Years. San Francisco, CA: Jossey-Bass (a reprint of the original 1968 work with minor updating).

 

Tobias, S. and Raphael, J. (1997). The Hidden Curriculum ó Faculty-Made Tests in Science, Part I Lower-Division Courses. New York, NY: Plenum, 139-141.