*Knipp, D. (2001). Knowledge Surveys: What do students bring to and take from a class? United States Air Force Academy Educator, Spring, 2001, was available until recently at http://www.usafa.af.mil/dfe/educator/S01/knipp0401.htm . When we learned that link was disabled in 2006, we reconstituted the original paper in slightly different web format and minus the U. S. Air Force Academy logos at this site. Users should continue to cite the Knipp article as above, but substitute the new link, http://www.isu.edu/ctl/facultydev/KnowS_files/KnippUSAFA/KSKNIPPUSAFA.html , for the now-defunct link above. The "Department of Physics" above refers to the Department of Physics at the United States Air Force Academy. Dr, Knipp's contact email is Delores.Knipp@usafa.af.mil.

Background knowledge surveys consist of a series of short, simple questions, each of which students can answer in a single-sentence, multiple-choice, and/or web-based "click" format.  These surveys can serve multiple purposes.  According to Dr. Ed Nuhfer (University of Colorado, Denver), the originator of knowledge surveys, such surveys are best used to "disclose" the breadth of the course material to students in a single setting and subsequently to summarize what students have learned  [Nuhfer, 1993].   A well-designed survey can also be used to help teachers determine the most appropriate level at which to begin instruction.  After learning about knowledge surveys at the  "Boot Camp for Professors÷2000" Workshop, I included a knowledge survey as part of my course assessment system.

Knowledge surveys can be especially useful in classes in which students have diverse backgrounds and knowledge levels and/or exhibit varying levels of preparation.  DFP's Fall 2000 astronomy course had a broad enrollment base (about one third physics majors, one third astronautical engineering majors, and one third basic science majors).  I wanted to know at the outset what level of collective knowledge my students possessed about astronomy and astrophysics.  I developed a very general conceptual knowledge survey and administered it on lesson 2.  The survey consisted of 45 questions covering course content from beginning to end.   On lesson 41 I asked the students to take the survey again so they could self-assess their learning at the end of the course.  Significantly, I implemented a tool for assessing the "before" and "after" effects of my course.

I adapted a geoscience survey provided by Dr. Nuhfer for web-based delivery in my astronomy course.  I chose an electronic delivery method so that students could participate in the survey at their own pace any time between lessons 2 and 3 and lessons 40 and 41 of the semester and so that they could access the material any time in between the survey offerings.

______________EXCERPT___________________

Physics 371 Knowledge Survey:

Instructions: This is a knowledge survey rather than a test.  Be sure to fill in your name and SSN.  The purpose of this survey is to evaluate the change produced in your knowledge of astronomy and astrophysics by this course. You will answer the same questions again near the end of the course.  Mark the first response if you feel confident that you can answer the question or perform the task indicated. Mark the second response to the question if you can now answer 50% of the question or you know precisely where you could get the information in 30 minutes or less. Mark the last response if you are at a loss as to how to answer the question. Do not try to actually answer the questions.

Last Name:  First Name:    

SSN:      Example 123456789

3. Briefly describe Kepler's three laws of orbital motion.

3 I know this

2 I know at least 50% of the answer or know exactly where to find the answer

1 I don't know

10. How are absorption and emission lines produced in a stellar spectrum?  What information might absorption lines in the spectrum of a star reveal about a cloud of cool gas lying between the star and us?

3 I know this

2 I know at least 50% of the answer or know exactly where to find the answer

1 I don't know

17. How did the temperature structure of the solar nebula determine planetary composition?

3 I know this

2 I know at least 50% of the answer or know exactly where to find the answer

1 I don't know

45. For how long was the universe dominated by radiation?  How hot was the universe when the dominance of radiation ended?

3 I know this

2 I know at least 50% of the answer or know exactly where to find the answer

1 I don't know

Figure 1.  Knowledge survey response values averaged over all students.  The dark, solid response level is from lesson 2 ("preliminary") and the grey response level is from lesson 42 ("end-of-course").  The questions were generally ordered by presentation sequence within the course.

The survey was a series of questions, approximately one per lesson, on specific knowledge and skills that students needed to possess to succeed in the course.  It queried physical concepts and relations, as well as terminology frequently used by astronomers and physicists when discussing astronomy at the undergraduate level.  Obviously, the survey was conceptual in nature, but I wanted to quantify the responses.  Fortunately, knowledge surveys can accommodate this apparent duality by assigning a value of "1" to a response of "I don't know" and a value of "3" to "I know this"  The middle ground value of "2" goes to subjective answers of "I know at least 50% of the answer or know exactly where to find the answer."  Ms. Carolyn Dull (DFE) helped with the conversion from student web-based "click" input to quantitative EXCELb output.

Figure 1 graphs the average responses I received for the 45 questions administered on lessons 2 and 41. Twenty-eight students participated in the surveys.  A visual inspection of Fig. 1 indicates that some students entered the course with specific bits of knowledge, but generally the students didn't know much about astronomy at the beginning of the semester.  The overall student self-assessed knowledge value was 1.46.  At lesson 41 the averaged student self-assessed knowledge level was at 2.45.  

What did I learn from the preliminary survey?  I gained insight into what background knowledge my students brought to the course.  The higher average response values on the first third of the questions indicated that many of the students entered the course with knowledge about orbital motion and electromagnetic radiation, consistent with material they'd seen in their core or majors courses.  Three questions with initial average response values higher than 2.4 were associated questions on planetary escape speed, the solar wind, and black holes. Students reported knowing about these topics from previous science courses.  The lower preliminary response values on the latter two thirds of the questions showed there was plenty of opportunity for students to acquire new knowledge.  The latter questions dealt more with stellar, galactic and cosmological concepts. 

The end-of-course survey indicated a significant, student-reported, increase in knowledge level.  The reported average was midway between "I know this" and "I know where to find the answer to this."  Responses on five of the questions remained rather low.  In reviewing these questions I realized that I had changed the content emphasis in approximately four of the lessons.  As a class, we had not covered those topics.  Nonetheless, some of the students remembered peripheral discussions or had run across the material in the textbook.

Since these are student self-reported results, it's rather natural to ask if the students really did know the material as well as they thought.  As a test I put several of the survey concepts on the final exam (other concepts had been tested on graded reviews, but I did not tabulate the results).  Figure 2 shows a normalized comparison of reported knowledge level and final examination results from twelve questions.

Some of the questions were presented on the final exam as verbatim short-answer questions.  Others were presented in work-out or multiple-choice format with minor wording changes appropriate to the format. 

A quick inspection of the comparison reveals that the students were slightly over confident about their knowledge on several of the questions from the first part of the course and less confident about their knowledge level of material toward the latter part of the course. 

The latter portion of the course covered material that was conceptually and mathematically new to the students.  They had less experience with this material.  On the other hand, because this material was the most-recently presented, it was fresh in their minds as they went into the final exam.

Besides providing insight into how to focus instructional efforts, this knowledge survey acted as a course road-map for the students and me.  The survey served as a vehicle to convey to students the conceptual knowledge they should possess at the end of the course.  At the end of the course the survey gave my students a qualitative measure of their knowledge gain over the semester.  I presented a summary of the results to the class at the end of the semester.  Additionally, I was able to verify that student impressions of their knowledge level were borne out by the final exam results.  Thus, it appears that first and second-class cadets are reasonably adept at assessing their own knowledge gains in a course. 

Figure 2. Comparison of normalized reported knowledge and final examination results.