The March Nutshell comes a bit later in the month than I had hoped. Both spring break and producing a report on the “Ideal Classroom” characteristics of light, color and furniture, now hot-linked at the CeTL home page, took time. Go to that link. As you’ll see, all who teach ISU students are invited to contribute to that resource.

As noted in the past Nutshell (v. 13, n. 2), individual test questions and challenges trigger responses from students to supply information or to use information. A test or test question ideally triggers a response that is representative of what a learner actually does know, but writing good questions that successfully trigger representative responses is not easy. Some simple statistical measures, like those shown in recent Nutshells, reveal that tests are not measures of “actual knowledge,” they are samples of knowledge. Likewise, test scores and grades are simply numerical expressions of samplings of knowledge. We hope that assigned grades result from good samples, but even the best tests are never perfect representations in terms of either reliability or validity. Consider the following challenge: “Tell me all you know.” Reflect for a moment on your own reaction to that question, before reading further.

The feeling that you have captures the affective experience that accompanies an encounter with a very bad test question. Does your response accurately represent what you do know? Surely, you possess an extraordinary amount of knowledge, but such a question is like putting water in a gas tank. It triggers the feeling of the brain’s equivalent to an engine seizure.

Good tests, first and foremost, are products of good teaching. As an analogy in the last issue, we noted that tests are akin to representing complex topography with a sampling of survey points. The topographic area could literally be the Earth, or it could be the apex of a single hilltop. The larger the area relative to sampling size, the less likely the sample can give a good representation of what lies within the area. Thus, the first quality of good teaching related to tests is the need to focus. University of South Florida’s Jim Eison is credited with an oft-cited admonition related to focus: “Teach less better.”

On what should we focus? This brings us to the link between tests and assessment: we focus primarily on content that achieves course goals and produces the stated learning outcomes. The assessment movement adds quite a new dimension to defining “good teaching.” It’s not enough to equate good teaching with high student ratings or “customer satisfaction”; one has to demonstrate that students learned—they actually changed in positive ways as result of taking the course. Further, it’s not even enough to show students are satisfied and they learned. One must demonstrate that what students learned meets truth in advertising: they need to master outcomes that justify the rationale behind offering the course. This is the reason that accreditation agencies do not accept grades as an assessment measure. Even if grades reflect learning, they don’t reveal whether tests provided solid representations of the planned and stated course outcomes (written outcomes being a requisite before one can even begin an assessment process). On what do we focus most? If we have a priority plan based on stated outcomes, then we’ll likely devote the most effort to cultivate success based on our higher priority outcomes. We won’t simply test once on those priorities. Instead, we’ll test highest priority outcomes repeatedly in different ways until we are assured that nearly every student in the class not bent on academic suicide through nonworking, nonattendance etc. has actually mastered that priority material. Thus, the second attribute of good testing is to test important outcomes in multiple ways. It’s the counterpart to teaching content material using multiple modalities.

Knowledge surveys are a wonderful tool through which to enact a plan, because goals and outcomes can be stated, and content questions and challenges written that map onto those goals and outcomes. Further, they disclose a detailed plan to students at the start of a class, permitting superb organization for them and for us. For example, a global goal such as “Understands the methods through which science produces knowledge about the physical world” could easily have a dozen test items through a science course that relate to outcomes that reveal understanding of that goal. Mapping these together reveals the degree to which the outcome was met.

In terms of best ways to write and to grade tests, convenient resources exist. For short-answer multiple choice tests, essay tests and grading, consult the links at the CeTL web site through http://www.isu.edu/ctl/facultydev/resources1.html. You’ll want the link to Kansas State University’s IDEA Papers. Specifically, you’ll want to download and print papers n.16-n.19.