Thursday, July 1, 2010

Multiple Choice

I have some friends who employ multiple choice tests in their philosophy classes. The primary motivation seems to be that such exams are much easier to grade, and I must admit that sometimes, when I'm about to dive into a pool of poorly organized, poorly thought out, illegible bluebooks, I am tempted to follow suit. For multiple choicers, exam day is no big deal; for bluebooksters, it is very very bad. With four classes, I lose an entire week. One friend got on the multiple choice track by giving exams where half was multiple choice and half was short answer/essay questions, and then noticed that scores were highly correlated with one another. So she cut out the essays. She says, "so why am I busting my balls to grade these fucking essays?" She was getting the same information from the multiple choice questions.

I dabbled in multiple choice once back in grad school, and it seems to me that the drawbacks of multiple choice are decisive. For one thing, they require a heavy upfront investment. It is much more difficult and time-consuming to write a multiple choice exam than it is to write an essay exam. You need at least four possible answers per question, where exactly one is definitely the right answer, and the other three are definitely not right, but they have to be close. I find I don't have the knack for this. I found that I ended up mostly writing questions where there was a trick or which were otherwise unfair.

And the other thing is, and this is more important, I worry that multiple choice tests emphasize the wrong things. While I think it is important for someone with training in philosophy to know, for example, what Descartes did and did not say, and to be capable of picking the thing he didn't say out of a lineup, it's just as important for that person to understand why Descartes said that stuff. And I think this is where philosophical training becomes useful to non-philosophers. Being able to understand what the arguments are, how they work, how their authors defend them, and what their critics think is wrong with them is a fundamental part of a philosophical education. And gaining a sufficiently deep understanding of how arguments and patterns of reasoning work that it is possible to apply the lessons to non-philosophical situations is an important benefit of success in philosophy classes. And it is more difficult to demonstrate this understanding in a multiple choice test, and students who are preparing for a multiple choice test are less likely to thereby develop these skills. I'm not saying it can't be done; I'm saying it has to be a very well-written exam, and it would be very difficult and time-consuming to write such an exam. And I've never seen a multiple choice exam that I thought would be effective at this.

--Mr. Zero


Matthew Pianalto said...

A couple thoughts:

1. My current intro-level exams are 1/2 multiple choice and 1/2 essay (in weight). What I did away with were "short answer" questions. MC questions, for me, get at what I tried to examine with the short answers. So they can help if you want to have an "objective" component to your exams.

2. The upfront investment pays off when you're grading! (The best policy is to have a mix of "easy" and "not-so-easy" questions. You have to do it a few times to get a rough feel for how to balance.)

3. I'm teaching an auditorium class (capped at 100 students) for the first time this fall, and have been told that it's fine if I want to eliminate the essay on my exams. (I might have an in-class assistant but no real TA's or graders.) I haven't decided about this (in part because the essay would be the only real chance students would get for any "personal" attention in such a large class). But I'm tempted to do it. You can, by the way, test why. (Suppose you've covered Descartes' conceivability argument for dualism; write a question offering several lines of reasoning that might have been the one RD used to conclude that he is not essentially his body. Students will have to read each answer carefully to identify the set that best characterizes how RD actually reasoned.) The one thing you can't really test with MC--and have a "right" answer--is what the students think about the various ideas and arguments, and thereby their ability to draw comparisons and contrasts, to synthesize, etc. But you could presumably get a good picture of which students would probably be able to do that from a carefully crafted MC exam. (At least that's what I'll be telling myself this fall, probably!)

Anonymous said...

I use two high-stakes exams (10 T/F, identify, short problem to solve or identify author of quote) and then a final paper. Some students bomb one of the exams, but the advantage for the instructor is that they are really easy to grade. You can offer extra credit to help out some of those students who bomb an exam. The nice thing about final essays is that you do not have to read them very carefully because students rarely ask for them back.

Ben said...

A bad multi-choice quiz tests factual recall (e.g. What year did Descartes write his Meditations? Which of these did he say?)

A well-worked test can however test understanding. Here's what I think is a pretty good example, though it's binary nature arguably makes guessing too easy:
(I assume the pass mark would need to be 80-90%)

Writing such a test would, I agree, involve a lot more up front work. And I'm not sure it could be re-used year after year (though if you set twenty random questions from a bank of 50, that might work)

Anonymous said...

Do you disagree with your friend that multiple choice and essay scores will roughly correlate?

If not, then how/why do you say that you can't devise a multiple choice test for things things like "*why* did Descartes say what he said?" If your friend is right, even factual/memorization type multiple choice questions should roughly correlate with deeper and more-difficult-to-grade essay questions.

Anonymous said...

I use multiple choice, and like others have said, yes, it takes the upfront work but in terms of overall time, this is easily paid for by the reduced grading.

It also definitely takes some development and buildup to get good exams. When I first did it, my questions were way too hard which I had to deal with using ridiculous curving. They're a lot better now. I think I'm also better at writing new questions, so that as I go along, I can create new or updated exams without having to gradually improve every question.

One thing that I think helps with the application, is to get your students to associate understanding the material with doing well. In a sense, the MC exam only tests for a student having memorized a generic outline of the material. This is why the exam seems cheap to us. But intro students can't memorize all the material if they don't understand it. I like to tell the class that the best way to remember the material is to understand it - it *really* helps. This may be why your friend found the correlation between MC and essay scores.

Anonymous said...

I think I agree with everyone else who's posted on this so far in that:

(a) if well worded, it can test understanding,
(b) writing such well worded tests is a LOT of upfront work,
(c) the questions have to have at least one, if not two, answers that might seem right on a casual read but are not, in fact, correct, in order to test said understanding.

Too hard for me, man. I use short answer and essay. Pain to grade, sure, but I just can't (or haven't yet) figured out how to write MC tests that conform to the crucial criteria.

Anonymous said...

Two years ago I completely reworked my Intro exams by including reading comprehension sections. Basically, I give a passage from one of the texts we read and ask various questions about that text. I have been very happy with the change.

Some of the questions involve mere recitation of facts (for example, I always include fill-in-the-blank questions asking for the author and title), but many of the questions--usually multiple choice--actually require engaging the students' reasoning skills. For example, I might ask them which answer best completes the line of reasoning presented in the passage, or which choice best represents a contradictory or opposing view to the one presented in the passage, or which general theory we discussed is being argued for, etc...

This way of doing exams does require a lot of up front work, and it is the case that the test can't be safely reused without significant modification. But I think Matthew Pianalto is right that the up front investment is paid off by the returns in grading. Also, I am not sure that I think it is much of a loss that they can't give their own views on the issues. They will have plenty of time in their lives to blow their own smoke, and should learn how to properly assess other views first (and they get a chance to have a bit of their own say in class discussion).

One last point--I purposely include trick questions in the multiple choice sections. Surely one of the most important skills any philosopher should have is the ability to read through difficult texts carefully, with a critical eye. I think a good way to test this skill (and to inculcate the skill, insofar as students who get caught out by the first exam will be more careful on subsequent exams) is to include various tricks that they should be able to catch if they are careful.

Anonymous said...

For my Intro-level, I design multiple choice quizzes weekly via Blackboard (even easier to grade--Blackboard does it for you: on a four-option MC quiz you tell Bb the right answer), each one (a 10-question MC quiz) takes me about an hour to 90 minutes to design.

The other parts of the grade are attendance (for which - if you miss more than 10% of the time - you would start to lose points but not gain them), mid-term (some t/f (10pts), some mc (10pts), some short-answer (15pts) and one long answer (15pts.)), and a final (same). There is extra-credit offered at the end so people can make up some points lost to absence or to poor performance on the exams.

I resorted to Blackboard and MC quizzes to get a sense of who was doing the reading, week-to-week. They were designed to be done in one's room, open-book, but not so easily designed that you could get the right answers without actually doing the readings. But if you did the readings with a pencil in hand you could ace the quiz.

This was an Intro to Philosophy class of 240, with 3 TAs to help with grading of mid-terms and finals, and who took small sections on their own once a week.

I agree with your goal (or 'outcome' as they like to say now) of wanting the students to be able to understand an argument (say, from Descartes), take it apart, criticize it, and put it back together again -- the skills this requires seem to me useful beyond the classroom - but my experience is that, lacking prior training, only about 20% will be able to learn how do this during a 1-semester course.

Especially if one is teaching an historical course (e.g., 300-level Ancient or Modern Phil.) it seems to me difficult to both communicate the content (the ideas of Descartes, Hume, Kant, et al.) while at the same time educating students as to the finer points of textual or argument analysis. I find even the ones who have aced symbolic logic (some of whom aced it with me as their teacher), can't transfer that knowledge to analyze an argument when they approach a text by Aristotle or Descartes or Spinoza.

Ideally, there would be a specialized critical thinking sort of course that would teach these skills, required of all students even before Logic 1 or Intro to Phil., but from all the trends I've seen in curriculum reform, that isn't going to happen because -- the gods only know why -- the geniuses behind curriculum reform don't seem to think this is a valuable skill.

Mr. Zero said...

to the person who left three comments, each of which was shorter a shorter version of the one prior: which one did you want me to approve? The short one, the long one, or the middle one?

Anonymous said...

One important reason not to use MC is that it does nothing improve student writing. And in my, admittedly limited, experience one of the best things we can do for students that take a philosophy class is (at least begin) to teach them how to write well: how to make a claim and argue for it in a semi-sustained way. This is a skill that will serve them well in whatever they do. My feeling is that repetition is the key to improved writing. As such, I only give essays *or* very short argument-reconstruction assignments (which I see as training students in the building blocks of writing a longer paper) since I don't think students get much out of just writing one, or even two, papers.

Having said that, I'm lucky to have classes of no more than 30, very strong students and no more than two classes per term. I can see that with a very large class (i.e. more than 50) and/or course load, only assigning papers simply isn't feasible.

Anonymous said...

The use of objective evaluation is a pragmatic matter for most teaching philosophers. Anon 5:44 is right--if we all had fewer that 60 students per semester then promoting good writing would be not just possible, but preferred. But I've never had less than 100 students, and doing nothing but evaluating writing is impractical. All my classes involve some writing, but it's simply not possible to evaluate students on that basis alone.

I was fortunate to have an undergrad experience with one great prof who also was inundated with students but solved the problem with exams principally based on MC questions. He taught me that such exams can be constructed to really test understanding. For example, provide a question that has several good and even correct partial answers, but also provide alternatives of conjunctions of given answers as the best answer. For example:

Hume believed that:
a. causes must have hidden powers
b. causes are based on psychological constant conjunction
c. causes can be accurately predicted in all cases based on the past
d. causes cannot be accurately predicted based on the past
e. a & c
f. b & d
g. a & d
h. a & b
i. c & d
j. none of the above

Here you even have the perogative to give partial credit for partially correct answers--so f. gets full credit but b or d gets some too. Of course using scantron or the like is out of the question (unless you have access to more sophisticated electronic correction than I do) if you take the partial credit option. My own old prof didn't--if you didn't answer f. above, you were toast. Does it take a lot of prep? Yes. On the other hand if the questions are crafted on well-understood interpretations of basic philosophical literature, then subjective factors of assessment are marginalized on these exams.

For later use of the exams, scramble questions/answer sequences, and insert some new questions. As time goes by, the combinations will conquer any effective attempt to cheat by recycling exams. (Or more cynically--collect them after review.)

FWIW--I think my undergrad prof gave me a fabulous grasp of the history of philosophy, and primarily with these MC exams.

(my verifiction--smst)

Anonymous said...

I agree with Anon @5:44 that one of the most important things we can do for the students is help them with their writing, but my first position involved teaching 4+ sections with 45+ students in each per semester. So, while I did have the students write two papers, I did not have them write as much as I think we should to really help them develop as writers. (Let's be honest, $32K only gets you so much.) I started using MC tests in addition to the papers because it forced students to do the reading. I think that's an important consideration. I don't know how much you really help the students if the students aren't reading the material and absorbing it, and one way to try to motivate them to read is to give them MC tests. They would bitch and moan about how horrible it was to have to take those tests, but they did read more, they attended more, and their papers were better for it.

From my experience, the test questions that were really good tested the implications of a view or forced students to identify an author's response to an objection. Those were the kinds of questions I could put on a MC test with a relatively clean conscience.

word verification: polly

As in polly want a cracker. As in, I guess it's not so bad having students parrot the stuff they read because then they actually read.

profb said...

anon 7:05 PM:
I don't want to hijack the thread to Hume interpretations, but I'm not sure I agree that d. is correct, esp. since it drops c.'s rather heavy-hand qualification "in all cases."

More importantly, tho, is the complexity of the question. I'm not for dumbing tests down, but psychological research has shown that most people's minds can only entertain around 7 distinct items at one time, and the less familiar the ideas, the fewer one can think about well. That question's set up strikes me as a rather thorny patch to make one's way thru, esp. for students learning the material for the first time and answering under stressful, timed conditions.

anon 3:23 AM:
I had the same thought about getting students to read more, but I found that the threat of quizzes only works well if they know how to read philosophical material, an unfounded assumption for many lower-level courses. So I switched to short writing assignments that try to teach them how to read by having them put around 4-5 pages of text into their own words. That way I know that they have read and thought about at least that section (always pick an important one), and it gives them exercise in reading comprehension and in writing. The gravy is that they're really easy to grade.

Here's a brief description of the assignment:

Anonymous said...

Anon 7.05: I find the Hume MC question completely nonsensical and the supposed correct answer doesn't show any understanding of Hume's thought at all. I don't want to make this into a Hume interpretation thread but questions like show the limitations of MC questions.

Anonymous said...

I like to give exams that are mainly essays, but also to give the possible essay questions beforehand--let's say 5 possible essays, from which they'll have to write on two. (They can prepare their essays beforehand, but the exams themselves are closed-book.) I like to think that, in at least some cases, these essays not only do a good job of assessing how well the students know the material, but that the process of preparing the essays for the exam helps the students learn the material better than they would if it were a multiple-choice exam.

Anonymous said...

I recently taught a summer course in which I used multiple choice exams. I did this with trepidation, but found that it was a successful experience.

Mr. Zero's worry about what students learn is a real one, however, and if the exam alone were my only point of assessment, then a multiple-choiuce exam would most likely be a terrible method to use. However, my strategy was to draw student attention away from exams and towards class discussion. I required each student to make multiple presentation, and I would ask them challenging questions of the kind that direct their attention to the reasons for, and importance of, an author's claims, especially when the students lapsed into merely summarizing. This led to vibrant discussions, in which I had a decent idea about who was able to egnage with the philosophical material in the right kind of way. The exams were relegated to a less significant place for my assessment, allowing me to see who got the mere basics.

I don't think that this strategy would be effective in a large class, but with a class of 20-30, it enabled me to turn my own attention away from exams and toward the important and challenging material that it really is the goal to teach.

Larger lectures, in my opinion, militate against really teaching students how to be philosophically engaged. Administrations love such courses, NOT because they are good for students or instructors, but because they are lucrative. As such, there are limitations inherent in such courses, and we ought to not consider them to have the same educational potential as a properly sized course.

Kimberly said...

Okay, I'm two months late to this discussion, but this is a good discussion of how to create better multiple choice questions:

Testing for Deep Understanding—Ben Eggleston