Tuesday, July 19, 2011

Multiple Guess

I am on record as being opposed to multiple choice philosophy tests. But suppose I have been tapped to teach a large introduction to philosophy course that will meet in a humongous lecture hall, and that I won't have any TAs or help with the grading, but I will have 150 students in this one ginormous class. I'm thinking that multiple choice might be the way to go. But I still sort of worry that these things are frowned upon. Suppose I get a nice interview at the APA, and the committee says, "I see you've got a humongous lecture class here. How'd you do it?" If I say, "Multiple choice tests..." will I be fucked? Or will the people be understanding and merciful? I plan to work in some in-class writing, but this wil have to be highly informal. What do the Smokers think?

--Mr. Zero

31 comments:

Alex said...

I wouldn't rely solely on multiple choice, especially if you're already weary about it.

Have you considered having students grade one anothers writing? If you give them a strong rubric to go off of, have two students grade each paper, and if it's done anonymously, it could save you a lot of the work AND expose students to a variety of alternate viewpoints.

You could also have them write ungraded reflections/answer a 1-2 page essay question at the end of each week and give them a small amount of points based on whether they completed it, and then at the end have them expand one essay of their choice into a much longer paper, addressing whether their view changed since then. They write a bunch, but you only have to grade one paper, then.

Anonymous said...

Why do you think there's something wrong with multiple choice tests in introductory Philosophy courses? I've never done it myself, but I don't see anything wrong with it in principle.

In my view, if you use a multiple choice test and are later asked about it, you have no good strategy other than to be straightforward and unapologetic about it, and explain what your reasons were for doing so.

You may have to be apologetic in another sense, though, in that you may need to preemptively answer objections to the use of multiple choice tests. I don't know what the objections are exactly, but if they come up in this thread I'll try to explain why I don't think the objections work. (Or I'll change my mind!)

I think you can use a multiple choice test very easily to determine whether students:

Understand more or less subtle distinctions between positions,

Understand what counts as a good reason for thinking what,

Are capable of keeping different authors' views straight and separate from each other

Understand the intellectual motivation for various authors' statements and views

Are capable of understanding which philosophical problems are dealt with under different branches of philosophy

And I'm sure there are many other things you can use a MC test for besides.

This seems to me like a good list of course objectives for Intro to Philosophy.

Anonymous said...

If I were interviewing you, I would ask about the nature of the class before I judge you.

For example, is the class a gen ed that has a writing requirement? Does the class count for *two* sections? Do you have grading help?

If you have a course management system, then I recommend giving short reading quizzes that you can grade online. This way, you don't have paper to deal with at all. That will make life easier on you too. You can make comments right on the from, input the grade, and forget about it.

Make thoughtful multiple choice exams, have copies to show at the interview, and I think you will be fine.

Mr. Zero said...

The main thing I worry about with MC tests is writing. Philosophy is mostly done via the written word, and skills involving clear formulation & communication of difficult ideas are among the primary benefits the study of philosophy has for non-philosophers. MC tests don't involve writing, and I worry that when students know they're studying for a MC test they don't practice those writing skills.

Anonymous said...

I don't think any sane search committee would dismiss you for using multiple choice exams in that setting, especially if you also do some sort of writing exercizes.

For the record, although the large majority of the grade in my intro classes is based on written assignments, I do use multiple choice questions (typically in short quizzes). It's the only way I know to check whether the students have read / understood *all* the readings. But I agree that students aren't really doing much philosophy if they aren't writing.

Anonymous said...

I do in fact run my Intro class as a writing intensive course. (But I don't have them write a paper. Instead I focus on the idea of concision, clarity and accuracy in brief write-ups.)

But in my view, writing is not a necessary component of an intro course. If my intro classes got very big, I'd probably stop concentrating on writing so much, not because it's "inconvenient" but rather because it's impossible to teach writing in a semester to more than four classes of 20 students or so.

So instead of writing, I'd try to help instill habits of carefully accurate, clear and concise thinking by writing a damn good objective test. It would require students to think clearly and accurately about more or less subtle distinctions between abstract concepts. Multiple choice would work great, and I could use Scantron!

Anonymous said...

This is a great opportunity to dispense with the traditional 2500 word essay and take up Eric Hoffer's challenge:

"Wordiness is a sickness of American writing. Too many words dilute and blur ideas. There is not an idea that cannot be expressed in 200 words. But the writer must know precisely what he wants to say. If you have nothing to say and want badly to say it, then all the words in the dictionaries will not suffice."

You, and your students, may be pleasantly surprised by the results.

FemFilosofer said...

I've used MCQ in conjunction with essays for a writing-intensive Intro course in the past. The reason I added the MCQ to the mix was because I did a great deal of writing development exercises that didn't always pull straight from course content.

I think MCQ can be a great way to assess students' comprehension of course content if the instructor puts the time in up front to write good questions. This is sometimes harder than it seems.

Also, I've done quizzes that were only True/False, but required the students to give a short reason why the statement was either true or false (supporting idea for T, correction of statement or reason for F). These were fairly easy to grade, and required students to connect ideas in a similar way to how they would when writing.

And, I think other Smokers are right; I can't imagine anyone being too hard on you for utilizing MCQ in a mega-section, as long as you can talk thoughtfully about your choice.

Anonymous said...

I went to a very small college with a one-person philosophy department. The professor mainly gave long MC exams that were amazingly tough and really weeded out the know-nots from the knows. Philosophy majors there did write quite a bit, but mostly only in independent studies (one on the early moderns required 8 10-page papers). From my 4-year stint 3 of 5 total majors (including me)went on to become professors of philosophy at state universities (I say that only because our college was a religious school but none of us stuck with that tradition as a possible "in" for future employment). So MC for many courses can be very effective--but they are a pain to design properly. FWIW.

Anonymous said...

Short reflective questions each week that count towards participation sounds like a great idea, but an accounting nightmare. Still, I would be willing to do this.

I had this very question to answer recently myself since I had the option of taking a job where I wouldn't have any TAs and would have large sections (thankfully, something else came along). But I was planning on using multiple choice, plus maybe some t/f and then perhaps some way of testing their knowledge of basic concepts by having them give a short set of definitions. Not ideal, but it has to be doable. I suspect search committees would be sympathetic.

The idea of having students grade each other's own papers seems atrocious at so many levels. Just say "no."

Anonymous said...

When I was tapped to do this my first year on the job, the chair explicitly encouraged me not to kill myself with grading. I used multiple choice (50 question) exams with weekly online quizzes. I gave them a few chances to write by having some post-exam bonus questions. The course was well-received by the students. The hard part is to write good questions that require some critical thinking, and not just regurgitation. But it can be done.

Anonymous said...

There are trade-offs here. Can you do writing assignments for 150 students and grade them fairly without it taking too much time away from your research? If so, great -- do it. If not, then you have to weigh how likely you think that question is to come up and how likely you think using multiple choice is to hurt you.

I suspect that the time can be put to better use.

Anonymous said...

To focus on the question about mentioning that you use multiple choice tests in an APA interview:

I would strongly advise that you not mention to anyone that you use MC test, nor should you include one, or in any way indicate the use of one, in your teaching portfolio.

Sane or not, I have met and spoke with multiple people in multiple departments who have extreme contempt for the use of multiple choice tests in philosophy classes. I think it is a fair guess that such people would judge your teaching negatively for using them. Such people are surely in the minority, but you have no idea where they are or who they are, so don't risk it.

Anonymous said...

I spent a few years teaching 4-6 courses per semester and struggled with my grading responsibilities (naturally). I used MC exams to make sure that students did the reading. It kept students engaged. I had students do an individual paper that I marked for them (that did take about a solid week of around the clock work) and then did a final paper that involved group presentations for the last two weeks of the class. This gave me a chance to give them instant feedback on rough drafts of their work and because we traded in lecture time for this, it was doable. I graded their second papers and if students wanted their comments back (less than 10%) they could come get them, but I didn't write up comments unless specifically asked. They already received some verbal feedback from myself and from students in the class. It was part of their final grade to see how well they were able to revise their papers in light of the questions that came up. Overall, it was something I could live with. The grading load was difficult, but there was still some time to get some of my work done. Since each student did write two papers over the course of the semester and received feedback on both, I thought I could live with that even if it wasn't ideal.

Anonymous said...

There are only two types of philosophy professor. Those who give multiple-choice tests and those who have never taught 100+ students in a semester.

BunnyHugger said...

I agree with Anon 7:03 that although multiple choice tests have a bad reputation, they can be used effectively to test the ability to distinguish between positions and claims, identify appropriate arguments, understand implications of positions, and so on. I don't use them when I teach Intro (but I've never had an Intro section that big). I do use them when I teach Business Ethics, which is (perhaps obviously) a "class for non-majors" at my institution. I also use them sometimes when teaching Moral Problems (i.e. "intro level moral and social problems/applied ethics class") although when I do that, I combine them with an essay question. I'm not really ashamed of it. I teach 180 students per semester.

Anonymous said...

I've taught a course with 200 students and used a test composed of short answers and essay questions. Of course, my TAs graded them.

wv: mulfoc: what you call someone whose pissed you off when you're too drunk to say it correctly.

Anonymous said...

Here's a way to use Multiple Choice questions that most likely will impress hiring committees: Use iClikers. Do multiple choice quizzes *in the classroom* at least once a week (if not in every class).

iClickers are best used for this purpose with large groups of students. If you don't know about them, ask the Instructional Development or IT at your university/college. See if they can set you up. You order clickers for students with regular textbook orders.

You don't need to present more than a small set of MC questions spread out over a class session. You can allow students to debate an answer among small groups or answer individually. The software records the answers and grades each quiz. The software also allows you to display the answers (pie charts, bar graphs) and everyone in the class can see the right answer and you can address the mistaken answers right then and there. The activity involved keeps students alert. If there's a quiz (a few questions) each class, they have more reason to keep up with readings.

By integrating MC into the class instead of on an exam, this leaves more time to grade written short essay or short answer exams.

Ben said...

I've heard stories of MC tests that focus on trivial details such as what year Descartes wrote the Meditations. I think such cases give MC testing a bad name. Though I've never used MC myself, I've come to see that it is possible to test understanding. Here's a fine example by Doug Portmore:

http://www.public.asu.edu/~dportmor/hau.htm

Anonymous said...

10:04 again:

I would absolutely not make a big deal about using clickers, unless you have been specifically asked about using new teaching technologies (or you are otherwise sure that the interviewers are interested). And if you do talk about them under such circumstances, make sure you balance out your discussion of them such that you note their weaknesses.

I mean this seriously and am not just trying to be the thread contrarian. There are a lot of people out there who are very conservative regarding such things, and who will think that clickers are frivolous and that their use is a sign of lazy or unserious teaching. Again, these people might be in the minority, but they are definitely out there--so why take the chance when pretty much everyone will be impressed with talk of good versions of regular old teaching?

Anonymous said...

By all means, try to design challenging multiple choice tests and use them. But at the end, if you are dissatisfied with the tests, I think it would be overly fastidious to blame yourself. The college or university that has 150 person classes with one professor is essentially demanding that you not assign papers. This is ONE class. How long would it take to grade (in a reasonable manner) 150 5 page papers?

Moreover, you are a visitor or an adjunct (I'm assuming). It would be ridiculous to expect you to grade that much for one class.(It would be _mostly_ ridiculous for a tenured or tenure-track person, but at least there the school is making some kind of commitment to you, and through you, to the students).

And if anyone who interviews you cannot appreciate the pedagogical situation that the school put you in, then you probably don't want to work there.

Anonymous said...

Exploitation to the max. At my university, tuition is about $1000 per 3 credit hour class.

150 students X $1000 = $150,000!!!

Now I understand that there are other costs, and I haven't calculated the wrap rate for colleges and universities, but the point is that *you* are paying for a lot of administrative costs with your one class and your measly salary.

As yourself this question?

How many tuition dollars do I generate a year as a teacher? Then ask if you are getting enough for what you do.

On average, I teach about 250 students a year as a tenured faculty member. My pay is pretty good, but it isn't anything close to the $250,000 in tuition my classes bring in a year. If the wrap rate for the university is 4x my salary, then we are breaking even, but I doubt it....

The problem is that much of the costs of the university are fixed and have been paid off for a long time.

A lot of people are making a shit pot of money off your labor. Don't kill yourself doing it.

Anonymous said...

The link to Portmore's exam is helpful. Other guides on designing good multiple-choice tests?

Anonymous said...

2:12:

One way is to be hard assed and give (say) four answers, one of which is correct but then add a couple of conjunctive combinations and an AOTA & NOTA, and then only accept the best answer (easy one):

Descartes was:

a. an empiricist
b. a rationalist
c. an idealist
d. a transcendentalist
e. b & d
f. a & c
g. all of the above
h. none of the above

Such questions can be used to not only test for grasp of content, but (for other similarly formed questions) for grasp of logic as well.

Speaking of which, if you've not tried Jim Propp's Self-Referential Apptitude Test on yourself or your logic students, you haven't lived, philosophically speaking:

http://www.drunkmenworkhere.org/170

wv: catios: the journal strictly for feline thinking

Anonymous said...

Many online systems (e.g., Blackboard) let you do quite sophisticated things with MC tests/quizzes. I've started using them for my big intro (140 students per section) and I always use the version where they have to click all the correct answer choices for each question, and then of course the number of correct answers varies.

(Nobody in the sciences would think twice about using MC exams. And they are actually taught to design and test their exams and exam questions. It's hard work--among other things, you will need to write many more questions in order to randomize which are presented, to prevent cheating.)

Anonymous said...

...you will need to write many more questions in order to randomize which are presented, to prevent cheating.

If you randomize, then one possible outcome is that some versions of the exam are more difficult than others.

But even aside from that, why not just create multiple orderings of the same set of questions? It's a bit of a pain to produce multiple versions of the same exam. But that's easier than 1) coming-up with additional questions and then 2) ensuring that every version of the exam is of equal difficulty.

Anonymous said...

But even aside from that, why not just create multiple orderings of the same set of questions?

Whether this will stop cheating depends on how much time they have, how many questions you use, whether they can save and continue, whether they can re-try, and so on.

The easiest thing is to write equal difficulty questions. The other thing is to break them down by difficulty and have it offer equal numbers from each category. You still need to write lots of questions; but you can re-use them year after year, just adding more to the question pool each year, while cutting loser questions.

zombie said...

I've used MC questions on tests, along with some short answer and short essay questions. I've never once in an interview been asked what kind of tests I give my students.

The benefits of MC: quick and easy to grade, objective answers. Drawback: You have to come up with a lot of questions, and at least four possible answers for each one. There's a lot of prep in writing a MC exam.

Other disciplines do it. Sufficiently challenging questions that truly test knowledge are quite possible. I always tell my students during the first lecture that I absolutely will not ask them about dates or historical events on a test.

zombie said...

You'd be surprised how many students don't bother to even guess when they don't know a MC answer. They'll leave the question blank.

Ben said...

I can't vouch for whether Zombie is right about students not guessing, but if that's your concern then you can always mitigate it with negative marking. Something like 2 points for each right answer and -1 for each wrong answer.

That way, a one-in-four guess has negative expected value, though a student who can narrow the choices down to one of two may be well advised to guess (but would do less well over a series of such guesses than someone who knew all the right answers).

Anonymous said...

I have taught a 100-student intro course many times and capitulated to using MC tests. The way I do it is by creating a large repository of questions and distributing a small minority of those questions randomly to each student (I have a complex way to keep the distribution fair)--all online. The exams are short but difficult, and I release them at the beginning of a unit so that students have something to focus on as they read and listen to lectures. Rather than being about minor reading details, the questions are about major concepts that are covered in the reading and the lectures--and about how concepts are mutually related. Let me emphasize that these are not knowledge exams like you would see in a science course but are comprehension exams to see if students understand complexities and subtleties of the material. Because every test is mostly distinct from the others but with significant conceptual overlap, students are allowed to help each other, and I allow students to ask about them in class. The results? A normal curve. Students like it, and I like it. I have them do informal writing in addition, which they submit online and that I grade credit/no credit.