Friday, June 4, 2010

Online Student Evals

My school recently announced that in the fall we will switch to a system where evaluations will be conducted online. I am against this for several reasons. For one thing, it means that students who barely attend class have the same chance to evaluate me as students who attend every class. But since students with poor attendance records have a poorer basis for evaluation and are likely to be doing poorly in the class, so their evaluations of my teaching are likely to be both bad and based on insufficient information.

For another thing, it means that I cannot control when evaluation day occurs. As an anonymous commenter recently pointed out, you should never conduct evaluations right after you return a major assignment. I no longer have any influence over whether my evaluations are conducted right after I have returned anything.

It also undermines my ability to more directly control their mood on evaluation day. Before I pass out the eval forms, I like to say a little bit about how much I enjoy teaching, how I approach my role as teacher, and how much we've accomplished so far in the class--I try to emphasize that we've accomplished a lot. I also emphasize that I take my evaluations very seriously, and how helpful I have found them to be in the past. I point out that on several occasions, suggestions that appeared in evaluations have directly resulted in clear improvements in the class, and I give an example of one such change. Then, in keeping with anon 9:43's suggestion, I tell a joke. Unlike 9:43, I make no effort to smoothly integrate this joke into the lecture. I say, "and now, here's a joke!" and then I tell the joke, get a big laugh, and leave the room.

Now I can't do any of that.

--Mr. Zero

29 comments:

m.a. program faculty member said...

My school went from in-class to online evaluations a few years ago, and I was worried about it a bit too, for the same sorts of reasons you give. But it turns out that it didn't have much of an impact on my scores (I expected a noticeable drop), and as far as I know didn't have much of an overall effect either.

My guess is that a lot of the people who seldom show up to class are the same people who won't bother to fill out an on-line evaluation. My school requires that you log into the evaluation page before you view your grades, but then you have the option of declining to fill it out, and a fair number of students opt for that.

And insofar as people can play the sorts of games you describe in order to improve their evaluations (not that I blame you for playing them!), while others don't know about those techniques, doing them all online is probably better--assuming that evaluations are at all informative.

Anonymous said...

Doesn't the importance of manipulating students' moods before evaluations highlight the pointlessness of the evaluation process? Shouldn't we be making this argument on our blogs and statements of teaching philosophy rather than complaining about the loss of such tactics or touting our excellent evaluation scores?

This comes from someone with consistently mediocre evaluations and may therefore be significantly biased.

Mr. Zero said...

... assuming that evaluations are at all informative.

I'm not sure I'm willing to make that assumption.

Doesn't the importance of manipulating students' moods before evaluations highlight the pointlessness of the evaluation process?

Yes.

Shouldn't we be making this argument on our blogs and statements of teaching philosophy rather than complaining about the loss of such tactics or touting our excellent evaluation scores?

It seems to me that we can do both. It's just that this time, I was doing one but not the other.

Anonymous said...

A school that I taught at last year introduced online evaluations in the fall then switched back to paper versions in the spring because too few students bothered to fill them out.

Anonymous said...

The same thing is happening at my campus this Fall. When I raised your concern at a faculty meeting, the chair suggested that it didn't matter, since everyone else in the department would presumably be affected in the same way. If all our numbers go down, then the merit pay standards will shift with them, so there's no real harm.

Except that adjuncts, VAPs, and junior faculty need good evals in order to secure jobs elsewhere. . . .

Anonymous said...

i am not sure class evaluations are good for much. clearly they are easily biased by our color, age, gender, cookies, jokes and etc, and we all remember the sad/funny fact that what students rate a prof they see a 30sec video of is very near what that prof usually gets at the end of the semester. i say all that and i have a bias: i get consistently excellent evaluations, despite having a gender (...) and and accent. but now an honest question: how else to gauge our teaching performance? classroom visits are stressful and thus often unrepresentative. we can't trust students to write letters for us, or can we? but then, class evaluations really are just a kind of popularity ratings. is the instructor fun? etc. sure you can be a tough grader and get good evals. but you have to 'make up for it' in a sense. i know i do. i work hard on accessibility - both of myself and of the material, and sometimes i feel like this is just a fancier way of pandering.

KateNorlock said...

My school went to online evaluations a couple of years ago, and we've noticed a deep drop in response rates ever since. I have to badger the kids with reminders in class to get enough to go home and fill it out so that I have a fifty-percent response rate. When it was done in paper form, I waited for a day when I had perfect- or near-perfect attendance, so that the numbers were more representative.

I used to be a skeptic but have been persuaded by scholars of education that there's just enough soc.sci. evidence to warrant the statement that the numbers mean something.

Also, happily, my averages have not gone down, they're consistent with the paper ones. Er,usually. Actually, they're consistently down about a tenth. But only years' worth constitutes a data-set anyway, which is why VAPs relying on one year's data -- or worse, hiring committees doing so -- is Bee Ess.

Anonymous said...

Re. Anonymous @ 12:26 -- My school had problems with a low response rate, to which they responded by offering the chance of winning prizes for completing the on-line eval's.

Hanuman said...

Three thoughts:

(1) One place I've been required students to complete online evaluations before they received their grades. Unlike M.A. Program Faculty Member's school, this school did not give the option to skip the evaluation.

(2) KateNorlock, I'm intrigued. Can you point us to some of the evaluation-supporting research you have in mind?

(3) Anon 1:48: Cookies!? That's fucking genius!

Anonymous said...

You guys all get to pick which day you do the evals?! I've always been required to do it at the end of the last day of the course.

I don't know about the soc.sci evidence concerning evaluation effectiveness, but I've generally taken them only this seriously: I aim to get my students to say the course was _both_ challenging _and_ rewarding. If I can get them to have learned _and_ to have had both of these feelings about the course, I think I've done a good job. (You might think it should be enough just if they've learned, but if they just learned but didn't feel challenged and didn't find the material engaging, then they're probably much less likely to do anything with what they've learned once the course is over.)

Anonymous said...

Anon @2:43, anon 1:48 here-- it's *all* in the bribes.

like i said, i get good evals, but i don't know what these say about my teaching. that i'm friendly, that they like cookies or jokes, sure. maybe i'm too skeptical, but i find it hard to believe they know when a class is a good one, versus a fun one. (but then who does, and how is it measured). i've also seen good friends get terrible evals along with having real fans among the students. since evaluations are just a part of the consumer satisfaction cycle, not a real measure of product quality (the 'product' here being our teaching), i really find it hard to take them seriously.

Anonymous said...

Possibly of interest to readers of this post:

http://therumpus.net/2010/06/re-online-course-evaluations/

And this one's an oldy but a goldy:

http://home.sprynet.com/~owl1/sef.htm

Anonymous said...

"My guess is that a lot of the people who seldom show up to class are the same people who won't bother to fill out an on-line evaluation."

The solution that they came up with at the last school I worked at was to require students to fill out evaluations in order to get their grades for the semester. My last semester at the school, I failed 10+ students for plagiarism. My scores dipped a bit.

Anonymous said...

@ Anon 1:48 re. how else to gauge our teaching performance?

I bought a small video camera and I record myself while teaching. I watch it and write up a self-assessment. Then I burn a dvd of a class session and let another faculty member review it. It's sort of brutal watching yourself teach, but I have never gotten information of this quality from even what I had considered my most helpful, critical colleagues. Now, I wouldn't do without it.

Anonymous said...

Anon 4am, that's serious!! Sounds almost too brutal, but I guess this is what I was looking for. Thank you.

- Anon 1:48

Anonymous said...

I taught under both systems at my last University: my courses for the University were old-fashioned paper evals, while my courses for the Honors college were on-line.

The students doing the online evals being Honors College students, they were more likely to fill them out, perhaps, but they also told me that, once the evals online went live, they were pestered via email about 3 times a day until they did it, so that seemed to work. I had a close to 100% completion rate on the evals.

I didn't find my scores differed all that much between the paper evals and the online evals. The Honors College students (as might be expected) were more critical in their verbal comments. The scores were about the same.

One advantage to the online evals, I found (and I don't think this has been mentioned yet) is to be considered for job seekers: the online evals come back to you as nice PDFs (at least mine did), with nice comparative charts (easily readable for future employers). So, if your online evals ARE good, they're a lot easier to package and send out (esp. with online apps) to prospective employers.

Let go the need to control. Maybe just find out what day the online evals go live and give a little speech to them sometime during the week before then that alerts them to the fact the evals are going live online, how important they are, and how much you value their feedback, yadda, yadda, yadda.

Anonymous said...

What does it mean when none of your students have evaluated you on the Rate My Professor website? I recall this guy who got a teaching award from Princeton a few years back and not one student had put anything up on that website about him. I found it kind of odd, since students are usually quite eager to either attack you or sing your praises on that site. And he was a good looking guy, so you'd think he'd have at least gotten a chili pepper. Usually the absence of any comments means indifference, and rarely do teachers who students are indifferent to get teaching awards. Odd.

Anonymous said...

i think the rate my profs issue is a matter of institutional culture/norms. i am now teaching someplace where students don't do much of that. in my graduate institution, my students would always write about their profs and TAs. where i taught in-between, they did too. where i am now, i got a single rat my profs entry, despite very good student evaluations. it's a smaller institution, and word of mouth and school forums account for most of this info transfer.

Anonymous said...

Also, many Ivy League schools now give teaching awards to their grad students to help them secure teaching-intensive positions, since there's such a paucity of research-intensive positions out there. With the exception of those who go green with pedigree envy, most search committees see through this BS. One way to counter this subterfuge is to check Ratemyprofessor.com to see if there are any comments. It's kind of obvious when the candidate has taught one class, received a teaching award and has no comments on the site that the teaching award is a steaming pile of BS.

KateNorlock said...

Sorry, Hanuman, I was off the grid for a couple days and just let the intrigue die away.

I'll just quote the report of our own committee, which cited some data like the biggies (Cashin 1995, Greenwald 1997), but noted that there were over 1500 articles at the time Cashin wrote! They summed up as much as they could survey with the statement,

"There seems to be general agreement in the literature that while the correlation is nowhere near perfect, a positive relationship exists between student evaluations and teaching effectiveness as measured by a variety of methods. As with most complex phenomena, the lack of stronger correlations on most dimensions can be partially explained by the fact that teacher effectiveness is only one factor that influences student learning. Among other determinants of student learning perhaps the most important would be student characteristics such as ability and motivation."

Very sensible, it seems.

Hanuman said...

Thanks, Kate.

For the curious, those two papers Kate cited are here (Cashin 1995) and here (Greenwald 1997).

Anonymous said...

Also, many Ivy League schools now give teaching awards to their grad students to help them secure teaching-intensive positions, since there's such a paucity of research-intensive positions out there.

This isn't true across the board (at my school, university-wide teaching awards are by undergraduate student nomination, and sometimes a junior faculty or TA wins it in their first semester or year, if they're good), and your evidence seems rather slim to slime this Princeton guy, let alone the practices of other top programs.

And while I wouldn't expect a search committee to take any award at face value, I doubt it's improved by looking at ratemyprofessors, which tends to draw comments from the extremes (love/hate) without *any* required input from all students, or even proof that the person was enrolled in the class or at the institution. It seems strange to value that more above in-class evaluations or above awards, both of which should be taken with a grain of salt; I mean, it's a site with chili peppers!!

I also trust that SLACs are smart enough to look at the whole package when they hire someone; they're not going to be fooled by this kind of practice, because they read teaching dossiers closely.

All we can conclude from the lack of a profile is that the students didn't feel a need to post anything, which can be explained by institutional norms as much as anything else.

Anonymous said...

Anon 8:37,

Exactly, ratemyprofessor.com tends to attract extremes, and if you're extremely good, that is, deserving of a teaching award, you will tend to get at least one superlative comment. The institutional norms argument sounds like it has a hidden premise, namely, a thinly-veiled form of determinism. I would not put much weight on ratemyprofessor.com comments, but I would take the total absence of comments as a sign that no one really cares enough about your teaching either way, positive or negative. Most teaching awards are decided by faculty/administrator committees, not by student polls. If your advisee was faced with the possibility of not securing a job after s/he earned a Ph.D. at Princeton (or Yale or Harvard), would you do everything you could to get your advisee a teaching award so that s/he had a chance in hell in this terrible market? Probably--sad, but true. Note: There was no personal attack on a specific person at Princeton, no individual was named, so stop being a troll.

carp said...

ratemyprofessor.com tends to attract extremes, and if you're extremely good, that is, deserving of a teaching award, you will tend to get at least one superlative comment.

How do you know this?

The best teacher in my department has no ratemyprof comments at all. Maybe this is contrary to some tendency, but I am very skeptical that you're in a position to know.

The institutional norms argument sounds like it has a hidden premise, namely, a thinly-veiled form of determinism.

Uh... WTF?

Mr. Zero said...

Thanks for the tips, everyone. I've been incorporating some of them and I can already tell the difference. I'm getting more questions, and I'm feeling more confident that when there aren't any, it's because there aren't any. I'm grateful to everyone who commented.

Also, the idea that you can know that a teaching award is bullshit because it isn't corroborated on RateMyProfessor is amazingly stupid.

Anonymous said...

Anon 1:12,

I have to concede the point in the face of your overwhelming evidence, sample size = 1 ("the best professor in my department").

Zero,

It is only stupid if you received a teaching award, happen to be a grad student with very little teaching experience and have no ratemyprofessor comments. If we took a poll, I would bet you a month's salary that anyone teaching 2 or more years has at least one comment on that site, positive or negative. Many of these grad students with teaching awards have less than a year of teaching experience. Their students have no reason to post a comment because they are perfectly indifferent to the instructor's teaching. The advisor has one reason to get his/her advisee an award. Help secure the advisee a coveted job at a teaching institution (given that these are the slim pickings left in this horrible job market).

carp said...

Dude, I didn't claim to have "overwhelming evidence" for anything. (I note, by the way, that you have exactly one fewer data point than my one data point.) I asked how you know what you asserted, and mentioned why I am skeptical. I guess you can't answer that question.

I'm with Mr. Zero on this one: "amazingly stupid" is apt.

Anonymous said...

Many of these grad students with teaching awards have less than a year of teaching experience.

It's many now? I thought we were talking about one guy. One might also think that the short teaching tenure might explain the lack of RMP comments.

But let's set the RMP thing aside, and let's suppose it's exactly as you say. You're wrong about the practices at at least one of the institutions you mentioned, but let's suppose that you're right, and all of these award winners are mediocre instructors with less than a year of teaching experience, and the awards are internal to the department. And their advisors are doing this in their second or third year so that in another three years, they can beat you out for a job.

Is this not going to be evident when it's clear that the award is departmental, and that the person's evaluations are, as we stipulated, mediocre? How naive do you think that search committees are?

More likely scenario: the person who wins a departmental award is probably quite good at teaching (after all, there are good teachers at these schools, too), and the SLAC takes departmental awards with a grain of salt.

Anonymous said...

My grad institution, not an Ivy but a very good R1, had a very well developed system of making all student comments from all course evaluations available online for those in the institution. So students virtually never bothered to use ratemyprofessor, since they had a much better way of accessing other students opinions.

So an absence on rmp can mean nothing at all.