Monday, July 02, 2012

Computer-Graded Essays


By now, everyone knows that the college-entrance tests require that students write essays.  Those essays can now be scored by any one of many software programs. 

One such program, Educational Testing Service's e-Rater, is described by Michael Winerip in a New York Times article.  His informant, an MIT writing instructor, easily trips up the program.  After all, it can only base its evaluation on "surface" features of the essay.  It gives higher scores to
  • Longer essays
  • Essays that use formal words such as however and moreover
  • Longer sentences
  • Longer paragraphs
  • Essays without sentences that begin with Or or And.
According to Winerip, defenders of the software argue that if you are smart enough to "game" it, you deserve the high score it gives.  They acknowledge that a student could excel by writing nonsense and/or using impressive words, long sentences, and long paragraphs.  The software cannot check facts or notice silliness.  Defenders add that, for graduate-school entrance exams, the software is "backed up" by human raters. 

Essay-scoring software is objectively discussed by another college prof who lays out the pros and cons and offers an up-to-date bibliography.

As e-Rater and its competitors gain popularity, will writing instructors encourage the high-scored characteristics? 

If so, would these longer documents with fancier words add up to obfuscation?  And if pedagogy and product changed to suit the software, would anyone notice?