A reader points me to this 2005 New York Times article, which makes two criticisms of the test. One is that length is an incredibly good predictor of score on the Writing test. Two is that the test does not punish factual errors.

On length: The article claims that, knowing only the length of an essay submitted to the scorers, you can guess the grade correctly 90 percent of the time. However, there are two possible interpretations of this: Smart kids write longer; or, graders really do just reward length. I would guess both might be true to some degree. The way to disentangle them would be to take a representative sample of essays, and give only the first (say) 100 words of each to some graders. If the “problem” is largely that smart kids write long, there should still be a significant relationship between the original length and the grade given. If not, the scorers are grading on length.

And to reiterate my broader point, two separate studies have concluded that not only does the SAT writing predict college grades, but does so better than the other, more objective SAT tests. Were the test so poorly administered that it only graded length, there is no way this could have happened.

On factual errors: I just don’t get what the big deal is. It’s a writing test, not a history test. Since the essay topics are not available beforehand, students cannot memorize a bunch of facts to use — they must rely completely on memory. Not only that, but to penalize any factual errors, a standardized test must penalize all factual errors, meaning that essay graders would have to check all facts. Having been a magazine intern at various times in my life, I can tell you this would at least double the amount of time grading would take.

UPDATE: Something else I just thought of. One of the most important things graders look for is that the respondent answered the question thoroughly. This is inherently linked to length. The article said the shortest essays tended to be only about 100 words — what can you even say in that amount of space? This update alone is more than three-fifths of that.