The Pennsylvania Department of Education is testing a computer technology that purportedly marks long-answer essay questions with the same accuracy as humans for its state-mandated exam, with the intention of potentially decreasing the number of markers needed and the time it takes to get results.

The technology is an artificial intelligence system called IntelliMetric, developed by Vantage Learning of Bucks County, Pa.

Jerry Bennett, school profiles manager for the Pennsylvania Department of Education's evaluation and reports division, said he was aware of the capabilities of artificial intelligence, though he was skeptical at first.

"I had to see it operate before I could believe it," Bennett said. "So far, the data I've looked at says it's as accurate as two human scorers."

Scott Elliott, chief operating officer of Vantage Learning, admitted that it seems hard to comprehend that a computer could automatically mark prose for focus, content, style, organization, and conventions—and be more accurate than humans.

"When we introduced this several years ago, there was much skepticism," Elliott said. But now, "the proof in the pudding is that, after 30 studies over three years, it's more accurate than expert scorers."

Before IntelliMetric can begin grading, it has to be "trained" to recognize what answers should look like for each possible score of each essay question.

"We would have experts score several hundred papers according to state standard rubrics, and then we would feed [this information] into the computer," Elliott said.

Assuming that essay answers are graded on a scale of one to five, IntelliMetric analyzes the previously marked responses, along with their scores, and it "learns" the pattern of the scores.

"The computer doesn't do any scoring per se, it learns what the pattern of a two or three or four is," Elliott said. IntelliMetric reflects the marking style of whoever marked the sample answers, since those are the answers it learns from.

The software "emulates the process an expert scorer would do," Elliott said. It analyzes the text the same way a human would for grammar, structure, and content.

Unlike typical assessments that take four to six months to return results, IntelliMetric provides an immediate response in three to six seconds.

That's one of the major attractions for the Pennsylvania System of School Assessment (PSSA), the state-mandated exam that is administered in the spring but not reported until the fall.

"We've been doing a number of studies to see how [IntelliMetric] works," Bennett said. In the fall of 1999, the department tested IntelliMetric using answers to an open-ended reading question, in which 11th graders read a passage and responded in writing. In the spring of 2000, about 5,000 11th graders responded to between one and three different open-ended questions.