In an age when district funding and even teacher pay raises are tied directly to student performance on state exams, it can be tempting for educators or school leaders to cheat the system, such as by giving students the answers to test questions in advance. Indeed, instances of fraud in the administration of high-stakes exams are on the rise since No Child Left Behind (NCLB) became law in 2001.
Now, in response to this trend, a growing number of states are contracting with outside firms to guard against such testing fraud.
When North Carolina recently hired security company Caveon Inc. of Utah to guard against cheating on standardized tests, it became at least the third state to outsource security services for its high-stakes testing to Caveon. South Carolina and Delaware have signed similar agreements with the company.
The security services that Caveon provides to guard against cheating on high-stakes tests are exhaustive. Some say such services are positive developments that will help ensure the fairness of high-stakes testing in the NCLB era. Others, however, believe that no amount of testing security can change a nationwide standard for testing that is itself critically flawed.
Caveon is the first company specifically to provide security services for high-stakes testing in K-12 education. Some people have made the analogy that hiring Caveon’s services is like having a sign in your yard warning that your house has an alarm: The sign alone acts as a deterrent, even if the system is not in place. The argument goes that by merely hiring an outside firm to investigate test security, cheating will be curbed.
Don Sorensen, a spokesman for Caveon, said that schools usually have some kind of security system in place for high-stakes testing. “The problem you have nowadays is that the methods of test fraud have become much more intertwined with technology, and the stakes are higher because of NCLB,” Sorensen said. “Whenever there are high stakes, there’s a reason to cheat.”
Although agreeing with his colleague, John Fremer, founder and senior director of testing security for Caveon, pointed out that educators are “really a pretty honorable crowd–you don’t go into teaching because you want to make the most amount of money you can.”
A primary benefit of the company’s services for state and school leaders is that “they’ll be able to clear things they might have had to worry about,” Fremer said. Areas of testing deployment that could be a cause for concern can be investigated, and administrators’ worries can be eased.
For example, “What if you go from the 35th to the 66th percentile?” Fremer asked rhetorically. “Was that because you had great educational collaboration, or is there something to worry about?”
Caveon provides security audits that examine high-stakes testing procedures to determine the security strengths and weaknesses of the educational body being examined. The company then helps design best practices for schools to improve security.
Fremer said that each contract begins with the company’s auditing service, which questions those involved in all aspects of test administration. He said Caveon officials interview “IT security people, test developers, file maintenance [workers], investigation report reviewers, state follow-up people,” and other administrative officials.
Auditors ask a series of questions and use the answers to evaluate the security situation on the ground. “For instance, what are the directions you give to test administrators? What are the directions you give when the administrator says he or she thinks there has been a problem? What do you permit teachers to do with [testing] materials?” Fremer said. “Or, these handheld devices–my grandchildren have cell phones–what sort of a security threat do they pose? Could you use them to cheat on the exams?”
Terry Siskind, state assessment director for the South Carolina State Department of Education, who was involved in the Caveon audit of that state’s high-stakes assessment system for high school students, said, “I think it is unique for a company to provide this kind of security assessment. In South Carolina, we had a pretty strong accountability assessment program for a number of years before NCLB.”
Siskind said South Carolina did not employ Caveon because of any specific instance of testing fraud in the state (though she did say South Carolina has “the typical problems”), but as the third audit in a series of optional audits built into the contract with its vendor for high-stakes tests, American Institutes for Research (AIR). South Carolina has carried out audits for the scaling of the exams and also the hand-scoring of the tests. In the third year of the five-year contract with AIR, South Carolina officials decided to audit security.
“We just wanted to be proactive,” Siskind said. “The test that we audited is our high school assessment program, and it is used to determine accountability for the state. But it is also our high school exit exam. We want to make certain it is fair.”
Siskind said Caveon interviewed AIR representatives and a subcontractor that works with the company. Caveon also interviewed Pearson Educational Measurement, the writer of the test, and others involved in the testing process.
Stan Bernkanopf, project director for the AIR South Carolina high-school assessment program, said most of the questions dealt with procedures that address security issues “physically and policy-wise.”
“Physically, the questions had to do with things like where we keep secure materials; policy-wise, what kinds of affidavits and agreements do people have to sign to get to see those secure materials?” Bernkanopf said.
“I was surprised at some of the detail they went into,” he added. According to Bernkanopf, Caveon asked what kinds of locks were used to protect secure materials and who had access to those locks–and company representatives even made a site visit to inspect the locks.
Bernkanopf said the Caveon team also asked questions about the electronic transfer of secured materials. “They wanted to make sure that all the materials were encrypted,” he said, “[and] that we sent nothing pertaining to kids’ names.” The questioners also wanted to know how long testing information might be available on an FTP web site if that ever became necessary. “We even had to arrange for a person to be at a fax machine before we sent a fax and get confirmation that the fax was received,” he said.
After the initial audit phase, Caveon moves on to the data forensics element of its security service to search for clues that cheating might have taken place. Caveon uses statistical tools designed to detect aberrant test-taking patterns. These tools notify administrators of irregularities in individual tests that might indicate isolated incidents of cheating.
“We look for things that are not logical,” Fremer said. “For example, students will get more easy items on the test correct than hard items. But what if the reverse is true? That’s trouble.”
Fremer continued, “What if it’s a computer-based test, and the amount of time it took to work on questions is not reasonable?” If an examinee moves quickly through a series of difficult questions, this could indicate that the student had the answers beforehand.
Another element of the company’s data forensics is web patrol. Caveon will look on the internet for evidence that protected test items have been illegally published online. Fremer said the longer the window of time for taking a test is open, the more likely materials from that test will be posted.
When all of these elements are complete, the company will offer a series of best practices for schools to follow. The audit summary provides a complete review of existing security measures and makes recommendations that are rated according to priority.
South Carolina’s Siskind said one of the most significant recommendations that Caveon made for her program was to compile all security regulations into one large “bible.” “The company wanted all the security regulations to be easily accessible to anyone involved,” she said.
Not everyone believes the way to make high-stakes testing more fair is to make it more secure. Angela Valenzuela, an associate professor of curriculum and instruction at the University of Texas and the editor of a collection of essays critical of NCLB, called “Leaving Children Behind,” said new systems for determining cheating on high-stakes tests miss the point: High-stakes testing under NCLB will always be corrupt.
“The assessment gets corrupted because in a numerical display of the scores, one could never parcel out the impact of all it took to get that score,” Valenzuela said. These factors could include “teaching to the test, teaching primarily those children who are likely to increase the school average up”–a practice Valenzuela likened to a kind of “academic triage”–or “otherwise manipulating the testing pool.”
Valenzuela called standardized testing “a factory-model approach that objectifies children into widgets on an assembly line with & the state determining children’s fates through testing.” She said this practice “further objectifies [students] and compounds the problem of a system that has never quite worked to their advantage.”
She added to her critique of testing practices in the United States that “however practiced and normalized teaching to the tests is in some contexts, this [practice] should appropriately be construed as institutionally approved cheating.”
Regardless of how relevant Valenzuela’s concerns about high-stakes testing might be, it appears the trend in outsourcing testing security will become more and more prominent unless major legislative changes affect the way testing is weighted.
“Whether Caveon will flourish, I can’t say,” Siskind said. “But I think the issue will be more important nationwide than it has been in the past.”
Bernkanopf agreed. “More and more states or entities (large school systems, for instance) will start paying more attention to test security issues and will begin to look for outside security,” he said.
Illustrating how large an issue testing fraud has become for educators, Bernkanopf pointed out that it was the first time he’d ever been asked to do a security audit. “It was certainly new to me,” he said. “I’ve been in the testing world for 30 years.”
South Carolina Board of Education
American Institutes for Research
“Leaving Children Behind,” Angela Valenzuela, ed.