Several states’ data systems found lacking, study says

Several statewide education systems are in danger of failing to meet higher standards for accountability as required by the No Child Left Behind Act (NCLB) come fall, according to a nationwide report released June 18.

Although nearly all states have implemented tests that measure students’ progress toward clearly defined standards, many still lack the infrastructure necessary to deliver results electronically and store them in multiyear databases for purposes of comparison, the report said.

The report is the first in an annual series on “Testing the Testers,” issued by the Princeton Review, a company that specializes in preparing students for high-stakes tests. The firm collected data from every state and the District of Columbia before ranking the testing programs of each. The rankings were based on 25 indicators under four key criteria: alignment with standards, test quality, accessibility of data, and policy or accountability systems. States received scores from 0 to 2 points for each indicator, depending on whether specific standards were met.

According to Steve Hodas, executive vice president for strategic development at the Princeton Review, most states do issue strong, high-quality tests aligned with state standards. But good tests achieve little if states are unable to qualify the results, he said. What’s alarming is the number of states that are unprepared to use test scores effectively to improve instruction.

“Issuing a good test is just the beginning,” he warned.

For example, the study found that many states do not provide educators, parents, and other stakeholders with informative, easily accessible testing information that can be used to achieve continuous improvement, Hodas said. This includes access to electronic databases, which can add accountability to results.

“Policy and openness are the areas where the rubber really hits the road,” he said. “You need to be able to translate the vast amounts of information from these tests into things that you can use. In the past, schools have done almost nothing useful with the data.”

Keeping in mind that most states already administer tests aligned with state standards, Hodas said the Princeton Review decided to use a weighted scale during its assessment in which 40 percent of states’ scores were derived from the quality of the tests themselves and 60 percent involved policy, accountability, and the openness of records.

Under its policy criterion, the study found 21 states did not distribute test results broken down by student and teacher to educators electronically, and results often were not complied in a way that allowed them to be linked to other state or school databases. According to the report, six states—Minnesota, Tennessee, Vermont, West Virginia, Hawaii, and Iowa—either did not respond to the company’s query or do not tabulate test results by teacher and student in any format.

The study also found that a number of states do not operate any form of data warehouse where educators, parents, and other stakeholders can turn to monitor testing progress and trends in schools.

Thirty states scored just 1 point on this indicator, while five states—Nebraska, New Mexico, Wyoming, Hawaii, and Iowa—scored a zero, meaning they make no such data available or the data are available only for a single year. Two-point scores were reserved for those states that supplied multiyear data that include item-response information and student performance figures broken out demographically.

“Data warehouses are important for policy makers to be able to track things over time,” Hodas said. But, he acknowledged, data warehousing does not come easily: “It take a lot of time and thought to create these warehouses and make them accessible.”

What’s more, the study showed that a number of states are a long way from complying with NCLB’s requirement for the disaggregation of test scores.

As early as next year, states and school systems will be held accountable for supplying test data and statistics broken down by different subgroups, including students’ race, gender, and economic status. According to the study, 11 states—Arizona, Maine, Alabama, Florida, Nevada, Tennessee, South Dakota, Wyoming, West Virginia, Hawaii, and Iowa—have no such metric in place. Eighteen states list disaggregated statistics for tests overall but do not list this information for each test question.

Hodas said test-score disaggregation isn’t likely to be included in next year’s study, because all states will have to meet this requirement to receive federal money. Still, he said, it’s surprising how many states have yet to make strides in that direction.

Not all educators agree with the Princeton Review’s assessment. Nebraska, for instance, ranked 39th overall and received no better than an average grade on any single criterion. But Doug Christensen, the state’s education commissioner, said the report gives a false impression that its schools are performing poorly.

According to Christensen, the study is biased on two accounts. First, it only accounts for performance on single state-level tests. Second, it values accountability measures more than school improvement measures.

“Nebraska is performing well,” he said. “The study makes an unfair presumption that state tests are the most important indicator.”

The Princeton Review responds that its study is meant only to show how well states are performing in relation to standards-based testing. The company said it agrees that a state’s testing procedures aren’t the only indicator of success, but these procedures have become more significant in light of new federal policies.

“Certainly many states were well along the path of developing and refining their own testing schema before this year, but NCLB created both new pressures and new templates for ramping up and systemizing those efforts,” the study said.

Hodas added: “The only way these programs are likely to improve is if you are aware of the flaws.”

West Virginia, which ranked 49th overall, is one state that is on its way, according to Karen Huffman, assistant director of the Office of Student Services and Assessment for West Virginia’s Department of Education.

“We are well on our way to making those changes,” Huffman said. “It just takes some time.”

Overall, states scored highest on test quality and alignment, the study indicated. The top five testing states were North Carolina, Texas, New York, Massachusetts, and Arizona. The worst performing states included Wyoming, Montana, West Virginia, Hawaii, and Iowa.

“Good testing programs will help improve education, and bad ones will make schools worse. These tests have increasingly serious consequences, so we should hold the testers to the same high standards that we have for schools and students,” said John Katzman, chief executive officer for the Princeton Review.


School Technology Buyer’s Guide

  • Company Search on “The Princeton Review”

“Testing the Testers 2002: An Annual Ranking of State Accountability Systems”

Nebraska Department of Education

West Virginia Department of Education

eSchool News Staff

Want to share a great resource? Let us know at