Computer-based assessment might seem like a cost-effective, less labor-intensive solution to meeting the increased testing demands of the No Child Left Behind Act (NCLB), but a new report from the University of Minnesota’s National Center on Education Outcomes (NCEO) warns that simply transferring a paper-and-pencil test to a computer will not do.

The Aug. 5 report, called “Access to Computer-Based Testing for Students with Disabilities,” says poorly designed computer-based tests could reduce the validity of assessment results and exclude some groups of students—especially persons with disabilities—from completing the exams.

“For some kids, computers might help them do better—but for some [they] might be a distraction,” said Rachel Quenemoen, a senior fellow for technical assistance and research at the center.

NCLB requires states to have annual math and reading exams in place for all students in grades three through eight by the end of the 2005-06 school year and science tests in place by 2007-08. All students must do well on these tests, because school funding is at stake.

To help meet this challenge, many states are looking at computer-based testing because it provides immediate feedback, is known to improve student writing, and offers the ability to build in assistive technologies such as text enlargement or text readers.

But compared with paper-and-pencil tests, computerized assessments have a number of challenges that still have to be addressed, according to the report.

“When you move these kinds of tests to a computerized environment, you’d think the dynamics would remain the same—but in fact they [do] not,” Quenemoen said.

In a computerized environment, students have to scroll and type instead of turning pages and writing. Text size also is an issue. Not every computer displays text and images equally, which can give some students an unfair advantage.

“The state has to be very careful in defining what large print is on a computer,” Quenemoen said. “This is a fairly big issue, and you have to talk to your developer about this before you begin [designing a computer-based test].”

To create a fair testing environment, the report says, all students must have equal computer speeds and internet connections. “Do your kids have to wait five minutes for a new question to load?” Quenemoen said. “That won’t work if you are doing a high-stakes test.”

Other challenges include poor computer skills such as typing and navigation, the fact that some students might tire faster by reading text on a computer screen, and difficulties in reading long passages on a computer.

The report also questions whether schools have the technology infrastructure to test a large number of students at once, or whether teachers are trained to help with any technical difficulties that might occur.

“The word ‘crash’ has taken on a whole new meaning in our technology-oriented world,” the report says.

Educators also have to ensure the security of online data, the report says.

Even if educators have resolved these technology infrastructure issues, the test’s design must be effective as well. The report makes the following suggestions for designing computer-based tests suitable for all students, including those with disabilities or for whom English is a second language:

First, test makers should follow the seven principles of universally designed assessments.

In June, NCEO released a report called “Universal Design Applied to Large-Scale Assessments,” which identified seven design elements educators should follow for creating a computer-based test that would suit a wide range of students.

Here are NCEO’s seven elements: “inclusive assessment population; precisely defined constructs; accessible, non-biased items; amendable to accommodations; simple, clear, and intuitive instructions and procedures; maximum readability and comprehensibility; maximum legibility.”

NCEO recommends several steps to produce satisfactory computer-based tests:

First, test makers should have a clear definition of what they want to measure, aim for maximum readability, and provide clear, simple, and intuitive instructions, Quenemoen said.

“Retrofitting a paper-and-pencil test to be done on a computer may not be the best idea,” she added. “We recommend starting from the beginning with things that work rather than retrofitting.”

Second, test makers should assemble a group of experts to guide the transformation.

The team should include a variety of perspectives, including experts in design, web accessibility, assistive technology, and curriculum, as well as state and local assessment personnel and educators.

Next, test makers should decide how assistive technologies will be incorporated into the computer-based test.

Quenemoen recommends starting by identifying what skills need to be measured. For instance, if you are testing reading ability, then permitting a text reader doesn’t make sense. If you are only testing comprehension, then a permitting a text reader would be suitable.

Also, make sure you’re testing students’ content knowledge, not their computer skills. If a student has to use an unfamiliar computer during a test, it can be a distraction. “You want to measure [whether] they know the constructs, not ‘Gee, I’ve never used this kind of mouse before’ or ‘I’ve never seen that button before,'” Quenemoen said.

Lastly, consider training implications for staff and students.

“It is important to note that making computer-based testing amenable to assistive technology does not mean that students will automatically know what to do. Educators, especially special educators, need to be competent in technology knowledge and use,” the report says.

The caution expressed by the report is justified for those who would throw together online assessment quickly, said Lan W. Neugent, assistant superintendent of technology at the Virginia Department of Education.

Virginia administered 15,000 course-end tests by computer this past spring.

“Our project, which began in 2000, has been successful because we planned carefully, tested deliverables through demonstration projects, and worked systematically through technical and assessment issues,” Neugent said.

Neugent said Virginia has learned from last year’s exams and is still planning and making adjustments to the project as it moves toward its goal of having all school districts capable of administering the tests by 2004.

Walter Haney, a Boston College professor and senior research associate at the school’s Center for the Study of Testing, Evaluation, and Education Policy, said the report raises issues that have existed since the dawn of pencil-and-paper assessments.

“This issue of minor differences in format—text size—has been a long-recognized issue,” Haney said.

For example, in the mid-80s a “reading anomaly” occurred when results from the 1986 National Assessment of Educational Progress showed a steep decline in reading scores. Reportedly, formatting problems—small changes in font size and spacing—were to blame.

“Formatting in general, not just on computerized tests, can have a dramatic impact on results,” Haney said.

Links:

National Center on Educational Outcomes
http://education.umn.edu/NCEO

“Access to Computer-Based Testing for Students with Disabilities”
http://education.umn.edu/NCEO/onlinepubs/synthesis45.html

NCEO Topic Area: Universally Designed Assessments
http://education.umn.edu/NCEO/TopicAreas/UnivDesign/UnivDesign_topic.htm

Virginia’s Web-based Standards of Learning Technology Initiative
http://www.pen.k12.va.us/VDOE/Technology/soltech/soltech.html

Center for the Study of Testing, Evaluation, and Education Policy
http://www.csteep.bc.edu