Princeton, N.J. — Sept. 7, 2007 — A nationwide study of 2,000 public school eighth grade students who answered extended, technology-based, problem-solving exercises, suggests promise for using computerized assessments "of, for and as" learning in K-12 classrooms, according to the primary author of the study. The skills measured include information not easily captured by paper-based testing.
The study, "Problem Solving in Technology-Rich Environments: A Report From the NAEP Technology-Based Assessment Project," was released recently by the National Center for Education Statistics (NCES) of the Department of Education. It provides results from the third and last field investigation in the NAEP Technology-Based Assessment Project, which explored the use of new technology in administering NAEP. The report was authored by Randy Elliot Bennett, Hilary Persky, Andrew R. Weiss, and Frank Jenkins.
The Technology Rich Environments (TRE) study used two extended physical-science scenarios to measure students´ ability to solve problems using technology. The TRE Search scenario required students to locate and synthesize information about scientific helium balloons from a simulated World Wide Web environment. The TRE simulation scenario required students to conduct experiments of increasing complexity about relationships among buoyancy, mass and volume. These scenarios were delivered on school computers or on laptop computers taken into the schools.
"Students were required to integrate technology and cognitive skills simultaneously, just as they must do in real-world academic and vocational settings," explains Bennett. "And, the scenarios were engaging, highly interactive and open-ended so as to capture skills not tapped by multiple-choice tests."
"The study illustrates the use of a new type of assessment that attempts to do three things," Bennett says. "Document what students know and can do (´of learning´), provide information about what they might do to improve (´for learning´), and make them a bit more knowledgeable simply as a function of taking the test (´as learning´). Because the techniques to cost-effectively develop, deliver, and score such assessments are still evolving, the routine use of assessments of, for, and as learning is still some years off."
One interesting result was that there were no measurable differences in performance between boys and girls, contrary to the stereotype of males being more computer proficient. However, statistically significant differences in performance were found between majority and minority group students, between students reporting their parents had attained different levels of education, and between students eligible and not eligible for lunch subsidy.
"The scoring of the TRE assessment accounted both for what students wrote in their constructed-response answers and what they did to arrive at those answers," explains Bennett. "By looking at what students did to arrive at their answers, a test can begin to identify not only what students know and can do, but also what those students might need to do to improve. For example, the study evaluated how effectively students searched for information by looking at whether they attempted to focus results by using advanced techniques, the quality of their queries, whether they visited relevant pages, and whether they used bookmarks to allow them to quickly revisit key websites. The TRE Search scenario produced a total score and two sub scores: scientific inquiry and computer skills. The results suggest that the TRE scenarios functioned reasonably well as assessment devices.
The analysis of TRE assessment results built upon the methodological procedures used in operational NAEP assessments, so it demonstrated how scenario-based assessments like those used in TRE might be analyzed and reported within a rigorous psychometric framework, like that used in NAEP," according to Bennett.
The executive summary of "Problem Solving in Technology-Rich Environments: A Report From the NAEP Technology-Based Assessment Project," can be downloaded from the National Center for Education Statistics at: http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007466. PDFs of the full report are also available free at that website.
ETS is a nonprofit institution with the mission to advance quality and equity in education by providing fair and valid assessments, research and related services for all people worldwide. In serving individuals, educational institutions and government agencies around the world, ETS customizes solutions to meet the need for teacher professional development products and services, classroom and end-of-course assessments, and research-based teaching and learning tools. Founded in 1947, ETS today develops, administers and scores more than 50 million tests annually in more than 180 countries, at over 9,000 locations worldwide. Additional information is available at www.ets.org.