"Effectiveness of Reading and Mathematics Software Products," a $10 million research study overseen by the U.S. Department of Education, suggests that using certain educational software programs to help teach reading and math did not lead to higher test scores after a year of implementation.
The study set out to examine the effectiveness of 15 classroom software programs in four categories: early reading (first grade); reading comprehension (fourth grade); pre-algebra (sixth grade); and algebra (ninth grade). Here’s a closer look at the study’s findings.

The first-grade study was based on five reading software products that were implemented in 11 districts and 43 schools. The sample included 158 teachers and 2,619 students. The five products were Destination Reading (published by Riverdeep), the Waterford Early Reading Program (published by Pearson Digital Learning), Headsprout (published by Headsprout), Plato Focus (published by Plato Learning), and the Academy of Reading (published by Autoskill). The study estimated the average licensing fees for the products to be about $100 a student for the school year, with a range of $53 to $124.

According to records maintained by the software, usage by individual students averaged about 30 hours a year, which the study estimated to be about 11 percent of reading instructional time. Some control group teachers used technology-based reading products that were not in the study, though these teachers reported using software about a fifth as frequently as treatment teachers reported using the products in the study.

First-grade reading products did not affect test scores by amounts that were statistically different from zero, according to the report. But researchers observed large differences in the effects among schools–and effects were larger when schools had smaller class sizes.

The fourth-grade study included four reading products that were implemented in nine districts and 43 schools. The sample included 118 teachers and 2,265 students. The four products were the Leaptrack Assessment System (published by LeapFrog SchoolHouse), Read 180 (published by Scholastic), Academy of Reading (published by Autoskill), and KnowledgeBox (published by Pearson Digital Learning). The study estimated the average licensing fees for the products to be about $96 per student for the school year, with a range of $18 to $184.

Annual usage by students for the two fourth-grade products that collected this measure in their databases was seven hours for one product and 20 for the other. Assuming a typical reading instruction period was 90 minutes, students used these products for less than 10 percent of the total reading instructional time, according to the report. The fourth-grade reading products did not affect test scores by amounts that were statistically different from zero–though effects were larger when teachers reported higher levels of product use.

The sixth-grade study included three products that were implemented in 10 districts and 28 schools. The sample included 81 teachers and 3,136 students. The three products were Larson Pre-Algebra (published by Houghton-Mifflin), Achieve Now (published by Plato Learning), and iLearn Math (published by iLearn). The study estimated the average licensing fees for the products to be about $18 per student for the school year, with a range of $9 to $30.

Student usage was about 17 hours a year, or about 11 percent of math instructional time, according to data from product records (available for two of the three products). The products did not affect test scores by amounts that were statistically different from zero.

The ninth-grade algebra study included three products that were implemented in 10 districts and 23 schools. The sample included 69 classrooms and 1,404 students. The three products were Cognitive Tutor Algebra (published by Carnegie Learning), Plato Algebra (published by Plato Learning), and Larson Algebra (published by Houghton-Mifflin). The study estimated the average licensing fees for the products to be about $15 per student for the school year, with a range of $7 to $30.

Product records showed that student usage was 15 hours for the overall sample, equivalent to about 10 percent of math instructional time. Usage averaged 5 to 28 hours, depending on the product. As with the other products studied, algebra products did not affect test scores by amounts that were statistically different from zero.

Minor technical difficulties, such as issues with students logging in or computers locking up, were fairly common throughout the study. However, most of those problems were easily corrected or worked around, according to the report.

When asked whether they would use the products again, nearly all teachers indicated that they would.

The report detailed the effectiveness of the products as a group and did not review the performance of particular programs. That was a point of contention among some critics of the research, and the Education Department says it will publish details about each program’s effectiveness after the second year of the study. (See "Delays, designs diminish ed-tech research": http://www.eschoolnews.com/news/showStory.cfm?ArticleID=6927.)
Congress called for a study on the effect of educational technology in the 2002 No Child Left Behind law, which aims to get all students reading and doing math on grade level.

Links:

"Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort"

http://ies.ed.gov/ncee/pubs/20074005/