Speaking as part of an internet-based telecast sponsored by the Consortium for School Networking on Oct. 16, top education officials attempted to define the meaning of “scientifically based research” by explaining what schools must do to comply with this portion of the No Child Left Behind Act (NCLB).

The law specifies that all federally funded education initiatives deployed in grades three to eight be proven effective by way of “scientifically based research.” So if a school district uses federal grant money to purchase reading software, for example, the software in question must be proven to work through rigorous analysis.

In a year marked by sweeping education reforms spurred by NCLB, few requirements have met with more confusion than the scientifically based research stipulation. While some educators complain that the term itself is broadly defined and subject to interpretation, still others contend that the federal government alienates solid, useful programs by imposing such a requirement.

Recognizing both criticisms, presenters used the conference to do two things: explain what constitutes scientifically based research, and address why the requirement rests at the very foundation of the Bush administration’s education policy.

Valerie Renya, a senior research advisor for the U.S. Department of Education (ED), said schools face several challenges in the struggle to implement NCLB, including making sure new programs are in line with state goals; disaggregating student data according to poverty status, race, ethnicity, and other indicators; and making sure all children achieve at “proficient” levels within the next 12 years.

According to her, scientifically based research is the lynchpin that holds all of these goals together. “It’s extremely important to know what works,” she said. “Scientifically based research should be the rationale for all of these educational approaches.”

Renya outlined four types of research educators can use to comply with the law.

The most comprehensive method is a randomized trial. Here, the research takes into account several variables—including teacher quality and student ability—by applying itself to a wide range of test subjects and demonstrating some degree of improvement at every level. Research also can take the form of a quasi-experiment with statistical controls, where the approaches are tested randomly within a single school. Another option is a correlation study without statistical controls. A final, acceptable method is the use of case studies, which may demonstrate a proven track record of success, Renya said.

Educators must be particularly careful when relying on case studies, however, because they often do not take into account certain variables that can shift from school to school, Renya warned.

Identifying the method of research used in a study is one challenge, but determining whether the data are scientifically based presents another problem, said John Bailey, ED’s director of educational technology.

Bailey said ED is well aware that school leaders are not statistical experts. Still, educators must familiarize themselves with the types of studies that are out there. “Schools need to ask better questions,” he said. “They need to make sense of the data.”

Bailey’s presentation included six key questions educators should ask when deciding if a study is meaningful:

  • What was the method of research used? Is it credible?

  • Was a rigorous data analysis performed?

  • Was the method of data collection valid and reliable?

  • Was the research design strong, or did it leave room for error or misreporting?

  • Are there detailed results available? Can the study be replicated with similar results?

  • Has the study been subject to scrutiny by other professionals, or possible critics?

One challenge the law creates for ed-tech specialists is that “there still is very little scientifically based research to gauge the effectiveness of technology” in the classroom, Bailey acknowledged.

Still, he said, the requirement should be seen as a potential boon for ed-tech use. He called it an opportunity for “educational technology people to stand up and be counted” and urged educators to demand more research-based evidence from school technology vendors.


U.S. Department of Education

Consortium for School Networking