New rules adopted recently by the U.S. Department of Education (ED) could impact whether your grant application results in cold, hard cash–or the cold shoulder.

On Feb. 24, new scientifically-based research regulations went into effect that allow ED program offices to give priority to grant applications that use randomized control trials in their evaluation activities. The language of the new rules is careful to state that offices are not required to give preference, but they are permitted to do so.

To fully understand the impact of these new regulations, I spoke with Berwood Yost, director of the Center for Opinion Research at Franklin and Marshall College. The center provides research services, technical assistance, and training to government agencies, private corporations, professional associations, the media, and other organizations.

Yost explained that there are two types of research designs: experimental and non-experimental. Experimental designs allow the researcher to have more control over the independent variables under study, the people being studied, the units of analysis, and the environment in which the experiment takes place. These experimental designs usually have five characteristics:

(1) There are experimental groups that receive an experimental stimulus (such as instruction by a certain educational program) and control groups that do not;

(2) The researcher determines the composition of the experimental and control groups by choosing the subjects (usually using random procedures) and assigning them to one of the two groups;

(3) The researcher controls the introduction of the stimulus;

(4) The researcher is able to measure the independent key variables before and after the stimulus treatment is given; and

(5) The researcher is able to control the environment of the subjects to control or exclude extraneous factors that might affect the experimental outcomes.

In randomized control trials, the subjects are randomly placed into the experimental and control groups by the researcher, which usually makes the important characteristics of each group roughly the same. Randomized control trials generally give researchers a clearer idea whether the effects of the stimulus are “real” and whether the stimulus makes a difference, which is why they are more desired.

But there are disadvantages to using randomized control trials, too. Yost noted that randomized control trials can be difficult to organize, expensive to establish, and hard to monitor.

For example, if a district wants to implement a new reading program for third-grade students using randomized control trials, then ideally the third-grade students in every school in the district would be placed randomly in either an experimental group or a control group. For randomized control trials to work, high-, medium-, and low-level readers would have to be assigned randomly to the experimental and control groups, meaning the district might have to reorganize its classrooms for reading. For those who plan to use an external evaluator, using randomized control trials also will result in higher evaluation costs.

There also might be some public-relations issues to take into consideration. For instance, parents might object if their children are not selected for the experimental group and are placed in the control group instead. Using the example I just outlined, this would mean half of the district’s third-graders would not be exposed to the new reading program at the beginning of the project.

The use of randomized control trials raises several questions for school district leaders. I would suggest that you discuss these questions with your project evaluator to determine whether it is necessary or even feasible to use randomized control trials. Before choosing to apply, you also should ask the program officer for any ED grant you intend to pursue whether the program gives priority to applications that use randomized control trials.

For more information about randomized control trials, Yost suggests reading Experimental and Quasi-experimental Designs for Research by Donald Campbell and Julian Stanley and Understanding Research Methods by Mildred Patten. For more information about this topic, you can reach Yost via eMail at berwood.yost@fandm.edu.

See this related link:

Center for Opinion Research
http://www.fandm.edu/opinionresearch.xml

Deborah Ward, CFRE, is an independent grant writing consultant. She welcomes questions at (717) 295-9437 or Debor21727@aol.com.