Today, all eyes in education are trained on the bottom linestudent achievement. It’s not a new story, but it’s happening in a new world.
The difference today is that educators are using technology to improve student achievement and, while they’re at it, applying the same tools to improving the business side of education. School districts are borrowing tools from the corporate world and renovating policies and practices in the classroom and central office.
Ironically, the key ingredient is something that has been as common in schools as textbooksdata. Schools have collected and reported data on everything from test scores, grades, and attendance to patterns of bus ridingall motivated primarily to fulfill government reporting requirements. Over time, data departments became the central location to send collected data and request printouts of routine information.
Meanwhile, in the corporate world, data were being used proactively. Customer buying habits and spending patterns were being analyzed to find out if dollars spent were actually increasing sales. Data were being pooled in a virtual warehouse, where companies would dig into the information to learn about their customers, their programs, and their products.
Now, schools use data warehousing to gather data in one place, analyze it, and take it straight to the bottom line.
This gives school systems the option of becoming proactive as they begin to mine information in the data warehouse. They can take known outcomes in education and analyze what is affecting them, then use that knowledge to cause change.
“We know who has dropped out,” explains Georgia Department of Education data mining specialist Steve Gabrielson, “but now we can throw in all of the variables and try to analyze … the ones that made an impact on the student that dropped out.”
A data warehouse is a central location where information from different applications is gathered, combined, and stored. The challenge in building a data warehouse is that the information is usually in different, incompatible formats. Information is received into the warehouse at frequent intervals and must be “cleansed” before it is in a format that’s valuable.
Looking deeply into that information, or data mining, shows relationships between data and reveals trends that otherwise would have remained hidden, according to Andrew Sipos, education sales representative for data warehousing specialist ECS Inc.: a link between teacher tenure and student performance, for instance, or between how often teachers are certified and how long they remain in the district. A server-based data warehouse on an intranet or the internet ensures accessibility to information on a web browser, making it available when and where needed.
Most school districts have managed to develop cumbersome processes to aggregate and separate data to get the information needed to prepare government reports and communicate to stakeholders. District employees are used to a long wait for any information they may need.
But with today’s technologies, creating a data warehouse can provide district employees with a vast repository of data to drill down into and, using pieces from different parts of district operations, instantly compare the data in ways that reveal relationships and trends valuable to managing a school district.
In school districts, the common problem is multiple applications and platforms that won’t allow valuable data to be easily integrated. Frustrations regarding this persistent problem paved the way for an educational software industry initiative called the Schools Interoperability Framework (SIF).
The SIF working group has been developing open specifications for K-12 instructional and administrative software applications. The initiative is planning to release developer version 1.0 of interoperability specifications this month. The new standards should provide relief and hope for school districts (see page 32.)
Florida’s Pinellas County Schools aligns its technology use to keep the primary focus on improving student achievement, making data available quickly at the classroom level. Al Swinyard, assistant superintendent for management information systems (MIS), has created a web-based infrastructure and uses Microsoft’s Access 2000 and Excel software for data warehousing and mining. Access 2000 includes built-in Microsoft SQL Server integration and allows an Access file to be converted easily to a full SQL Server database, via a “wizard” feature common to Microsoft applications.
Swinyard identified the 100 most popular data elements and packaged them for district administrators and teachers. He finds that the most common information elements needed are student test scores, grades, and demographics. But more obscure data also are available, such as information on a student’s progresscomparing performance from the past to the current school year.
Pinellas County assesses a student’s progress three times a year, using a district-developed test that mirrors the Florida assessment test. The test is administered to students in September, December, and May, and scores are sent to the district server via an intranet. Teachers can use the data to get a clear picture of student performance.
At Azalea Elementary in Pinellas County, Danny Bigham, once a teacher and now the school’s data manager, facilitates that process. Bigham maintains the Apple G3 Server at the school and uses Microsoft Access and Visual Basic to gather information into the data warehouse. He combines classroom, school, district, and state data to give teachers at Azalea real-time information that will benefit students immediately. Teachers use test scores to diagnose weaknesses or locate areas of the curriculum that have not been mastered and address them before the school year ends.
“Taking the information from the Pinellas assessment, we can see how Azalea students are performing individually, by classroom and by grade,” Bigham said. They don’t stop there, but also combine county and state information into the database, because “we can think our students are progressing well and then compare countywide or statewide and be very surprised,” he said.
This year marks the first time Azalea has used an intranet model. In the past, analysis was restricted by the inaccessibility of raw data, and Bigham always began the school year with the arduous task of manually inputting student’s names and addresses.
This particular type of data still is not accessible over an intranet, but the district data manager now can send the information on a disk in a compatible form that loads directly into Bigham’s data warehouse. Although the transformation is not complete in Pinellas County, the district will continue to move everything to a platform for intranet data collection, making it accessible for more extensive data mining.
At the district level, Pinellas County is looking for new applications that will enable its teachers to connect to the district’s mainframe to access data directly from their desktop computers, according to Michele Smith, MIS team leader. Creating these systems and making data accessible to decision makers and teachers throughout the district is a different experience for the data-processing department.
“In data processing, usually we’re just throwing out requirements to the schools, but this is for the kidsto help them improveand it’s kind of exciting,” Smith said.
And the students in Pinellas County are a part of the data analysis team as well. This unique approachgiving students a clear understanding of what the state standards are for their grade levelhas transformed the learning process, explains Judy Ambler, assistant superintendent for instructional technology.
“It’s not ‘I think’ when you get data, because you know,” Ambler said. “Students know where they stand individually” and how they compare to the expectations for their grade level.
Azalea Principal Brenda Clark calls it “in-your-face data.” When data are presented that track progress, and goals are clearly defined, students seem to take personal responsibility for their learning. In a visit to Azalea, Ambler spoke with a third-grade student who displayed his portfolio and explained, “Here’s where I am and here’s where the class isI want to be up here, too.”
Ambler finds as much value in the positive information that is revealed when analyzing data as she does in the negative information. Teachers can find “best practice” techniqueswith proven resultswhen reviewing school-wide or district-wide data, she said. Wherever they find areas of weakness, they also find students or classes that are excelling, and that’s information they can use as well.
Teachers don’t fear a punitive outcome when reviewing data, but instead look for answers that will help them help students. Therefore, they are willing and motivated to borrow and share ideasand integrate what works. Finding new ways to communicate and share ideas, Ambler said, is “bound to have an influence on how we teach.”
Keeping student portfolios
North Carolina’s Charlotte-Mecklenburg Schools (CMS) also is moving toward the goal of making data accessible and useable for everyone in the district. Rick Rozelle, chief information officer, said, “The first priority is to build a reliable infrastructure.” T1 lines now connect the schools and administration, but the goal is to connect all classrooms to administrative offices.
The district’s 2000-02 technology plan calls for creating a technology environment “that facilitates the easy implementation, distribution, and support of new classroom and administrative software across the district.” With this in mind, the district chose to focus on web-enabled software running on PCs (minimum Pentium 100 processor) after deciding that a server/thin client configuration did not have the level of reliability the district wanted.
Data warehousing and mining are just beginning at CMS, and Rozelle carefully has chosen a suite of software applications by Lawson Software to ensure interoperability between all business and instructional applications. The software includes Financial and Human Resources suites, which work together seamlessly and are accessible using an internet browser. Rozelle plans to modify the entire district’s business processes using the new applications, improving the overall efficiency of the central office.
Dr. Susan Agrusso, assistant superintendent for instructional accountability at CMS, looks forward to having widespread access to data. Supplying teachers with information on their studentswhat they know and what they need to work on to meet state standardsis critical.
“If any child is having trouble, we want to help immediately,” she said.
The district keeps a portfolio on students throughout their school careers, tracking them year to year on assessment scores and providing good data on their progress toward meeting state standards. Agrusso visualizes giving teachers access to curriculum online, including strategies to meet state standards, and then giving them direct access to data analysis tools to determine what changes are needed in classroom instruction.
A computer program recently developed by the district could facilitate this process. The program is able to provide information as simple as a single child’s test score, or as complicated as chemistry scores for all white males who missed eight to 14 days of school, school officials said.
Called TOP II (Testing Outcome Profile), the program can call up more than 300 million test score combinations. The program currently is being adapted to middle and elementary schools.
The program, which parents can access through their child’s teacher or counselor, is simple to use: Type in a student’s name and up come all that student’s end-of-course scores, from English I to U.S. History.
It tells if the student passed his reading and math competency tests and how he did on the PSAT exam.
Parents can request specific scores only for their children, and they will not be able to access the program except through a teacher, counselor, or principal.
The program’s strength lies in its combination possibilities. Scores can be compiled by individual schools, by gender, by students who qualify for free or reduced-price lunches, and other ways.
“A few years back, we realized we have a lot of data that the community and parents get on an annual basis, but most parents don’t keep the records,” said Geri Ross, director of high school instructional accountability. “We thought it would be nice to give them a comparison, to give them the average scores.”
Mike Huggins, an instructional accountability programmer for CMS, developed the first phase of TOP II in 1998.
That program can graph a student’s history on end-of-grade results for third through eighth grades. It can compare the student’s scores with the grade-level standard, tell how many absences and suspensions the student had the previous year, and reveal whether the student is classified as exceptional.
It also includes answers the student gave to a school system survey about how many hours of homework they do a night and how much television they watch. Parents and district officials can compare this information to a child’s performance to see if they should recommend a change in habits.
Testing with technology
Test scores are an integral piece of the raw data used to analyze students’ progressand now testing is being adapted and improved with technology as well.
One new method uses a powerful database and computerized problem-solving assessments to show teachers a “map” of how each student, and the class as a whole, solved the problem. Research immunologist Ron Stevens designed the assessments, called Interactive Multimedia Exercises (IMMEX), 12 years ago to test University of California at Los Angeles medical students. A grant from the National Science Foundation expanded the project to include K-12 schools in southern California.
IMMEX is Windows-based and allows “problem set” scenarios to be developed without formal software programming. One problem set for a science class, for example, might state that “a dead fish washes up on shore and the students must find the source of the pollution that killed the fish,” Stevens said.
As students work on the problems in small groups, the mapping part of the software creates a linear representation of the steps they take in attempting to solve each problem. The teacher then can review the process students used to solve the problems, with different patterns revealing the depth of the students’ grasp of the subject matter. Timely access to the map information lets teachers quickly adjust strategies for struggling students.
Marcia Sprang, Advanced Placement chemistry teacher at Esperanza High School in southern California, uses IMMEX as “a means for students to synthesize knowledge.” In order to solve one of her chemistry problem sets, students must know the subject and have completed the lab work.
Sprang believes that students using the IMMEX system understand the material better than if they had used only a textbook. As the author of the problem sets her students use, she knows they were “written to drive content understanding and deeper thinking.”
This type of teaching and testing tool allows students to learn a subject more thoroughly, she said, because it provides the teacher with a view of how the information is taken in, how it is processed and in what order. Sprang also finds students asking questions “when you thought you had taught them that information,” but the IMMEX problem set has them look at the information differently.
“You get an insight into what they understand well and where they are fuzzy,” she said. “This is not the only way to teach, but it’s a good adjunct to good teaching and an alternative to regular testing,” and it presents yet another way to look at students’ mastery of content.
There is a 70-percent to 80-percent passing rate for students from her class who take the Advanced Placement chemistry test, Sprang said, but she is still trying to gather data on whether using the IMMEX technology has had an impact on these test scores.
Computerized adaptive testing
Educators rely heavily on achievement testing to measure how students are mastering the new state and local curriculum standards. Combining advances in measurement theory and technology, computerized adaptive testing (CAT) can provide more information about the actual level of a student’s ability than paper-and-pencil tests, researchers say.
CAT has the ability to customize a test to a student’s achievement level. Such tests begin by asking a question at the average level for a student’s grade. If the question is answered correctly, the computer increases the level of difficulty on the next question. When a question is answered incorrectly, the next question presented will be at or below the level of the previous question. In this way, the computer is able to hone in on the student’s actual level of ability very quickly.
Students at different levels of achievement in the same classroom can take different versions of the same test, targeted to their own achievement level. While testing is done several times a year, CAT will “remember” archival information for each student to prevent repeat test questions and allow for more accurate testing of a student’s progress.
The Northwest Evaluation Association (NWEA), a 25-year-old nonprofit organization, now offers a CAT version of its standard Achievement Level Tests. The CAT version requires a Novell or Windows NT network server and Windows-based workstations with Pentium 100 or better processors.
Executive Director Allan Olson said that beta testing is under way for using the CAT version on an internet platform. “This will allow site-collected data to be merged into a common reporting system, with access available to the school district and to NWEA,” he said.
The group is developing a longitudinal database to explore which district characteristics and instructional programs improve academic achievement. Currently, there are CAT versions customized to seven states’ curriculum standards, Olson said. When a school district adopts the program, further customization of local standards is adapted into the program.
A school would need 25 workstations per 350 students to administer the test efficiently, Olson said, so migrating completely from paper and pencil to CAT will be gradual for most districts.
California’s Capistrano Unified School District uses CAT and paper-and-pencil versions of its achievement tests, but the district has plans to use CAT in some key areas. It’s being used to identifying the level of proficiency in math and English for new students, for example. By this summer, Jeff Bristow, Capistrano’s director for testing and evaluation, sees all at-risk students being tested by CAT so assessments can be done more often than in the spring and fall.
Using CAT saves time, he explains, because “once the students have finished the test, they have a score.” Also, on a paper-and-pencil test, if there are too many wrong answersor too many correct onesthe student must take a retest at a more appropriate level. CAT’s ability to adapt the level of the questions as they are presented is “a very efficient use of time,” Bristow said, and “every test is conclusive.”
Olson believes CAT’s purpose is to improve measures of accuracy, timeliness of data, and alignment with curriculum standards. “Our expectation is that the data will be used to change behavior and methods for improvement,” he said.
Attracting and retaining good teachers
The business of managing school districts’ personnel issues also has changed for the better by employing technology tools. The internet connections that are linking schools within a district and districts within a state have helped streamline administrative functions such as payroll, evaluations, teacher information, and recruitment.
In some districts, technology is replacing almost archaic practices. Pinellas County recently moved its records of teachers on leave of absence from 3-inch by 5-inch index cards to an electronic database, according to Personnel Director Harriet Konstantinides.
Konstantinides has moved many of her department’s processes onto web-based applications and is turning to data warehousing and mining. The department is developing several databases using Microsoft Access on a Windows NT platform, but still encounters the typical roadblock created by incompatible software applications.
Recognizing the increased value of comparing all its data, the department is developing a program to allow its databases to share information. Konstantinides would like to be able to “prioritize data and look at it longitudinally, assessing the [district’s] requirementwho needs what [and] when? We need to keep evaluatingare we where we want to be?”
The district also uses the internet for recruitment, but at first, it proved to be too much of a good thing. Applications came in at a higher volume than ever experienced, and the district initially wasn’t prepared for the onslaught in terms of its systems or manpower.
Swinyard says Florida soon will facilitate online recruitment statewide, because “there is a lot of competition for districts to attract teachers, and if a district can respond quickly to a candidate, they are more competitive.”
Georgia has recruited online since 1996, helping districts compete nationally for candidates. Stephanie Carreker, human resources manager for the state’s Department of Education, describes its recruitment site, teachgeorgia.org, and the site’s impact on hiring processes.
“Originally, we offered a phone recording of openings and posted 1,500 jobs in 1996,” she said. “Last year, on the ‘teachgeorgia’ site, we had 4,000 job postings and 25,000 applicants.”
The site includes online applications, statewide job searching, a résumé database, and a place that lets candidates profile the position they’re looking for and request eMail notification if a position opens. Schools can search the résumé databank for applicants to fill their positions.
The web site has changed the state’s pool of applicants, Carreker noted, with 25 percent coming from outside Georgiaand some from outside the country. “I’ve had schools tell me they found candidates through ‘teachgeorgia’ that they would never have found before,” she said.
The department continually educates its schools on the benefits of using the web site, relaying success stories from progressive districts to those that are initially resistant. Carreker sees districts that suddenly are motivated to use technology for recruiting in order to compete for candidates: “Things changed when the pressure of filling teaching positions increased.”
The site also has helped the department reduce its staff from four full-time positions in 1996 to one full-time and one part-time position today. “Personnel and telecommunications costs are way downwe have very little paper correspondence now that everything is done through eMail,” Carreker said.
As the public demands the kind of accountability from school districts that stakeholders have demanded for years from corporations, educators are borrowing tools from the business world to improve the business of education. The result, experts agree, is better teaching and learning, more accurate assessment of student abilities, and streamlined performance of schools’ daily functions.
Pinellas County Schools
Georgia Department of Education
Northwest Evaluation Association