How did young students perform with a computer-based writing assessment pilot?
A new pilot study reveals that fourth grade students are well-equipped to successfully take computer-based writing assessments.
The National Center for Education Statistics conducted the National Assessment of Educational Progress (NAEP) Grade 4 Writing Computer-Based Assessment (WCBA) study to determine if fourth-grade students were in fact able to effectively use computers to complete a writing assessment.
“We found that fourth-graders were able to organize and convey their thoughts well enough to be assessed against the NAEP rubric,” said Cornelia Orr, executive director of the National Assessment Governing Board, during a webinar briefing that addressed the results.
Results showed that students typed enough words to enable evaluators to evaluate their writing skills, she added.
“The significance of writing right now is something that we have to hold onto, and because of that, the notion of what it means to prepare young people as writers is ever more significant,” said Elyse Eidman-Aadahl, executive director of the National Writing Project. Eidman-Aadahl served as a member on the 2011 NAEP Writing Framework committee.
(Next page: Results from the writing pilot test)
“For most people right now, they will be writing in a digital environment, and certainly for fourth graders, they’re growing into a world where our interactions are in a digital environment,” she said.
In fact, moving writing assessments to a digital platform has increased relevance for young students today.
“Writing in a computer-based environment is part of the construct of what writing is–so looking at how young people are able to move into, with comfort, writing in a digital framework both tells us something about the move toward putting more assessment opportunities into the computer, but it also tells us something extremely significant about preparing young people to be writers at all–because to be a writer today is to write in digital environments,” Eidman-Aadahl said.
The study consisted of two parts:
- Small-scale usability testing to improve development of the assessment platform for fourth-grade students
- A pilot writing assessment administered to a sample of 13,000 students nationwide
The pilot examined students’ ability to organize and write typed responses to prompts, with each prompt accompanied by either 30 minutes or 20 minutes of writing time.
Students with 30 minutes of writing time were scored on a 6-point rubric, with 1 being the lowest and 6 being the highest.
Thirty-nine percent of students received a 1 or a 2, indicating little to no marginal writing skills; 47 percent of students received a 3 or 4, indicating developing or adequate writing skills; and 14 percent of students received a 5 or 6 and demonstrated competent or effective writing skills.
The average word count was 110 words per response. Thirty-one percent of students typed 1,001 keystrokes or more, and 71 percent of the highest-scoring student responses used 1,001 keystrokes or more, compared to 9 percent of the lowest-scoring responses.
Fifty-five percent used the backspace key fewer than 100 times, and 39 percent of students used the spell check tool at least four times.
When writing for different purposes:
- 17 percent of student responses received a score of 5 or 6 when asked to “convey experience”
- 15 percent of responses scored a 5 or 6 when asked to “explain”
- 10 percent of student responses received a 5 or 6 when asked to “persuade”
The usability study helped to inform the pilot assessment administered to students nationwide.
A number of lessons emerged from those usability tests, including:
1. Simplifying directions will help students understand the objective.
- Students had trouble reading and following directions to adjust the computer screen. The test was altered to deliver one step at a time, accompanied by a recorded voice-over.
- Students didn’t spend enough time reading the general directions (32 percent), therefore the directions were shortened and were read aloud.
2. Tools should be easy to locate and well-labeled
- Students used icons more often than drop-down menus, so more icons were added to the toolbar in order to help students see the available editing options
- More than half of students (68 percent) had trouble finding the writing panel, so the buttons to locate the panel were enlarged
- Some students (25 percent) misinterpreted the original text-to-speech icon, so it was changed and a rollover label was added
- Nearly one-third of students (32 percent) thought a “reset” button erased all answers instead of the answer to their current question. That button was changed to a “clear answer” button.
- Friday 5: Universal Design for Learning - April 19, 2024
- Educators love their edtech, but want more training - April 18, 2024
- Friday 5: College and career readiness - April 12, 2024
