Generalizability of Automated Scores of Writing Quality in Grades 3-5
Wilson, J., Chen, D., Sandbank, M. P., & Hebert, M. (2019). Generalizability of automated scores of writing quality in grades 3-5. Journal of Educational Psychology, 111(4), 619-640. https://doi.org/10.1037/edu0000311
This study examined the reliability of automated essay scoring for elementary students' writing quality across grades 3-5. Using Project Essay Grade and generalizability theory, the authors evaluated variation due to prompts, genres, and student-by-facet interactions, with separate analyses for struggling and nonstruggling writers.
Key takeaways
- Reliable low-stakes decisions could be made using one prompt per genre for typical samples, or two prompts per genre for struggling writers.
- Reliable high-stakes decisions required more prompts: about two per genre for full-range samples and four to five per genre for struggling writers.
- Prompt and genre sampling meaningfully affected score reliability, highlighting the importance of assessment design.
- Automated essay scoring may be a practical option for response to intervention in writing, but additional work on construct validity is still needed.
Why it matters
Provides evidence on how many prompts are needed for reliable automated writing scores in elementary grades, especially for decision-making within response to intervention in writing.
Coauthors
Joshua Wilson, Dandan Chen, Micheal P. Sandbank, Michael Hebert
Topic tags
External links