Accuracy and fairness of generative AI in automated essay scoring: Comparing GPT-4o, feature-based models, and human raters
Huang, Y., Palermo, C., & Wilson, J. (2026). Accuracy and fairness of generative AI in automated essay scoring: Comparing GPT-4o, feature-based models, and human raters. Assessing Writing, 101047. https://doi.org/10.1016/j.asw.2026.101047
This article evaluates the accuracy and fairness of GPT-4o for automated essay scoring in Grades 3-4 writing, comparing multiple prompting strategies and a fine-tuned configuration against a feature-based scoring engine and human raters. The study examines both holistic and trait-level scoring performance as well as subgroup score differences.
Key takeaways
- Well-designed prompts improved GPT-4o's scoring accuracy.
- The fine-tuned GPT-4o configuration showed the closest alignment with human-human agreement.
- Generative AI and feature-based models showed complementary strengths at the trait level.
- No model was entirely free of subgroup differences, so fairness remains a concern.
Why it matters
Provides an early-grade, validity- and fairness-focused comparison of generative AI, feature-based AES, and human scoring, showing that GPT-4o can approximate human scoring when carefully calibrated but does not eliminate fairness concerns.
Coauthors
Yue Huang, Corey Palermo, Joshua Wilson
Topic tags
External links