Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm
Myers, M. C., & Wilson, J. (2023). Evaluating the construct validity of an automated writing evaluation system with a randomization algorithm. International Journal of Artificial Intelligence in Education, 33(3), 609-634. https://doi.org/10.1007/s40593-022-00301-6
This article examines the construct validity of the MI Write automated writing evaluation system by randomizing student persuasive essays at the sentence level and comparing trait scores for original and randomized versions. The study focuses on whether higher-level traits such as idea development and organization decrease when essay coherence is disrupted.
Key takeaways
- Sentence-level randomization did not consistently lower MI Write scores for higher-level writing traits.
- More than one-third of the essays showed significant increases in one or both higher-level traits after randomization.
- The findings suggest a possible disconnect between MI Write's formative feedback and the constructs it claims to measure.
- The study raises important validity concerns for consumers and developers of automated writing evaluation systems.
Why it matters
Provides an empirical construct-validity test of an automated writing evaluation system in a formative-feedback context using randomized essay manipulation.
Coauthors
Matthew C. Myers, Joshua Wilson
Topic tags
External links