Scholarship

An interactive viewof Dr. Wilson's research

This page offers an interactive way to explore the scale, thematic structure, and scope of Dr. Wilson's scholarship. It highlights major research themes, provides summary metrics, and makes it easy to browse publications by theme and year.

Why this scholarship matters

Dr. Joshua Wilson's scholarship has helped shape the field's understanding of automated writing evaluation and the effective use of AI in writing instruction and assessment. His research shows that the value of these tools depends not only on what they can do technically, but on how they are taken up by teachers and students and whether classroom and school conditions allow them to support meaningful learning.

Total publications

56

Peer-reviewed journal articles, book chapters, and books published by Dr. Wilson.

Book chapters

7

Chapters in influential books such as The Routledge International Handbook of Automated Essay Evaluation, The Handbook of Writing Research, and Best Practices in Writing Instruction.

Featured handbook

The Routledge International Handbook of Automated Essay Evaluation (2024)

This Handbook, edited by Dr. Mark Shermis and Dr. Joshua Wilson, brings together foundational and emerging perspectives on automated essay evaluation.

View handbook

Google Scholar

October 2026

3,886

Citations

32

h-index

50

i10-index

These metrics are updated periodically from Dr. Wilson's Google Scholar profile.

h-index: The h-index is the largest number h such that at least h publications have been cited at least h times. For example, an h-index of 32 means 32 publications have each been cited at least 32 times.

i10-index: The i10-index counts the number of publications with at least 10 citations.

Thematic profile

Breadth and concentration across major themes

This profile visualizes how Dr. Wilson's scholarship is distributed across the major research themes that organize his body of work.

Automated WritingEvaluation & FeedbackWriting Assessment,Validity & MeasurementAI, Automation &EducationalMeasurementWriting Instruction &Pedagogical DesignImplementation,Teachers & EducationalSystemsEquity, EnglishLearners & StudentSupportWriting Development,Motivation & LearnerExperience

Explore the major themes

Select one or more themes to focus the publication results below. Publications can belong to multiple strands when they contribute across areas.

Year filter

Select a year to focus the publication results below.

Publications

Filtered publication results

Showing all 56 publications.

Click any publication card below to open the full record, summary, and links for that specific publication.

2026journal article

Accuracy and fairness of generative AI in automated essay scoring: Comparing GPT-4o, feature-based models, and human raters

Yue Huang, Corey Palermo, Joshua Wilson

This article evaluates the accuracy and fairness of GPT-4o for automated essay scoring in Grades 3-4 writing, comparing multiple prompting strategies and a fine-tuned configuration against a feature-based scoring engine and human raters. The study examines both holistic and trait-level scoring performance as well as subgroup score differences.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2026journal article

Exploring ChatGPT as a Simulated Client: A Feasibility Study in Undergraduate Counselor Training

Kayla E. Hayes, Rebecca L. Wilson, Joshua Wilson

This exploratory multiple-case study examined ChatGPT-4o as a simulated client and feedback provider for five upper-level undergraduate human services students learning counseling skills. Each student completed a single text-based counseling simulation, reviewed AI feedback, and participated in an interview. Trained raters generally judged the client portrayal and feedback accurate, with stronger performance for emotional tone, empathy, and solution-focused responses than for rapport, summarizing, and the pacing of an initial session. Students valued the low-pressure practice and time to reflect, while noting limitations of text-only interaction. The findings support feasibility under these conditions; they do not establish skill improvement or readiness to work with real clients.

AI, Automation & Educational MeasurementWriting Instruction & Pedagogical Design
View publication details
2026journal articleFeatured

Impact of Automated Writing Evaluation (AWE) on Middle School Students' Writing Outcomes

Joshua Wilson, Corey Palermo, Matthew C. Myers, Tania Cruz Cordero, Halley Eacker, Andrew Potter, Jessica Coles, Henry May, Fan Zhang

This randomized controlled trial examined whether the MI Write automated writing evaluation system improved middle school students' argumentative writing performance, writing self-efficacy, enjoyment of writing, and beliefs about the recursive nature of writing, with particular attention to students from historically underserved groups. The study found no significant overall treatment effects, but it did identify district-level differences in outcomes and generally positive perceptions of the system.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2026journal article

Writing growth among non-English learners and English learners in grades 6-8 using automated writing evaluation

Joshua Wilson, Corey Palermo, Arianto Wibowo, Fan Zhang, Adiba Nusrat

This study examined MI Write as an automated writing evaluation benchmark for monitoring middle-school writing growth in grades 6-8. Using multilevel growth models with 3,299 students, the authors found modest overall gains across the school year, with similar total growth for English learners and non-English learners but different growth trajectories and feature-level patterns.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2026journal article

Writing Motivation Profiles Among Diverse Adolescents: A Mixed Methods Study

Tania Cruz Cordero, Joshua Wilson

This mixed methods study in Scientific Studies of Reading used latent profile analysis with 2,534 middle school students and interviews with 27 students to identify distinct writing motivation profiles. The article reports four profiles-personal interest, self-expression, performance, and non-specific-and shows that these groups differ in demographic composition and writing performance.

Writing Development, Motivation & Learner ExperienceWriting Instruction & Pedagogical DesignEquity, English Learners & Student Support
View publication details
2025journal article

A comparative study on monologic vs. dialogic technology-mediated feedback modalities: Students' uptake and perspectives

Maryam Kazemi, Mahboobeh Saadat, Joshua Wilson, Mohammad Rahimi

This study compares screencast feedback and one-on-one video-conference feedback for L2 academic writing revisions. Using literature-review assignments from 10 students, the authors examined revision uptake and student perspectives and found stronger uptake for the dialogic video-conference condition on several writing features.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2025journal article

AI-Integrated Scaffolding to Enhance Agency and Creativity in K-12 English Language Learners: A Systematic Review

Molly Li, Joshua Wilson

This systematic review synthesizes 14 peer-reviewed empirical studies from 2018 to 2024 on how AI-integrated tools support K-12 English language learners' writing. Using thematic synthesis, the article examines AI as cognitive support, creative collaborator, and language-enhancement tool while emphasizing the importance of preserving student agency and creativity.

AI, Automation & Educational MeasurementWriting Instruction & Pedagogical DesignEquity, English Learners & Student SupportWriting Development, Motivation & Learner Experience
View publication details
2025journal article

Assessing the Accuracy of Automated Writing Evaluation in Predicting English Language Arts Proficiency for Middle-Grade English Language Learners and Non-English Language Learners

Fan Zhang, Joshua Wilson

This study examined how well fall, winter, and spring MI Write benchmark writing scores predicted non-proficiency on the Smarter Balanced ELA assessment for middle-grade students. Using ROC curve analyses, the authors compared classification accuracy across grade levels and language status for English learners and non-English learners.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2025journal article

Exploring the Effectiveness of Large-Scale Automated Writing Evaluation Implementation on State Test Performance Using Generalised Boosted Modelling

Yue Huang, Joshua Wilson

This study examined the first-year statewide implementation of the Utah Compose automated writing evaluation system and its association with students' English Language Arts state test performance. Using statewide data from students in Grades 4-11 in Utah (N = 337,473) and a quasi-experimental design with generalized boosted modelling for propensity score weighting, the authors found that more frequent AWE use was associated with improved ELA performance, with some variation across student demographic groups.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementImplementation, Teachers & Educational Systems
View publication details
2025journal article

Identifying active ingredients and uptake patterns in the implementation of an AI-based writing support tool: Insights from a randomized controlled trial

Yue Huang, Corey Palermo, Joshua Wilson

This secondary analysis used treatment-group data from a randomized controlled trial of MI Write, an automated writing evaluation system used by 902 middle school students across three U.S. districts, to examine which system components and usage patterns were associated with writing outcomes.

AI, Automation & Educational MeasurementWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2025journal article

Recommendations for Integrating AWE with Evidence-Based Practices

Joshua Wilson, Tania Cruz Cordero, Andrew Potter, Matthew Myers, Charles A. MacArthur, Gaysha Beard, Emily A. Fudge, Alexandria Raiche, Cristina Ahrendt

This review synthesizes research-based recommendations for integrating automated writing evaluation with evidence-based writing instruction. It argues that AWE is most effective when students learn the evaluation criteria, receive strategy instruction for planning and revising, benefit from teacher feedback in addition to automated feedback, and engage in goal setting and progress monitoring.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2024journal article

A Latent Dirichlet Allocation approach to understanding students' perceptions of Automated Writing Evaluation

Joshua Wilson, Saimou Zhang, Corey Palermo, Tania Cruz Cordero, Fan Zhang, Matthew C. Myers, Andrew Potter, Halley Eacker, Jessica Coles

This article examines U.S. middle school students' perceptions of the MI Write automated writing evaluation system using survey responses and open-ended comments. The authors apply latent Dirichlet allocation and follow-up qualitative analysis to identify common themes in students' feedback and compare topic prevalence by perceived usefulness of the tool.

Automated Writing Evaluation & FeedbackAI, Automation & Educational MeasurementWriting Development, Motivation & Learner Experience
View publication details
2024journal article

Elementary English learners' engagement with automated feedback

Joshua Wilson, Corey Palermo, Arianto Wibowo

This study examined how elementary English learners interacted with feedback from the MI Write automated writing evaluation system. Using data from 3,459 students in grades 3-5, the authors compared non-ELs, proficient ELs, and non-proficient ELs on access to feedback, writing improvement, revision behavior, and perceptions of the system.

Automated Writing Evaluation & FeedbackEquity, English Learners & Student SupportWriting Development, Motivation & Learner Experience
View publication details
2024journal article

Exploring the Long-Term Effects of the Statewide Implementation of an Automated Writing Evaluation System on Students' State Test ELA Performance

Yue Huang, Joshua Wilson, Henry May

This article examines the long-term effects of statewide implementation of Utah Compose, an automated writing evaluation system, on Grades 4-10 students' state English Language Arts performance in Utah using a quasi-experimental inverse probability of treatment weighting design.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementImplementation, Teachers & Educational Systems
View publication details
2024book chapter

Exploring the Role of Automated Writing Evaluation as a Formative Assessment Tool Supporting Self-Regulated Learning in Writing

Joshua Wilson, Charles A. MacArthur

This chapter reviews whether automated writing evaluation can function as a formative assessment tool that supports self-regulated learning in writing. Using Black and Wiliam's framework, it examines how automated writing evaluation may help clarify success criteria, elicit writing practice, provide feedback, and support revision and other self-regulatory processes in classroom writing instruction.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignAI, Automation & Educational Measurement
View publication details
2024book chapter

Introduction to Automated Evaluation

Mark D. Shermis, Joshua Wilson

This introductory chapter traces the development of automated essay scoring, automated essay evaluation, and automated writing evaluation. It distinguishes these terms, explains how the field expanded from summative scoring to formative feedback and writing support, and situates current applications in both large-scale assessment and classroom contexts.

Automated Writing Evaluation & FeedbackAI, Automation & Educational Measurement
View publication details
2024journal articleFeatured

Middle School Teachers' Implementation and Perceptions of Automated Writing Evaluation

Joshua Wilson, Amanda Delgado, Corey Palermo, Tania M. Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles, Saimou Zhang

This mixed-methods study examined how middle school teachers implemented and perceived the MI Write automated writing evaluation system during a randomized controlled trial. Teachers generally reported positive attitudes toward MI Write, but implementation varied and was shaped by contextual factors such as curricular alignment, workload, and administrative support.

Automated Writing Evaluation & FeedbackImplementation, Teachers & Educational Systems
View publication details
2024journal article

Predictors of middle school students' perceptions of automated writing evaluation

Joshua Wilson, Fan Zhang, Corey Palermo, Tania Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles

This study examined middle school students' perceptions of the MI Write automated writing evaluation system, focusing on usability, usefulness, and desirability. Using hierarchical regression models, the authors tested district, classroom, demographic, writing-belief, affective, and writing-proficiency predictors of students' perceptions.

Automated Writing Evaluation & FeedbackWriting Development, Motivation & Learner Experience
View publication details
2024book chapter

Relationships Between Middle-School Teachers' Perceptions and Application of Automated Writing Evaluation and Student Performance

Amanda Delgado, Joshua Wilson, Corey Palermo, Tania M. Cruz Cordero, Matthew C. Myers, Halley Eacker, Andrew Potter, Jessica Coles, Saimou Zhang

This handbook chapter examines how middle school teachers' perceptions of automated writing evaluation relate to their implementation of the technology and, in turn, to student writing outcomes. It argues that the effects of AWE depend partly on how teachers understand, value, and enact the system in classroom practice.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2024journal articleFeatured

The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges

Okan Bulut, Maggie Beiting-Parrish, Jodi M. Casabianca, Sharon C. Slater, Hong Jiao, Dan Song, Christopher Ormerod, Deborah Gbemisola Fabiyi, Rodica Ivan, Cole Walsh, Oscar Rios, Joshua Wilson, Seyma N. Yildirim-Erbasli, Tarid Wongvorachan, Joyce Xinle Liu, Bin Tan, Polina Morilova

This article reviews how artificial intelligence is changing educational measurement through automated scoring, content analysis, and personalized feedback, while also examining ethical concerns related to validity, reliability, transparency, fairness, equity, automation bias, and environmental impact.

Writing Assessment, Validity & MeasurementAI, Automation & Educational Measurement
View publication details
2024book

The Routledge International Handbook of Automated Essay Evaluation

Mark D. Shermis, Joshua Wilson

This edited handbook offers a comprehensive overview of automated essay evaluation across large-scale assessment and classroom-based teaching and learning contexts. It synthesizes current applications, ethical and psychometric concerns, and emerging technological directions shaping the field.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementAI, Automation & Educational MeasurementWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2024journal articleFeatured

Validity of automated essay scores for elementary-age English language learners: Evidence of bias?

Joshua Wilson, Yue Huang

This study examined whether automated essay scores from the MI Write system show evidence of bias for elementary-age English language learners. Using data from 2,829 students in grades 3-5, the authors compared the predictive validity of automated and human scores on an English-learner-focused writing test and a state writing test, and found that automated scoring was not uniquely biased relative to human scoring, though both methods were less predictive for English language learners than for non-ELL students on the state test.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2023book chapter

Computer-Based Writing Instruction

Andrew Potter, Joshua Wilson, Rod D. Roscoe, Tracy Arner, Danielle S. McNamara

This chapter reviews research on computer-based writing instruction, including technologies that provide computer-generated feedback and instructional support for writers. It argues that these tools are most effective when paired with evidence-based writing practices and authentic writing communities, and it closes by considering emerging opportunities and risks associated with generative AI.

Writing Instruction & Pedagogical DesignAI, Automation & Educational Measurement
View publication details
2023journal article

Evaluating the Construct Validity of an Automated Writing Evaluation System with a Randomization Algorithm

Matthew C. Myers, Joshua Wilson

This article examines the construct validity of the MI Write automated writing evaluation system by randomizing student persuasive essays at the sentence level and comparing trait scores for original and randomized versions. The study focuses on whether higher-level traits such as idea development and organization decrease when essay coherence is disrupted.

Writing Assessment, Validity & MeasurementAI, Automation & Educational Measurement
View publication details
2023journal article

Student engagement with teacher and automated written corrective feedback on L2 writing: A multiple case study

Sara Afifi, Mohammad Rahimi, Joshua Wilson

This multiple-case study examined how four Iranian EFL learners engaged behaviorally, cognitively, and affectively with teacher written corrective feedback and automated written corrective feedback from Writing Mentor in an academic writing course. The study compared engagement patterns across the two feedback sources and considered how learner beliefs, attitudes, and writing proficiency related to engagement.

Automated Writing Evaluation & FeedbackEquity, English Learners & Student SupportWriting Development, Motivation & Learner Experience
View publication details
2023journal article

Writing motivation and ability profiles and transition during a technology-based writing intervention

Tania Cruz Cordero, Joshua Wilson, Matthew C. Myers, Corey Palermo, Halley Eacker, Andrew Potter, Jessica Coles

This article uses latent profile and latent transition analysis to examine how middle-school students participating in an automated writing evaluation intervention differed in writing motivation and ability, and how those profiles changed over time. The study identifies four profiles and evaluates whether treatment assignment or student characteristics predicted profile transitions.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2022journal article

Examining Human and Automated Ratings of Elementary Students' Writing Quality: A Multivariate Generalizability Theory Application

Dandan Chen, Michael Hebert, Joshua Wilson

This article uses multivariate generalizability theory to compare the reliability of human hand-scoring and automated essay scoring for elementary students' writing. Using essays from grade 3-5 students across multiple genres, the study examines how the two scoring methods perform for struggling and non-struggling writers and considers how they might be combined to improve writing assessment.

Writing Assessment, Validity & MeasurementAI, Automation & Educational Measurement
View publication details
2022journal articleFeatured

Integrating Goal-Setting and Automated Feedback

Joshua Wilson, Andrew Potter, Tania Cruz Cordero, Matthew C. Myers

This pilot study tested an intervention that combined goal setting and reflection with automated writing evaluation to improve middle school students' writing outcomes. Students improved in goal calibration, confidence in meeting goals, self-efficacy for writing self-regulation, and writing performance, while also reporting that the intervention was usable, useful, and desirable.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2022journal articleFeatured

Investigating the Promise of Automated Writing Evaluation for Supporting Formative Writing Assessment at Scale

Joshua Wilson, Matthew C. Myers, Andrew Potter

This study investigates whether MI Write can support formative writing assessment at scale in elementary grades by examining teachers' perceptions and implementation alongside changes in student writing performance across the school year. Teachers generally viewed the system as usable and useful, but implemented it in limited ways, and student performance gains were uneven across grades and genres.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2022journal article

Upper-Elementary Students' Metacognitive Knowledge About Writing and Its Relationship to Writing Outcomes Across Genres

Joshua Wilson, Huijing Wen

This study investigated fourth- and fifth-grade students' metacognitive knowledge about writing using a metawriting task and examined how that knowledge related to writing quality and length across genres. Students were most aware of general characteristics of good writing and production procedures, while knowledge of substantive processes and genre was less developed, and only some dimensions uniquely predicted writing outcomes.

Writing Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2021journal article

Automated Feedback and Automated Scoring in the Elementary Grades: Usage, Attitudes, and Associations with Writing Outcomes in a Districtwide Implementation of MI Write

Joshua Wilson, Yue Huang, Corey Palermo, Gaysha Beard, Charles A. MacArthur

This article reports on a districtwide implementation of the MI Write automated writing evaluation system in elementary schools. It examines how the system was used, teacher and student attitudes toward it, and whether usage patterns were associated with student writing outcomes.

Automated Writing Evaluation & FeedbackAI, Automation & Educational MeasurementWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2021journal article

Elementary teachers' perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation

Joshua Wilson, Cristina Ahrendt, Emily A. Fudge, Alexandria Raiche, Gaysha Beard, Charles A. MacArthur

This qualitative study examined elementary writing teachers' experiences with the MI Write automated writing evaluation system during a districtwide implementation in grades 3-5. Using focus groups and activity theory, the authors explored how teachers perceived automated feedback and scoring as influencing instruction, student writing behavior, and classroom practice.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2021journal article

Statewide implementation of automated writing evaluation: Analyzing usage and associations with state test performance in grades 4-11

Andrew Potter, Joshua Wilson

This study examined a statewide implementation of an automated writing evaluation system used by more than 114,000 students in grades 4-11. The authors analyzed feature usage, possible equity and access differences in use, and whether AWE usage was associated with performance on a large-scale state writing assessment.

Automated Writing Evaluation & FeedbackAI, Automation & Educational MeasurementImplementation, Teachers & Educational Systems
View publication details
2021journal article

The Sensitivity of a Scenario-Based Assessment of Written Argumentation to School Differences in Curriculum and Instruction

Paul Deane, Joshua Wilson, Mo Zhang, Chen Li, Peter van Rijn, Hongwen Guo, Amanda Roth, Eowyn Winchester, Theresa Richter

This article examines whether a scenario-based assessment of written argumentation, supplemented with automated writing evaluation features and keystroke-log process measures, is sensitive to differences in curriculum and instruction across middle schools. Using data from one suburban and two urban schools, the authors found significant score gains and distinctive improvement patterns in the suburban school, suggesting the approach can provide actionable interim information about student progress.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementWriting Instruction & Pedagogical Design
View publication details
2021journal article

Using automated feedback to develop writing proficiency

Yue Huang, Joshua Wilson

This study examined whether use of the automated writing evaluation system MI Write was associated with growth in writing quality among 431 students in grades 4-5. Using growth-curve and gain-score models, the authors found non-linear improvement across successive revisions, no evidence of unaided transfer to first-draft writing quality, and evidence of aided transfer in students' revision growth rates.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical Design
View publication details
2020journal article

Automated Writing Evaluation and Feedback: Multiple Metrics of Efficacy

Joshua Wilson, Rod D. Roscoe

This study evaluated automated writing evaluation using multiple outcomes, including writing self-efficacy, holistic writing quality, state English language arts performance, and teachers' perceptions of social validity. After controlling for pretest measures, students using AWE showed stronger writing self-efficacy and higher state test performance, even though holistic writing quality did not differ from the comparison condition.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & Measurement
View publication details
2020journal article

Classification accuracy and efficiency of writing screening using automated essay scoring

Joshua Wilson, Jessica Rodrigues

This study tested whether automated essay scoring could support universal screening for writing by classifying elementary students as at risk for later English language arts failure. It compared a multi-prompt automated essay scoring screener, shorter automated essay scoring versions with fewer prompts, and word-count scoring using ROC-based classification accuracy analyses.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2020journal article

Implementing Automated Writing Evaluation in Different Instructional Contexts: A Mixed-Methods Study

Corey Palermo, Joshua Wilson

This mixed-methods study examined how the automated writing evaluation system MI Write was implemented in two middle-school writing contexts: a traditional process approach and strategy instruction based on Self-Regulated Strategy Development. The study assessed student growth in first-draft writing quality across essays and gathered teacher and student perceptions of teaching and learning with AWE.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2020journal article

Modeling Student Evaluations of Writing and Authors as a Function of Writing Errors

Rod D. Roscoe, Joshua Wilson, Melissa Patchan, Dandan Chen, Adam C. Johnson

This study examined how college students evaluate essays containing different kinds of writing errors and how those judgments extend to perceptions of the author. Using a within-subjects design and exploratory factor analysis, the authors found that ratings of writing quality and author characteristics clustered into two correlated latent constructs rather than remaining fully separate.

Writing Assessment, Validity & MeasurementWriting Development, Motivation & Learner Experience
View publication details
2019book chapter

Assessing Writing

Joshua Wilson

This chapter frames writing assessment through an assessment-for-learning lens, asking what students need to learn, where they are now, and how instruction can move them forward. It emphasizes that strong writing assessment depends on drawing accurate inferences from student work and using those inferences to guide teaching.

Writing Assessment, Validity & MeasurementWriting Instruction & Pedagogical Design
View publication details
2019journal article

Generalizability of Automated Scores of Writing Quality in Grades 3-5

Joshua Wilson, Dandan Chen, Micheal P. Sandbank, Michael Hebert

This study examined the reliability of automated essay scoring for elementary students' writing quality across grades 3-5. Using Project Essay Grade and generalizability theory, the authors evaluated variation due to prompts, genres, and student-by-facet interactions, with separate analyses for struggling and nonstruggling writers.

Writing Assessment, Validity & MeasurementAI, Automation & Educational Measurement
View publication details
2018journal article

Content and alignment of state writing standards and assessments as predictors of student writing achievement: An analysis of 2007 National Assessment of Educational Progress data

Gary A. Troia, Natalie G. Olinghouse, Mingcai Zhang, Joshua Wilson, Kelly A. Stewart, Ya Mo, Lisa Hawkins

This article examined whether the content of state writing standards and assessments, and the alignment between them, predicted student writing achievement on the 2007 NAEP writing assessment after controlling for student, school, and state characteristics. The authors found that student demographics explained the most between-state variation and that greater alignment between state writing tests and NAEP was associated with higher writing scores.

Writing Assessment, Validity & MeasurementWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2018journal article

Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4

This study examined whether a direct assessment of writing scored with automated essay scoring could be used as a universal screener to identify third- and fourth-grade students at risk of failing the Smarter Balanced English language arts assessment. Using ROC curve analysis and logistic regression, the study evaluated classification accuracy for fall and spring screening data.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2017journal article

Associated effects of automated essay evaluation software on growth in writing quality for students with and without disabilities

This study examined growth in writing quality associated with the automated essay evaluation system PEG Writing using matched samples of students with disabilities and typically developing students. Students with disabilities began with lower-quality drafts but improved at a faster rate across revisions, with some effects shaped by school quality and access to internet-connected devices. No evidence of transfer to a new writing prompt was found.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementEquity, English Learners & Student Support
View publication details
2017journal articleFeatured

Automated Formative Writing Assessment Using a Levels of Language Framework

Joshua Wilson, Rod D. Roscoe, Yusra Ahmed

This study tests whether automated Coh-Metrix measures can represent three levels of language in student writing - word choice, syntax, and cohesion/discourse - and whether those latent factors predict performance on state writing assessments for grades 6 and 8.

Automated Writing Evaluation & FeedbackWriting Assessment, Validity & MeasurementWriting Instruction & Pedagogical Design
View publication details
2017journal article

College Student Perceptions of Writing Errors, Text Quality, and Author Characteristics

Adam C. Johnson, Joshua Wilson, Rod D. Roscoe

This within-subjects study examined how lower-level and higher-level writing errors influenced college students' judgments of text quality and author characteristics. Students detected both types of errors, but they were much more sensitive to lower-level errors such as spelling and grammar than to higher-level issues such as organization and argument quality.

Writing Assessment, Validity & MeasurementWriting Development, Motivation & Learner Experience
View publication details
2017journal article

Presentation, expectations, and experience: Sources of student perceptions of automated writing evaluation

Rod D. Roscoe, Joshua Wilson, Adam C. Johnson, Christopher R. Mayra

This article examines how students form perceptions of automated writing evaluation systems and how those perceptions relate to writing performance, revision behavior, and intentions to use the technology again. The study finds that framing and presentation shape expectations somewhat, but direct experience with the software is the strongest driver of student perceptions.

Automated Writing Evaluation & FeedbackWriting Development, Motivation & Learner Experience
View publication details
2016journal article

Automated Essay Evaluation Software in English Language Arts Classrooms: Effects on Teacher Feedback, Student Motivation, and Writing Quality

Joshua Wilson, Amanda Czik

This quasi-experimental study examined whether combining automated essay evaluation with teacher feedback changed the feedback students received, their writing motivation, and their writing quality in eighth-grade English language arts classrooms. Teachers in the combined-feedback condition gave proportionately more higher-level feedback and reported notable time savings, while students showed greater writing persistence but no advantage in final-draft writing quality.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2016journal article

Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment

Joshua Wilson, Natalie G. Olinghouse, D. Betsy McCoach, Tanya Santangelo, Gilbert N. Andrada

This study compared several writing scoring methods for their ability to identify sixth graders at risk of failing a state writing assessment. Using benchmark writing data from fall 2012 and state assessment results from spring 2013, the authors evaluated prior writing achievement, human holistic scoring, automated essay scoring, and related writing indicators using ROC and AUC classification analyses.

Writing Assessment, Validity & MeasurementAI, Automation & Educational MeasurementEquity, English Learners & Student Support
View publication details
2016journal article

The Common Core Writing Standards: A Descriptive Study of Content and Alignment with a Sample of Former State Standards

Gary A. Troia, Natalie G. Olinghouse, Joshua Wilson, Kelly A. Stewart, Ya Mo, Lisa Hawkins, Rachel A. Kopke

This article compares the content of the Common Core State Standards for writing and language with a sample of former state writing standards. It evaluates breadth, frequency, balance, and alignment, showing that the Common Core offers concise and relatively balanced coverage while prior state standards varied considerably and aligned only weakly.

Writing Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2016book chapter

Using Automated Feedback to Improve Writing Quality: Opportunities and Challenges

Joshua Wilson, Gilbert Andrada

This chapter examines the promise and limitations of automated feedback in writing contexts, with attention to where technology can support instruction and where human judgment remains essential. It situates automated feedback within the broader automated essay evaluation literature and discusses both practical opportunities and persistent concerns.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical Design
View publication details
2015journal article

Academic Standards for Writing: To What Degree Do Standards Signpost Evidence-Based Instructional Practices and Interventions?

Gary A. Troia, Natalie G. Olinghouse, Ya Mo, Lisa Hawkins, Rachel A. Kopke, Angela Chen, Joshua Wilson, Kelly A. Stewart

This article examines writing standards from a representative sample of states and the Common Core State Standards to determine the extent to which they signpost evidence-based practices for writing instruction. The authors found that some practices were included fairly consistently, but others appeared rarely, with substantial variation across standards and limited overlap between many states' standards and the CCSS.

Writing Instruction & Pedagogical DesignImplementation, Teachers & Educational Systems
View publication details
2014journal articleFeatured

Does Automated Feedback Improve Writing Quality?

Joshua Wilson, Natalie G. Olinghouse, Gilbert N. Andrada

This study examines whether automated feedback in a statewide computer-based benchmark writing assessment was associated with improved writing quality across revisions and with transfer to later writing tasks. Writing quality improved across revisions, but the gains decelerated over time and the study found no strong transfer effects to subsequent first-draft performance.

Automated Writing Evaluation & FeedbackWriting Instruction & Pedagogical Design
View publication details
2014journal article

Writing Performance of Students with Emotional and/or Behavioral Disabilities

Nicholas A. Gage, Joshua Wilson, Ashley S. MacSuga-Gage

This study examined the writing performance of students with emotional and/or behavioral disabilities and compared their results with both a full sample of typical peers and a propensity-score-matched comparison group. The authors also tested whether reading performance mediated differences in writing outcomes.

Writing Assessment, Validity & MeasurementEquity, English Learners & Student SupportWriting Development, Motivation & Learner Experience
View publication details
2013journal article

The Relationship Between Vocabulary and Writing Quality in Three Genres

Natalie G. Olinghouse, Joshua Wilson

This study examined how vocabulary relates to writing quality across narrative, persuasive, and informative writing in fifth grade. Students wrote three compositions on a shared topic, and the authors analyzed several vocabulary measures alongside holistic writing quality.

Writing Assessment, Validity & MeasurementWriting Instruction & Pedagogical DesignWriting Development, Motivation & Learner Experience
View publication details
2012book chapter

Examining the Validity of Single-Occasion, Single-Genre, Holistically Scored Writing Assessments

Natalie G. Olinghouse, Tanya Santangelo, Joshua Wilson

This chapter examines whether single-occasion, single-genre, holistically scored writing assessments provide valid evidence about students' broader writing ability. Using writing from fifth graders across story, persuasive, and informative genres, the study questions whether a single genre and a single holistic score can adequately represent the range of skills embodied in writing standards.

Writing Assessment, Validity & MeasurementWriting Instruction & Pedagogical Design
View publication details