Identifying active ingredients and uptake patterns in the implementation of an AI-based writing support tool: Insights from a randomized controlled trial

While AI-based educational technologies offer many features designed to support learning, it remains unclear which specific components actually contribute to improved student outcomes. Identifying these active ingredients—the subset of an intervention's core components that are demonstrably associated with learning gains—is essential. Yet, evidence from experimental studies remains limited. To address this gap, we conducted a secondary analysis of data from the treatment group of a randomized controlled trial involving 902 middle school students across three U.S. school districts who used MI Write, an automated writing evaluation (AWE) system. Drawing on implementation science, we examined which core components of MI Write, and which uptake patterns, were most strongly associated with student outcomes. Hierarchical linear modeling identified the amount of writing and revisions completed within MI Write as active ingredients, as both were significantly associated with improvements in students' argumentative writing performance. However, uptake of these active ingredients varied across districts, with greater engagement correlating with improved outcomes in one district. Student-level factors, such as baseline writing performance, perceptions of teacher caring, and demographic characteristics, influenced writing task completion and revisions, while teacher-level factors primarily explained variability in writing task completion. These findings emphasize the importance of reaching specific uptake benchmarks in writing and revision to optimize AWE's impact. Furthermore, fostering positive classroom climates may enhance engagement, particularly among underserved student populations. This study advances the understanding of AWE's active ingredients and contextual dependencies, offering actionable insights for improving the design, implementation, and scalability of AI-based writing support tools.