Evaluating the potential of ChatGPT-reformulated essays as written feedback in L2 writing

Reformulation is a form of written corrective feedback to help second language (L2) learners improve their writing. This study examined whether ChatGPT could produce reformulations that (1) retain the meanings of the original essays and (2) are linguistically more developed than learners’ original essays. In addition, three types of ChatGPT prompts were compared to see which type yielded better reformulations. One thousand two hundred argumentative essays written for the TOEFL iBT® independent writing task were submitted to ChatGPT. ROUGE-L scores, used as a proxy for meaning retention, showed that ChatGPT reformulations largely retained the meaning of the original essays. A qualitative examination was conducted to examine the major types of changes ChatGPT made. For linguistic features, the ChatGPT reformulations were compared with the original essays for syntactic complexity, lexical sophistication, lexical diversity, and cohesion. Results showed that while ChatGPT reformulations were more developed for most linguistic features than the original essays, the reformulations did worse in cohesion. ChatGPT prompts with specific instructions produced reformulations with more developed linguistic features than a generic prompt. Findings were discussed in terms of how to use ChatGPT to generate reformulations and how to use the reformulations to improve L2 writing.