Assessing how accurately large language models encode and apply the common European framework of reference for languages
Abstract
Large Language Models (LLMs) can have a transformative effect on a variety of domains, including education, and it is therefore pressing to understand whether these models have knowledge of – or, in other words, how they have encoded – the specific pedagogical requirements of different educational domains, and whether they use this when performing educational tasks. In this work, we propose an approach to evaluate the knowledge – or encoding – that the LLMs have of the Common European Framework of Reference for Languages (CEFR), and use it to evaluate five modern LLMs. Our study shows that the suite of tasks we propose is quite challenging for all the LLMs, and they often provide results which are not satisfactory and would be unusable in educational applications, suggesting that – even if they encode some information about the CEFR – this knowledge is not really leveraged when performing downstream tasks.
Identifier Metadata
| Identifier | 110.0769/CON.2026.00740 |
| Canonical | mdoi:110.0769/CON.2026.00740 |
| Resolver URL | https://mdoi.org/110.0769/CON.2026.00740 |
| Resource URL | Open resource |
| Document URL | Open document |
| Content Type | Article |
| Authors | Luca Benedetto, Gabrielle Gaudeau, Andrew Caines, Paula Buttery |
| Year | 2025 |
| Depositor | Convergence Chronicles Organisation |
| Prefix | 110.0769 |
| Registered | July 23, 2026 |
| Updated | July 23, 2026 |
| Status | Active |
| Visibility | Public |
Cite This Identifier
APA 7th Edition
Click to copy
MLA 9th Edition
Click to copy
Chicago 17th Edition
Click to copy
BibTeX
Click to copy
Persistent Identifier
mdoi:110.0769/CON.2026.00740Click to copy