A Bayesian active learning approach to comparative judgement within education assessment

Assessment is a crucial part of education. Traditional marking is a source of inconsistencies and unconscious bias, placing a high cognitive load on the assessors. One approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items of work, and asked to select the better one. Following a series of comparisons, a rank for any item may be derived using a ranking model, for example, the Bradley-Terry model, based on the pairwise comparisons. While CJ is considered to be a reliable method for conducting marking, there are concerns surrounding its transparency, and the ideal number of pairwise comparisons to generate a reliable estimation of the rank order is not known. Additionally, there have been attempts to generate a method of selecting pairs that should be compared next in an informative manner, but some existing methods are known to have created their own bias within results inflating the reliability metric used within the process. As a consequence, a random selection approach is usually deployed. In this paper, we propose a novel Bayesian approach to CJ (which we call BCJ) for determining the ranks of a range of items under scrutiny alongside a new way to select the pairs to present to the marker(s) using active learning, addressing the key shortcomings of traditional CJ. Furthermore, we demonstrate how the entire approach may provide transparency by providing the user insights into how it is making its decisions and, at the same time, being more efficient. Results from our synthetic experiments confirm that the proposed BCJ combined with entropy-driven active learning pair-selection method is superior (i.e. always equal to or significantly better) than other alternatives, for example, the traditional CJ method with differing selection methods such as uniformly random, or the popular no repeating pairs where pairs are selected in a round-robin fashion. We also find that the more comparisons that are conducted, the more accurate BCJ becomes, which solves the issue the current method has of the model deteriorating if too many comparisons are performed. As our approach can generate the complete predicted rank distribution for an item, we also show how this can be utilised in probabilistically devising a predicted grade, guided by the choice of the assessor. Finally, we demonstrate our approach on a real dataset on assessing GCSE (UK school-level) essays, highlighting the advantages of BCJ over CJ.