Rationale Behind Essay Scores: Enhancing S-LLM’s Multi-Trait Essay Scoring with Rationale Generated by LLMs

Traditional automated essay scoring (AES) focused on holistic scoring, predicting an overall score using handcrafted features and linear regression models Taghipour and Ng (2016); Dong and Zhang (2016); Dong et al. (2017); Cozma et al. (2018). Particularly, transformer-based models like BERT Devlin et al. (2018) significantly improved AES by capturing detailed language information Yang et al. (2020); Wang et al. (2022); Mayfield and Black (2020). These models enhanced scoring accuracy but were primarily used for holistic scoring. Extending them to multi-trait scoring is inefficient due to the need for multiple models for different traits, increasing computational costs Kumar et al. (2021); Do et al. (2024).

Multi-trait AES evaluates essays across various dimensions with respect to different features existing in essays such as Content, Organization, and Conventions. Existing models used multiple linear layers or separate models for each trait, which require intensive resources Mathias and Bhattacharyya (2020); Ridley et al. (2021). Recent approaches introduced multi-task learning frameworks with shared models and trait-specific layers, improving efficiency Kumar et al. (2021). However, handling trait dependencies and requiring specialized modules remains challenging. The autoregressive multi-trait scoring (ArTS) model addressed this by using a pre-trained T5 model to sequentially generate trait scores, leveraging inter-dependencies for better accuracy Do et al. (2024). Yet, it still relied solely on essay texts alone for score prediction.

Recently, LLMs have been used to evaluate essays alongside assessment rubrics, showing competitive performance. For example, one study Lee et al. (2024) divided criteria into multiple traits and generated sub-criteria for scoring, achieving moderate results. Another study Li et al. (2023) used rubrics to score short answers and generated rationales, which were then used as labels to fine-tune the T5 model Raffel et al. (2020) to produce both scores and rationales. However, this approach did not outperform fine-tuned models like BERT Devlin et al. (2018) and Longformer Beltagy et al. (2020a), which were trained using only scores as labels. Our approach differs by directly extracting rubric-based rationales from essays using LLMs and feeding them into a pre-trained S-LLM. This method explicitly considers detailed scoring criteria, improving alignment with human evaluators and enhancing both the reliability and transparency of automated essay scoring.

As illustrated in Figure 2, individual trait-specific prompts are constructed using the essay and the rubric corresponding to each trait. Each trait-specific prompt is then provided to a separate LLM agent dedicated to that trait. This approach, referred to as the LLM-based trait-wise rationale generation system, relies on the LLM’s demonstrated ability to effectively evaluate essays, as supported by prior research Lee et al. (2024); Ho et al. (2022); Li et al. (2023). We have adopted the prompts used in Lee et al. (2024) as a basis for the task description and modified them to fit our context. We have also added trait-specific rubric to them. Our trait-wise LLM agents generate qualitative assessments based on the rubric, producing rationales in a text form. This method enables the generation of detailed, text-based rationales that are directly tied to the rubric, facilitating a subsequent S-LLM to decide the final numeric score in a more accurate manner.

Given that the decoder of the S-LLM used in RMTS predicts subsequent tokens based on previous ones, the rationale was also constructed in a sequence that evaluates sub-dimensional, constituent traits of the essay first (e.g., Content, Organization, Style), followed by the overall trait. This approach capitalizes on the model’s capability to boost predictive performance by replicating the sequential nature of human assessors when evaluating traits Do et al. (2024).

In the current study, we utilize various pre-trained encoder-decoder S-LLMs for scoring multi-traits of essays. We include five widely used models—T5, Flan-T5, BART, Pegasus, and LED (Longformer Encoder-Decoder model) Raffel et al. (2020); Chung et al. (2024); Lewis et al. (2019); Zhang et al. (2019); Beltagy et al. (2020b)—as the S-LLMs for essay scoring. Figure 3 shows the RMTS architecture. Each component in RMTS framework’s essay scoring model corresponds to the respective component of the individual S-LLM.

In RMTS, both the essay and the generated rationales are fed into a single encoder to extract their respective representations, which means that the two texts share a common encoder, allowing their representations to be projected into the same vector space. Inspired by Do et al. (2024), we add the prompt "Score the essay of the prompt N" to the essay text to improve model inference. Special tokens, such as "<Essay>" and "<Rationale>," are inserted before the essay and rationale to help the tokenizer distinguish between the two. We also introduce tokens for multi-trait names (e.g., <Content>) to prevent them from being split into sub-words, preserving their meanings.

The encoder processes this combined input to generate dense representations, which are integrated into a unified feature vector by a linear layer for scoring. This vector is passed to a decoder, which predicts trait-specific scores. By leveraging both the essay and rationale, the model delivers detailed multi-trait scoring.

Since we use S-LLMs, which are sequence-to-sequence models, we predict and generate scores for multiple traits alongside their respective names from each essay one at a time, based on techniques from Do et al. (2024). The generated string of scores is transformed into a dictionary format, where trait names serve as keys to extract the scores. For accurate evaluation, we disregard predictions for traits whose ground truth values are NaN.

In our main experiment, we utilized the ASAP³³3https://www.kaggle.com/competitions/asap-aes/data and ASAP++⁴⁴4https://lwsam.github.io/ASAP++/lrec2018.html Mathias and Bhattacharyya (2020) datasets, comprising English essays from American high school students (grades 7–10) across eight prompts. The ASAP dataset provides overall scores for all essays, but only prompts 7 and 8 have trait-specific scores. Thus, we included ASAP++ for rated trait scores on the remaining prompts, and this combined dataset will be referred to as "ASAP/ASAP++" throughout the paper. Additionally, the Feedback Prize dataset⁵⁵5https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data, which consists of argumentative essays written by American students (grades 6–12) and labeled with six traits, was used without distinguishing between prompts to examine the generalizability of the incremental effect of using essays and rationales together on vanilla S-LLMs. Due to space constraints, the dataset descriptions are provided in Table 1.

To evaluate rationale quality, we performed various analyses. We evaluated the similarity of the generated rationales using ROUGE-L Lin (2004) on a sample of 100 essays to analyze the diversity in how LLMs generate them. Additionally, we measured the faithfulness of LLM-generated rationales to the predicted multi-trait scores, using a proxy method from prior studies Wiegreffe et al. (2020); Jain et al. (2020); Li et al. (2023). Specifically, we fine-tuned S-LLMs to predict multi-trait scores using only the rationales as input to the models.

To compare performance across the two datasets described earlier, we used five widely adopted vanilla S-LLMs, all encoder-decoder models Raffel et al. (2020); Chung et al. (2024); Lewis et al. (2019); Zhang et al. (2019); Beltagy et al. (2020b) designed for text generation tasks. We also included baseline models with a string kernel based model and RNN-based architectures from the referenced papers: HISK Cozma et al. (2018), STL-LSTM Dong et al. (2017) and MTL-BiLSTM Kumar et al. (2021), which align with our model’s characteristics (see Appendix A for details on each baseline). For a fair comparison with the traditional benchmark dataset (ASAP/ASAP++), we used the performance data of the four baseline models—HISK Cozma et al. (2018), STL-LSTM Dong et al. (2017), MTL-BiLSTM Kumar et al. (2021), and ArTS Do et al. (2024)—as reported in Do et al. (2024). ArTS is a model that employs the vanilla T5-base model for scoring multi-trait essays.

In this study, we employed GPT-3.5-Turbo⁶⁶6https://openai.com/index/openai-api/ and Llama-3.1-8B-Instruct⁷⁷7https://llama.meta.com/responsible-use-guide/ for rationale generation based on a prompt-engineering technique, and fine-tuned pre-trained S-LLMs from Huggingface⁸⁸8https://huggingface.co/. Using the Seq2SeqTrainer from the same platform, models were trained over 15 epochs with a batch size of 4, and evaluations took place every 5000 steps, applying early stopping with a patience of 2. All experiments were conducted on a single NVIDIA A100 GPU using the PyTorch framework.

To ensure consistent evaluation, we utilized 5-fold cross-validation across all models, employing a 60/20/20 split for training, validation, and testing, following the methodology of Taghipour and Ng (2016) and Kumar et al. (2021) with the combined ASAP and ASAP++ dataset. For the Feedback Prize dataset, we applied the same 5-fold process but with stratified splitting based on label distribution. Assessment was conducted using quadratic weighted kappa (QWK) Cohen (1968), the dataset’s designated metric, which effectively measures score disparities between human raters and model predictions. We chose the top two models from each fold and reported the highest QWK as the final result Do et al. (2024).

We focus on analyzing the rationales from the ASAP/ASAP++ and Feedback Prize datasets in terms of similarity and faithfulness.

To address the second research question, we first evaluate RMTS against the baseline models and vanilla S-LLMs on the ASAP/ASAP++ dataset, the standard for essay scoring. We then assess RMTS against vanilla S-LLMs on the Feedback Prize dataset to demonstrate its broader applicability. Finally, to gain deeper insights into the role of rationales, we fine-tuned three S-LLMs with rationales and essays, removing one trait at a time during the process to observe performance variations.

To assess the effectiveness of trait-specific rationales, we conducted an ablation study by removing a trait commonly shared across prompts, as illustrated in Figure 7. We focused on traits present in at least four prompts. Specifically, Content is evaluated in all prompts, while Organization and Conventions are scored in Prompts 1, 2, 7, and 8. Prompt Adherence, Language, and Narrativity are assessed in Prompts 3 to 6. For comparison, we divided the prompts into two groups and excluded Word Choice, Sentence Fluency, Style, and Voice since they appear in fewer prompts. We used T5, Flan-T5, and BART in the experiments, fine-tuning each model by removing one trait at a time.

A consistent decline in performance is observed across all traits when the corresponding rationale is removed, confirming that trait rationales significantly influence their respective assessments. For instance, removing the rationale for Conventions results in a performance drop for that trait, particularly when compared to RMTS, which utilizes rationales for all traits. Although the performance of models without a trait rationale lag behind RMTS, it still outperforms vanilla models without any rationale input. This suggests that trait rationales not only influence their own assessments but also interact with and affect the evaluation of other traits Canale and Swain (1980). For example, when the rationale for Conventions is removed, performance still surpass that of vanilla models.

Interestingly, performance does not always drop significantly when certain trait rationales are removed, particularly in the case of T5 when Organization and Prompt Adherence rationales are excluded. This implies that the effectiveness of rationales can vary by trait and model.

Overall, the results show that rationales generally have an incremental effect on performance. Essay scoring performs worse when trait rationales are removed, highlighting the crucial role these rationales play in predicting trait scores.