Automated scoring of the Ambiguous Intentions Hostility Questionnaire using fine-tuned large language models

TL;DR

This study demonstrates that fine-tuned large language models can accurately automate the scoring of open-ended responses in the Ambiguous Intentions Hostility Questionnaire, reducing the need for time-consuming human ratings.

Contribution

We developed and validated a fine-tuned large language model approach for automated scoring of AIHQ responses, showing high alignment with human ratings and generalization across datasets.

Findings

Model ratings aligned with human ratings for hostility and aggression.

Fine-tuned models outperformed baseline models in accuracy.

Models generalized well to an independent nonclinical dataset.

Abstract

Hostile attribution bias is the tendency to interpret social interactions as intentionally hostile. The Ambiguous Intentions Hostility Questionnaire (AIHQ) is commonly used to measure hostile attribution bias, and includes open-ended questions where participants describe the perceived intentions behind a negative social situation and how they would respond. While these questions provide insights into the contents of hostile attributions, they require time-intensive scoring by human raters. In this study, we assessed whether large language models can automate the scoring of AIHQ open-ended responses. We used a previously collected dataset in which individuals with traumatic brain injury (TBI) and healthy controls (HC) completed the AIHQ and had their open-ended responses rated by trained human raters. We used half of these responses to fine-tune the two models on human-generated ratings, and tested the fine-tuned models on the remaining half of AIHQ responses. Results showed that model-generated ratings aligned with human ratings for both attributions of hostility and aggression responses, with fine-tuned models showing higher alignment. This alignment was consistent across ambiguous, intentional, and accidental scenario types, and replicated previous findings on group differences in attributions of hostility and aggression responses between TBI and HC groups. The fine-tuned models also generalized well to an independent nonclinical dataset. To support broader adoption, we provide an accessible scoring interface that includes both local and cloud-based options. Together, our findings suggest that large language models can streamline AIHQ scoring in both research and clinical contexts, revealing their potential to facilitate psychological assessments across different populations.