Fairness Evaluation of Large Language Models in Academic Library Reference Services

TL;DR

This study assesses whether large language models provide equitable responses in academic library reference services, finding minimal bias and nuanced role accommodation, indicating their potential for fair support.

Contribution

It is the first comprehensive evaluation of LLM fairness in academic library contexts, analyzing biases related to user demographics and social roles.

Findings

No evidence of racial or ethnic bias in responses.

Minor stereotypical bias against women in one model.

LLMs adapt language based on institutional roles, reflecting norms rather than bias.

Abstract

As libraries explore large language models (LLMs) for use in virtual reference services, a key question arises: Can LLMs serve all users equitably, regardless of demographics or social status? While they offer great potential for scalable support, LLMs may also reproduce societal biases embedded in their training data, risking the integrity of libraries' commitment to equitable service. To address this concern, we evaluate whether LLMs differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role. We find no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model. LLMs demonstrate nuanced accommodation of institutional roles through the use of linguistic choices related to formality, politeness, and domain-specific vocabularies, reflecting professional norms rather than discriminatory treatment. These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services.