HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations

TL;DR

This paper introduces HumBEL, a human-in-the-loop framework for evaluating how well large language models adapt to demographic factors like age and gender in human-machine conversations, using clinical techniques and automated methods.

Contribution

It presents a novel evaluation approach combining clinical speech pathology techniques with automated tools to assess demographic compatibility of language models.

Findings

GPT-3.5 mimics 6-15 year olds in inference tasks

Outperforms a 21-year-old in memorization tasks

Less than 50% of pragmatic skills are exhibited by GPT-3.5

Abstract

While demographic factors like age and gender change the way people talk, and in particular, the way people talk to machines, there is little investigation into how large pre-trained language models (LMs) can adapt to these changes. To remedy this gap, we consider how demographic factors in LM language skills can be measured to determine compatibility with a target demographic. We suggest clinical techniques from Speech Language Pathology, which has norms for acquisition of language skills in humans. We conduct evaluation with a domain expert (i.e., a clinically licensed speech language pathologist), and also propose automated techniques to complement clinical evaluation at scale. Empirically, we focus on age, finding LM capability varies widely depending on task: GPT-3.5 mimics the ability of humans ranging from age 6-15 at tasks requiring inference, and simultaneously, outperforms a typical 21 year old at memorization. GPT-3.5 also has trouble with social language use, exhibiting less than 50% of the tested pragmatic skills. Findings affirm the importance of considering demographic alignment and conversational goals when using LMs as public-facing tools. Code, data, and a package will be available.