Reading Users' Minds from What They Say: An Investigation into LLM-based Empathic Mental Inference

TL;DR

This paper explores how Large Language Models can infer users' goals and psychological needs to improve empathic understanding in human-centered design, achieving performance comparable to human designers.

Contribution

It introduces a benchmark dataset and an empathic accuracy metric for evaluating LLMs' mental inference capabilities in user experience research.

Findings

LLMs can infer user goals and needs with accuracy similar to humans.

Zero-shot prompt engineering enhances LLMs' empathic inference performance.

Potential for scalable, in-depth user understanding in design processes.

Abstract

In human-centered design, developing a comprehensive and in-depth understanding of user experiences, i.e., empathic understanding, is paramount for designing products that truly meet human needs. Nevertheless, accurately comprehending the real underlying mental states of a large human population remains a significant challenge today. This difficulty mainly arises from the trade-off between depth and scale of user experience research: gaining in-depth insights from a small group of users does not easily scale to a larger population, and vice versa. This paper investigates the use of Large Language Models (LLMs) for performing mental inference tasks, specifically inferring users' underlying goals and fundamental psychological needs (FPNs). Baseline and benchmark datasets were collected from human users and designers to develop an empathic accuracy metric for measuring the mental inference performance of LLMs. The empathic accuracy of inferring goals and FPNs of different LLMs with varied zero-shot prompt engineering techniques are experimented against that of human designers. Experimental results suggest that LLMs can infer and understand the underlying goals and FPNs of users with performance comparable to that of human designers, suggesting a promising avenue for enhancing the scalability of empathic design approaches through the integration of advanced artificial intelligence technologies. This work has the potential to significantly augment the toolkit available to designers during human-centered design, enabling the development of both large-scale and in-depth understanding of users' experiences.