TouchAI: Exploring human-AI perceptual alignment in touch through language model representations

TL;DR

This paper investigates how well large language models align with human touch perceptions using a textile comparison task, revealing partial alignment that varies across textile types and highlighting areas for improvement.

Contribution

It introduces a novel 'Guess What Textile' task to assess perceptual alignment between LLMs and human touch experiences, an unexplored area in AI alignment research.

Findings

LLMs show some perceptual alignment with human touch descriptions.

Alignment varies significantly among different textile samples.

Participants' perceptions did not closely match LLM predictions.

Abstract

Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This work investigates how well LLMs align with human touch experiences using the "textile hand" task. We created a "Guess What Textile" interaction in which participants were given two textile samples -- a target and a reference -- to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space. Our results suggest that a degree of perceptual alignment exists, however varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants didn't perceive their textile experiences closely matched by the LLM predictions. This is only the first exploration into perceptual alignment around touch, exemplified through textile hand. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks.