Can Language Models Learn to Listen?

TL;DR

This paper introduces a novel framework that enables language models to generate contextually appropriate facial responses in social interactions by integrating gesture prediction with language understanding, leveraging pre-trained models for improved performance.

Contribution

The work presents a new method combining language and gesture modeling using a transformer with VQ-VAE quantization, enhancing listener response generation in social interactions.

Findings

Generated responses are fluent and semantically relevant.

Pre-trained language models significantly improve response quality.

Model effectively captures temporal and semantic aspects of speech.

Abstract

We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Given an input transcription of the speaker's words with their timestamps, our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. Since gesture is a language component, we propose treating the quantized atomic motion elements as additional language token inputs to a transformer-based large language model. Initializing our transformer with the weights of a language model pre-trained only on text results in significantly higher quality listener responses than training a transformer from scratch. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study. In our evaluation, we analyze the model's ability to utilize temporal and semantic aspects of spoken text. Project page: https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/