LUCY: Linguistic Understanding and Control Yielding Early Stage of Her

TL;DR

LUCY is an advanced end-to-end speech model that understands and responds to emotional cues in human speech, producing natural, emotionally aware responses and integrating external tools for real-time inquiries.

Contribution

It introduces a novel E2E speech system capable of emotion sensing, natural response generation, and external tool integration, advancing emotional and functional capabilities of AI audio agents.

Findings

LUCY outperforms peer models in emotion control.

LUCY generates more natural responses judged by external models.

LUCY effectively uses external tools for real-time question answering.

Abstract

The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-time responses that are natural, informative and sensitive to emotional subtleties. Moving one step toward more sophisticated audio agent from recent advancement in end-to-end (E2E) speech systems, we propose LUCY, a E2E speech model that (1) senses and responds to user's emotion, (2) deliver responses in a succinct and natural style, and (3) use external tool to answer real-time inquiries. Experiment results show that LUCY is better at emotion control than peer models, generating emotional responses based on linguistic emotional instructions and responding to paralinguistic emotional cues. Lucy is also able to generate responses in a more natural style, as judged by external language models, without sacrificing much performance on general question answering. Finally, LUCY can leverage function calls to answer questions that are out of its knowledge scope.