"Yeah Right!" -- Do LLMs Exhibit Multimodal Feature Transfer?

TL;DR

This paper investigates whether large language models trained on speech and text, especially those focused on human conversations, can transfer multimodal communication skills like detecting covert deception, finding speech+text models have an advantage.

Contribution

It demonstrates that speech+text LLMs and conversation-trained models outperform unimodal models in detecting covert deception without special prompts.

Findings

Speech+text LLMs outperform unimodal models in deception detection.

Models trained on human conversations have enhanced multimodal transfer skills.

No special prompting needed for speech+text models to excel.

Abstract

Human communication is a multifaceted and multimodal skill. Communication requires an understanding of both the surface-level textual content and the connotative intent of a piece of communication. In humans, learning to go beyond the surface level starts by learning communicative intent in speech. Once humans acquire these skills in spoken communication, they transfer those skills to written communication. In this paper, we assess the ability of speech+text models and text models trained with special emphasis on human-to-human conversations to make this multimodal transfer of skill. We specifically test these models on their ability to detect covert deceptive communication. We find that with no special prompting speech+text LLMs have an advantage over unimodal LLMs in performing this task. Likewise, we find that human-to-human conversation-trained LLMs are also advantaged in this skill.