A Simple Method to Enhance Pre-trained Language Models with Speech Tokens for Classification

TL;DR

This paper introduces a straightforward approach to augment pre-trained language models with speech tokens for classification, improving performance by selecting key audio features through a simple feature selection method.

Contribution

The method combines speech tokenization with feature selection and self-supervised learning to enhance language models with speech information efficiently.

Findings

Improved classification performance over unimodal models and larger SpeechLMs.

Even random audio token selection can enhance unimodal models.

Effective on Argumentative Fallacy Detection and affective computing tasks.

Abstract

This paper presents a simple method that allows to easily enhance textual pre-trained large language models with speech information, when fine-tuned for a specific classification task. A classical issue with the fusion of many embeddings from audio with text is the large length of the audio sequence compared to the text one. Our method benefits from an existing speech tokenizer trained for Audio Speech Recognition that output long sequences of tokens from a large vocabulary, making it difficult to integrate it at low cost in a large language model. By applying a simple lasso-based feature selection on multimodal Bag-of-Words representation, we retain only the most important audio tokens for the task, and adapt the language model to them with a self-supervised language modeling objective, before fine-tuning it on the downstream task. We show this helps to improve the performances compared to an unimodal model, to a bigger SpeechLM or to integrating audio via a learned representation. We demonstrate its effectiveness on Argumentative Fallacy Detection and Classification tasks where audio was previously believed counterproductive, and affective computing tasks on a widely-used dataset. We also provide an in-depth analysis of the method, showing that even a random audio token selection helps enhancing the unimodal model. Our code is available [online](https://github.com/salocinc/EACL26SpeechTokFallacy/).