State-Space Large Audio Language Models

TL;DR

This paper introduces the first state-space-based Large Audio Language Model, replacing transformers in perception and language modules, achieving competitive performance with fewer parameters and addressing computational challenges.

Contribution

It pioneers the use of state-space models in large audio language models, replacing transformers for perception and language understanding modules.

Findings

State-space LALM performs competitively on close-ended tasks.

Significantly fewer parameters than transformer-based models.

Addresses computational scalability issues.

Abstract

Large Audio Language Models (LALM) combine the audio perception models and the Large Language Models (LLM) and show a remarkable ability to reason about the input audio, infer the meaning, and understand the intent. However, these systems rely on Transformers which scale quadratically with the input sequence lengths which poses computational challenges in deploying these systems in memory and time-constrained scenarios. Recently, the state-space models (SSMs) have emerged as an alternative to transformer networks. While there have been successful attempts to replace transformer-based audio perception models with state-space ones, state-space-based LALMs remain unexplored. First, we begin by replacing the transformer-based audio perception module and then replace the transformer-based LLM and propose the first state-space-based LALM. Experimental results demonstrate that space-based LALM despite having a significantly lower number of parameters performs competitively with transformer-based LALMs on close-ended tasks on a variety of datasets.