MiDashengLM: Efficient Audio Understanding with General Audio Captions

TL;DR

MiDashengLM is an open, efficient audio-language model that uses general audio captions for comprehensive understanding, achieving faster processing and higher throughput than existing models, while relying solely on publicly available datasets.

Contribution

Introduces MiDashengLM, a novel open-source audio-language model that processes diverse audio types with a new training dataset, emphasizing transparency and efficiency.

Findings

Up to 4x faster in time-to-first-token

Up to 20x higher throughput than comparable models

Uses publicly available datasets for training and fine-tuning

Abstract

Current approaches for large audio language models (LALMs) often rely on closed data sources or proprietary models, limiting their generalization and accessibility. This paper introduces MiDashengLM, a novel open audio-language model designed for efficient and comprehensive audio understanding through the use of general audio captions using our novel ACAVCaps training dataset. MiDashengLM exclusively relies on publicly available pretraining and supervised fine-tuning (SFT) datasets, ensuring full transparency and reproducibility. At its core, MiDashengLM integrates Dasheng, an open-source audio encoder, specifically engineered to process diverse auditory information effectively. Unlike previous works primarily focused on Automatic Speech Recognition (ASR) based audio-text alignment, our strategy centers on general audio captions, fusing speech, sound and music information into one textual representation, enabling a holistic textual representation of complex audio scenes. Lastly, MiDashengLM provides an up to 4x speedup in terms of time-to-first-token (TTFT) and up to 20x higher throughput than comparable models. Checkpoints are available online at https://huggingface.co/mispeech/midashenglm-7b and https://github.com/xiaomi-research/dasheng-lm.