The CMU-AIST submission for the ICME 2025 Audio Encoder Challenge

TL;DR

This paper presents an advanced audio encoding system for the ICME 2025 challenge, combining large-scale pre-training, model scaling, and ensembling techniques to improve performance over existing baselines.

Contribution

It introduces a scaled-up BEATs model trained on diverse data and a novel ensembling method that outperforms previous models and baselines.

Findings

Ensembling improves audio encoding quality.

Pre-training on diverse data enhances performance.

Our system surpasses baseline and Dasheng 1.2B models.

Abstract

This technical report describes our submission to the ICME 2025 audio encoder challenge. Our submitted system is built on BEATs, a masked speech token prediction based audio encoder. We extend the BEATs model using 74,000 hours of data derived from various speech, music, and sound corpora and scale its architecture upto 300 million parameters. We experiment with speech-heavy and balanced pre-training mixtures to study the impact of different domains on final performance. Our submitted system consists of an ensemble of the Dasheng 1.2 billion model with two custom scaled-up BEATs models trained on the aforementioned pre-training data mixtures. We also propose a simple ensembling technique that retains the best capabilities of constituent models and surpasses both the baseline and Dasheng 1.2B. For open science, we publicly release our trained checkpoints via huggingface at https://huggingface.co/shikhar7ssu/OpenBEATs-ICME-SOUND and https://huggingface.co/shikhar7ssu/OpenBEATs-ICME.