Vision Language Models Are Few-Shot Audio Spectrogram Classifiers

TL;DR

This paper shows that vision language models can effectively classify audio spectrograms in a few-shot setting, outperforming existing audio models and even human experts on specific tasks.

Contribution

It introduces a novel approach of using vision language models for audio classification via spectrogram images, demonstrating strong performance in few-shot scenarios.

Findings

VLMs achieve 59% accuracy on ESC-10 dataset.

VLMs outperform Gemini-1.5 audio model in classification accuracy.

VLMs perform slightly better than human experts on spectrogram classification.

Abstract

We demonstrate that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images. Specifically, we instruct VLMs to perform audio classification tasks in a few-shot setting by prompting them to classify a spectrogram image given example spectrogram images of each class. By carefully designing the spectrogram image representation and selecting good few-shot examples, we show that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset. Moreover, we demonstrate that VLMs currently outperform the only available commercial audio language model with audio understanding capabilities (Gemini-1.5) on the equivalent audio classification task (59.00% vs. 49.62%), and even perform slightly better than human experts on visual spectrogram classification (73.75% vs. 72.50% on first fold). We envision two potential use cases for these findings: (1) combining the spectrogram and language understanding capabilities of VLMs for audio caption augmentation, and (2) posing visual spectrogram classification as a challenge task for VLMs.