Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding

TL;DR

AcuLa is a post-training framework that aligns audio models with medical language models to imbue clinical understanding, significantly improving diagnostic performance in medical audio tasks.

Contribution

The paper introduces AcuLa, a novel method for aligning audio encoders with medical language models using large-scale clinical report generation, enhancing semantic understanding in medical audio analysis.

Findings

Achieved state-of-the-art results on 18 cardio-respiratory tasks.

Improved AUROC from 0.68 to 0.79 on classification benchmarks.

Boosted COVID-19 cough detection AUROC from 0.55 to 0.89.

Abstract

Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance, limiting their use and performance in diagnostic tasks. To bridge this gap, we introduce AcuLa (Audio-Clinical Understanding via Language Alignment), a lightweight post-training framework that instills semantic understanding into any audio encoder by aligning it with a medical language model, which acts as a "semantic teacher." To enable alignment at scale, we construct a large-scale dataset by leveraging off-the-shelf large language models to translate the rich, structured metadata accompanying existing audio recordings into coherent clinical reports. Our alignment strategy combines a representation-level contrastive objective with a self-supervised modeling, ensuring that the model learns clinical semantics while preserving fine-grained temporal cues. AcuLa achieves state-of-the-art results across 18 diverse cardio-respiratory tasks from 10 different datasets, improving the mean AUROC on classification benchmarks from 0.68 to 0.79 and, on the most challenging COVID-19 cough detection task, boosting the AUROC from 0.55 to 0.89. Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools, establishing a novel paradigm for enhancing physiological understanding in audio-based health monitoring.