Multilingual Audio Captioning using machine translated data

TL;DR

This paper explores multilingual automated audio captioning by translating datasets into multiple languages, training monolingual and multilingual models, and demonstrating the effectiveness of building native language systems over translation-based approaches.

Contribution

It introduces a multilingual AAC approach using machine translation of datasets and develops a multilingual model that performs comparably to monolingual systems with fewer parameters.

Findings

Monolingual systems achieved about 75% CIDEr on AudioCaps and 43% on Clotho.

French manual captions improved results over translated outputs, supporting native language system development.

Multilingual model performed comparably to monolingual models with fewer parameters.

Abstract

Automated Audio Captioning (AAC) systems attempt to generate a natural language sentence, a caption, that describes the content of an audio recording, in terms of sound events. Existing datasets provide audio-caption pairs, with captions written in English only. In this work, we explore multilingual AAC, using machine translated captions. We translated automatically two prominent AAC datasets, AudioCaps and Clotho, from English to French, German and Spanish. We trained and evaluated monolingual systems in the four languages, on AudioCaps and Clotho. In all cases, the models achieved similar performance, about 75% CIDEr on AudioCaps and 43% on Clotho. In French, we acquired manual captions of the AudioCaps eval subset. The French system, trained on the machine translated version of AudioCaps, achieved significantly better results on the manual eval subset, compared to the English system for which we automatically translated the outputs to French. This advocates in favor of building systems in a target language instead of simply translating to a target language the English captions from the English system. Finally, we built a multilingual model, which achieved results in each language comparable to each monolingual system, while using much less parameters than using a collection of monolingual systems.