Curriculum Audiovisual Learning

TL;DR

This paper introduces a curriculum learning-based audiovisual model that effectively associates sounds with their sources, enabling improved sound localization and separation with minimal supervision.

Contribution

It proposes a novel curriculum learning strategy combined with a soft-clustering module for audiovisual association, enhancing training efficiency and model performance.

Findings

Significantly outperforms existing sound localization methods.

Achieves comparable sound separation performance without external visual supervision.

Provides effective unimodal and cross-modal representations.

Abstract

Associating sound and its producer in complex audiovisual scene is a challenging task, especially when we are lack of annotated training data. In this paper, we present a flexible audiovisual model that introduces a soft-clustering module as the audio and visual content detector, and regards the pervasive property of audiovisual concurrency as the latent supervision for inferring the correlation among detected contents. To ease the difficulty of audiovisual learning, we propose a novel curriculum learning strategy that trains the model from simple to complex scene. We show that such ordered learning procedure rewards the model the merits of easy training and fast convergence. Meanwhile, our audiovisual model can also provide effective unimodal representation and cross-modal alignment performance. We further deploy the well-trained model into practical audiovisual sound localization and separation task. We show that our localization model significantly outperforms existing methods, based on which we show comparable performance in sound separation without referring external visual supervision. Our video demo can be found at https://youtu.be/kuClfGG0cFU.