Sound and Visual Representation Learning with Multiple Pretraining Tasks

TL;DR

This paper introduces a Multi-SSL approach combining multiple self-supervised tasks for binaural sound and image data, demonstrating improved downstream task performance over single SSL and supervised models.

Contribution

It proposes a novel Multi-SSL framework with incremental learning that enhances feature representations across modalities, outperforming existing methods.

Findings

Multi-SSL with incremental learning outperforms single SSL and supervised models in binaural sound tasks.

Multi-SSL surpasses recent image SSL methods on VOC07 classification and COCO detection.

The approach generalizes well across different data modalities, improving downstream task results.

Abstract

Different self-supervised tasks (SSL) reveal different features from the data. The learned feature representations can exhibit different performance for each downstream task. In this light, this work aims to combine Multiple SSL tasks (Multi-SSL) that generalizes well for all downstream tasks. Specifically, for this study, we investigate binaural sounds and image data in isolation. For binaural sounds, we propose three SSL tasks namely, spatial alignment, temporal synchronization of foreground objects and binaural audio and temporal gap prediction. We investigate several approaches of Multi-SSL and give insights into the downstream task performance on video retrieval, spatial sound super resolution, and semantic prediction on the OmniAudio dataset. Our experiments on binaural sound representations demonstrate that Multi-SSL via incremental learning (IL) of SSL tasks outperforms single SSL task models and fully supervised models in the downstream task performance. As a check of applicability on other modality, we also formulate our Multi-SSL models for image representation learning and we use the recently proposed SSL tasks, MoCov2 and DenseCL. Here, Multi-SSL surpasses recent methods such as MoCov2, DenseCL and DetCo by 2.06%, 3.27% and 1.19% on VOC07 classification and +2.83, +1.56 and +1.61 AP on COCO detection. Code will be made publicly available.