Merlin L48 Spectrogram Dataset

TL;DR

This paper introduces the L48 dataset, a real-world, fine-grained bird sound dataset for single-positive multi-label learning, highlighting the limitations of synthetic datasets and benchmarking existing methods on this challenging new benchmark.

Contribution

The paper presents the L48 dataset, a realistic, fine-grained bird sound dataset for SPML, and evaluates existing methods, revealing their weaknesses and the need for more challenging benchmarks.

Findings

Significant performance gaps between synthetic and real-world datasets.

Existing SPML methods underperform on the L48 dataset.

Benchmarking shows the necessity for more robust SPML approaches.

Abstract

In the single-positive multi-label (SPML) setting, each image in a dataset is labeled with the presence of a single class, while the true presence of other classes remains unknown. The challenge is to narrow the performance gap between this partially-labeled setting and fully-supervised learning, which often requires a significant annotation budget. Prior SPML methods were developed and benchmarked on synthetic datasets created by randomly sampling single positive labels from fully-annotated datasets like Pascal VOC, COCO, NUS-WIDE, and CUB200. However, this synthetic approach does not reflect real-world scenarios and fails to capture the fine-grained complexities that can lead to difficult misclassifications. In this work, we introduce the L48 dataset, a fine-grained, real-world multi-label dataset derived from recordings of bird sounds. L48 provides a natural SPML setting with single-positive annotations on a challenging, fine-grained domain, as well as two extended settings in which domain priors give access to additional negative labels. We benchmark existing SPML methods on L48 and observe significant performance differences compared to synthetic datasets and analyze method weaknesses, underscoring the need for more realistic and difficult benchmarks.