NELS -- Never-Ending Learner of Sounds

TL;DR

NELS is a continuously learning system that automatically analyzes, indexes, and improves sound recognition models by leveraging web data, addressing large-scale sound understanding challenges without relying on predefined references.

Contribution

The paper introduces NELS, a novel system for ongoing learning of sounds and their relations to language from web data, enabling scalable and autonomous sound knowledge acquisition.

Findings

NELS improves sound recognition accuracy over time.

The system effectively learns from web-based sound-language relations.

Large-scale evaluation demonstrates autonomous learning capabilities.

Abstract

Sounds are essential to how humans perceive and interact with the world and are captured in recordings and shared on the Internet on a minute-by-minute basis. These recordings, which are predominantly videos, constitute the largest archive of sounds we know. However, most of these recordings have undescribed content making necessary methods for automatic sound analysis, indexing and retrieval. These methods have to address multiple challenges, such as the relation between sounds and language, numerous and diverse sound classes, and large-scale evaluation. We propose a system that continuously learns from the web relations between sounds and language, improves sound recognition models over time and evaluates its learning competency in the large-scale without references. We introduce the Never-Ending Learner of Sounds (NELS), a project for continuously learning of sounds and their associated knowledge, available on line in nels.cs.cmu.edu