Multi-label Scene Classification for Autonomous Vehicles: Acquiring and Accumulating Knowledge from Diverse Datasets

TL;DR

This paper presents a novel deep learning approach combining Knowledge Acquisition and Accumulation with Consistency-based Active Learning to improve multi-label scene classification for autonomous vehicles, effectively utilizing heterogeneous datasets and recognizing unseen attributes.

Contribution

It introduces a new method that leverages single-label datasets to enhance multi-label scene classification, reducing data requirements and enabling recognition of unseen attributes.

Findings

56.1% improvement over baseline on DSI dataset

Outperforms state-of-the-art methods on BDD100K and HSD datasets

Achieves high accuracy with 85% less data

Abstract

Driving scenes are inherently heterogeneous and dynamic. Multi-attribute scene identification, as a high-level visual perception capability, provides autonomous vehicles (AVs) with essential contextual awareness to understand, reason through, and interact with complex driving environments. Although scene identification is best modeled as a multi-label classification problem via multitask learning, it faces two major challenges: the difficulty of acquiring balanced, comprehensively annotated datasets and the need to re-annotate all training data when new attributes emerge. To address these challenges, this paper introduces a novel deep learning method that integrates Knowledge Acquisition and Accumulation (KAA) with Consistency-based Active Learning (CAL). KAA leverages monotask learning on heterogeneous single-label datasets to build a knowledge foundation, while CAL bridges the gap between single- and multi-label data, adapting the foundation model for multi-label scene classification. An ablation study on the newly developed Driving Scene Identification (DSI) dataset demonstrates a 56.1% improvement over an ImageNet-pretrained baseline. Moreover, KAA-CAL outperforms state-of-the-art multi-label classification methods on the BDD100K and HSD datasets, achieving this with 85% less data and even recognizing attributes unseen during foundation model training. The DSI dataset and KAA-CAL implementation code are publicly available at https://github.com/KELISBU/KAA-CAL .