Label Anything: An Interpretable, High-Fidelity and Prompt-Free Annotator

TL;DR

The paper introduces LAM, a prompt-free, interpretable model that leverages a pretrained Vision Transformer and minimal training data to generate high-fidelity annotations for street scene datasets, reducing manual labeling costs.

Contribution

The novel LAM framework combines a Vision Transformer, a semantic class adapter, and an optimization-based unrolling algorithm to produce accurate annotations with minimal training data and high interpretability.

Findings

Achieves nearly 100% mIoU on multiple datasets

Requires only a single seed image for training

Demonstrates high-fidelity annotations across real-world and simulated datasets

Abstract

Learning-based street scene semantic understanding in autonomous driving (AD) has advanced significantly recently, but the performance of the AD model is heavily dependent on the quantity and quality of the annotated training data. However, traditional manual labeling involves high cost to annotate the vast amount of required data for training robust model. To mitigate this cost of manual labeling, we propose a Label Anything Model (denoted as LAM), serving as an interpretable, high-fidelity, and prompt-free data annotator. Specifically, we firstly incorporate a pretrained Vision Transformer (ViT) to extract the latent features. On top of ViT, we propose a semantic class adapter (SCA) and an optimization-oriented unrolling algorithm (OptOU), both with a quite small number of trainable parameters. SCA is proposed to fuse ViT-extracted features to consolidate the basis of the subsequent automatic annotation. OptOU consists of multiple cascading layers and each layer contains an optimization formulation to align its output with the ground truth as closely as possible, though which OptOU acts as being interpretable rather than learning-based blackbox nature. In addition, training SCA and OptOU requires only a single pre-annotated RGB seed image, owing to their small volume of learnable parameters. Extensive experiments clearly demonstrate that the proposed LAM can generate high-fidelity annotations (almost 100% in mIoU) for multiple real-world datasets (i.e., Camvid, Cityscapes, and Apolloscapes) and CARLA simulation dataset.