Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding

TL;DR

This paper introduces a zero-shot classifier editing technique for fine-grained video understanding, enabling the refinement of coarse categories into subcategories without additional data, improving accuracy and adaptability.

Contribution

The paper proposes a novel zero-shot editing method for classifiers that leverages latent structure to refine categories, along with a new benchmark for category splitting in videos.

Findings

Outperforms vision-language baselines in category splitting tasks

Significantly improves accuracy on newly split categories

Zero-shot initialization benefits low-shot fine-tuning

Abstract

Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: https://kaitingliu.github.io/Category-Splitting/.