RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification

TL;DR

This paper introduces RB-FT, a two-stage self-improvement method that enhances vision language models for domain-specific video classification by generating and fine-tuning on self-created rationales, reducing the need for new annotations.

Contribution

The paper presents a novel rationale-based fine-tuning approach that improves domain adaptation of VLMs for video classification without additional annotations.

Findings

Significant performance improvements over direct supervised fine-tuning.

Effective use of self-generated rationales as intermediate supervision.

Validated across diverse video datasets.

Abstract

Vision Language Models (VLMs) are becoming increasingly integral to multimedia understanding; however, they often struggle with domain-specific video classification tasks, particularly in cases with limited data. This stems from a critical \textit{rationale gap}, where sparse domain data is insufficient to bridge the semantic distance between complex spatio-temporal content and abstract classification labels. We propose a two-stage self-improvement paradigm to bridge this gap without new annotations. First, we prompt the VLMs to generate detailed textual rationales for each video, compelling them to articulate the domain-specific logic. The VLM is then fine-tuned on these self-generated rationales, utilizing this intermediate supervision to align its representations with the nuances of the target domain. Second, conventional supervised fine-tuning (SFT) is performed on the task labels, achieving markedly higher effectiveness as a result of the model's pre-acquired domain reasoning. Extensive experiments on diverse datasets demonstrate that our method significantly outperforms direct SFT, validating self-generated rationale as an effective, annotation-efficient paradigm for adapting VLMs to domain-specific video analysis.