AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioning

TL;DR

The AnyCap Project introduces a comprehensive framework, dataset, and benchmark for controllable omni-modal captioning, significantly improving caption quality and controllability across various models and modalities.

Contribution

It presents a novel lightweight framework (ACM), a large-scale dataset (ACD), and a new evaluation benchmark (AnyCapEval) for enhanced controllable captioning.

Findings

ACM improves caption quality across multiple models.

ACM-8B boosts GPT-4o's content scores by 45%.

The dataset covers three modalities and 28 instruction types.

Abstract

Controllable captioning is essential for precise multimodal alignment and instruction following, yet existing models often lack fine-grained control and reliable evaluation protocols. To address this gap, we present the AnyCap Project, an integrated solution spanning model, dataset, and evaluation. We introduce AnyCapModel (ACM), a lightweight plug-and-play framework that enhances the controllability of existing foundation models for omni-modal captioning without retraining the base model. ACM reuses the original captions from base models while incorporating user instructions and modality features to generate improved captions. To remedy the data scarcity in controllable multimodal captioning, we build AnyCapDataset (ACD), covering three modalities, 28 user-instruction types, and 300\,k high-quality data entries. We further propose AnyCapEval, a new benchmark that provides more reliable evaluation metrics for controllable captioning by decoupling content accuracy and stylistic fidelity. ACM markedly improves caption quality across a diverse set of base models on AnyCapEval. Notably, ACM-8B raises GPT-4o\'s content scores by 45\% and style scores by 12\%, and it also achieves substantial gains on widely used benchmarks such as MIA-Bench and VidCapBench.