LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Models

TL;DR

This paper addresses the challenge of generating detailed long captions for long videos using Large Multimodal Models, proposing a new data synthesis framework and benchmark to improve and evaluate long caption generation.

Contribution

We introduce LongCaption-Agent, a hierarchical data synthesis framework, and create LongCaption-10K and LongCaption-Bench datasets to enhance and assess long video captioning capabilities.

Findings

LMMs struggle to generate captions over 300 words for long videos.

Training with synthesized long-caption data enables models to produce over 1,000 words.

Our approach achieves state-of-the-art results on long captioning benchmarks.

Abstract

Large Multimodal Models (LMMs) have demonstrated exceptional performance in video captioning tasks, particularly for short videos. However, as the length of the video increases, generating long, detailed captions becomes a significant challenge. In this paper, we investigate the limitations of LMMs in generating long captions for long videos. Our analysis reveals that open-source LMMs struggle to consistently produce outputs exceeding 300 words, leading to incomplete or overly concise descriptions of the visual content. This limitation hinders the ability of LMMs to provide comprehensive and detailed captions for long videos, ultimately missing important visual information. Through controlled experiments, we find that the scarcity of paired examples with long-captions during training is the primary factor limiting the model's output length. However, manually annotating long-caption examples for long-form videos is time-consuming and expensive. To overcome the annotation bottleneck, we propose the LongCaption-Agent, a framework that synthesizes long caption data by hierarchical semantic aggregation. % aggregating multi-level descriptions. Using LongCaption-Agent, we curated a new long-caption dataset, LongCaption-10K. We also develop LongCaption-Bench, a benchmark designed to comprehensively evaluate the quality of long captions generated by LMMs. By incorporating LongCaption-10K into training, we enable LMMs to generate captions exceeding 1,000 words for long-form videos, while maintaining high output quality. In LongCaption-Bench, our model achieved State-of-The-Art performance, even surpassing larger proprietary models like GPT4o.