VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning

TL;DR

VidChain introduces a novel framework that decomposes dense video captioning into sub-tasks and aligns training with evaluation metrics, significantly enhancing fine-grained video understanding in large language models.

Contribution

The paper proposes VidChain, combining Chain-of-Tasks and Metric-based Direct Preference Optimization to improve dense video captioning and temporal grounding.

Findings

Outperforms previous VideoLLMs on DVC benchmarks

Enhances fine-grained video understanding

Improves temporal video grounding results

Abstract

Despite the advancements of Video Large Language Models (VideoLLMs) in various tasks, they struggle with fine-grained temporal understanding, such as Dense Video Captioning (DVC). DVC is a complicated task of describing all events within a video while also temporally localizing them, which integrates multiple fine-grained tasks, including video segmentation, video captioning, and temporal video grounding. Previous VideoLLMs attempt to solve DVC in a single step, failing to utilize their reasoning capability. Moreover, previous training objectives for VideoLLMs do not fully reflect the evaluation metrics, therefore not providing supervision directly aligned to target tasks. To address such a problem, we propose a novel framework named VidChain comprised of Chain-of-Tasks (CoTasks) and Metric-based Direct Preference Optimization (M-DPO). CoTasks decompose a complex task into a sequence of sub-tasks, allowing VideoLLMs to leverage their reasoning capabilities more effectively. M-DPO aligns a VideoLLM with evaluation metrics, providing fine-grained supervision to each task that is well-aligned with metrics. Applied to two different VideoLLMs, VidChain consistently improves their fine-grained video understanding, thereby outperforming previous VideoLLMs on two different DVC benchmarks and also on the temporal video grounding task. Code is available at \url{https://github.com/mlvlab/VidChain}.