DreamFrame: Enhancing Video Understanding via Automatically Generated QA and Style-Consistent Keyframes

TL;DR

DreamFrame introduces an automated framework for generating style-consistent keyframes and QA pairs, creating a large dataset to improve video understanding in LVLMs and enabling effective downstream fine-tuning.

Contribution

It presents a novel three-stage method for automatic dataset creation with style consistency, enhancing LVLM instruction tuning without extensive manual annotation.

Findings

DreamFrame dataset contains ~1k stylized videos and 100k QA pairs.

Fine-tuned LVLMs using DreamFrame outperform previous models on benchmarks.

DreamFrame-7B surpasses similar-sized LVLMs in various evaluations.

Abstract

Recent large vision-language models (LVLMs) for video understanding are primarily fine-tuned with various videos scraped from online platforms. Existing datasets, such as ActivityNet, require considerable human labor for structuring and annotation before effectively utilized for tuning LVLMs. While current LVLMs are primarily trained on existing datasets in broad, general-purpose settings, adapting them to specific downstream scenarios remains challenging, as collecting and annotating task-specific videos is highly labor-intensive and time-consuming. To address this issue, we propose a three-stage framework named DreamFrame for automatically generating style-consistent keyframes and corresponding question-answer (QA) pairs to support LVLM instruction tuning. DreamFrame generates datasets in a movie-like manner. First, we utilize an LLM to generate structured movie plots including movie prior information (like overview and style), frame descriptions and plot-related QA pairs, with a story expansion strategy to mitigate context length limitations.Then, to ensure visual consistency across generated frames, we design a Style Immobilization Process which maintains consistent style through an embedding learning strategy. Finally, frame descriptions and style embeddings are integrated to produce coherent keyframes. Using DreamFrame, we construct a dataset comprising approximately 1k stylized keyframe-like videos and 100k diverse QA pairs. Extensive fine-tuned experiments on various LVLM architectures demonstrate the effectiveness of the proposed dataset. Furthermore, based on the proposed dataset, we fine-tune a new LVLM named DreamFrame-7B, which significantly surpasses the previous similar-sized LVLMs across different benchmarks.