CoS: Chain-of-Shot Prompting for Long Video Understanding

TL;DR

This paper introduces Chain-of-Shot prompting (CoS), a method that adaptively selects relevant shots in long videos to improve multi-modal large language model understanding, addressing the challenge of processing lengthy visual content.

Contribution

The paper proposes a novel shot selection framework using test-time prompt optimization, including pseudo temporal grounding and binary coding for task-relevant shot identification.

Findings

CoS improves long video understanding across multiple datasets.

The method effectively balances relevant and irrelevant shot selection.

Experiments show enhanced performance over baseline models.

Abstract

Multi-modal Large Language Models (MLLMs) struggle with long videos due to the need for excessive visual tokens. These tokens exceed massively the context length of MLLMs, resulting in filled by redundant task-irrelevant shots. How to select shots is an unsolved critical problem: sparse sampling risks missing key details, while exhaustive sampling overwhelms the model with irrelevant content, leading to video misunderstanding. To solve this problem, we propose Chain-of-Shot prompting (CoS). The key idea is to frame shot selection as test-time visual prompt optimisation, choosing shots adaptive to video understanding semantic task by optimising shots-task alignment. CoS has two key parts: (1) a binary video summary mechanism that performs pseudo temporal grounding, discovering a binary coding to identify task-relevant shots, and (2) a video co-reasoning module that deploys the binary coding to pair (learning to align) task-relevant positive shots with irrelevant negative shots. It embeds the optimised shot selections into the original video, facilitating a focus on relevant context to optimize long video understanding. Experiments across three baselines and five datasets demonstrate the effectiveness and adaptability of CoS. Code given in https://lwpyh.github.io/CoS.