Shotluck Holmes: A Family of Efficient Small-Scale Large Language Vision Models For Video Captioning and Summarization

TL;DR

Shotluck Holmes introduces a family of efficient small-scale large language vision models that significantly improve video captioning and summarization by understanding shot-by-shot semantic information with less computational cost.

Contribution

The paper presents a novel family of small, efficient LLVMs that extend visual understanding from images to videos, enhancing captioning and summarization capabilities.

Findings

Outperforms state-of-the-art on Shot2Story task

Uses less computational resources than larger models

Achieves better accuracy in video understanding

Abstract

Video is an increasingly prominent and information-dense medium, yet it poses substantial challenges for language models. A typical video consists of a sequence of shorter segments, or shots, that collectively form a coherent narrative. Each shot is analogous to a word in a sentence where multiple data streams of information (such as visual and auditory data) must be processed simultaneously. Comprehension of the entire video requires not only understanding the visual-audio information of each shot but also requires that the model links the ideas between each shot to generate a larger, all-encompassing story. Despite significant progress in the field, current works often overlook videos' more granular shot-by-shot semantic information. In this project, we propose a family of efficient large language vision models (LLVMs) to boost video summarization and captioning called Shotluck Holmes. By leveraging better pretraining and data collection strategies, we extend the abilities of existing small LLVMs from being able to understand a picture to being able to understand a sequence of frames. Specifically, we show that Shotluck Holmes achieves better performance than state-of-the-art results on the Shot2Story video captioning and summary task with significantly smaller and more computationally efficient models.