Pegasus-v1 Technical Report

TL;DR

Pegasus-1 is a multimodal language model tailored for understanding and interacting with video content, addressing challenges like spatiotemporal interpretation to enhance video comprehension tasks.

Contribution

This report introduces Pegasus-1, a novel multimodal model specifically designed for video content understanding and interaction, with detailed architecture and training strategies.

Findings

State-of-the-art performance on video conversation benchmarks

Effective zero-shot video question answering capabilities

Strong results in video summarization tasks

Abstract

This technical report introduces Pegasus-1, a multimodal language model specialized in video content understanding and interaction through natural language. Pegasus-1 is designed to address the unique challenges posed by video data, such as interpreting spatiotemporal information, to offer nuanced video content comprehension across various lengths. This technical report overviews Pegasus-1's architecture, training strategies, and its performance in benchmarks on video conversation, zero-shot video question answering, and video summarization. We also explore qualitative characteristics of Pegasus-1 , demonstrating its capabilities as well as its limitations, in order to provide readers a balanced view of its current state and its future direction.