PerspectiveNet: Multi-View Perception for Dynamic Scene Understanding

TL;DR

PerspectiveNet is a lightweight, multi-view perception model that combines visual encoders, a connector module, and large language models to generate detailed descriptions of dynamic scenes from multiple camera viewpoints.

Contribution

The paper introduces PerspectiveNet, a novel architecture that effectively integrates visual features and LLMs for multi-view scene understanding, with a focus on efficiency and detailed description generation.

Findings

Achieves accurate scene descriptions from multiple camera views.

Efficient training and inference with a lightweight architecture.

Effective in the Traffic Safety Description and Analysis task.

Abstract

Generating detailed descriptions from multiple cameras and viewpoints is challenging due to the complex and inconsistent nature of visual data. In this paper, we introduce PerspectiveNet, a lightweight yet efficient model for generating long descriptions across multiple camera views. Our approach utilizes a vision encoder, a compact connector module to convert visual features into a fixed-size tensor, and large language models (LLMs) to harness the strong natural language generation capabilities of LLMs. The connector module is designed with three main goals: mapping visual features onto LLM embeddings, emphasizing key information needed for description generation, and producing a fixed-size feature matrix. Additionally, we augment our solution with a secondary task, the correct frame sequence detection, enabling the model to search for the correct sequence of frames to generate descriptions. Finally, we integrate the connector module, the secondary task, the LLM, and a visual feature extraction model into a single architecture, which is trained for the Traffic Safety Description and Analysis task. This task requires generating detailed, fine-grained descriptions of events from multiple cameras and viewpoints. The resulting model is lightweight, ensuring efficient training and inference, while remaining highly effective.