Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer

TL;DR

This paper introduces FlowLens, a novel transformer-based architecture for online video inpainting that extends camera field-of-view in autonomous vehicles, improving scene perception and safety by reconstructing unseen areas from past video streams.

Contribution

The paper presents FlowLens, a new clip-recurrent transformer architecture with innovative attention and fusion modules for effective beyond-FoV scene reconstruction in autonomous driving.

Findings

FlowLens achieves state-of-the-art performance in beyond-FoV scene reconstruction.

Reconstructed scenes enhance perception accuracy within the camera's original field of view.

The method improves object detection and semantic understanding in extended-view scenarios.

Abstract

Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) FlowLens includes a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time. 2) It integrates a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. To facilitate training and evaluation, we derive the KITTI360 dataset with various FoV mask, which covers both outer- and inner FoV expansion scenarios. We also conduct both quantitative assessments and qualitative comparisons of beyond-FoV semantics and beyond-FoV object detection across different models. We illustrate that employing FlowLens to reconstruct unseen scenes even enhances perception within the field of view by providing reliable semantic context. Extensive experiments and user studies involving offline and online video inpainting, as well as beyond-FoV perception tasks, demonstrate that FlowLens achieves state-of-the-art performance. The source code and dataset are made publicly available at https://github.com/MasterHow/FlowLens.