Efficient Bayer-Domain Video Computer Vision with Fast Motion Estimation and Learned Perception Residual

TL;DR

This paper introduces an efficient Bayer-domain video computer vision framework that reduces computational load by directly processing Bayer raw data, employing fast motion estimation, and using perception residuals to maintain accuracy.

Contribution

It proposes a joint front-end and back-end optimization, removing traditional ISP, implementing fast motion estimation, and learning perception residuals for improved efficiency.

Findings

Achieves significant acceleration with minimal accuracy loss

Eliminates the need for traditional ISP processing

Employs lightweight perception residual networks

Abstract

Video computer vision systems face substantial computational burdens arising from two fundamental challenges: eliminating unnecessary processing and reducing temporal redundancy in back-end inference while maintaining accuracy with minimal extra computation. To address these issues, we propose an efficient video computer vision framework that jointly optimizes both the front end and back end of the pipeline. On the front end, we remove the traditional image signal processor (ISP) and feed Bayer raw measurements directly into Bayer-domain vision models, avoiding costly human-oriented ISP operations. On the back end, we introduce a fast and highly parallel motion estimation algorithm that extracts inter-frame temporal correspondence to avoid redundant computation. To mitigate artifacts caused by motion inaccuracies, we further employ lightweight perception residual networks that directly learn perception-level residuals and refine the propagated features. Experiments across multiple models and tasks demonstrate that our system achieves substantial acceleration with only minor performance degradation.