Leveraging Motion Estimation for Efficient Bayer-Domain Computer Vision

TL;DR

This paper introduces MEVC, a novel framework that accelerates video vision tasks directly in the Bayer domain by integrating motion estimation into convolutional layers, reducing computation by over 70% with minimal accuracy loss.

Contribution

The paper presents the first effective method to reuse motion estimation for accelerating video vision directly from raw Bayer sensor data, eliminating the need for traditional ISP processing.

Findings

Achieves over 70% reduction in FLOPs across multiple benchmarks.

Supports raw Bayer input for various vision tasks.

Maintains high accuracy with minimal degradation.

Abstract

Existing computer vision processing pipeline acquires visual information using an image sensor that captures pixel information in the Bayer pattern. The raw sensor data are then processed using an image signal processor (ISP) that first converts Bayer pixel data to RGB on a pixel by pixel basis, followed by video convolutional network (VCN) processing on a frame by frame basis. Both ISP and VCN are computationally expensive with high power consumption and latency. In this paper, we propose a novel framework that eliminates the ISP and leverages motion estimation to accelerate video vision tasks directly in the Bayer domain. We introduce Motion Estimation-based Video Convolution (MEVC), which integrates sliding-window motion estimation into each convolutional layer, enabling prediction and residual-based refinement that reduces redundant computations across frames. This design bridges the structural gap between block-based motion estimation and spatial convolution, enabling accurate, low-cost processing. Our end-to-end pipeline supports raw Bayer input and achieves over 70\% reduction in FLOPs with minimal accuracy degradation across video semantic segmentation, depth estimation, and object detection benchmarks, using both synthetic Bayer-converted and real Bayer video datasets. This framework generalizes across convolution-based models and marks the first effective reuse of motion estimation for accelerating video computer vision directly from raw sensor data.