MotionDeltaCNN: Sparse CNN Inference of Frame Differences in Moving Camera Videos

TL;DR

MotionDeltaCNN is a novel sparse CNN inference framework that efficiently processes moving camera videos by fusing new and old regions without increasing memory, outperforming previous methods significantly.

Contribution

It introduces spherical buffers and padded convolutions to enable efficient fusion of image regions in moving camera videos, extending DeltaCNN's capabilities.

Findings

Outperforms DeltaCNN by up to 90% on moving camera videos

Supports seamless fusion of new and old regions without extra memory overhead

Effectively handles camera motion without knowing extrinsics

Abstract

Convolutional neural network inference on video input is computationally expensive and requires high memory bandwidth. Recently, DeltaCNN managed to reduce the cost by only processing pixels with significant updates over the previous frame. However, DeltaCNN relies on static camera input. Moving cameras add new challenges in how to fuse newly unveiled image regions with already processed regions efficiently to minimize the update rate - without increasing memory overhead and without knowing the camera extrinsics of future frames. In this work, we propose MotionDeltaCNN, a sparse CNN inference framework that supports moving cameras. We introduce spherical buffers and padded convolutions to enable seamless fusion of newly unveiled regions and previously processed regions -- without increasing memory footprint. Our evaluation shows that we outperform DeltaCNN by up to 90% for moving camera videos.