Recurrent Residual Module for Fast Inference in Videos

TL;DR

This paper introduces the Recurrent Residual Module (RRM), a novel framework that significantly accelerates CNN inference on videos by leveraging feature map similarities, achieving up to 500x speedup while maintaining accuracy.

Contribution

The paper presents a new RRM framework that reduces redundant computation in CNNs for video recognition, enabling precise feature map computation and substantial speed improvements.

Findings

Achieves 2x to 12x acceleration on standard CNNs

Attains 500x speedup on binary networks like XNOR-Nets

Maintains recognition performance while significantly speeding up inference

Abstract

Deep convolutional neural networks (CNNs) have made impressive progress in many video recognition tasks such as video pose estimation and video object detection. However, CNN inference on video is computationally expensive due to processing dense frames individually. In this work, we propose a framework called Recurrent Residual Module (RRM) to accelerate the CNN inference for video recognition tasks. This framework has a novel design of using the similarity of the intermediate feature maps of two consecutive frames, to largely reduce the redundant computation. One unique property of the proposed method compared to previous work is that feature maps of each frame are precisely computed. The experiments show that, while maintaining the similar recognition performance, our RRM yields averagely 2x acceleration on the commonly used CNNs such as AlexNet, ResNet, deep compression model (thus 8-12x faster than the original dense models using the efficient inference engine), and impressively 9x acceleration on some binary networks such as XNOR-Nets (thus 500x faster than the original model). We further verify the effectiveness of the RRM on speeding up CNNs for video pose estimation and video object detection.