Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design

TL;DR

This paper introduces Dy-DCA, a dynamic neural network framework with content-aware data processing and compilation optimizations for efficient on-device video super-resolution, achieving real-time performance and reduced resource usage.

Contribution

The paper presents Dy-DCA, a novel dynamic neural network approach with compilation techniques that significantly reduce model complexity and resource consumption for on-device super-resolution.

Findings

Achieves 33 FPS on mobile devices.

Reduces memory usage by up to 1.61 times.

Improves PSNR compared to baseline models.

Abstract

Deep neural networks (DNNs) are frequently employed in a variety of computer vision applications. Nowadays, an emerging trend in the current video distribution system is to take advantage of DNN's overfitting properties to perform video resolution upscaling. By splitting videos into chunks and applying a super-resolution (SR) model to overfit each chunk, this scheme of SR models plus video chunks is able to replace traditional video transmission to enhance video quality and transmission efficiency. However, many models and chunks are needed to guarantee high performance, which leads to tremendous overhead on model switching and memory footprints at the user end. To resolve such problems, we propose a Dynamic Deep neural network assisted by a Content-Aware data processing pipeline to reduce the model number down to one (Dy-DCA), which helps promote performance while conserving computational resources. Additionally, to achieve real acceleration on the user end, we designed a framework that optimizes dynamic features (e.g., dynamic shapes, sizes, and control flow) in Dy-DCA to enable a series of compilation optimizations, including fused code generation, static execution planning, etc. By employing such techniques, our method achieves better PSNR and real-time performance (33 FPS) on an off-the-shelf mobile phone. Meanwhile, assisted by our compilation optimization, we achieve a 1.7$\times$ speedup while saving up to 1.61$\times$ memory consumption. Code available in https://github.com/coulsonlee/Dy-DCA-ECCV2024.