Sparsity-guided Network Design for Frame Interpolation

TL;DR

This paper introduces a compression-driven, sparsity-guided network design for frame interpolation that significantly reduces model size while maintaining or improving performance, making it suitable for resource-limited systems.

Contribution

The authors propose a novel compression framework using sparsity-inducing optimization for frame interpolation networks, achieving high compression ratios with enhanced visual quality.

Findings

10x compressed AdaCoF performs similarly to the original.

Multi-resolution warping improves visual consistency.

Model outperforms state-of-the-art methods on various datasets.

Abstract

DNN-based frame interpolation, which generates intermediate frames from two consecutive frames, is often dependent on model architectures with a large number of features, preventing their deployment on systems with limited resources, such as mobile devices. We present a compression-driven network design for frame interpolation that leverages model pruning through sparsity-inducing optimization to greatly reduce the model size while attaining higher performance. Concretely, we begin by compressing the recently proposed AdaCoF model and demonstrating that a 10 times compressed AdaCoF performs similarly to its original counterpart, where different strategies for using layerwise sparsity information as a guide are comprehensively investigated under a variety of hyperparameter settings. We then enhance this compressed model by introducing a multi-resolution warping module, which improves visual consistency with multi-level details. As a result, we achieve a considerable performance gain with a quarter of the size of the original AdaCoF. In addition, our model performs favorably against other state-of-the-art approaches on a wide variety of datasets. We note that the suggested compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithms. The source code is available at https://github.com/tding1/CDFI.