ST-MFNet Mini: Knowledge Distillation-Driven Frame Interpolation

TL;DR

This paper introduces ST-MFNet Mini, a highly compressed and efficient video frame interpolation model achieved through network pruning and knowledge distillation, maintaining competitive performance with significantly reduced complexity.

Contribution

The paper proposes a novel two-stage workflow combining network pruning and knowledge distillation to create a compact, high-performance VFI model, ST-MFNet Mini.

Findings

91% reduction in model parameters

35% increase in processing speed

Competitive performance with high-complexity models

Abstract

Currently, one of the major challenges in deep learning-based video frame interpolation (VFI) is the large model sizes and high computational complexity associated with many high performance VFI approaches. In this paper, we present a distillation-based two-stage workflow for obtaining compressed VFI models which perform competitively to the state of the arts, at a greatly reduced model size and complexity. Specifically, an optimisation-based network pruning method is first applied to a recently proposed frame interpolation model, ST-MFNet, which outperforms many other VFI methods but suffers from large model size. The resulting new network architecture achieves a 91% reduction in parameters and 35% increase in speed. Secondly, the performance of the new network is further enhanced through a teacher-student knowledge distillation training process using a Laplacian distillation loss. The final low complexity model, ST-MFNet Mini, achieves a comparable performance to most existing high-complex VFI methods, only outperformed by the original ST-MFNet. Our source code is available at https://github.com/crispianm/ST-MFNet-Mini