Learnable Sampling 3D Convolution for Video Enhancement and Action Recognition

TL;DR

This paper introduces LS3D-Conv, a learnable sampling 3D convolution module that adaptively fuses multi-level features across frames, improving robustness and performance in video enhancement and action recognition tasks.

Contribution

The paper proposes a novel LS3D-Conv module with learnable offsets for flexible, task-specific sampling in 3D convolution, enhancing existing networks for video processing.

Findings

Improves performance in video interpolation, super-resolution, denoising, and action recognition.

Demonstrates robustness over traditional correspondence-based methods.

Flexible replacement for standard 3D convolution layers.

Abstract

A key challenge in video enhancement and action recognition is to fuse useful information from neighboring frames. Recent works suggest establishing accurate correspondences between neighboring frames before fusing temporal information. However, the generated results heavily depend on the quality of correspondence estimation. In this paper, we propose a more robust solution: \emph{sampling and fusing multi-level features} across neighborhood frames to generate the results. Based on this idea, we introduce a new module to improve the capability of 3D convolution, namely, learnable sampling 3D convolution (\emph{LS3D-Conv}). We add learnable 2D offsets to 3D convolution which aims to sample locations on spatial feature maps across frames. The offsets can be learned for specific tasks. The \emph{LS3D-Conv} can flexibly replace 3D convolution layers in existing 3D networks and get new architectures, which learns the sampling at multiple feature levels. The experiments on video interpolation, video super-resolution, video denoising, and action recognition demonstrate the effectiveness of our approach.