# Single-frame Regularization for Temporally Stable CNNs

## TL;DR

This paper introduces a novel regularization technique for CNNs that enhances temporal stability in video applications without relying on optical flow or recurrent structures, improving smoothness and generalization.

## Contribution

The authors propose a frame-regularization method that enforces temporal consistency during CNN training, avoiding the need for motion estimation or architectural changes.

## Key findings

- Significant improvement in temporal smoothness of CNN outputs.
- Enhanced generalization performance on small datasets.
- Effective as a fine-tuning strategy without architectural modifications.

## Abstract

Convolutional neural networks (CNNs) can model complicated non-linear relations between images. However, they are notoriously sensitive to small changes in the input. Most CNNs trained to describe image-to-image mappings generate temporally unstable results when applied to video sequences, leading to flickering artifacts and other inconsistencies over time. In order to use CNNs for video material, previous methods have relied on estimating dense frame-to-frame motion information (optical flow) in the training and/or the inference phase, or by exploring recurrent learning structures. We take a different approach to the problem, posing temporal stability as a regularization of the cost function. The regularization is formulated to account for different types of motion that can occur between frames, so that temporally stable CNNs can be trained without the need for video material or expensive motion estimation. The training can be performed as a fine-tuning operation, without architectural modifications of the CNN. Our evaluation shows that the training strategy leads to large improvements in temporal smoothness. Moreover, for small datasets the regularization can help in boosting the generalization performance to a much larger extent than what is possible with na\"ive augmentation strategies.