Complementing Brightness Constancy with Deep Networks for Optical Flow Prediction

TL;DR

This paper introduces COMBO, a deep network for optical flow that combines brightness constancy with data-driven learning, improving accuracy and simplifying training.

Contribution

The novel COMBO network explicitly integrates brightness constancy with deep learning, including uncertainty quantification and a joint training scheme for supervised and semi-supervised learning.

Findings

Outperforms state-of-the-art supervised networks like RAFT

Achieves top results on multiple benchmarks

Simplifies training with semi-supervised approach

Abstract

State-of-the-art methods for optical flow estimation rely on deep learning, which require complex sequential training schemes to reach optimal performances on real-world data. In this work, we introduce the COMBO deep network that explicitly exploits the brightness constancy (BC) model used in traditional methods. Since BC is an approximate physical model violated in several situations, we propose to train a physically-constrained network complemented with a data-driven network. We introduce a unique and meaningful flow decomposition between the physical prior and the data-driven complement, including an uncertainty quantification of the BC model. We derive a joint training scheme for learning the different components of the decomposition ensuring an optimal cooperation, in a supervised but also in a semi-supervised context. Experiments show that COMBO can improve performances over state-of-the-art supervised networks, e.g. RAFT, reaching state-of-the-art results on several benchmarks. We highlight how COMBO can leverage the BC model and adapt to its limitations. Finally, we show that our semi-supervised method can significantly simplify the training procedure.