UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

TL;DR

UnFlow introduces an unsupervised deep learning approach for optical flow estimation that leverages bidirectional flow and census transform to outperform previous methods and even surpass some supervised models on benchmark datasets.

Contribution

It presents a novel unsupervised loss function for optical flow that does not require ground truth, inspired by classical energy-based methods and robust census transform.

Findings

Outperforms previous unsupervised deep networks on KITTI benchmarks.

Achieves accuracy comparable to supervised methods trained on synthetic data.

Enables effective pre-training for supervised optical flow networks.

Abstract

In the era of end-to-end deep learning, many advances in computer vision are driven by large amounts of labeled data. In the optical flow setting, however, obtaining dense per-pixel ground truth for real scenes is difficult and thus such data is rare. Therefore, recent end-to-end convolutional networks for optical flow rely on synthetic datasets for supervision, but the domain mismatch between training and test scenarios continues to be a challenge. Inspired by classical energy-based optical flow methods, we design an unsupervised loss based on occlusion-aware bidirectional flow estimation and the robust census transform to circumvent the need for ground truth flow. On the KITTI benchmarks, our unsupervised approach outperforms previous unsupervised deep networks by a large margin, and is even more accurate than similar supervised methods trained on synthetic datasets alone. By optionally fine-tuning on the KITTI training data, our method achieves competitive optical flow accuracy on the KITTI 2012 and 2015 benchmarks, thus in addition enabling generic pre-training of supervised networks for datasets with limited amounts of ground truth.