The Devil in the Details: Simple and Effective Optical Flow Synthetic Data Generation

TL;DR

This paper introduces a simple, effective synthetic data generation method for optical flow that improves model performance on real benchmarks by focusing on essential dataset characteristics and utilizing occlusion masks.

Contribution

A novel synthetic data generation approach using elementary operations and occlusion masks, enhancing optical flow model generalization to real-world data.

Findings

Outperforms original RAFT on MPI Sintel and KITTI 2015 benchmarks

Simpler synthetic datasets can achieve competitive realism

Occlusion mask utilization improves training effectiveness

Abstract

Recent work on dense optical flow has shown significant progress, primarily in a supervised learning manner requiring a large amount of labeled data. Due to the expensiveness of obtaining large scale real-world data, computer graphics are typically leveraged for constructing datasets. However, there is a common belief that synthetic-to-real domain gaps limit generalization to real scenes. In this paper, we show that the required characteristics in an optical flow dataset are rather simple and present a simpler synthetic data generation method that achieves a certain level of realism with compositions of elementary operations. With 2D motion-based datasets, we systematically analyze the simplest yet critical factors for generating synthetic datasets. Furthermore, we propose a novel method of utilizing occlusion masks in a supervised method and observe that suppressing gradients on occluded regions serves as a powerful initial state in the curriculum learning sense. The RAFT network initially trained on our dataset outperforms the original RAFT on the two most challenging online benchmarks, MPI Sintel and KITTI 2015.