# Automatic Generation of Dense Non-rigid Optical Flow

## TL;DR

This paper introduces an automated framework for generating dense non-rigid optical flow from real-world videos, overcoming the lack of large-scale datasets and improving training outcomes for optical flow estimation models.

## Contribution

The authors present a novel automatic method to generate dense optical flow for non-rigid motion from real videos, eliminating manual annotation and synthetic data limitations.

## Key findings

- Training with generated optical flow improves model performance.
- The method outperforms models trained on synthetic rigid data.
- Datasets and code are publicly available.

## Abstract

There hardly exists any large-scale datasets with dense optical flow of non-rigid motion from real-world imagery as of today. The reason lies mainly in the required setup to derive ground truth optical flows: a series of images with known camera poses along its trajectory, and an accurate 3D model from a textured scene. Human annotation is not only too tedious for large databases, it can simply hardly contribute to accurate optical flow. To circumvent the need for manual annotation, we propose a framework to automatically generate optical flow from real-world videos. The method extracts and matches objects from video frames to compute initial constraints, and applies a deformation over the objects of interest to obtain dense optical flow fields. We propose several ways to augment the optical flow variations. Extensive experimental results show that training on our automatically generated optical flow outperforms methods that are trained on rigid synthetic data using FlowNet-S, LiteFlowNet, PWC-Net, and RAFT. Datasets and implementation of our optical flow generation framework are released at https://github.com/lhoangan/arap_flow