Oflib: Facilitating Operations with and on Optical Flow Fields in Python

TL;DR

This paper introduces Oflib, a comprehensive Python library for the mathematical characterization and manipulation of optical flow fields, supporting deep learning applications with differentiable operations.

Contribution

It provides a rigorous theoretical framework and a Python implementation for optical flow operations, including flow composition and inversion, with support for back-propagation in deep learning.

Findings

Verified flow composition empirically

Demonstrated application to synthetic data creation

Supported deep learning with differentiable flow operations

Abstract

We present a robust theoretical framework for the characterisation and manipulation of optical flow, i.e 2D vector fields, in the context of their use in motion estimation algorithms and beyond. The definition of two frames of reference guides the mathematical derivation of flow field application, inversion, evaluation, and composition operations. This structured approach is then used as the foundation for an implementation in Python 3, with the fully differentiable PyTorch version oflibpytorch supporting back-propagation as required for deep learning. We verify the flow composition method empirically and provide a working example for its application to optical flow ground truth in synthetic training data creation. All code is publicly available.