STaRFlow: A SpatioTemporal Recurrent Cell for Lightweight Multi-Frame Optical Flow Estimation

TL;DR

STaRFlow is a lightweight CNN-based multi-frame optical flow algorithm that uses a novel spatio-temporal recurrent cell to achieve state-of-the-art results with fewer parameters.

Contribution

It introduces a double recurrence mechanism with a spatio-temporal recurrent cell, combining feature-based temporal recurrence and occlusion detection for efficient optical flow estimation.

Findings

Achieves state-of-the-art performance on MPI Sintel and Kitti2015 datasets.

Uses significantly fewer parameters than comparable methods.

Incorporates a novel occlusion detection process with minimal additional complexity.

Abstract

We present a new lightweight CNN-based algorithm for multi-frame optical flow estimation. Our solution introduces a double recurrence over spatial scale and time through repeated use of a generic "STaR" (SpatioTemporal Recurrent) cell. It includes (i) a temporal recurrence based on conveying learned features rather than optical flow estimates; (ii) an occlusion detection process which is coupled with optical flow estimation and therefore uses a very limited number of extra parameters. The resulting STaRFlow algorithm gives state-of-the-art performances on MPI Sintel and Kitti2015 and involves significantly less parameters than all other methods with comparable results.