FlowFormer: A Transformer Architecture and Its Masked Cost Volume Autoencoding for Optical Flow

TL;DR

FlowFormer introduces a transformer-based architecture with Masked Cost Volume AutoEncoding for improved optical flow estimation, achieving state-of-the-art results and better generalization on benchmark datasets.

Contribution

The paper presents a novel transformer architecture for optical flow and a pretraining scheme that leverages unlabeled data to enhance performance.

Findings

FlowFormer achieves 1.16 and 2.09 AEPE on Sintel, surpassing previous methods.

MCVA pretraining further improves FlowFormer's accuracy, reducing errors by over 7%.

FlowFormer+MCVA ranks 1st on Sintel and KITTI benchmarks.

Abstract

This paper introduces a novel transformer-based network architecture, FlowFormer, along with the Masked Cost Volume AutoEncoding (MCVA) for pretraining it to tackle the problem of optical flow estimation. FlowFormer tokenizes the 4D cost-volume built from the source-target image pair and iteratively refines flow estimation with a cost-volume encoder-decoder architecture. The cost-volume encoder derives a cost memory with alternate-group transformer~(AGT) layers in a latent space and the decoder recurrently decodes flow from the cost memory with dynamic positional cost queries. On the Sintel benchmark, FlowFormer architecture achieves 1.16 and 2.09 average end-point-error~(AEPE) on the clean and final pass, a 16.5\% and 15.5\% error reduction from the GMA~(1.388 and 2.47). MCVA enhances FlowFormer by pretraining the cost-volume encoder with a masked autoencoding scheme, which further unleashes the capability of FlowFormer with unlabeled data. This is especially critical in optical flow estimation because ground truth flows are more expensive to acquire than labels in other vision tasks. MCVA improves FlowFormer all-sided and FlowFormer+MCVA ranks 1st among all published methods on both Sintel and KITTI-2015 benchmarks and achieves the best generalization performance. Specifically, FlowFormer+MCVA achieves 1.07 and 1.94 AEPE on the Sintel benchmark, leading to 7.76\% and 7.18\% error reductions from FlowFormer.