Local Attention Transformers for High-Detail Optical Flow Upsampling

TL;DR

This paper introduces local attention-based convex upsampling methods for optical flow, addressing limitations of traditional convex upsampling, and demonstrates improved accuracy on standard benchmarks.

Contribution

It proposes a novel attention-based convex upsampler with larger masks and an alternative training scheme, enhancing optical flow upsampling accuracy.

Findings

Reduced Sintel end-point-error for RAFT, GMA, and FlowFormer models.

Demonstrated effectiveness of local attention masks in convex upsampling.

Improved optical flow accuracy by solely modifying the upsampling component.

Abstract

Most recent works on optical flow use convex upsampling as the last step to obtain high-resolution flow. In this work, we show and discuss several issues and limitations of this currently widely adopted convex upsampling approach. We propose a series of changes, in an attempt to resolve current issues. First, we propose to decouple the weights for the final convex upsampler, making it easier to find the correct convex combination. For the same reason, we also provide extra contextual features to the convex upsampler. Then, we increase the convex mask size by using an attention-based alternative convex upsampler; Transformers for Convex Upsampling. This upsampler is based on the observation that convex upsampling can be reformulated as attention, and we propose to use local attention masks as a drop-in replacement for convex masks to increase the mask size. We provide empirical evidence that a larger mask size increases the likelihood of the existence of the convex combination. Lastly, we propose an alternative training scheme to remove bilinear interpolation artifacts from the model output. Our proposed ideas could theoretically be applied to almost every current state-of-the-art optical flow architecture. On the FlyingChairs + FlyingThings3D training setting we reduce the Sintel Clean training end-point-error of RAFT from 1.42 to 1.26, GMA from 1.31 to 1.18, and that of FlowFormer from 0.94 to 0.90, by solely adapting the convex upsampler.