FlowFeat: Pixel-Dense Embedding of Motion Profiles

TL;DR

FlowFeat introduces a high-resolution, multi-task dense image representation that embeds motion profiles, improving performance across various dense prediction tasks with a self-supervised, computationally efficient approach.

Contribution

We propose FlowFeat, a novel dense feature embedding technique using a self-supervised distillation method to incorporate motion profiles, enhancing dense prediction tasks.

Findings

Significantly improves performance on video object segmentation, depth estimation, and semantic segmentation.

Effective even with unsupervised flow networks and low computational cost.

Provides high spatial detail and temporal consistency in dense image representations.

Abstract

Dense and versatile image representations underpin the success of virtually all computer vision applications. However, state-of-the-art networks, such as transformers, produce low-resolution feature grids, which are suboptimal for dense prediction tasks. To address this limitation, we present FlowFeat, a high-resolution and multi-task feature representation. The key ingredient behind FlowFeat is a novel distillation technique that embeds a distribution of plausible apparent motions, or motion profiles. By leveraging optical flow networks and diverse video data, we develop an effective self-supervised training framework that statistically approximates the apparent motion. With its remarkable level of spatial detail, FlowFeat encodes a compelling degree of geometric and semantic cues while exhibiting high temporal consistency. Empirically, FlowFeat significantly enhances the representational power of five state-of-the-art encoders and alternative upsampling strategies across three dense tasks: video object segmentation, monocular depth estimation and semantic segmentation. Training FlowFeat is computationally inexpensive and robust to inaccurate flow estimation, remaining highly effective even when using unsupervised flow networks. Our work takes a step forward towards reliable and versatile dense image representations.