Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties

TL;DR

This paper introduces uncertainty-based confidence maps and tailored loss functions to improve optical flow and stereo depth estimation by focusing on challenging pixels and regions, including occlusions.

Contribution

It proposes the Difficulty Balancing and Occlusion Avoiding losses, which adapt training to pixel difficulty and occlusion challenges, enhancing model performance.

Findings

Significant performance improvements on optical flow tasks.

Effective handling of occlusions and difficult regions.

Enhanced focus on challenging pixels during training.

Abstract

Conventional training for optical flow and stereo depth models typically employs a uniform loss function across all pixels. However, this one-size-fits-all approach often overlooks the significant variations in learning difficulty among individual pixels and contextual regions. This paper investigates the uncertainty-based confidence maps which capture these spatially varying learning difficulties and introduces tailored solutions to address them. We first present the Difficulty Balancing (DB) loss, which utilizes an error-based confidence measure to encourage the network to focus more on challenging pixels and regions. Moreover, we identify that some difficult pixels and regions are affected by occlusions, resulting from the inherently ill-posed matching problem in the absence of real correspondences. To address this, we propose the Occlusion Avoiding (OA) loss, designed to guide the network into cycle consistency-based confident regions, where feature matching is more reliable. By combining the DB and OA losses, we effectively manage various types of challenging pixels and regions during training. Experiments on both optical flow and stereo depth tasks consistently demonstrate significant performance improvements when applying our proposed combination of the DB and OA losses.