Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world

TL;DR

This paper introduces Self-Assessed Generation (SAG), a self-supervised framework that enhances the generalization of optical flow and stereo matching models to real-world data by generating and validating datasets without changing existing methods.

Contribution

SAG is a novel data-driven self-supervised framework that constructs datasets using advanced reconstruction and confidence assessment, improving real-world generalization for optical flow and stereo tasks.

Findings

SAG significantly improves model generalization on real-world datasets.

It enhances the robustness of state-of-the-art networks without modifying their architectures.

SAG is cost-effective and yields more accurate results compared to previous methods.

Abstract

A significant challenge facing current optical flow and stereo methods is the difficulty in generalizing them well to the real world. This is mainly due to the high costs required to produce datasets, and the limitations of existing self-supervised methods on fuzzy results and complex model training problems. To address the above challenges, we propose a unified self-supervised generalization framework for optical flow and stereo tasks: Self-Assessed Generation (SAG). Unlike previous self-supervised methods, SAG is data-driven, using advanced reconstruction techniques to construct a reconstruction field from RGB images and generate datasets based on it. Afterward, we quantified the confidence level of the generated results from multiple perspectives, such as reconstruction field distribution, geometric consistency, and structural similarity, to eliminate inevitable defects in the generation process. We also designed a 3D flight foreground automatic rendering pipeline in SAG to encourage the network to learn occlusion and motion foreground. Experimentally, because SAG does not involve changes to methods or loss functions, it can directly self-supervised train the state-of-the-art deep networks, greatly improving the generalization performance of self-supervised methods on current mainstream optical flow and stereo-matching datasets. Compared to previous training modes, SAG is more generalized, cost-effective, and accurate.