Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks

TL;DR

This paper introduces TwInS, a bio-inspired joint learning framework that simultaneously improves scene parsing and geometric vision tasks through interactive, multi-level feature fusion and semi-supervised training, outperforming existing methods.

Contribution

The novel TwInS framework integrates two interactive streams for scene parsing and geometric vision, utilizing a cross-task adapter and semi-supervised learning to enhance performance without relying on costly annotations.

Findings

Outperforms state-of-the-art methods on three datasets.

Effectively leverages multi-view data with semi-supervised training.

Demonstrates the benefit of interactive, multi-level feature fusion.

Abstract

Inspired by the human visual system, which operates on two parallel yet interactive streams for contextual and spatial understanding, this article presents Two Interactive Streams (TwInS), a novel bio-inspired joint learning framework capable of simultaneously performing scene parsing and geometric vision tasks. TwInS adopts a unified, general-purpose architecture in which multi-level contextual features from the scene parsing stream are infused into the geometric vision stream to guide its iterative refinement. In the reverse direction, decoded geometric features are projected into the contextual feature space for selective heterogeneous feature fusion via a novel cross-task adapter, which leverages rich cross-view geometric cues to enhance scene parsing. To eliminate the dependence on costly human-annotated correspondence ground truth, TwInS is further equipped with a tailored semi-supervised training strategy, which unleashes the potential of large-scale multi-view data and enables continuous self-evolution without requiring ground-truth correspondences. Extensive experiments conducted on three public datasets validate the effectiveness of TwInS's core components and demonstrate its superior performance over existing state-of-the-art approaches. The source code will be made publicly available upon publication.