Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance

TL;DR

This paper introduces an efficient multi-task RGB-D scene understanding model that enhances feature fusion, employs adaptive loss functions, and achieves superior accuracy and speed across multiple datasets.

Contribution

It proposes a novel multi-task adaptive learning framework with enhanced fusion and feature interaction modules for improved RGB-D scene understanding.

Findings

Outperforms existing methods in accuracy and speed on NYUv2, SUN RGB-D, and Cityscapes datasets.

Effectively leverages RGB and depth data through an enhanced fusion encoder.

Demonstrates superior segmentation and scene classification performance.

Abstract

Scene understanding plays a critical role in enabling intelligence and autonomy in robotic systems. Traditional approaches often face challenges, including occlusions, ambiguous boundaries, and the inability to adapt attention based on task-specific requirements and sample variations. To address these limitations, this paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification. The proposed model incorporates an enhanced fusion encoder, which effectively leverages redundant information from both RGB and depth inputs. For semantic segmentation, we introduce normalized focus channel layers and a context feature interaction layer, designed to mitigate issues such as shallow feature misguidance and insufficient local-global feature representation. The instance segmentation task benefits from a non-bottleneck 1D structure, which achieves superior contour representation with fewer parameters. Additionally, we propose a multi-task adaptive loss function that dynamically adjusts the learning strategy for different tasks based on scene variations. Extensive experiments on the NYUv2, SUN RGB-D, and Cityscapes datasets demonstrate that our approach outperforms existing methods in both segmentation accuracy and processing speed.