A Cross-Scale Decoder with Token Refinement for Off-Road Semantic Segmentation

TL;DR

This paper introduces a novel cross-scale decoder with token refinement for off-road semantic segmentation, effectively handling irregular terrains, ambiguous boundaries, and sparse structures with improved robustness and efficiency.

Contribution

It proposes a new decoder architecture with three mechanisms: token refinement, gated detail bridge, and class-aware point refinement, tailored for challenging off-road environments.

Findings

Achieves improved boundary preservation and structural detail recovery.

Demonstrates consistent performance gains over prior methods on benchmarks.

Maintains computational efficiency suitable for deployment.

Abstract

Off-road semantic segmentation is fundamentally challenged by irregular terrain, vegetation clutter, and inherent annotation ambiguity. Unlike urban scenes with crisp object boundaries, off-road environments exhibit strong class-level similarity among terrain categories, resulting in thick and uncertain transition regions that degrade boundary coherence and destabilize training. Rare or thin structures, such as narrow traversable gaps or isolated obstacles, further receive sparse and unreliable supervision and are easily overwhelmed by dominant background textures. Existing decoder designs either rely on low-scale bottlenecks that oversmooth fine structural details, or repeatedly fuse high-detail features, which tends to amplify annotation noise and incur substantial computational cost. We present a cross-scale decoder that explicitly addresses these challenges through three complementary mechanisms. First, a global--local token refinement module consolidates semantic context on a compact bottleneck lattice, guided by boundary-aware regularization to remain robust under ambiguous supervision. Second, a gated detail bridge selectively injects fine-scale structural cues only once through cross-scale attention, preserving boundary and texture information while avoiding noise accumulation. Third, an uncertainty-guided class-aware point refinement selectively updates the least reliable pixels, improving rare and ambiguous structures with minimal computational overhead. The resulting framework achieves noise-robust and boundary-preserving segmentation tailored to off-road environments, recovering fine structural details while maintaining deployment-friendly efficiency. Experimental results on standard off-road benchmarks demonstrate consistent improvements over prior approaches without resorting to heavy dense feature fusion.