Spatial-Aware Conditioned Fusion for Audio-Visual Navigation

TL;DR

This paper introduces SACF, a novel method for audio-visual navigation that discretizes target position and uses spatial descriptors to improve navigation efficiency and generalization.

Contribution

The paper proposes a new spatial-aware fusion technique that explicitly models target position and enhances feature fusion for better navigation performance.

Findings

SACF achieves higher navigation efficiency.

SACF generalizes well to unseen sounds.

SACF reduces computational overhead.

Abstract

Audio-visual navigation tasks require agents to locate and navigate toward continuously vocalizing targets using only visual observations and acoustic cues. However, existing methods mainly rely on simple feature concatenation or late fusion, and lack an explicit discrete representation of the target's relative position, which limits learning efficiency and generalization. We propose Spatial-Aware Conditioned Fusion (SACF). SACF first discretizes the target's relative direction and distance from audio-visual cues, predicts their distributions, and encodes them as a compact descriptor for policy conditioning and state modeling. Then, SACF uses audio embeddings and spatial descriptors to generate channel-wise scaling and bias to modulate visual features via conditional linear transformation, producing target-oriented fused representations. SACF improves navigation efficiency with lower computational overhead and generalizes well to unheard target sounds.