Asymmetric Hierarchical Anchoring for Audio-Visual Joint Representation: Resolving Information Allocation Ambiguity for Robust Cross-Modal Generalization

TL;DR

This paper introduces Asymmetric Hierarchical Anchoring (AHA), a novel framework for audio-visual joint representation learning that addresses information allocation ambiguity and improves cross-modal generalization by enforcing directional semantic anchoring.

Contribution

The paper proposes AHA, which uses hierarchical semantic anchors and adversarial decoupling to enhance cross-modal transfer and reduce semantic leakage in audio-visual representations.

Findings

AHA outperforms symmetric baselines on AVE and AVVP benchmarks.

The framework improves semantic consistency and disentanglement in learned representations.

AHA demonstrates broader applicability in tasks like talking-face disentanglement.

Abstract

Audio-visual joint representation learning under Cross-Modal Generalization (CMG) aims to transfer knowledge from a labeled source modality to an unlabeled target modality through a unified discrete representation space. Existing symmetric frameworks often suffer from information allocation ambiguity, where the absence of structural inductive bias leads to semantic-specific leakage across modalities. We propose Asymmetric Hierarchical Anchoring (AHA), which enforces directional information allocation by designating a structured semantic anchor within a shared hierarchy. In our instantiation, we exploit the hierarchical discrete representations induced by audio Residual Vector Quantization (RVQ) to guide video feature distillation into a shared semantic space. To ensure representational purity, we replace fragile mutual information estimators with a GRL-based adversarial decoupler that explicitly suppresses semantic leakage in modality-specific branches, and introduce Local Sliding Alignment (LSA) to encourage fine-grained temporal alignment across modalities. Extensive experiments on AVE and AVVP benchmarks demonstrate that AHA consistently outperforms symmetric baselines in cross-modal transfer. Additional analyses on talking-face disentanglement experiment further validate that the learned representations exhibit improved semantic consistency and disentanglement, indicating the broader applicability of the proposed framework.