MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality

TL;DR

MUSE introduces a topological orthogonality framework to decouple and optimize structural and semantic aspects in visual tokenization, overcoming the traditional trade-off between pixel fidelity and conceptual invariance.

Contribution

It proposes a novel decoupling method using topological orthogonality to improve both reconstruction quality and semantic perception in visual tokenization.

Findings

Achieves state-of-the-art generation quality with gFID 3.08.

Surpasses teacher InternViT-300M in linear probing accuracy (85.2%).

Demonstrates that structural alignment enhances semantic perception.

Abstract

Unified visual tokenization faces a fundamental trade-off between high-fidelity pixel reconstruction (spatial equivariance) and semantic abstraction (conceptual invariance). We attribute this conflict to Manifold Misalignment: naive joint optimization induces opposing gradients, creating a zero-sum game between reconstruction and perception. To address this, we propose MUSE, a framework based on Topological Orthogonality. By treating Structure as an orthogonal bridge, MUSE decouples optimization within Transformers: structural gradients refine attention topology, while semantic gradients update feature values. This turns destructive interference into Mutual Reinforcement. Experiments show that MUSE breaks the trade-off, achieving state-of-the-art generation quality (gFID 3.08) and surpassing its teacher InternViT-300M in linear probing (85.2\% vs. 82.5\%), demonstrating that structurally aligned reconstruction can enhance semantic perception. Code is available at https://github.com/PanqiYang1/MUSE.