Any2Any: Unified Arbitrary Modality Translation for Remote Sensing

TL;DR

Any2Any introduces a unified diffusion-based framework for arbitrary modality translation in remote sensing, leveraging a shared latent space to improve generalization and efficiency across multiple sensing modalities.

Contribution

It formulates multi-modal translation as inference over a shared latent space, enabling flexible, zero-shot translation among any modalities with a unified model.

Findings

Outperforms pairwise translation methods on 14 tasks.

Exhibits strong zero-shot generalization to unseen modality pairs.

Introduces RST-1M, a large-scale remote sensing dataset with five modalities.

Abstract

Multi-modal remote sensing imagery provides complementary observations of the same geographic scene, yet such observations are frequently incomplete in practice. Existing cross-modal translation methods treat each modality pair as an independent task, resulting in quadratic complexity and limited generalization to unseen modality combinations. We formulate Any-to-Any translation as inference over a shared latent representation of the scene, where different modalities correspond to partial observations of the same underlying semantics. Based on this formulation, we propose Any2Any, a unified latent diffusion framework that projects heterogeneous inputs into a geometrically aligned latent space. Such structure performs anchored latent regression with a shared backbone, decoupling modality-specific representation learning from semantic mapping. Moreover, lightweight target-specific residual adapters are used to correct systematic latent mismatches without increasing inference complexity. To support learning under sparse but connected supervision, we introduce RST-1M, the first million-scale remote sensing dataset with paired observations across five sensing modalities, providing supervision anchors for any-to-any translation. Experiments across 14 translation tasks show that Any2Any consistently outperforms pairwise translation methods and exhibits strong zero-shot generalization to unseen modality pairs. Code and models will be available at https://github.com/MiliLab/Any2Any.