Alignment among Language, Vision and Action Representations

TL;DR

This study demonstrates that language, vision, and action representations in AI systems tend to converge towards shared semantic structures, even across different training modalities and data types, indicating potential for cross-domain transfer.

Contribution

It provides empirical evidence of robust cross-modal alignment among language, vision, and action representations in embodied AI models, revealing unexpected convergence.

Findings

Action representations align strongly with language models and BLIP.

Alignment between action and language models approaches that among language models.

Cross-modal alignment suggests shared semantic structures across modalities.

Abstract

A fundamental question in cognitive science and AI concerns whether different learning modalities: language, vision, and action, give rise to distinct or shared internal representations. Traditional views assume that models trained on different data types develop specialized, non-transferable representations. However, recent evidence suggests unexpected convergence: models optimized for distinct tasks may develop similar representational geometries. We investigate whether this convergence extends to embodied action learning by training a transformer-based agent to execute goal-directed behaviors in response to natural language instructions. Using behavioral cloning on the BabyAI platform, we generated action-grounded language embeddings shaped exclusively by sensorimotor control requirements. We then compared these representations with those extracted from state-of-the-art large language models (LLaMA, Qwen, DeepSeek, BERT) and vision-language models (CLIP, BLIP). Despite substantial differences in training data, modality, and objectives, we observed robust cross-modal alignment. Action representations aligned strongly with decoder-only language models and BLIP (precision@15: 0.70-0.73), approaching the alignment observed among language models themselves. Alignment with CLIP and BERT was significantly weaker. These findings indicate that linguistic, visual, and action representations converge toward partially shared semantic structures, supporting modality-independent semantic organization and highlighting potential for cross-domain transfer in embodied AI systems.