Discrete JEPA: Learning Discrete Token Representations without Reconstruction

TL;DR

Discrete-JEPA introduces a novel semantic tokenization framework that enhances symbolic reasoning and systematic inference in AI, outperforming existing methods on visual symbolic prediction tasks and fostering emergent structured patterns.

Contribution

It extends latent predictive coding with semantic tokenization and new objectives, enabling robust symbolic reasoning without reconstruction, a significant advancement over prior tokenization approaches.

Findings

Outperforms baselines on visual symbolic prediction tasks

Emergence of systematic patterns in learned semantic token space

Potential to advance symbolic world modeling and planning in AI

Abstract

The cornerstone of cognitive intelligence lies in extracting hidden patterns from observations and leveraging these principles to systematically predict future outcomes. However, current image tokenization methods demonstrate significant limitations in tasks requiring symbolic abstraction and logical reasoning capabilities essential for systematic inference. To address this challenge, we propose Discrete-JEPA, extending the latent predictive coding framework with semantic tokenization and novel complementary objectives to create robust tokenization for symbolic reasoning tasks. Discrete-JEPA dramatically outperforms baselines on visual symbolic prediction tasks, while striking visual evidence reveals the spontaneous emergence of deliberate systematic patterns within the learned semantic token space. Though an initial model, our approach promises a significant impact for advancing Symbolic world modeling and planning capabilities in artificial intelligence systems.