Neural Fields as World Models

TL;DR

This paper introduces isomorphic neural field models that preserve sensory topology for more accurate physics prediction, demonstrating advantages in transfer learning and emergent body representations.

Contribution

It proposes neural field architectures with motor-gated channels that maintain sensory topology, enabling geometric physics prediction and emergent body encoding.

Findings

Local connectivity suffices for learning ballistic physics

Imagination-trained policies transfer better to real physics

Motor-gated channels develop body-selective encoding

Abstract

How does the brain predict physical outcomes while acting in the world? Machine learning world models compress visual input into latent spaces, discarding the spatial structure that characterizes sensory cortex. We propose isomorphic world models: architectures preserving sensory topology so that physics prediction becomes geometric propagation rather than abstract state transition. We implement this using neural fields with motor-gated channels, where activity evolves through local lateral connectivity and motor commands multiplicatively modulate specific populations. Three experiments support this approach: (1) local connectivity is sufficient to learn ballistic physics, with predictions traversing intermediate locations rather than "teleporting"; (2) policies trained entirely in imagination transfer to real physics at nearly twice the rate of latent-space alternatives; and (3) motor-gated channels spontaneously develop body-selective encoding through visuomotor prediction alone. These findings suggest intuitive physics and body schema may share a common origin in spatially structured neural dynamics.