Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning

TL;DR

This paper investigates how neural operators lose geometric information with depth, formalizes the geometric forgetting hypothesis, and proposes a lightweight memory mechanism to mitigate this issue.

Contribution

It introduces the geometric forgetting hypothesis, demonstrates systematic geometric loss in neural operators, and proposes a memory injection method to preserve geometric fidelity.

Findings

Neural operators systematically lose geometric fidelity as depth increases.

The geometric forgetting degrades accuracy, stability, and generalization.

A simple memory injection restores geometric constraints and mitigates forgetting.

Abstract

Neural operators perform well on structured domains, yet their behaviour on irregular geometries remains poorly understood. We show that this limitation is not merely an encoding issue, but a depth-wise failure mode inherent to deep operator architectures. We formalise the Geometric Forgetting Hypothesis: due to the Markovian structure of operator layers and their reliance on global mixing mechanisms, neural operators progressively lose access to domain geometry as depth increases. Using layer-wise geometric probing, we demonstrate that both spectral and attention-based operators systematically lose geometric fidelity. We show that this geometric forgetting degrades accuracy, stability, and generalisation. To counteract it, we introduce a lightweight geometry memory injection mechanism that restores geometric constraints at intermediate depths with minimal architectural overhead. This simple intervention consistently mitigates forgetting and exposes a geometric shortcut instability in transformer-based operators, revealing that geometric retention is a structural requirement rather than a design choice.