Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalization

TL;DR

This paper investigates the limitations and mechanisms of low-dimension-to-high-dimension generalization, revealing the necessity of prior knowledge and inductive bias, and proposes new position embedding methods to improve length generalization.

Contribution

It provides a theoretical analysis of LDHD generalization, links it to length generalization, and introduces RPE-Square, a novel position embedding to enhance model generalization capabilities.

Findings

LDHD generalization is generally unattainable without prior knowledge.

Different architectures converge to min-degree interpolators based on independent sets.

The proposed RPE-Square improves position encoding for better generalization.

Abstract

Low-Dimension-to-High-Dimension (LDHD) generalization is a special case of Out-of-Distribution (OOD) generalization, where the training data are restricted to a low-dimensional subspace of the high-dimensional testing space. Assuming that each instance is generated from a latent variable and the dimension of the latent variable reflects the problem scale, the inherent scaling challenge in length generalization can be captured by the LDHD generalization in the latent space. We theoretically demonstrate that LDHD generalization is generally unattainable without exploiting prior knowledge to provide appropriate inductive bias. Specifically, we explore LDHD generalization in Boolean functions. We verify that different architectures trained with (S)GD converge to \emph{min-degree interpolators w.r.t. different independent sets}. LDHD generalization is achievable if and only if the target function coincides with this inductive bias. Applying the insights from LDHD generalization to length generalization, we explain the effectiveness of CoT as changing the structure latent space to enable better LDHD generalization. We also propose a principle for position embedding design to handle both the inherent LDHD generalization and the nuisances such as the data format. Following the principle, we propose a novel position embedding called RPE-Square that remedies the RPE for dealing with the data format nuisance.