On Transferring Transferability: Towards a Theory for Size Generalization

TL;DR

This paper develops a theoretical framework for understanding and improving the transferability of models across different input sizes, especially in graph neural networks, by linking transferability to continuity in a limit space.

Contribution

It introduces a general framework for transferability across dimensions, connecting it to continuity in a limit space, and provides design principles for creating models with better size generalization.

Findings

Transferability is equivalent to continuity in a limit space.

Existing architectures can be adapted for transferability using the proposed framework.

Numerical experiments validate the theoretical insights.

Abstract

Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs are graphs, sets, and point clouds. Recent work on graph neural networks has explored whether a model trained on low-dimensional data can transfer its performance to higher-dimensional inputs. We extend this body of work by introducing a general framework for transferability across dimensions. We show that transferability corresponds precisely to continuity in a limit space formed by identifying small problem instances with equivalent large ones. This identification is driven by the data and the learning task. We instantiate our framework on existing architectures, and implement the necessary changes to ensure their transferability. Finally, we provide design principles for designing new transferable models. Numerical experiments support our findings.