Approximation-Generalization Trade-offs under (Approximate) Group Equivariance

TL;DR

This paper provides a formal analysis of how incorporating task-specific symmetries, including approximate and partial symmetries, affects the generalization ability of neural networks, offering bounds and conditions for optimal model-data symmetry alignment.

Contribution

It introduces the most general quantitative bounds linking symmetry incorporation to generalization, applicable to approximate and partial symmetries, and analyzes model-data symmetry mis-specification.

Findings

Models with task-specific symmetries improve generalization.

Bounds are valid for approximate and partial symmetries, not just exact group symmetries.

Conditions for optimal model-data symmetry alignment are established.

Abstract

The explicit incorporation of task-specific inductive biases through symmetry has emerged as a general design precept in the development of high-performance machine learning models. For example, group equivariant neural networks have demonstrated impressive performance across various domains and applications such as protein and drug design. A prevalent intuition about such models is that the integration of relevant symmetry results in enhanced generalization. Moreover, it is posited that when the data and/or the model may only exhibit $\textit{approximate}$ or $\textit{partial}$ symmetry, the optimal or best-performing model is one where the model symmetry aligns with the data symmetry. In this paper, we conduct a formal unified investigation of these intuitions. To begin, we present general quantitative bounds that demonstrate how models capturing task-specific symmetries lead to improved generalization. In fact, our results do not require the transformations to be finite or even form a group and can work with partial or approximate equivariance. Utilizing this quantification, we examine the more general question of model mis-specification i.e. when the model symmetries don't align with the data symmetries. We establish, for a given symmetry group, a quantitative comparison between the approximate/partial equivariance of the model and that of the data distribution, precisely connecting model equivariance error and data equivariance error. Our result delineates conditions under which the model equivariance error is optimal, thereby yielding the best-performing model for the given task and data. Our results are the most general results of their type in the literature.