Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry

TL;DR

This paper demonstrates that enforcing approximate symmetry in machine learning models is exponentially easier than enforcing exact symmetry, providing a theoretical foundation for the practical preference of approximate symmetry.

Contribution

The paper introduces averaging complexity to quantify symmetry enforcement costs and proves an exponential separation between exact and approximate symmetry enforcement.

Findings

Exact symmetry requires linear averaging complexity.

Approximate symmetry can be achieved with logarithmic complexity.

This is the first theoretical separation between exact and approximate symmetry enforcement.

Abstract

Enforcing exact symmetry in machine learning models often yields significant gains in scientific applications, serving as a powerful inductive bias. However, recent work suggests that relying on approximate symmetry can offer greater flexibility and robustness. Despite promising empirical evidence, there has been little theoretical understanding, and in particular, a direct comparison between exact and approximate symmetry is missing from the literature. In this paper, we initiate this study by asking: What is the cost of enforcing exact versus approximate symmetry? To address this question, we introduce averaging complexity, a framework for quantifying the cost of enforcing symmetry via averaging. Our main result is an exponential separation: under standard conditions, exact symmetry requires linear averaging complexity, whereas approximate symmetry can be attained with only logarithmic complexity in the group size. To the best of our knowledge, this provides the first theoretical separation of these two cases, formally justifying why approximate symmetry may be preferable in practice. Beyond this, our tools and techniques may be of independent interest for the broader study of symmetries in machine learning.