On the spectral theory of groups of affine transformations of compact nilmanifolds

TL;DR

This paper characterizes when subgroups of affine transformations on compact nilmanifolds have spectral gaps, linking spectral properties to invariant subtori and using decay of matrix coefficients to relate spectral gaps with strong ergodicity.

Contribution

It provides a complete characterization of spectral gaps for subgroup actions on nilmanifolds, connecting spectral theory, ergodic properties, and the structure of invariant subtori.

Findings

Spectral gap exists iff no proper invariant subtorus with abelian quotient.

Spectral gap is equivalent to strong ergodicity of the action.

Ergodicity and strong mixing are characterized by actions on the maximal torus.

Abstract

Let $N$ be a connected and simply connected nilpotent Lie group, $\Lambda$ a lattice in $N$, and $X=N/\Lambda$ the corresponding nilmanifold. Let $Aff(X)$ be the group of affine transformations of $X$. We characterize the countable subgroups $H$ of $Aff(X)$ for which the action of $H$ on $X$ has a spectral gap, that is, such that the associated unitary representation $U$ of $H$ on the space of functions from $L^2(X)$ with zero mean does not weakly contain the trivial representation. Denote by $T$ the maximal torus factor associated to $X$. We show that the action of $H$ on $X$ has a spectral gap if and only if there exists no proper $H$-invariant subtorus $S$ of $T$ such that the projection of $H$ on $Aut (T/S)$ has an abelian subgroup of finite index. We first establish the result in the case where $X$ is a torus. In the case of a general nilmanifold, we study the asymptotic behaviour of matrix coefficients of $U$ using decay properties of metaplectic representations of symplectic groups. The result shows that the existence of a spectral gap for subgroups of $Aff(X)$ is equivalent to strong ergodicity in the sense of K.Schmidt. Moreover, we show that the action of $H$ on $X$ is ergodic (or strongly mixing) if and only if the corresponding action of $H$ on $T$ is ergodic (or strongly mixing).