Generalized Whittaker quotients of Schwartz functions on G-spaces

TL;DR

This paper investigates the existence and properties of generalized Whittaker quotients for Schwartz functions on G-spaces over local fields, extending previous results and providing new bounds and criteria for distinguished representations.

Contribution

It generalizes recent results on Whittaker quotients, characterizes when distinguished representations exist, and extends theorems to Archimedean cases, with applications to automorphic forms.

Findings

The set of nilpotent elements with non-vanishing Whittaker quotients contains the nilpotent part of the moment map image.

Existence of infinite-dimensional H-distinguished representations correlates with the non-compactness of the associated real reductive group.

Sharp bounds on wave-front sets of distinguished representations in the non-Archimedean case.

Abstract

Let $G$ be a reductive group over a local field $F$ of characteristic zero, Archimedean or not. Let $X$ be a $G$-space. In this paper we study the existence of generalized Whittaker quotients for the space of Schwartz functions on $X$, considered as a representation of $G$. We show that the set of nilpotent elements of the dual space to the Lie algebra such that the corresponding generalized Whittaker quotient does not vanish contains the nilpotent part of the image of the moment map, and lies in the closure of this image. This generalizes recent results of Prasad and Sakellaridis. Applying our theorems to symmetric pairs $(G,H)$ we show that there exists an infinite-dimensional $H$-distinguished representation of $G$ if and only if the real reductive group corresponding to the pair $(G,H)$ is non-compact. For quasi-split $G$ we also extend to the Archimedean case the theorem of Prasad stating that there exists a generic $H$-distinguished representation of $G$ if and only if the real reductive group corresponding to the pair $(G,H)$ is quasi-split. In the non-Archimedean case our result also gives rather sharp bounds on the wave-front sets of distinguished representations. The results in the present paper can be used to recover many of the vanishing results on periods of automorphic forms proved by Ash-Ginzburg-Rallis. This follows from our Corollary H when combined with the restrictions on the Whittaker support of cuspidal automorphic representations proven in [GGS21].