Characterisation of the poles of the $\ell$-modular Asai $L$-factor

TL;DR

This paper characterizes when the $ ext{L}$-factor of $ ext{ell}$-modular cuspidal representations of $ ext{GL}(n,E)$ has a pole at 1, linking it to a new notion of relative banality and providing explicit computations and criteria.

Contribution

It introduces the concept of relatively banal distinguished representations for $ ext{ell}$-modular cuspidal representations and characterizes poles of the Asai $ ext{L}$-factor in this context.

Findings

Pole at X=1 occurs iff the representation is relatively banal distinguished.

Computed Asai L-factors for all $ ext{ell}$-modular cuspidal representations.

Established criteria for non-vanishing of the $ ext{GL}(n,F)$-period on Whittaker models.

Abstract

Let $E/F$ be a quadratic extension of non-archimedean local fields, and let $\ell$ be a prime number different from the residual characteristic of $F$. For a complex cuspidal representation $\pi$ of $GL(n,E)$, the Asai $L$-factor $L^+(X,\pi)$ has a pole at $X=1$ if and only if $\pi$ is $GL(n,F)$-distinguished. In this paper we solve the problem of characterising the occurrence of a pole at $X=1$ of $L^+(X,\pi)$ when $\pi$ is an $\ell$-modular cuspidal representation of $GL(n,E)$: we show that $L^+(X,\pi)$ has a pole at $X=1$ if and only if $\pi$ is a relatively banal distinguished representation; namely $\pi$ is $GL(n,F)$-distinguished but not $\vert\det(~ )|_{F}$-distinguished. This notion turns out to be an exact analogue for the symmetric space $GL(n,E)/GL(n,F)$ of M\' inguez and S\'echerre's notion of banal cuspidal $\overline{\mathbb{F}}_\ell$-representation of $GL(n,F)$. Along the way we compute the Asai $L$-factor of all cuspidal $\ell$-modular representations of $GL(n,E)$ in terms of type theory, and prove new results concerning lifting and reduction modulo $\ell$ of distinguished cuspidal representations. Finally, we determine when the natural $GL(n,F)$-period on the Whittaker model of a distinguished cuspidal representation of $GL(n,E)$ is nonzero.

Equations

$${\mathcal{L}}_{\pi }:\mathrm{W}\mapsto {\int }_{{\mathrm{Z}}_{\mathsf{o}}{\mathrm{N}}_{\mathsf{o}}\backslash {\mathrm{G}}_{\mathsf{o}}}\mathrm{W}(h)dh.$$

$$\dim ({\mathrm{Hom}}_{{\mathrm{GL}}_n({\mathrm{F}}_{\mathsf{o}})}(\pi ,\mathrm{R}))⩽1,$$

$$v\mapsto (g\mapsto L(\pi (g)v))$$

$$\{\mathrm{k}\text{-regular characters of }{\mathrm{l}}^{\times }\to {\overline{{\mathbb{Q}}_{\ell }}}^{\times }\}$$

$$\theta$$

$$\{\text{supercuspidal Qℓ-representations of }\mathcal{G}\}/\simeq$$

$$\rho (\theta )$$

$$i:{\Gamma }_{\mathsf{o}}\hookrightarrow \Gamma ,i:\gamma \mapsto \gamma \circ {\mathrm{N}}_{\mathrm{l}/{\mathrm{l}}_{\mathsf{o}}},$$

$${\Gamma }^{+}=\{\gamma \in \Gamma :{\gamma }^{\tilde{\sigma }}=\gamma \}.$$

$${\Gamma }^{-}$$

$$\{\rho ({\theta }_r\mu ):\mu \in {\Gamma }_s,{\theta }_r\mu \mathrm{k}-\text{regular}\}/\simeq .$$

$$\{\rho ({\theta }_r\mu ):\mu \in {\Gamma }_s\cap {\Gamma }^{-},{\theta }_r\mu \mathrm{k}-\text{regular}\}/\simeq$$

$$i^{\prime }:\mathrm{Hom}(\mathrm{Gal}({\mathrm{l}}_{\mathsf{o}}^{\prime }/\mathrm{k}),{\overline{{\mathbb{Q}}_{\ell }}}^{\times })\hookrightarrow \Gamma ,i^{\prime }:\gamma \mapsto \gamma \circ {\mathrm{N}}_{\mathrm{l}/{\mathrm{l}}_{\mathsf{o}}^{\prime }},$$

$${\Gamma }_{+}=\{\gamma \in \Gamma :{\gamma }^{{\sigma }^{\prime }}=\gamma \}.$$

$${\Gamma }_{-}=\{\gamma \in \Gamma :{\gamma }^{{\sigma }^{\prime }}={\gamma }^{-1}\},$$

$$\{\rho ({\theta }_r\mu ):\mu \in {\Gamma }_s\cap {\Gamma }_{-},{\theta }_r\mu \text{ regular}\}/\simeq$$

$$\pi \simeq {\mathrm{ind}}_{\mathbf{J}}^{\mathrm{G}}(\mathbf{\lambda }).$$

$$d(\pi )=d(r_{\ell }(\pi )),e(\pi )=e(r_{\ell }(\pi )),f(\pi )=f(r_{\ell }(\pi )),m(\pi )=m(r_{\ell }(\pi )).$$

$$e_{\sigma }(\pi )=e(\mathrm{E}/{\mathrm{E}}_{\mathsf{o}}).$$

$$e_{\mathsf{o}}(\pi )=\{\begin{array}{ll}2e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})& \text{if }{e}_{\sigma }(\pi )=2\text{ and }m(\pi )\ne 1;\\ e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})& \text{otherwise.}\end{array}$$

$$e_{\sigma }(\pi )=e_{\sigma }(r_{\ell }(\pi )),e_{\mathsf{o}}(\pi )=e_{\mathsf{o}}(r_{\ell }(\pi )).$$

$${\mathrm{Hom}}_{({\mathrm{J}}^1{)}^{\sigma }}(\mathbf{\kappa },\mathrm{R})={\mathrm{Hom}}_{{\mathbf{J}}^{\sigma }}(\mathbf{\kappa },{\chi }_{\mathbf{\kappa }}),$$

$${\mathrm{Hom}}_{{\mathbf{J}}^{\sigma }}(\mathbf{\kappa },{\chi }_{\mathbf{\kappa }})\otimes {\mathrm{Hom}}_{{\mathbf{J}}^{\sigma }}(\mathbf{\tau },{\chi }_{\mathbf{\kappa }}^{-1})\to {\mathrm{Hom}}_{{\mathbf{J}}^{\sigma }}(\mathbf{\lambda },\mathrm{R})$$

$${\mathrm{Hom}}_{\mathrm{N}\cap \mathbf{J}}(\mathbf{\lambda },\psi )\ne \{0\}.$$

$${\mathrm{X}}_{\mathsf{o}}(\pi )=\{{\mu }_{\mathsf{o}}\in \mathrm{Hom}({\mathrm{G}}_{\mathsf{o}},{\mathrm{R}}^{\times }):{\mu }_{\mathsf{o}}\text{is unramified},{\mathrm{Hom}}_{{\mathrm{G}}_{\mathsf{o}}}(\pi ,{\mu }_{\mathsf{o}})\ne \{0\}\}.$$

$${\chi }_{\mathsf{o}}(\det (w^{\prime }))=\chi (\det ({\varpi }_E))=\chi ({\mathrm{N}}_{\mathrm{E}/\mathrm{F}}({\varpi }_{\mathrm{E}}){)}^{m(\pi )}=\chi (\varpi {)}^{f(\mathrm{E}/F)m(\pi )}=\chi (\varpi {)}^{n/e(\mathrm{E}/\mathrm{F})}$$

$$=\chi (\varpi {)}^{ne(\mathrm{F}/{\mathrm{F}}_{\mathsf{o}})/e_{\sigma }(\pi )e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})}=\chi (\varpi {)}^{n/e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})}=\chi ({\varpi }_{\mathsf{o}}{)}^{n/2e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})}=\chi ({\varpi }_{\mathsf{o}}{)}^{n/e_{\mathsf{o}}(\pi )}$$

$${\chi }_{\mathsf{o}}(\det (w^{\prime }))={\chi }_{\mathsf{o}}({\mathrm{N}}_{\mathrm{E}/\mathrm{F}}({\varpi }_{E_{\mathsf{o}}}){)}^{m(\pi )}={\chi }_{\mathsf{o}}({\mathrm{N}}_{{\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}}}({\varpi }_{E_{\mathsf{o}}}){)}^{m(\pi )}={\chi }_{\mathsf{o}}({\varpi }_{\mathsf{o}}{)}^{f({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})m(\pi )}$$

$$={\chi }_{\mathsf{o}}({\varpi }_{\mathsf{o}}{)}^{n/e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})}=\chi ({\varpi }_{\mathsf{o}}{)}^{n/e_{\mathsf{o}}(\pi )}$$

$$r_{\ell }:{\mu }_{\mathsf{o}}\mapsto r_{\ell }({\mu }_{\mathsf{o}})$$

$$e_{\mathsf{o}}(\pi )=e({\mathrm{E}}_{\mathsf{o}}/{\mathrm{F}}_{\mathsf{o}})=e(\mathrm{E}/{\mathrm{F}}_{\mathsf{o}})/e_{\sigma }(\pi )=e(\mathrm{E}/{\mathrm{F}}_{\mathsf{o}})=e(\mathrm{E}/\mathrm{F})e(\mathrm{F}/{\mathrm{F}}_{\mathsf{o}})=e(\pi )e(\mathrm{F}/{\mathrm{F}}_{\mathsf{o}}).$$

Full text

1. Introduction

Let ${\rm F}/{\rm F}_{\mathsf{o}}$ be a quadratic extension of non-archimedean local fields of residual characteristic $p\neq 2$, and set ${\rm G}=\operatorname{GL}_{n}({\rm F})$ and ${\rm G}_{\mathsf{o}}=\operatorname{GL}_{n}({\rm F}_{\mathsf{o}})$. An irreducible representation of ${\rm G}$ is said to be distinguished by ${\rm G}_{\mathsf{o}}$ if it possesses a nonzero ${\rm G}_{\mathsf{o}}$-invariant linear form. In the case of complex representations, the equality of the Asai ${\rm L}$-factor defined by the Rankin–Selberg method and its Galois avatar ([2], [21]) provides a bridge between functorial lifting from the quasi-split unitary group $\mathrm{U}_{n}({\rm F}/{\rm F}_{\mathsf{o}})$ and ${\rm G}_{\mathsf{o}}$-distinction of discrete series representations of ${\rm G}$: a discrete series of ${\rm G}$ is a (stable or unstable depending on the parity of $n$) lift of a (necessarily discrete series) representation of $\mathrm{U}_{n}({\rm F}/{\rm F}_{\mathsf{o}})$ if and only if the Asai ${\rm L}$-factor of its Galois parameter has a pole at ${\rm X}=1$ ([25], [10]), whereas it is ${\rm G}_{\mathsf{o}}$-distinguished if and only if its Asai ${\rm L}$-factor obtained by the Rankin-Selberg method has a pole at ${\rm X}=1$ ([14], [1]).

Recently, motivated by the study of congruences between automorphic representations, there has been great interest in studying representations of ${\rm G}$ on vector spaces over fields of positive characteristic $\ell$. There are two very different cases, when $\ell=p$ and when $\ell\neq p$. This article focuses on the latter $\ell\neq p$ case, where there is a theory of Haar measure which allows us to define Asai ${\rm L}$-factors via the Rankin–Selberg method as in the complex case (Section 7).

The aim of this article is to show that in this case a connection remains between the poles of Asai ${\rm L}$-factor and distinction, however this connection no longer characterises distinction, but a more subtle notion which we call relatively banal distinction. The easiest way to state that a cuspidal distinguished $\ell$-modular representation is relatively banal is to say that it is not $|\mathrm{det}(\ \cdot\ )|_{{\rm F}_{\mathsf{o}}}$-distinguished, where $|\mathrm{det}(\ \cdot\ )|_{{\rm F}_{\mathsf{o}}}$ is considered as an $\overline{\mathbb{F}_{\ell}}$-valued character, but other compact definitions can also be given in terms of type theory as well as in terms of its supercuspidal lifts:

Proposition 1.1** **(Defintion 6.2, Theorem 6.11 and

Corollary 6.3).

Let $\pi$ be an $\ell$-modular cuspidal distinguished representation of ${\rm G}$. Then the following are equivalent, and when they are satisfied we say that $\pi$ is relatively banal:

1. (i)

$\pi$* is not $|\mathrm{det}(\ \cdot\ )|_{{\rm F}_{\mathsf{o}}}$-distinguished.* 2. (ii)

All supercuspidal lifts of $\pi$ are distinguished by an unramified character of ${\rm G}_{\mathsf{o}}$. 3. (iii)

$q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\not\equiv 1[\ell]$, where $e_{\mathsf{o}}(\pi)$ denotes the invariant associated to $\pi$ in **[3, Lemma 5.10]** (see Section 5.2).

Relatively banal for ${\rm G}_{\mathsf{o}}$-distinguished cuspidal representations turns out to be the exact analogue of the definition of banal cuspidal representations of ${\rm G}_{\mathsf{o}}$ (see [24, Remark 8.15] and [23]) after one identifies the cuspidal (irreducible) representations of ${\rm G}_{\mathsf{o}}$ with the $\Delta({\rm G}_{\mathsf{o}})$-distinguished cuspidal (irreducible) representations of ${\rm G}_{\mathsf{o}}\times{\rm G}_{\mathsf{o}}$, where $\Delta:{\rm G}_{\mathsf{o}}\rightarrow{\rm G}_{\mathsf{o}}\times{\rm G}_{\mathsf{o}}$ is the diagonal embedding, as we explain in Section 8.3.

The main theorem of this paper characterises the poles of the Asai ${\rm L}$-factor:

Theorem 1.2** (Theorem 8.1).**

Let $\pi$ be a cuspidal $\ell$-modular representation, then ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ has a pole at ${\rm X}=1$ if and only if $\pi$ is relatively banal distinguished.

Note that the proof of the above theorem is completely different from the proof of characterisation theorem in the complex case (see Remark 8.2 for more on the comparison of the proofs). Here, we show that the Asai ${\rm L}$-factor of a cuspidal $\ell$-modular representation is equal to $1$ whenever $\pi$ is not the unramified twist of a relatively banal representation using Theorem 6.11, which is the characterisation of relatively banal in terms of supercuspidal lifts. Then when $\pi$ is the unramified twist of relatively banal representation, following our paper [18], we get an explicit formula for ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ in Theorem 7.10 from the test vector computation of [3], which thanks to the relatively banal assumption (more precisely its type theory version) reduces modulo $\ell$ without vanishing. We then deduce Theorem 1.2 from this computation, together with the computation of the group of unramified characters $\mu$ of ${\rm G}_{\mathsf{o}}$ such that $\pi$ is $\mu$-distinguished (Corollary 5.17).

Finally, denoting by ${\rm N}$ the unipotent radical of the group of upper triangular matrices in ${\rm G}$,  ${\rm Z}_{\mathsf{o}}$ the centre of ${\rm G}_{\mathsf{o}}$ and ${\rm N}_{\mathsf{o}}$ the group ${\rm N}\cap{\rm G}_{\mathsf{o}}$, the most natural ${\rm G}_{\mathsf{o}}$-invariant linear form to consider on the Whittaker model of an $\ell$-modular cuspidal representation $\pi$ with respect to a distinguished non degenerate character of ${\rm N}$ trivial on ${\rm N}_{\mathsf{o}}$ is the local period

[TABLE]

In fact this period plays an essential role in the proof of Theorem 1.2 over the field of complex numbers (see Remark 8.2). One of the main differences in the $\ell$-modular setting is that $\mathcal{L}_{\pi}$ can be zero even when $\pi$ is distinguished. Not only do we show that it can vanish but we say exactly when it does:

Theorem 1.3** (Theorem 8.3).**

Let $\pi$ be a cuspidal distinguished $\ell$-modular representation of $\operatorname{GL}_{n}({\rm F})$. Then the local period $\mathcal{L}_{\pi}$ is nonzero if and only if the following two properties are satisfied:

1. (i)

$\pi$* is relatively banal.* 2. (ii)

$\ell$* does not divide $e_{\mathsf{o}}(\pi)$, in other words if $\widetilde{\pi}$ is a lift of $\pi$, the $\ell$-adic valuation of $n$ is the same as the $\ell$-adic valuation of the number of $\ell$-adic unramified characters $\widetilde{\mu}_{\mathsf{o}}$ of ${\rm G}_{\mathsf{o}}$ such $\widetilde{\pi}$ is $\widetilde{\mu}_{\mathsf{o}}$-distinguished.*

This theorem is related to the vanishing modulo $\ell$ of a rather interesting and subtle scalar related, after fixing an isomorphism $\mathbb{C}\simeq\overline{\mathbb{Q}_{\ell}}$, to a quotient of the formal degree of a complex cuspidal representation of a unitary group by the formal degree of its base change to $\operatorname{GL}_{n}({\rm F})$, see Remark 8.4 for a precise statement.

In light of Theorem 1.2, the role of the Asai ${\rm L}$-factor in the study of distinguished representations will be less important in the $\ell$-modular setting as some $\ell$-modular distinguished representations have Asai ${\rm L}$-factors equal to $1$ in the cuspidal case already, and new ideas will be required already for non-relatively banal distinguished cuspidal representations. We will focus on the general case of distinguished irreducible $\ell$-modular representations restricting to small rank in the paper [19].

Finally, we mention that this paper heavily relies on the results from [3] and [27], and can be seen as a natural continuation of the themes developed in these two papers. In particular our section on lifting distinction for cuspidal representations of finite general linear groups uses the same techniques as [27], and the statements which we obtain here were known to the author of [27].

Acknowledgements**.**

It is a pleasure to thank Vincent Sécherre for motivating discussions and very useful explanations concerning his paper [27]. We thank David Helm and Alberto Mínguez for their interest and useful conversations.

This work has benefited from support from GDRI Representation Theory 2016-2020 and the EPSRC grant EP/R009279/1, and the Heilbronn Institute for Mathematical Research. Parts of the paper were written while the second named author was at the conferences “On the Langlands Program: Endoscopy and Beyond” held at the IMS of the National University of Singapore in 2018/2019 and “Automorphic forms, automorphic representations and related topics” held at the RIMS of the University of Kyoto in 2019. He would like to thank the organisers of both conferences, especially Wee Teck Gan and Shunsuke Yamana, and both institutions for their hospitality.

2. Notation

Let ${\rm F}/{\rm F}_{\mathsf{o}}$ be a quadratic extension of non-archimedean local fields of odd residual characteristic $p$. For any finite extension ${\rm E}/{\rm F}_{\mathsf{o}}$ we let $|~{}|_{\rm E}$ be the absolute value, $\operatorname{val}_{\rm E}$ the additive valuation, $\mathscr{O}_{{\rm E}}$ denote the ring of integers of ${\rm E}$, with maximal ideal $\mathscr{P}_{\rm E}$, residue field $\mathrm{k}_{\rm E}$, and put $q_{\rm E}=\#\mathrm{k}_{\rm E}$. We put $|~{}|=|~{}|_{\rm F}$, $\operatorname{val}=\operatorname{val}_{\rm F}$, $\mathscr{O}=\mathscr{O}_{{\rm F}}$, $\mathscr{P}=\mathscr{P}_{\rm F}$, $\mathrm{k}=\mathrm{k}_{\rm F}$, $q=q_{\rm F}$, $|~{}|_{\mathsf{o}}=|~{}|_{{\rm F}_{\mathsf{o}}}$, $\operatorname{val}_{\mathsf{o}}=\operatorname{val}_{{\rm F}_{\mathsf{o}}}$, $\mathscr{O}_{\mathsf{o}}=\mathscr{O}_{{\rm F}_{\mathsf{o}}}$, $\mathscr{P}_{\mathsf{o}}=\mathscr{P}_{{\rm F}_{\mathsf{o}}}$, $\mathrm{k}_{\mathsf{o}}=\mathrm{k}_{{\rm F}_{\mathsf{o}}}$ and $q_{\mathsf{o}}=q_{{\rm F}_{\mathsf{o}}}$.

We let $\ell$ denote a prime not equal to $p$. Set $\overline{\mathbb{Q}_{\ell}}$ to be an algebraic closure of the $\ell$-adic numbers, $\overline{\mathbb{Z}_{\ell}}$ its ring of integers, and $\overline{\mathbb{F}_{\ell}}$ its residue field.

Let ${\rm G}$ be the ${\rm F}$-points of a connected reductive group defined over ${\rm F}$ and $\mathcal{G}$ be the $\mathrm{k}$-points of a connected reductive group defined over $\mathrm{k}$.

All representations considered are assumed to be smooth. We consider representations of ${\rm G}$ and $\mathcal{G}$ and their subgroups on $\overline{\mathbb{Q}_{\ell}}$ and $\overline{\mathbb{F}_{\ell}}$-vector spaces and the relations between them. We let ${\rm R}$ denote either $\overline{\mathbb{Q}_{\ell}}$ or $\overline{\mathbb{F}_{\ell}}$, so that we can make statements valid in both cases more briefly. By an ${\rm R}$-representation we mean a representation on an ${\rm R}$-vector space.

An ${\rm R}$-representation of ${\rm G}$ or $\mathcal{G}$ is called cuspidal if it is irreducible and does not appear as a quotient of any representation parabolically induced from an irreducible representation of a proper Levi subgroup. It is called supercuspidal if it is irreducible and does not appear as a subquotient of any representation parabolically induced from an irreducible representation of a proper Levi subgroup. Over $\overline{\mathbb{Q}_{\ell}}$ a representation of ${\rm G}$ or $\mathcal{G}$ is cuspidal if and only if it is supercuspidal, however this is not the case over $\overline{\mathbb{F}_{\ell}}$, see [28, III] and [16] for examples of cuspidal non-supercuspidal representations.

3. Background on integral representations and distinction

Definition 3.1**.**

Let ${\rm G}$ be a locally profinite group and ${\rm H}$ be a closed subgroup of ${\rm G}$. Let $\pi$ be an ${\rm R}$-representation of ${\rm G}$ and $\chi:{\rm H}\rightarrow{\rm R}^{\times}$ be a character. We say that $\pi$ is $\chi$-distinguished if $\operatorname{Hom}_{\rm H}(\pi,\chi)\neq 0$. We say that $\pi$ is distinguished if it is $1$-distinguished where $1$ denotes the trivial character of ${\rm H}$.

We will mainly consider cases where ${\rm H}$ is the group of fixed points ${\rm G}^{\sigma}=\{g\in{\rm G}:\sigma(g)=g\}$ of an involution $\sigma$. In this case for any subset ${\rm X}\subset{\rm G}$, we set ${\rm X}^{\sigma}={\rm X}\cap{\rm G}^{\sigma}$.

Definition 3.2**.**

We call a triple $({\rm G},{\rm H},\sigma)$ an ${\rm F}$-symmetric pair when

1. (i)

${\rm G}=\mathbb{G}({\rm F})$ with $\mathbb{G}$ a connected reductive group defined over ${\rm F}$, and $\sigma$ is an involution of $\mathbb{G}$ defined over ${\rm F}$; 2. (ii)

${\rm H}$ is an open subgroup of the group ${\rm G}^{\sigma}$.

The main symmetric pair of interest in this note will be $(\operatorname{GL}_{n}({\rm F}),\operatorname{GL}_{n}({\rm F}_{\mathsf{o}}),\sigma)$, where $\sigma$ is the involution induced from the nontrivial element of $\operatorname{Gal}({\rm F}/{\rm F}_{\mathsf{o}})$. Two important basic results concerning this pair, that we shall use later are the following ([9], [26] for $\overline{\mathbb{Q}_{\ell}}$-representations, extended to ${\rm R}$-representations in [27, Theorem 3.1]):

Proposition 3.3**.**

Let $\pi$ be an irreducible ${\rm R}$-representation of $\operatorname{GL}_{n}({\rm F})$, then

[TABLE]

moreover if this dimension is equal to one, then $\pi^{\vee}\simeq\pi^{\sigma}$.

Let ${\rm K}$ be a locally profinite group. An irreducible $\overline{\mathbb{Q}_{\ell}}$-representation $\pi$ of ${\rm K}$ is called integral if it stabilises a $\overline{\mathbb{Z}_{\ell}}$-lattice in its vector space. An integral irreducible $\overline{\mathbb{Q}_{\ell}}$-representation $\pi$ which stabilises a lattice ${\rm L}$ induces an $\overline{\mathbb{F}_{\ell}}$-representation on the space ${\rm L}\otimes_{\overline{\mathbb{Z}_{\ell}}}\overline{\mathbb{F}_{\ell}}$. When ${\rm K}$ is either a profinite group or the $F$-points of a connected reductive ${\rm F}$-group, the semisimplification $r_{\ell}(\pi)=$ of ${\rm L}\otimes_{\overline{\mathbb{Z}_{\ell}}}\overline{\mathbb{F}_{\ell}}$ is independent of the choice of ${\rm L}$ and called the reduction modulo $\ell$ of $\pi$ ([28, 9.6] in the profinite setting where all representations are automatically defined over a finite extension of $\overline{\mathbb{F}_{\ell}}$, or [30, Theorem 1] in the context of reductive groups). Given an irreducible $\overline{\mathbb{F}_{\ell}}$-representation $\overline{\pi}$ of ${\rm K}$, we will call any integral irreducible $\overline{\mathbb{Q}_{\ell}}$-representation $\pi$ of ${\rm K}$ which satisfies $r_{\ell}(\pi)=\overline{\pi}$ a lift of $\overline{\pi}$.

We shall see later that distinction of cuspidal representations of ${\rm G}$ does not always lift, i.e. that an $\ell$-modular cuspidal distinguished representation may have no distinguished lifts. However, we have the following general result which shows that distinction reduces modulo $\ell$:

Theorem 3.4**.**

Let $({\rm G},{\rm H},\sigma)$ be an ${\rm F}$-symmetric pair. Let $\pi$ be an integral $\ell$-adic supercuspidal representation if ${\rm G}$, and let $\chi$ be an integral character of ${\rm H}$. Then if $\pi$ is $\chi$-distinguished, the representation $r_{\ell}(\pi)$ is $r_{\ell}(\chi)$-distinguished.

Proof.

Note that $\chi$ coincides with the central character of $\pi$ restricted to ${\rm H}$ which is also integral on ${\rm Z}_{\rm G}\cap{\rm H}$ (where ${\rm Z}_{\rm G}$ is the centre of ${\rm G}$) hence we extend it to a character still denoted $\chi$ to ${\rm Z}_{\rm G}{\rm H}$: $\chi(zh)=c_{\pi}(z)\chi(h)$. Note that $\operatorname{Hom}_{{\rm H}}(\pi,\chi)=\operatorname{Hom}_{{\rm Z}_{\rm G}{\rm H}}(\pi,\chi)$. By [15, Proposition 8.1] for $\chi=1$, extended to general $\chi$ in [8, Theorem 4.4], for $L$ a nonzero element of $\operatorname{Hom}_{{\rm H}}(\pi,\chi)$, the map

[TABLE]

embeds $\pi$ as a submodule of $\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi)$. Now by [30, Proposition II.3], $\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi,\overline{\mathbb{Z}_{\ell}})$ is an integral structure in $\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi)$, hence its intersection $\pi_{e}$ with $\pi$ is an integral structure of $\pi$ by [28, 9.3] (note that Vignéras works over a finite extension of $\overline{\mathbb{F}_{\ell}}$, but her results apply here because both $\pi$ and $\chi$, hence both $\pi$ and $\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi)$ have $E$-structures by [28, Section 4]). So $\pi_{e}\subset\operatorname{ind}_{Z_{\rm G}{\rm H}}^{\rm G}(\chi,\overline{\mathbb{Z}_{\ell}})$ but the map $\Lambda:f\mapsto f(1_{\rm G})$ is an element of $\operatorname{Hom}_{{\rm H}}(\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi,\overline{\mathbb{Z}_{\ell}}),\chi)$ which is nonzero on any submodule of $\operatorname{ind}_{{\rm Z}_{\rm G}{\rm H}}^{\rm G}(\chi,\overline{\mathbb{Z}_{\ell}})$, in particular on $\pi_{e}$. Up to multiplying $\Lambda$ by an appropriate nonzero scalar in $\overline{\mathbb{Q}_{\ell}}$, we can suppose that $\Lambda(\pi_{e})=\overline{\mathbb{Z}_{\ell}}$, and $\Lambda$ induces a nonzero element of $\operatorname{Hom}_{{\rm H}}(\pi_{e}\otimes\overline{\mathbb{F}_{\ell}},r_{\ell}(\chi))$. The result follows. ∎

Remark 3.5**.**

If ${\rm K}^{\prime}$ is a closed subgroup of a profinite group ${\rm K}$, (smooth) finite dimensional $\overline{\mathbb{Q}_{\ell}}$-representations of ${\rm K}$ are always integral and the image of a lattice by a nonzero linear form on such a representation is obviously a lattice of $\overline{\mathbb{Q}_{\ell}}$, so the reduction modulo $\ell$ of a $({\rm K}^{\prime},\chi)$-distinguished finite dimensional $\overline{\mathbb{Q}_{\ell}}$-representation of ${\rm K}$ is $({\rm K}^{\prime},r_{\ell}(\chi))$-distinguished.

Remark 3.6**.**

The following observation sheds more light on Theorem 3.4 when ${\rm G}=\operatorname{GL}_{n}({\rm F})$. Let $\pi$ be an integral supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of $\operatorname{GL}_{n}({\rm F})$, then its reduction modulo $\ell$ is (irreducible and) cuspidal, by [28, III 4.25]. This is however not true in general, see [16] for an example of an integral supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation whose reduction modulo $\ell$ is reducible.

Let ${\rm K}$ be a locally profinite group and ${\rm K}^{\prime}$ a closed subgroup. While in general it appears a subtle question to ascertain when the distinction of $\overline{\mathbb{F}_{\ell}}$-representations of ${\rm K}$ lifts, there is however one elementary case where it does: when the subgroup for which we want to study distinction ${\rm K}^{\prime}$ is profinite of pro-order prime to $\ell$. In this case, an $\ell$-modular finite dimensional (smooth) representation of ${\rm K}^{\prime}$ is semisimple and reduction modulo $\ell$ defines a bijection between the set of isomorphism classes of integral irreducible $\overline{\mathbb{Q}_{\ell}}$-representations of ${\rm K}^{\prime}$ and the set of isomorphism classes irreducible $\overline{\mathbb{F}_{\ell}}$-representations of ${\rm K}^{\prime}$, and we have:

Lemma 3.7**.**

Let ${\rm K}$ be a locally profinite group and ${\rm K}^{\prime}$ be a compact subgroup of ${\rm K}$. Suppose that the pro-order of ${\rm K}^{\prime}$ is prime to $\ell$. Let $\rho$ be an finite dimensional integral $\overline{\mathbb{Q}_{\ell}}$-representation of ${\rm K}$ and $\chi$ be a character of ${\rm K}^{\prime}$. Then $\rho$ is $\chi$-distinguished if and only if $r_{\ell}(\rho)$ is $r_{\ell}(\chi)$-distinguished.

Remark 3.8**.**

If ${\rm K}$ is compact modulo centre, an irreducible $\overline{\mathbb{Q}_{\ell}}$-representation of ${\rm K}$ is always finite dimensional and is integral if and only if its central character is integral.

4. Distinction for finite $\operatorname{GL}_{n}$

For the rest of this section, we set $\mathcal{G}=\operatorname{GL}_{n}(\mathrm{k})$, where (as before) $\mathrm{k}$ denotes a finite field of odd cardinality $q$. If $\mathrm{k}/\mathrm{k}_{\mathsf{o}}$ is a quadratic extension of $\mathrm{k}_{\mathsf{o}}$ we denote by $\sigma$ the non trivial element of $\operatorname{Gal}(\mathrm{k}/\mathrm{k}_{\mathsf{o}})$ and set $\mathcal{G}_{\mathsf{o}}=\operatorname{GL}_{n}(\mathrm{k}_{\mathsf{o}})$.

We recall the definitions of self-dual and $\sigma$-self-dual representations of $\mathcal{G}$:

Definition 4.1**.**

1. (i)

Suppose $\mathrm{k}/\mathrm{k}_{\mathsf{o}}$ is a quadratic extension of finite fields, then a representation $\rho$ of $\mathcal{G}$, over $\overline{\mathbb{Q}_{\ell}}$ or $\overline{\mathbb{F}_{\ell}}$, is called $\sigma$-self-dual if $\rho^{\sigma}\simeq\rho^{\vee}$. 2. (ii)

A representation $\rho$ of $\mathcal{G}$, over $\overline{\mathbb{Q}_{\ell}}$ or $\overline{\mathbb{F}_{\ell}}$, is called self-dual if $\rho\simeq\rho^{\vee}$.

4.1. Basic results on distinction

The following multiplicity one results are [27, Remark 3.2 with the adhoc modification in the proof of Theorem 3.1, Proposition 6.10 and Remark 6.11]:

Proposition 4.2**.**

Let $\rho$ be an irreducible ${\rm R}$-representation of $\mathcal{G}$:

1. (i)

If $\mathrm{k}/\mathrm{k}_{\mathsf{o}}$ is a quadratic extension of finite fields, then $\operatorname{dim}(\operatorname{Hom}_{\mathcal{G}_{\mathsf{o}}}(\rho,\chi))\leq 1$ for any character $\chi$ of $\mathcal{G}_{\mathsf{o}}$. 2. (ii)

If $\rho$ is cuspidal and $r$ and $s$ are two nonnegative integers such $r+s=n\geq 2$. Then $\operatorname{dim}(\operatorname{Hom}_{(\operatorname{GL}_{r}\times\operatorname{GL}_{s})(\mathrm{k})}(\rho,\chi))\leq 1$ for any character $\chi$ of $\operatorname{GL}_{r}\times\operatorname{GL}_{s}$ and this dimension is equal to zero if $r$ and $s$ are positive and $r\neq s$.

The final goal of this section is to understand when a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of a finite general linear group which is distinguished by a maximal Levi subgroup or by a Galois involution has a lift which does not share the same distinction property.

The connection between $(\sigma)$-self-dual representations and distinction comes from:

Lemma 4.3**.**

1. (i)

A $\operatorname{GL}_{n}(\mathrm{k}_{\mathsf{o}})$-distinguished irreducible ${\rm R}$-representation of $\operatorname{GL}_{n}(\mathrm{k})$ is $\sigma$-self-dual. Moreover if $\rho$ is supercuspidal, we have an equivalence: $\rho$ is $\sigma$ self-dual if and only if it is $\operatorname{GL}_{n}(\mathrm{k}_{\mathsf{o}})$-distinguished. 2. (ii)

A supercuspidal representation of $\operatorname{GL}_{n}(\mathrm{k})$ is self-dual if and only if either $n=1$ and it a quadratic character, or if $n$ is even and it is $(\operatorname{GL}_{n/2}\times\operatorname{GL}_{n/2})(\mathrm{k})$-distinguished.

Proof.

The first assertion of (i) follows from [27, Remark 3.2], and the second from [27, Lemma 8.3]. The second assertion follows from [27, Lemmas 7.1 and 7.3]. ∎

4.2. Self-dual and $\sigma$-self-dual cuspidal representations via the Green–Dipper–James parametrisation

In this subsection either $\mathrm{k}$ is an arbitrary finite field and we consider self-dual representations of $\operatorname{GL}_{n}(\mathrm{k})$ or $\mathrm{k}/\mathrm{k}_{\mathsf{o}}$ is quadratic and we consider $\sigma$-self-dual representations of $\operatorname{GL}_{n}(\mathrm{k})$ where $\langle\sigma\rangle=\operatorname{Gal}(\mathrm{k}/\mathrm{k}_{\mathsf{o}})$.

Let $\mathrm{l}/\mathrm{k}$ be a degree $n$ extension of $\mathrm{k}$. A character $\theta:\mathrm{l}^{\times}\rightarrow\overline{\mathbb{Q}_{\ell}}^{\times}$ is called $\mathrm{k}$-regular if $\#\{\theta^{\tau}:\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})\}=n$, i.e. the orbit of $\theta$ under $\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ has maximal sise. By [11], there is a surjective map

[TABLE]

The character formula given in [11] also implies:

1. (i)

Two such cuspidal representations $\rho(\theta)$ and $\rho(\theta^{\prime})$ are isomorphic if and only if there exists $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ such that $\theta^{\prime}=\theta^{\tau}$. 2. (ii)

The dual $\rho(\theta)^{\vee}$ is isomorphic to $\rho(\theta^{-1})$. 3. (iii)

If $\mathrm{k}/\mathrm{k}^{\prime}$ is a finite extension and $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}^{\prime})$, we have $\rho(\theta^{\tau})\simeq\rho(\theta)^{\tau}$.

The following is well-known, and a similar proof to ours can be found in [27, Lemmas 7.1 & 8.1] in the greater generality of supercuspidal ${\rm R}$-representations, we provide a proof as a warm-up:

Lemma 4.4**.**

1. (i)

If there exists a $\sigma$-self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of $\mathcal{G}$, then $n$ is odd. 2. (ii)

If there exists a self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of $\mathcal{G}$, then $n$ is either one or even.

Proof.

1. (i)

Suppose that $\rho$ is a $\sigma$-self-dual cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation, and write $\rho=\rho(\theta)$ for a $\mathrm{k}$-regular character $\theta$. Choose an extension of $\sigma$ to $\widetilde{\sigma}\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$. Then as $\rho(\theta^{-1})\simeq\rho(\theta)^{\vee}\simeq\rho(\theta)^{\sigma}$, necessarily $\theta^{\widetilde{\sigma}}=(\theta^{-1})^{\tau}$ for some $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$. Hence $(\tau^{-1}\circ\widetilde{\sigma})^{2}$ fixes $\theta$, so it is $1$ as $\theta$ is regular. This implies that $\tau^{-1}\circ\widetilde{\sigma}$ is a $\mathrm{k}_{\mathsf{o}}$-linear involution of $\mathrm{l}$ which extends $\sigma$. However the cyclic group $\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$ contains a unique element of order $2$. If $n$ was even, $\tau^{-1}\circ\widetilde{\sigma}$ would belong to $\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ and this is absurd as it extends $\sigma$ which is not $\mathrm{k}$-linear. 2. (ii)

Suppose that $\rho$ is a self-dual cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation, and write $\rho=\rho(\theta)$ for a $\mathrm{k}$-regular character $\theta$. In this case, reasoning as before, necessarily $\theta=\tau(\theta^{-1})$ for some $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ and it follows that $\tau^{2}=1$. Either $\tau=1$ hence $\theta^{2}=1$, but there is a unique non-trivial quadratic character of $\mathrm{l}^{\times}$ which is thus fixed by all $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ and the trivial character of $\mathrm{l}^{\times}$ is also $\operatorname{Gal}(\mathrm{l}/\mathrm{k})$-invariant, so we deduce that $n=1$ as $\theta$ is regular. Or $\tau$ has order $2$ hence $n=\#\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ is even.

∎

We now recall the classification of cuspidal $\overline{\mathbb{F}_{\ell}}$-representations of James [13]. We have a surjective map

[TABLE]

given by reduction modulo $\ell$.

Given a character $\theta:\mathrm{l}^{\times}\rightarrow\overline{\mathbb{Q}_{\ell}}^{\times}$ we can uniquely write $\theta=\theta_{r}\theta_{s}$ with $\theta_{r}$ of order prime to $\ell$ and $\theta_{s}$ of order a power of $\ell$. The parametrisation of James enjoys the following properties:

1. (i)

Two supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations $\rho(\theta),\rho(\theta^{\prime})$ have isomorphic reductions modulo $\ell$ if and only if there exists $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ such that $\theta^{\prime}_{r}=\theta_{r}^{\tau}$. 2. (ii)

$\overline{\rho(\theta)}$ is supercuspidal if and only if $\theta_{r}$ is regular.

4.3. $\sigma$-self-dual lifts of cuspidal $\overline{\mathbb{F}_{\ell}}$-representations

We now specialise to the case $\mathrm{k}/\mathrm{k}_{\mathsf{o}}$ is quadratic. Write $\Gamma=\operatorname{Hom}(\mathrm{l}^{\times},\overline{\mathbb{Q}_{\ell}}^{\times})$, then $\Gamma=\Gamma_{s}\times\Gamma_{r}$ where $\Gamma_{s}$ consists of the characters of $\ell$-power order, and $\Gamma_{r}$ consists of the characters with order prime to $\ell$.

We study $\sigma$-self-dual lifts of cuspidal $\overline{\mathbb{F}_{\ell}}$-representations, and when $n$ is even there are no $\sigma$-self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations by Lemma 4.4. Hence without loss of generality, we can assume that $n$ is odd. Moreover as reduction modulo $\ell$ commutes with taking contragredients and with the action of $\sigma$, this implies that when the cuspidal representation $\overline{\rho}$ of $\mathcal{G}$ is not $\sigma$-self-dual, it has no $\sigma$-self-dual lifts, so we suppose that it is from now on.

For $\gamma\in\Gamma$ we set $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\gamma}=\{\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}):\gamma^{\tau}=\gamma\}$. Let $\theta\in\Gamma$, we can decompose $\theta=\theta_{r}\theta_{s}$, and we have $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta}=\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}\cap\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{s}}$. In particular, $\theta$ is regular if and only if $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}\cap\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{s}}=\{1\}.$

Let $\mathrm{l}_{\mathsf{o}}$, be the unique subextension of $\mathrm{l}/\mathrm{k}_{\mathsf{o}}$ of degree $n$ as an extension of $\mathrm{k}_{\mathsf{o}}$ and put $\Gamma_{\mathsf{o}}=\operatorname{Hom}(\mathrm{l}_{\mathsf{o}}^{\times},\overline{\mathbb{Q}_{\ell}}^{\times})$. We have an embedding

[TABLE]

by surjectivity of the norm. Hence $\Gamma^{+}=i(\Gamma_{\mathsf{o}})$ is unique subgroup of the cyclic group $\Gamma$ of order $q_{\mathsf{o}}^{n}-1$. Write $\widetilde{\sigma}$ for the unique involution in $\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$, which extends $\sigma$ (as $n$ is odd). By Hilbert’s theorem 90, we have

[TABLE]

On the other hand, the unique subgroup of the cyclic group $\Gamma$ of order $q_{\mathsf{o}}^{n}+1$ is

[TABLE]

as the norm map is surjective. Notice that $(q_{\mathsf{o}}^{n}+1,q_{\mathsf{o}}^{n}-1)=2$ because $q$ is odd, so $\Gamma^{+}\cap\Gamma^{-}=\{1,\eta\}$, where $\eta$ denotes the unique quadratic character in $\Gamma$. Moreover if $\ell$ is odd:

1. (i)

If $\ell\mid q_{\mathsf{o}}^{n}-1$, then $\Gamma_{s}\subseteq\Gamma^{+}$; 2. (ii)

If $\ell\mid q_{\mathsf{o}}^{n}+1$, then $\Gamma_{s}\subseteq\Gamma^{-}$.

Before giving the full solution of the lifting $\sigma$-self-duality for $\ell$-modular cuspidal representations, we characterise $\ell$-modular cuspidal $\sigma$-self-duality in terms of the Dipper and James parametrisation.

Proposition 4.5**.**

Let $\overline{\rho}$ be a cuspidal representation of $\mathcal{G}$ and suppose that $n$ and $\ell$ are odd, then $\overline{\rho}$ is $\sigma$-self-dual if and only if $\theta_{r}^{\widetilde{\sigma}}=\theta_{r}^{-1}$.

Proof.

Write $\overline{\rho}=\overline{\rho(\theta)}$ for a $\mathrm{k}$-regular character $\theta$, and let $\widetilde{\sigma}\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$ be the unique involution extending $\sigma$. One implication is obvious, for the other we thus suppose that $\overline{\rho(\theta)}$ is $\sigma$-self-dual. Then there exists $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ such that $\theta_{r}^{\widetilde{\sigma}\tau}=\theta_{r}^{-1}$. This implies that $\tau^{2}=(\widetilde{\sigma}\tau)^{2}$ belongs to $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}$. On the other hand the order of $\tau$ is odd because $n$ is, hence $\tau$ as well belongs to $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}$, so $\theta_{r}^{\widetilde{\sigma}}=\theta_{r}^{-1}$. ∎

We have the following complete result when $\ell$ is odd.

Proposition 4.6**.**

Assume that $n$ and $\ell$ are odd. Let $\overline{\rho}$ be a $\sigma$-self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of $\mathcal{G}$.

1. (i)

Suppose that $\ell$ is prime with $q^{n}-1$. Then the unique supercuspidal lift of $\overline{\rho}$ is $\sigma$-self-dual. 2. (ii)

Suppose that $\ell\mid q_{\mathsf{o}}^{n}-1$.

1. (a)

If $\overline{\rho}$ is supercuspidal and $\ell^{a}$ is the highest power of $\ell$ dividing $q^{n}-1$, then there is a unique $\sigma$-self-dual supercuspidal lift amongst the $\ell^{a}$ supercuspidal lifts of $\overline{\rho}$. In terms, of Green’s parameterisation of supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations, if $\rho(\theta)$ is a lift of $\overline{\rho}$ then $\rho(\theta_{r})$ is the unique $\sigma$-self-dual supercuspidal lift of $\overline{\rho}$. 2. (b)

If $\overline{\rho}$ is cuspidal non-supercuspidal, then none of its supercuspidal lifts are $\sigma$-self-dual. 3. (iii)

Suppose that $\ell\mid q_{\mathsf{o}}^{n}+1$. Then all supercuspidal lifts of $\overline{\rho}$ are $\sigma$-self-dual.

Proof.

Write $\overline{\rho}=\overline{\rho(\theta)}$ for a $\mathrm{k}$-regular character $\theta$ of $\mathrm{l}^{\times}$, and let $\widetilde{\sigma}\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$ be the unique involution extending $\sigma$.

The set of isomorphism classes of supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations lifting $\overline{\rho}$ is then

[TABLE]

Such a representation $\rho(\theta_{r}\mu)$ is $\sigma$-self-dual if and only if there exists $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ such that $(\theta_{r}\mu)^{\widetilde{\sigma}}=(\theta_{r}^{-1}\mu^{-1})^{\tau}$. As $\theta_{r}\mu$ is regular, this condition implies that $\tau^{2}=(\widetilde{\sigma}\tau)^{2}$ is the identity, so that $\tau=\mathrm{Id}$ as $n$ is odd. So $\rho(\theta_{r}\mu)$ is $\sigma$-self-dual if and only if $\theta_{r}^{\widetilde{\sigma}}=\theta_{r}^{-1}$ and $\mu^{\widetilde{\sigma}}=\mu^{-1}$, and the set of $\sigma$-self-dual lifts of $\overline{\rho}$ is equal to

[TABLE]

because the condition $\theta_{r}^{\widetilde{\sigma}}=\theta_{r}^{-1}$ is always satisfied thanks to Proposition 4.5. In particular when $\theta_{r}$ is regular, then all $\theta_{r}\mu$ must be regular as well and the cardinality of the set of $\sigma$-selfdual lifts of $\overline{\rho}$ is that of $\Gamma_{s}$, namely the highest power of $\ell$ dividing $q^{n}-1$.

In particular if $\ell$ is prime to $q^{n}-1$ then $\Gamma_{s}$ is trivial and this proves (i).

If $\ell\mid q_{\mathsf{o}}^{n}-1$. Then $\Gamma_{s}\subseteq\Gamma^{+}$, and $\Gamma_{s}\cap\Gamma^{-}=\Gamma_{s}\cap\Gamma^{-}\cap\Gamma^{+}=\{1\}$ because $\Gamma^{+}\cap\Gamma^{-}=\{1,\eta\}$ and $\eta\not\in\Gamma_{s}$. Hence if $\overline{\rho}$ is supercuspidal i.e. if $\theta_{r}$ is regular, then $\rho(\theta_{r})$ is the unique $\sigma$-self-dual supercuspidal lift of $\overline{\rho}$ whereas if $\overline{\rho}$ is cuspidal non-supercuspidal, then it has no $\sigma$-self-dual supercuspidal lift, and we have shown (ii).

Finally suppose that $\ell\mid q_{\mathsf{o}}^{n}+1$, then $\Gamma_{s}\subseteq\Gamma^{-}$ so all supercuspidal lifts of $\overline{\rho}$ are $\sigma$-self-dual, and we have shown (iii). ∎

In the case $\ell=2$ we have:

Proposition 4.7**.**

Assume that $n$ is odd and $\ell=2$. Let $\overline{\rho}$ be a $\sigma$-self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of $\mathcal{G}$, then it has a non $\sigma$-self-dual lift.

Proof.

Write $\overline{\rho}=\overline{\rho(\theta)}$ for a regular character $\theta$, and let $\widetilde{\sigma}\in\operatorname{Gal}(\mathrm{l}/\mathrm{k}_{\mathsf{o}})$ be the unique involution extending $\sigma$.

First note that $q^{n}-1=(q_{\mathsf{o}}^{n}-1)(q_{\mathsf{o}}^{n}+1)$ so the highest power of $2$ dividing $q^{n}-1$ which is the order of $\Gamma_{s}$ does not divide $q_{\mathsf{o}}^{n}+1$ as $2$ also divides $q_{\mathsf{o}}^{n}-1$. In particular $\Gamma_{s}$ is not a subgroup of $\Gamma^{-}$, so if $\mu_{0}$ is a generator of $\Gamma_{s}$, then $\widetilde{\sigma}(\mu_{0})\neq\mu_{0}^{-1}$. Now we claim that $\rho(\theta_{r}\mu_{0})$ is a non $\sigma$-self-dual lift of $\overline{\rho(\theta)}$. First it is supercuspidal: indeed $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\mu_{0}}\subset\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{s}}$ because $\theta_{s}$ is a power of $\mu_{0}$, hence $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}\cap\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\mu_{0}}$ is trivial because $\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{r}}\cap\operatorname{Gal}(\mathrm{l}/\mathrm{k})_{\theta_{s}}$ is. Moreover suppose that $\rho(\theta_{r}\mu_{0})$ was $\sigma$-self-dual, then following the beginning of the proof of Proposition 4.6, this would imply that both $\theta_{r}$ and $\mu_{0}$ belong to $\Gamma^{-}$, which is absurd. ∎

4.4. Self-dual lifts of self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representations

If there exists a self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of $\mathcal{G}$ then $n$ is one or even by Lemma 4.4. The case $n=1$ is straightforward, a character is self-dual if and only if it is quadratic and we treat it separately:

Proposition 4.8**.**

Suppose that $n=1$. Then $1,\eta$ are the unique self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations of $\operatorname{GL}_{1}(\mathrm{k})$. The reductions $\overline{1},\overline{\eta}$ of $1,\eta$ respectively are the unique self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representations of $\operatorname{GL}_{1}(\mathrm{k})$.

1. (i)

Suppose that $\ell\nmid q-1$. Then $\overline{1},\overline{\eta}$ have $1,\eta$ respectively as unique lift. 2. (ii)

Suppose that $\ell\mid q-1$ and let $\ell^{a}$ be the highest power of $\ell$ dividing $q-1$. Then $\overline{1},\overline{\eta}$ each have $\ell^{a}$-supercuspidal lifts of which $1,\eta$ (respectively) is the unique self-dual supercuspidal lift.

Note that case (ii) contains the case $\ell=2$, in which case $\overline{1}=\overline{\eta}$. So in particular in case (ii) non trivial lifts of the trivial character of ${\kappa}^{\times}$ always exist.

Hence for the rest of this section we assume that $n$ is even. Let $\sigma^{\prime}$ denote the unique involution in $\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ and $\mathrm{l}_{\mathsf{o}}^{\prime}=\mathrm{l}^{\sigma^{\prime}}$ denote the $\sigma^{\prime}$-fixed subfield. Then we have an embedding

[TABLE]

by surjectivity of the norm so its image is the unique subgroup of $\Gamma$ of order $q^{n/2}-1$:

[TABLE]

The unique subgroup of $\Gamma$ of order $q^{n/2}+1$ is thus

[TABLE]

their intersection given by $\Gamma_{+}\cap\Gamma_{-}=\{1,\eta\}$ as $q$ is odd. As $(q^{n/2}+1,q^{n/2}-1)=2$, we deduce that if $\ell$ is odd:

1. (i)

If $\ell\mid q^{n/2}-1$, then $\Gamma_{s}\subseteq\Gamma_{+}$; 2. (ii)

If $\ell\mid q^{n/2}+1$, then $\Gamma_{s}\subseteq\Gamma_{-}$.

The results concerning self-duality look very similar to those concerning $\sigma$-self-duality, however in one case we only consider lifting of distinction.

Proposition 4.9**.**

Suppose that $n=2m\geqslant 2$ is even and that $\ell$ is odd. Let $\overline{\rho}$ be a self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of $\mathcal{G}$.

1. (i)

Suppose that $\ell$ is prime with $q^{n}-1$. Then the unique supercuspidal lift of $\overline{\rho}$ is self-dual. 2. (ii)

Suppose that $\ell\mid q^{n/2}-1$.

1. (a)

If $\overline{\rho}$ is supercuspidal and $\ell^{a}$ is the highest power of $\ell$ dividing $q^{n}-1$, then there is a unique self-dual lift of $\overline{\rho}$ amongst its $\ell^{a}$ supercuspidal lifts. In terms, of Green’s parameterisation of supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representations, if $\rho(\theta)$ is a lift of $\overline{\rho}$ then $\rho(\theta_{r})$ is the unique self-dual supercuspidal lift of $\overline{\rho}$. 2. (b)

If $\overline{\rho}$ is cuspidal non-supercuspidal, then none of its supercuspidal lifts are self-dual. 3. (iii)

Suppose that $\ell\mid q^{n/2}+1$ and that $\overline{\rho}$ is $\operatorname{GL}_{m}(\mathrm{k})\times\operatorname{GL}_{m}(\mathrm{k})$-distinguished. Then all supercuspidal lifts of $\overline{\rho}$ are self-dual.

Proof.

A lift $\rho(\theta_{r}\mu)$ with $\mu\in\Gamma_{s}$ is self-dual if and only if there exists $\tau\in\operatorname{Gal}(\mathrm{l}/\mathrm{k})$ such that $\theta_{r}\mu=(\theta_{r}^{-1}\mu^{-1})^{\tau}$ hence $\tau^{2}=\mathrm{Id}$, i.e. $\tau=\mathrm{Id}$ or $\sigma^{\prime}$. The first case is impossible as it would imply that $\theta_{r}\mu$ is the quadratic character in $\Gamma$, which would contradict its regularity. Hence $\rho(\theta_{r}\mu)$ is self-dual if and only if $\theta_{r}^{\sigma^{\prime}}=\theta_{r}^{-1}$ and $\mu^{\sigma^{\prime}}=\mu^{-1}$. Thus, if $\theta_{r}^{\sigma^{\prime}}=\theta_{r}^{-1}$, then the set

[TABLE]

is a full set of representatives for the isomorphism classes of self-dual lifts of $\overline{\rho}$ (and there are no self-dual lifts if $\theta_{r}^{\sigma^{\prime}}\neq\theta_{r}^{-1}$).

For Parts (i), (ii)(a) as $\theta_{r}$ is regular, the self-duality of $\overline{\rho}$ implies that $\theta_{r}^{\sigma^{\prime}}=\theta_{r}^{-1}$. Hence Parts (i), (ii)(a) follow in the same way as their analogues in Proposition 4.6. Part (ii)(b) is obvious as $\Gamma_{s}\cap\Gamma_{-}$ is trivial in this case. Finally (iii) holds because distinction lifts in this case by Lemma 3.7.

∎

When $\ell=2$, we have the exact analogue of Proposition 4.7 with the same proof, replacing $q_{\mathsf{o}}^{n}$ by $q^{n/2}$.

Proposition 4.10**.**

Assume that $n\geqslant 2$ is even and $\ell=2$. Let $\overline{\rho}$ be a self-dual cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of $\mathcal{G}$, then it has a non self-dual lift.

5. Type theory and distinction

From now on we set ${\rm G}=\operatorname{GL}_{n}({\rm F})$, ${\rm G}_{\mathsf{o}}=\operatorname{GL}_{n}({\rm F}_{\mathsf{o}})$, and $\sigma$ the Galois involution. We use the Bushnell–Kutzko construction of cuspidal representations of ${\rm G}$ [7], extended by Vignéras to the setting of cuspidal ${\rm R}$-representations [28, §III]. We summarise its properties that we will use, and refer the reader to [7] and [28] for more details on this construction.

5.1. Properties of types

Let $\pi$ be a cuspidal ${\rm R}$-representation of ${\rm G}$. Then associated to it is a family of explicitly constructed pairs $(\mathbf{J},\boldsymbol{\lambda})$, called extended maximal simple types, where $\mathbf{J}$ is a compact open subgroup of ${\rm G}$ containing the centre ${\rm Z}_{\rm G}$ of ${\rm G}$ with $\mathbf{J}/{\rm Z}_{\rm G}$ compact, and $\boldsymbol{\lambda}$ is an irreducible (hence finite dimensional) representation of $\mathbf{J}$ such that

[TABLE]

We abbreviate extended maximal simple type to type for the rest of the paper, and will say ${\rm R}$-type when we wish to specify the field ${\rm R}$ considered.

Let $(\mathbf{J},\boldsymbol{\lambda})$ be an ${\rm R}$-type in $\pi$, i.e. associated to $\pi$ as described above. Types enjoy the following key properties:

1. (T-1)

Two types in $\pi$ are conjugate in ${\rm G}$, [7, 6.2.4] and [28, III 5.3] 2. (T-2)

The group $\mathbf{J}$ has a unique maximal compact subgroup ${\rm J}$, and ${\rm J}$ has a unique maximal normal pro-$p$-subgroup ${\rm J}^{1}$, cf. [7, §3.1] for the definitions of these groups. 3. (T-3)

There is a subfield ${\rm E}$ of $\mathcal{M}_{n}({\rm F})$ containing ${\rm F}$, the multiplicative group of which normalises ${\rm J}$, and $\mathbf{J}={\rm E}^{\times}{\rm J}$. (In fact, we have summarised this construction in reverse; the extension ${\rm E}/{\rm F}$ is part of the original data used to construct the type.) The quotient ${\rm J}/{\rm J}^{1}$ is isomorphic to $\operatorname{GL}_{m}(\mathrm{k}_{E})$ with $m=n/[{\rm E}:{\rm F}]$. Moreover, ${\rm E}^{\times}\cap{\rm J}=\mathscr{O}_{{\rm E}}^{\times}$, hence $\mathbf{J}=\langle\varpi_{\rm E}\rangle{\rm J}$ is the semi-direct product of ${\rm J}$ with the group generated by $\varpi_{\rm E}$. 4. (T-4)

Let $(\mathbf{J},\boldsymbol{\lambda})$ be a type, let $\boldsymbol{\lambda}^{\prime}$ be a representation of $\mathbf{J}$, set $\pi=\operatorname{ind}_{\mathbf{J}}^{{\rm G}}(\boldsymbol{\lambda})$ and $\pi^{\prime}=\operatorname{ind}_{\mathbf{J}}^{{\rm G}}(\boldsymbol{\lambda}^{\prime})$. If $\boldsymbol{\lambda}^{\prime}\mid_{{\rm J}}\simeq\boldsymbol{\lambda}\mid_{{\rm J}}$, then $(\mathbf{J},\boldsymbol{\lambda}^{\prime})$ is a type and the cuspidal representation $\pi^{\prime}$ is an unramified twist of $\pi$. Conversely if $(\mathbf{J},\boldsymbol{\lambda}^{\prime})$ is a type and the cuspidal representation $\pi^{\prime}$ is an unramified twist of $\pi$, then $\boldsymbol{\lambda}^{\prime}\mid_{{\rm J}}\simeq\boldsymbol{\lambda}\mid_{{\rm J}}$. 5. (T-5)

The representation $\boldsymbol{\lambda}$ (by construction) decomposes (non-uniquely) as a tensor product $\boldsymbol{\kappa}\otimes\boldsymbol{\tau}$, where:

• •

$\boldsymbol{\tau}$ is a representation of $\mathbf{J}$ trivial on ${\rm J}^{1}$ which restricts irreducibly to ${\rm J}$, and the representation of ${\rm J}/{\rm J}^{1}$ induced by $\boldsymbol{\tau}$ identifies with a cuspidal representation of $\operatorname{GL}_{m}(\mathrm{k}_{\rm E})$.

• •

$\boldsymbol{\kappa}$ is a representation of $\mathbf{J}$ which restricts irreducibly to ${\rm J}^{1}$. 6. (T-6)

The representation $\pi$ is supercuspidal if and only if $\boldsymbol{\tau}$ induces a supercuspidal restriction on ${\rm J}/{\rm J}^{1}$, [28, III 5.14]. 7. (T-7)

The pair $(\mathbf{J},\boldsymbol{\kappa}\otimes\boldsymbol{\tau}^{\prime})$ is another type in $\pi$ if and only if $\boldsymbol{\tau}\simeq\boldsymbol{\tau}^{\prime}$. 8. (T-8)

When ${\rm R}=\overline{\mathbb{Q}_{\ell}}$, the representation $\pi$ is integral if and only if $\boldsymbol{\lambda}$ is integral. 9. (T-9)

The construction is compatible with reduction modulo $\ell$ in the following sense: given a $\overline{\mathbb{Q}_{\ell}}$-type $(\mathbf{J},\boldsymbol{\lambda})$ with $\boldsymbol{\lambda}$ integral, $(\mathbf{J},r_{\ell}(\boldsymbol{\lambda}))$ is an $\overline{\mathbb{F}_{\ell}}$-type and $r_{\ell}(\operatorname{ind}_{\mathbf{J}}^{\rm G}(\boldsymbol{\lambda}))=\operatorname{ind}_{\mathbf{J}}^{\rm G}(r_{\ell}(\boldsymbol{\lambda}))$ is a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation, [28, III 4.25]. 10. (T-10)

The construction lifts [28, III 4.29]: given an $\overline{\mathbb{F}_{\ell}}$-type $(\mathbf{J},\boldsymbol{\kappa}\otimes\boldsymbol{\tau})$, there is a unique irreducible $\overline{\mathbb{Q}_{\ell}}$-representation $\widetilde{\eta}$ of ${\rm J}^{1}$ which lifts $\boldsymbol{\kappa}\mid_{{\rm J}^{1}}$ and we can fix a extension $\widetilde{\boldsymbol{\kappa}}$ of $\widetilde{\eta}$ with $r_{\ell}(\widetilde{\boldsymbol{\kappa}})=\boldsymbol{\kappa}$. As all of the extensions of $\widetilde{\eta}$ to $\mathbf{J}$ are related by twisting by a character trivial on ${\rm J}^{1}$, Property (T-7) implies that the set of isomorphism classes of lifts of $\pi=\operatorname{ind}_{\mathbf{J}}^{\rm G}(\boldsymbol{\kappa}\otimes\boldsymbol{\tau})$ is in bijection with the set of isomorphism classes of lifts of $\boldsymbol{\tau}$ by $\widetilde{\boldsymbol{\tau}}\mapsto\operatorname{ind}_{\mathbf{J}}^{\rm G}(\widetilde{\boldsymbol{\kappa}}\otimes\widetilde{\boldsymbol{\tau}})$. 11. (T-11)

We call a field extension ${\rm E}/{\rm F}$ associated to a type in $\pi$ as in (T- 3) a parameter field for $\pi$. While there are potentially many choices, the ramification index $e({\rm E}/{\rm F})$, inertial degree $f({\rm E}/{\rm F})$, and (hence) the degree $[{\rm E}:{\rm F}]$ are invariants of $\pi$ as follows from [7, 3.5.1]. As such, we write $d(\pi)=[{\rm E}:{\rm F}]$, $e(\pi)=e({\rm E}/{\rm F})$ and $f(\pi)=f({\rm E}/{\rm F})$. We write $m(\pi)$ for $n/d(\pi)$. These invariants are compatible with reduction modulo $\ell$: for $\pi$ an integral cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation we have

[TABLE]

5.2. Galois-self-dual types

It has recently been showed in [3] and [27] that the construction of types also enjoys good compatibility properties with $\sigma$-self-duality and distinction. Indeed, according to [3, §4], if $\pi$ is a cuspidal ${\rm R}$-representation of ${\rm G}$ which is $\sigma$-self-dual, then one can chose a type $(\mathbf{J},\boldsymbol{\lambda})$ in $\pi$ such that:

1. (SSDT-I)

$\mathbf{J}$ (hence ${\rm J}$ and ${\rm J}^{1}$) and ${\rm E}$ are $\sigma$-stable, and $\boldsymbol{\lambda}^{\vee}\simeq\boldsymbol{\lambda}^{\sigma}$. 2. (SSDT-II)

Set ${\rm E}_{\mathsf{o}}={\rm E}^{\sigma}$, then ${\rm E}/{\rm E}_{\mathsf{o}}$ is a quadratic extension and we can choose a uniformiser $\varpi_{\rm E}$ with $\sigma(\varpi_{\rm E})=\varpi_{\rm E}$ if ${\rm E}/{\rm E}_{\mathsf{o}}$ is unramified and $\sigma(\varpi_{E})=-\varpi_{E}$ if ${\rm E}/{\rm E}_{\mathsf{o}}$ ramified as in [27, (5.2)]. 3. (SSDT-III)

$\boldsymbol{\kappa}$ hence $\boldsymbol{\tau}$ are $\sigma$-self-dual ([27, Lemma 8.9]).

The ramification index $e({\rm E}/{\rm E}_{\mathsf{o}})\in\{1,\ 2\}$ (and is equal to $1$ if ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified) is an invariant of $\pi$ and we write

[TABLE]

Remark 5.1**.**

This latter invariant is also equal to the ramification index of the extension ${\rm T}/{\rm T}^{\sigma}$, where ${\rm T}$ is the maximal tamely ramified extension of ${\rm F}$ contained in ${\rm E}$ thanks to [27, Remark 4.15 (2)]. We use this fact when referring to some results of [27].

In [3, §6.2], another invariant, a positive integer $e_{\mathsf{o}}(\pi)$ dividing $n$ defined only in terms of the $\sigma$-stable group $\mathbf{J}$ is associated to $\pi$. By [3, Lemma 5.10], we have the following description:

[TABLE]

Again, these invariants are compatible with reduction modulo $\ell$: for $\pi$ an integral $\sigma$-self-dual supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation we have

[TABLE]

Definition 5.2**.**

We call a type $\boldsymbol{\lambda}=\boldsymbol{\kappa}\otimes\boldsymbol{\tau}$ satisfying Conditions (SSDT-I) to (SSDT-III) a $\sigma$-self-dual type.

Note that an immediate consequence of the existence of a $\sigma$-self-dual cuspidal type in a $\sigma$-self-dual cuspidal representation is that in some cases there are no $\sigma$-self-dual cuspidal representations, as follows from [27, Lemma 6.9 and Lemma 8.1]:

Lemma 5.3**.**

Let $\pi$ be $\sigma$-self-dual cuspidal ${\rm R}$-representation.

1. (i)

If $e_{\sigma}(\pi)=1$ and $\pi$ is supercuspidal, then $m(\pi)$ is odd. 2. (ii)

If $e_{\sigma}(\pi)=2$, then $m(\pi)$ is either equal to $1$ or even.

A crucial property of $\sigma$-self-dual types is the following:

Proposition 5.4**.**

[27, Lemma 5.19]** Let $(\mathbf{J},\boldsymbol{\lambda})$ be a $\sigma$-self-dual type. Then there exists a unique character $\chi_{\boldsymbol{\kappa}}$ of $\mathbf{J}^{\sigma}$ trivial on $({\rm J}^{1})^{\sigma}$ such that

[TABLE]

and the canonical map

[TABLE]

is an isomorphism.

In many cases, it is shown in [27] that one can choose $\chi_{\boldsymbol{\kappa}}=1$ above, including the supercuspidal case:

Proposition 5.5**.**

Let $\pi$ be a $\sigma$-self-dual supercuspidal ${\rm R}$-representation and $(\mathbf{J},\boldsymbol{\lambda})$ be a $\sigma$-self-dual type of $\pi$, then one can choose $\boldsymbol{\kappa}$ such that $\chi_{\boldsymbol{\kappa}}=1$.

Proof.

The only cases to consider are those which are not ruled out by Lemma 5.3, and the assertion then follows from [27, Propositions 6.15 and 8.10]. ∎

Remark 5.6**.**

Note that if ${\rm R}=\overline{\mathbb{Q}_{\ell}}$ above, then $\pi$ is integral and $(\mathbf{J},r_{\ell}(\boldsymbol{\tau}))$ is a $\sigma$-self-dual type for $r_{\ell}(\pi)$. Moreover $\chi_{r_{\ell}(\boldsymbol{\kappa})}=1$ if $\chi_{\boldsymbol{\kappa}}=1$. Indeed, thanks to Remark 3.5 applied to $(\mathbf{J}/{\rm F}_{\mathsf{o}}^{\times},\boldsymbol{\kappa})$, the representation $r_{\ell}(\boldsymbol{\kappa})$ is distinguished, hence $\chi_{r_{\ell}(\boldsymbol{\kappa})}=1$ by the first part of Proposition 5.4.

5.3. Generic types and distinguished types

There are in general more than one ${\rm G}_{\mathsf{o}}$-conjugacy class of $\sigma$-self-dual types in a $\sigma$-self-dual cuspidal ${\rm R}$-representation (see [27, Section 1.11]). However there is only one ${\rm G}_{\mathsf{o}}$-conjugacy class among those which contain a generic type in the following sense.

We denote by ${\rm N}$ be the maximal unipotent subgroup of the subgroup of upper triangular matrices in ${\rm G}$, and ${\rm N}_{\mathsf{o}}={\rm N}^{\sigma}$. Let $\psi$ a nondegenerate character of ${\rm N}$. Note that such a character is always integral with nondegenerate reduction modulo $\ell$ as ${\rm N}$ is exhausted by its pro-$p$-subgroups.

Definition 5.7**.**

Let $(\mathbf{J},\boldsymbol{\lambda})$ be an ${\rm R}$-type, we say that $(\mathbf{J},\boldsymbol{\lambda})$ is a $\psi$-generic type if

[TABLE]

We say that it is generic if it is $\psi$-generic for some nondegenerate character of ${\rm N}$.

Remark 5.8**.**

Note that if $\mu$ is a character of ${\rm G}$ and $(\mathbf{J},\boldsymbol{\lambda})$ is a $\psi$-generic type, then $(\mathbf{J},\mu\mid_{\mathbf{J}}\otimes\boldsymbol{\lambda})$ is also $\psi$-generic.

We will also use the following observation later.

Lemma 5.9**.**

If $(\mathbf{J},\boldsymbol{\lambda})$ is an integral $\overline{\mathbb{Q}_{\ell}}$-type and $\psi$ be a nondegenerate character of ${\rm N}$. If $(r_{\ell}(\mathbf{J}),r_{\ell}(\boldsymbol{\lambda}))$ is $r_{\ell}(\psi)$-generic, then $(\mathbf{J},\boldsymbol{\lambda})$ is $\psi$-generic.

Proof.

It is a consequence of Lemma 3.7, once one observes that $\mathbf{J}\cap{\rm N}$ is a (compact) pro-$p$ group. ∎

If the type we consider is moreover $\sigma$-self-dual, we will only consider distinguished nondegenerate characters $\psi$ of ${\rm N}$, i.e. those which are trivial on ${\rm N}_{\mathsf{o}}$:

Definition 5.10**.**

A $\sigma$-self-dual ${\rm R}$-type is called generic if it is $\psi$-generic with respect to a distinguished nondegenerate character $\psi$ of ${\rm N}$.

Our definition of a generic $\sigma$-self-dual type coincides with the definition given in [27, Definition 9.1] (see the discussion after [3, Definition 5.7]). There are two fundamental facts about these types, first they always occur in $\sigma$-self-dual cuspidal representations.

Proposition 5.11**.**

[3, Proposition 5.5]** Let $\pi$ be a $\sigma$-self-dual cuspidal ${\rm R}$-representation of ${\rm G}$ and let $\psi$ be a distinguished nondegenerate character of ${\rm N}$, then $\pi$ has a $\psi$-generic $\sigma$-self-dual type, which is moreover unique up to ${\rm N}_{\mathsf{o}}$-conjugacy.

The second one concerns distinguished representations, which are $\sigma$-self-dual thanks to Proposition 3.3.

Theorem 5.12**.**

[3, Corollary 6.6]**, [27, Theorem 9.3] Let $\pi$ be a $\sigma$-self-dual cuspidal ${\rm R}$-representation and $(\mathbf{J},\boldsymbol{\lambda})$ a generic $\sigma$-self-dual type of $\pi$. Then $\pi$ is distinguished if and only if $\boldsymbol{\lambda}$ is $\mathbf{J}^{\sigma}$-distinguished.

Definition 5.13**.**

We call a $\sigma$-self-dual type $(\mathbf{J},\boldsymbol{\lambda})$ a distinguished type if $\boldsymbol{\lambda}$ is $\mathbf{J}^{\sigma}$-distinguished.

We have the following surprising result, which completes Theorem 5.12 and is evidence of the interplay between genericity and Galois distinction for $\operatorname{GL}_{n}({\rm F})$:

Lemma 5.14**.**

A distinguished ${\rm R}$-type is automatically $\sigma$-self-dual generic.

Proof.

Take a $\sigma$-self-dual type $(\mathbf{J},\boldsymbol{\lambda})$ such that $\boldsymbol{\lambda}$ is $\mathbf{J}^{\sigma}$-distinguished. Then $\operatorname{ind}_{\mathbf{J}}^{{\rm G}}(\boldsymbol{\lambda})$ is distinguished by Mackey theory and Frobenius reciprocity. But then by [3, Remark 6.7] (and the equivalence of our definition of generic type with that given in [27, Defintion 9.1]), the type $(\mathbf{J},\boldsymbol{\lambda})$ must be $\psi$-generic for some character distinguished nondegenerate character $\psi$ of ${\rm N}$. ∎

We end this section with the following important corollary of Proposition 5.4:

Corollary 5.15**.**

A $\sigma$-self-dual type $(\mathbf{J},\boldsymbol{\lambda})$ is a distinguished type if and only if $\operatorname{Hom}_{{\mathbf{J}}^{\sigma}}(\boldsymbol{\kappa},\chi_{\boldsymbol{\kappa}})$ and $\operatorname{Hom}_{{\mathbf{J}}^{\sigma}}(\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})$ are nonzero, in which case both are one dimensional.

5.4. The relative torsion group of a distinguished representation

For the rest of this section we fix $\pi$ a distinguished cuspidal ${\rm R}$-representation and $(\mathbf{J},\boldsymbol{\lambda})$ a distinguished type in $\pi$. We set $\varpi_{{\rm E}_{\mathsf{o}}}=\varpi_{{\rm E}}^{2}$ if ${\rm E}/{\rm E}_{\mathsf{o}}$ is ramified and $\varpi_{{\rm E}_{\mathsf{o}}}=\varpi_{{\rm E}}$ if ${\rm E}/{\rm E}_{\mathsf{o}}$ is unramified. When $e_{\sigma}(\pi)=2$ and $m(\pi)=2r$ is even, we denote by $w$ the element of $\mathbf{J}$ corresponding to $\left(\begin{smallmatrix}0&1_{r}\\ 1_{r}&0\end{smallmatrix}\right)$ as in [27, Lemma 6.19]. We define the relative torsion group of $\pi$ to be the following group:

[TABLE]

Theorem 5.16**.**

Let $\pi$ be a cuspidal distinguished ${\rm R}$-representation of ${\rm G}$ and set $\varpi^{\prime}=\varpi_{E}w$ if  $e_{\sigma}(\pi)=2$ and $m(\pi)$ is even and $\varpi^{\prime}=\varpi_{{\rm E}_{\mathsf{o}}}$ otherwise. Then we have:

1. (i)

Let $\mu_{\mathsf{o}}$ be an unramified character of ${\rm G}_{\mathsf{o}}$, then $\mu_{\mathsf{o}}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if $\mu_{\mathsf{o}}(\varpi^{\prime})=1$. 2. (ii)

Let $\chi_{\mathsf{o}}$ be an unramified character of ${\rm F}_{\mathsf{o}}^{\times}$, then $\chi_{\mathsf{o}}\circ\mathrm{det}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if $\chi_{\mathsf{o}}(\varpi_{\mathsf{o}})^{n/e_{\mathsf{o}}(\pi)}=1$.

Proof.

Note that if $e_{\sigma}(\pi)=2$, then $m(\pi)$ is even or equal to $1$ thanks to Lemma 5.3. Then $\mathbf{J}^{\sigma}$ is generated by $\varpi^{\prime}$ and ${\rm J}^{\sigma}$, thanks to [27, Lemmas 6.18, 6.19 and 8.7]. Let $\mu_{\mathsf{o}}$ be an unramified character of ${\rm G}_{\mathsf{o}}$ and denote by $\mu$ an unramified extension of it to ${\rm G}$, hence $\mu_{\mathsf{o}}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if $\mu\otimes\pi$ is distinguished. Suppose that $\mu\otimes\pi$ is distinguished, then $(\mathbf{J},\boldsymbol{\kappa}\otimes(\mu\otimes\boldsymbol{\tau}))$ is a $\sigma$-self-dual type which is in fact a distinguished type thanks to Remark 5.8 and Theorem 5.12, and conversely if $(\mathbf{J},\boldsymbol{\kappa}\otimes(\mu\otimes\boldsymbol{\tau}))$ is a distinguished type, then $\mu\otimes\pi$ is distinguished. So $\mu_{\mathsf{o}}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if $(\mathbf{J},\boldsymbol{\kappa}\otimes(\mu\otimes\boldsymbol{\tau}))$ is a distinguished type, which is if and only if $\operatorname{Hom}_{\mathbf{J}^{\sigma}}(\mu\otimes\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})$ has dimension $1$ according to Corollary 5.15.

However $\operatorname{Hom}_{\mathbf{J}^{\sigma}}(\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})$ has dimension $1$ by the same corollary, but $\operatorname{Hom}_{{\rm J}^{\sigma}}(\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})$ is already at most one dimensional thanks to Proposition 4.2, so from these multiplicity one statements we deduce that $\mu_{\mathsf{o}}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if $\operatorname{Hom}_{\mathbf{J}^{\sigma}}(\mu\otimes\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})=\operatorname{Hom}_{\mathbf{J}^{\sigma}}(\boldsymbol{\tau},\chi_{\boldsymbol{\kappa}}^{-1})$. Finally this translates as: $\mu_{\mathsf{o}}\in{\rm X}_{\mathsf{o}}(\pi)$ if and only if  $\mu(\varpi^{\prime})=1$ and proves (i).

Now let $\chi_{\mathsf{o}}$ be an unramified chareter of ${\rm F}_{\mathsf{o}}^{\times}$ and let $\chi$ be an unramified character of ${\rm F}^{\times}$ extending it. If $e_{\sigma}(\pi)=2$ then ${\rm F}/{\rm F}_{\mathsf{o}}$ is ramified according to [27, Lemma 4.14].

Suppose that $e_{\sigma}(\pi)=2$ and  $m(\pi)$ is even, we have:

[TABLE]

[TABLE]

thanks to [3, Lemma 5.10]. Otherwise we have:

[TABLE]

[TABLE]

thanks to [3, Lemma 5.10] again. ∎

It has the following two corollaries.

Corollary 5.17**.**

Let $\pi$ be a distinguished cuspidal ${\rm R}$-representation of ${\rm G}$. Write $n/e_{\mathsf{o}}(\pi)=a(\pi)\ell^{r}$ with $a(\pi)$ prime to $\ell$. Then ${\rm X}_{\mathsf{o}}(\pi)$ is a cyclic group, of order $n/e_{\mathsf{o}}(\pi)$ if ${\rm R}=\overline{\mathbb{Q}_{\ell}}$, and of order $a(\pi)$ if ${\rm R}=\overline{\mathbb{F}_{\ell}}$.

Corollary 5.18**.**

Let $\pi$ be a distinguished cuspidal (hence integral) $\overline{\mathbb{Q}_{\ell}}$-representation of ${\rm G}$. Then the homomorphism

[TABLE]

is surjective from ${\rm X}_{\mathsf{o}}(\pi)$ to ${\rm X}_{\mathsf{o}}(r_{\ell}(\pi))$, and its kernel is the $\ell$-singular part of ${\rm X}_{\mathsf{o}}(\pi)$.

Proof.

It suffices to verify the assertion on the kernel. It is clear that the $\ell$-singular part of ${\rm X}_{\mathsf{o}}(\pi)$ belongs to the kernel of $r_{\ell}$. Conversely if $r_{\ell}(\mu_{\mathsf{o}})=1$, write $\mu_{\mathsf{o}}=(\mu_{\mathsf{o}})_{r}(\mu_{\mathsf{o}})_{s}$ with $(\mu_{\mathsf{o}})_{r}$ of order prime to $\ell$ and $(\mu_{\mathsf{o}})_{s}$ of order a power of $\ell$, then $r_{\ell}((\mu_{\mathsf{o}})_{r})=r_{\ell}(\mu_{\mathsf{o}})=1$ so $(\mu_{\mathsf{o}})_{r}=1$ because $r_{\ell}$ induces a bijection between the group of roots of unity of order prime to $\ell$ in $\overline{\mathbb{Q}_{\ell}}^{\times}$ and the group of roots of unity in $\overline{\mathbb{F}_{\ell}}^{\times}$. ∎

6. Relatively banal cuspidal representations of $p$-adic $\operatorname{GL}_{n}$

In [24] and [23], Mínguez and Sécherre single out a class of irreducible representations called banal for which the Zelevinski classification works particularly nicely. For cuspidal representations, the following definition can be given ([24, Remarque 8.15] and [24, Lemme 5.3]).

Definition 6.1**.**

A cuspidal $\overline{\mathbb{F}_{\ell}}$-representation $\pi$ is called banal if $q^{n/e(\pi)}\not\equiv 1[\ell]$.

The following definition is new and is motivated by our cuspidal ${\rm L}$-factor computation later and an analogy with banal cuspidal representations and the Rankin–Selberg computation of [18]. We show in Section 8.3 how it is a natural analogue of banal for the symmetric pair $({\rm G},{\rm G}_{\mathsf{o}},\sigma)$.

Definition 6.2**.**

Let $\pi$ be a distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representation. We say that it is relatively banal if $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\not\equiv 1[\ell]$.

Theorem 5.16 (ii) has the following third consequence:

Corollary 6.3** (of Theorem 5.16).**

Let $\pi$ be a distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representation, it is relatively banal if and only if it is not $|\mathrm{det}(\ \cdot\ )|_{\mathsf{o}}$-distinguished.

Before stating the next lemma, we make the following observation which shows that the statement of the lemma in question (Lemma 6.6) is indeed complete.

Remark 6.4**.**

For any character $\chi$ of ${\rm G}_{\mathsf{o}}$, there are no $\chi$-distinguished cuspidal ${\rm R}$-representations $\pi$ of ${\rm G}$ with $e_{\sigma}(\pi)=2$ when $m(\pi)\geqslant 3$ is odd, by Proposition 5.4 and [27, Lemma 6.9].

While we have defined relatively banal distinguished representations in terms of the invariant $e_{\mathsf{o}}(~{})$, we will use the following equivalent formulation:

Lemma 6.5**.**

Let $\pi$ be a $\sigma$-self-dual cuspidal ${\rm R}$-representation of ${\rm G}$. Let ${\rm E}$ be a $\sigma$-self-dual parameter field for $\pi$. Then

1. (i)

If $e_{\sigma}(\pi)=1$, then $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=q_{{\rm E}_{\mathsf{o}}}^{m(\pi)}$ (and is also equal to $q_{\mathsf{o}}^{n/e(\pi)}$ if ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified and $q^{n/2e(\pi)}$ if ${\rm F}/{\rm F}_{\mathsf{o}}$ is ramified). 2. (ii)

If $e_{\sigma}(\pi)=2$ and $m(\pi)=1$, then $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=q_{{\rm E}_{\mathsf{o}}}$ (and is also equal to $q^{n/e(\pi)}=q_{{\rm E}}$). 3. (iii)

If $e_{\sigma}(\pi)=2$ and $m(\pi)\geqslant 2$ is even, then $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=q_{{\rm E}_{\mathsf{o}}}^{m(\pi)/2}$ (and is also equal to $q^{n/2e(\pi)}=q_{\rm E}^{m(\pi)/2}$).

Proof.

In all cases, we have $q^{n/e(\pi)}=q_{\rm E}^{m(\pi)}$. In case (i), $q_{{\rm E}_{\mathsf{o}}}^{m}$ is the positive square root of $q_{\rm E}^{m}$. However by [3, Lemma 5.10], we have

[TABLE]

If ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified, then $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=q_{\mathsf{o}}^{n/e(\pi)}$ is the positive square root of $q^{n/e(\pi)}$. Now if ${\rm F}/{\rm F}_{\mathsf{o}}$ is ramified, then $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=q^{n/2e(\pi)}_{\mathsf{o}}$ and (i) is proved.

In case (ii) by [3, Lemma 5.10] again, we have

[TABLE]

However $e({\rm F}/{\rm F}_{\mathsf{o}})=2$ by [27, Lemma 4.14] so $e_{\mathsf{o}}(\pi)=e(\pi)$ and $q=q_{\mathsf{o}}$ which proves case (ii).

Finally in case (iii) by [3, Lemma 5.10] again, we have

[TABLE]

and  $e({\rm F}/{\rm F}_{\mathsf{o}})=2$ by [27, Lemma 4.14] so $e_{\mathsf{o}}(\pi)=2e(\pi)$ and $q=q_{\mathsf{o}}$ which proves case (iii). ∎

Immediately, from Remark 6.4 and Lemma 6.5, we have:

Corollary 6.6**.**

A banal distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of ${\rm G}$ is relatively banal.

Remark 6.7**.**

A banal cuspidal $\overline{\mathbb{F}_{\ell}}$-representation is supercuspidal. However, there are relatively banal distinguished cuspidal non-supercuspidal $\overline{\mathbb{F}_{\ell}}$-representations. For example when $n=3$ and $\ell\neq 2$, the non-normalised parabolic induction of the trivial representation of the Borel subgroup has a cuspidal subquotient $\operatorname{St}(3)$ when $q_{\mathsf{o}}^{3}\equiv-1[\ell]$, and when ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified it is relatively banal distinguished (see [19]).

Before proving the main result of this section, it will be useful to know that there are no relatively banal distinguished cuspidal representations $\pi$ when $e_{\sigma}(\pi)=1$ and $m(\pi)$ is even:

Lemma 6.8**.**

Let $\pi$ be a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of ${\rm G}$ which is $\sigma$-self-dual. Suppose that $e_{\sigma}(\pi)=1$, that $m(\pi)$ is even, and $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\not\equiv 1[\ell]$, then $\pi$ is not distinguished.

Proof.

Let $(\mathbf{J},\boldsymbol{\kappa}\otimes\boldsymbol{\tau})$ be a $\sigma$-self-dual generic $\overline{\mathbb{F}_{\ell}}$-type for $\pi$ (Proposition 5.11) with $\sigma$-stable parameter field ${\rm E}$. Suppose $\pi$ is distinguished. Then by Theorem 5.12 we can suppose that $\boldsymbol{\kappa}\otimes\boldsymbol{\tau}$ is distinguished as well. By Proposition 5.4, $\boldsymbol{\tau}$ is $\chi_{\boldsymbol{\kappa}}^{-1}$-distinguished hence $\rho=\boldsymbol{\tau}\mid_{{\rm J}}$ seen as a representation of $\operatorname{GL}_{m(\pi)}(\mathrm{k}_{\rm E})$ is $\chi_{\boldsymbol{\kappa}}^{-1}$-distinguished by the group $\operatorname{GL}_{m(\pi)}(\mathrm{k}_{{\rm E}_{\mathsf{o}}})$, i.e. that $\rho^{\prime}=\chi\otimes\rho$ is distinguished for an extension $\chi$ of $\chi_{\boldsymbol{\kappa}}$ to $\mathrm{k}_{{\rm E}}^{\times}$. Now by Lemma 6.5 and Lemma 3.7, $\rho^{\prime}$ has a distinguished lift, which contradicts Lemma 4.4. ∎

Remark 6.9**.**

Notice that the statement of Lemma 6.8 is not empty as $\sigma$-self-dual representations $\pi$ exist under the hypothesis $e_{\sigma}(\pi)=1$ and $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\not\equiv 1[\ell]$: for example when $n=2$ and ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified the non-normalised parabolic induction of the trivial representation of the Borel subgroup has a cuspidal subquotient $\operatorname{St}(2)$ when $q\equiv-1[\ell]$ which is $\sigma$-self-dual and $e_{\sigma}(\operatorname{St}(2))=1$ as ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified.

Lemma 6.10**.**

Let $(\mathbf{J},\boldsymbol{\lambda})$ be an ${\rm R}$-type such that $\mathbf{J}$ is $\sigma$-stable and put $\pi=\operatorname{ind}_{\mathbf{J}}^{{\rm G}}(\boldsymbol{\lambda})$. If $\boldsymbol{\lambda}\mid_{\rm J}$ is distinguished, then $\pi$ is the unramified twist of a distinguished representation. Conversely suppose that moreover $\boldsymbol{\lambda}=\boldsymbol{\kappa}\otimes\boldsymbol{\tau}$ is generic and that $\boldsymbol{\kappa}$ is distinguished and $\sigma$-self-dual, if $\pi$ is the unramified twist of a distinguished representation, then $\boldsymbol{\tau}\mid_{{\rm J}}$ is distinguished.

Proof.

If $\boldsymbol{\lambda}\mid_{{\rm J}}$ is distinguished, then we can extend $\boldsymbol{\lambda}$ to a distinguished representation $\boldsymbol{\lambda}_{\rm F}$ of ${\rm F}^{\times}{\rm J}$ by setting $\boldsymbol{\lambda}_{\rm F}(\varpi_{\rm F})=1$. The induced representation $\operatorname{ind}_{{\rm F}^{\times}{\rm J}}^{{\rm E}^{\times}{\rm J}}(\boldsymbol{\lambda}_{\rm F})$ is distinguished, and because $\mathbf{J}/{\rm F}^{\times}{\rm J}\simeq\langle\varpi_{\rm E}\rangle/\langle\varpi_{\rm F}\rangle$ is cyclic, all of its irreducible subquotients extend $\boldsymbol{\lambda}_{\rm F}$ by Clifford theory, so one extension $\boldsymbol{\lambda}_{\rm E}$ of $\boldsymbol{\lambda}_{\rm F}$ to $\mathbf{J}$ is distinguished. Hence $\operatorname{ind}_{\mathbf{J}}^{\rm G}(\boldsymbol{\lambda}_{\rm E})$ is distinguished and an unramified twist of $\operatorname{ind}_{\mathbf{J}}^{\rm G}(\boldsymbol{\lambda})$ by Property (T-4).

For the partial converse, by twisting by an unramified character without loss of generality we can suppose that $\pi$ is distinguished (and $\boldsymbol{\kappa}$ is the same). Then $\boldsymbol{\tau}$ is $\sigma$-self-dual thanks to Property (T-7) hence $\boldsymbol{\lambda}$ as well, and it is distinguished because of Theorem 5.12. Then Proposition 5.4 implies that $\boldsymbol{\tau}$, hence $\boldsymbol{\tau}\mid_{\rm J}$ is distinguished. ∎

Relatively banal distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representations enjoy very nice lifting properties:

Theorem 6.11**.**

Let $\pi$ be a cuspidal and distinguished $\overline{\mathbb{F}_{\ell}}$-representation of ${\rm G}$.

1. (i)

Then $\pi$ is relatively banal if and only if all of its lifts are unramified twists of distinguished representations. 2. (ii)

If it is relatively banal, then it has a distinguished lift.

Proof.

Suppose that $\pi$ is relatively banal distinguished. Choose a distinguished type $(\mathbf{J},\boldsymbol{\lambda})$ in $\pi$ and let $\widetilde{\pi}$ be a lift of $\pi$. We can choose a type in $\widetilde{\pi}$ of the form $(\mathbf{J},\widetilde{\boldsymbol{\lambda}})$ with $r_{\ell}(\widetilde{\boldsymbol{\lambda}})=\boldsymbol{\lambda}$ by property (T-10). As $\ell$ is coprime to ${\rm J}^{\sigma}$, we can apply Lemma 3.7 and $\widetilde{\boldsymbol{\lambda}}\mid_{{\rm J}}$ is distinguished because so is $\boldsymbol{\lambda}\mid_{\rm J}$. Hence $\widetilde{\pi}$ is a unramified twist of a distinguished representation by Lemma 6.10 and this proves one implication in (i).

We now prove (ii). Suppose that $\pi$ is relatively banal. By the implication already proved in (i) we know that $\pi$ has a lift $\widetilde{\pi}$ which is $\widetilde{\mu_{\mathsf{o}}}$-distinguished for $\widetilde{\mu_{\mathsf{o}}}$ an unramified character of ${\rm G}_{\mathsf{o}}$. Let $\widetilde{\mu}$ be an unramified character of ${\rm G}$ extending $\widetilde{\mu_{\mathsf{o}}}$, then $\widetilde{\mu}^{-1}\otimes\widetilde{\pi}$ is distinguished. However, because $\pi$ is distinguished, setting $\mu=r_{\ell}(\widetilde{\mu})$, the representation $\mu^{-1}\otimes\pi$ is $\mu_{\mathsf{o}}^{-1}$-distinguished for $\mu_{\mathsf{o}}=r_{\ell}(\widetilde{\mu_{\mathsf{o}}})=\mu\mid_{{\rm G}_{\mathsf{o}}}$. Thanks to Corollary 5.18, $\mu_{\mathsf{o}}$ has a lift $\widetilde{\mu_{\mathsf{o}}}^{\prime}\in{\rm X}_{\mathsf{o}}(\widetilde{\mu}^{-1}\otimes\widetilde{\pi})$. Writing $\mu=\chi\circ\mathrm{det}$, it is possible to extend $\widetilde{\mu_{\mathsf{o}}^{\prime}}$ to an unramified character $\widetilde{\mu^{\prime}}$ of ${\rm G}$ such that if $\mu^{\prime}=r_{\ell}(\mu)=\chi^{\prime}\circ\mathrm{det}$, then $\chi^{\prime}(\varpi)=\chi(\varpi)$: indeed as $\mu$ and $\mu^{\prime}$ both extend $\mu_{\mathsf{o}}$, this is automatic if ${\rm F}/{\rm F}_{\mathsf{o}}$ is unramified, whereas if ${\rm F}/{\rm F}_{\mathsf{o}}$ is ramified $\chi^{\prime}(\varpi)=\pm\chi(\varpi)$ is automatic, and we can always change $\widetilde{\mu^{\prime}}$ so that this sign is $+$. With such choices, the representation $\widetilde{\mu^{\prime}}\widetilde{\mu}^{-1}\otimes\widetilde{\pi}$ is a distinguished lift of $\pi$.

It remains to prove the second implication of (i). Suppose that $\pi$ is not relatively banal, i.e. $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\equiv 1[\ell]$. Suppose, for the sake of contradiction, that all lifts of $\pi$ are distinguished up to an unramified twist and let $\widetilde{\pi}$ be a lift of $\pi$. Under this assumption the argument used to prove (ii) shows that $\pi$ has a distinguished supercuspidal lift $\widetilde{\pi}$. This lift has a distinguished type $(\mathbf{J},\widetilde{\boldsymbol{\kappa}}\otimes\widetilde{\boldsymbol{\tau}})$ with $\chi_{\widetilde{\boldsymbol{\kappa}}}=1$ thanks to Theorem 5.12 and Proposition 5.5, and we set $\boldsymbol{\kappa}=r_{\ell}(\widetilde{\boldsymbol{\kappa}})$ and $\boldsymbol{\tau}=r_{\ell}(\widetilde{\boldsymbol{\tau}})$ so in particular $(\mathbf{J},\boldsymbol{\kappa}\otimes\boldsymbol{\tau})$ is a distinguished type thanks to Remark 3.5 (and also Lemma 5.14). Proposition 5.4 together with Lemma 4.4 imply that if $e_{\sigma}(\pi)=2$, then either $m(\pi)=1$ or it is even, and if $e_{\sigma}(\pi)=1$, then $m(\pi)$ is odd. Then $\boldsymbol{\tau}\mid_{\rm J}$ is distinguished according to Remark 3.5, but the assumption $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}\equiv 1[\ell]$ translated in terms of $\operatorname{GL}_{m(\pi)}(\mathrm{k}_{\rm E})$ thanks to Lemma 6.5 together with Propositions 4.6 (ii), 4.7, 4.8 (ii), 4.6 (ii) and 4.10 imply that $\boldsymbol{\tau}\mid_{\rm J}$ has a non distinguished lift $\widetilde{\boldsymbol{\tau}^{\prime}}$. This lift extends to $\langle\varpi_{\mathsf{o}}\rangle{\rm J}$ to a lift of $\boldsymbol{\tau}\mid_{\langle\varpi_{\mathsf{o}}\rangle{\rm J}}$ by setting $\widetilde{\boldsymbol{\tau}^{\prime}}(\varpi_{\mathsf{o}})=1$. Then by Clifford theory, because the quotient $\mathbf{J}/\langle\varpi_{\mathsf{o}}\rangle{\rm J}\simeq\langle\varpi_{\rm E}\rangle/\langle\varpi_{\mathsf{o}}\rangle$ is cyclic, the representation $\operatorname{ind}_{\langle\varpi_{\mathsf{o}}\rangle{\rm J}}^{\mathbf{J}}(\widetilde{\boldsymbol{\tau}^{\prime}})$ contains a lift of $\boldsymbol{\tau}$ which extends $\widetilde{\boldsymbol{\tau}^{\prime}}$, and we again denote it $\widetilde{\boldsymbol{\tau}^{\prime}}$. Then the representation $\pi^{\prime}=\operatorname{ind}_{\mathbf{J}}^{{\rm G}}(\widetilde{\boldsymbol{\kappa}}\otimes\widetilde{\boldsymbol{\tau}}^{\prime})$ is then a supercuspidal lift of $\pi$. As $\widetilde{\boldsymbol{\kappa}}\otimes\widetilde{\boldsymbol{\tau}}^{\prime}$ reduces to the generic type $\boldsymbol{\kappa}\otimes\boldsymbol{\tau}$, it is generic thanks to Lemma 5.9, hence it can’t be an unramified twist of a distinguished representation according to the second part of Lemma 6.10. ∎

Remark 6.12**.**

Note that as an unramified character of ${\rm G}_{\mathsf{o}}$ always has an unramified extension to ${\rm G}$, Part (i) of Theorem 6.11 can also be stated as: $\pi$ is relatively banal if and only if all its lifts are distinguished by an unramified character.

7. Asai ${\rm L}$-factors of cuspidal representations

7.1. Asai ${\rm L}$-factors

Let ${\rm N}$ be the maximal unipotent subgroup of the subgroup of upper triangular matrices in ${\rm G}$, and ${\rm N}_{\mathsf{o}}={\rm N}^{\sigma}$. Let $\psi$ be a non-degenerate ${\rm R}$-valued character of ${\rm N}$ trivial on ${\rm N}_{\mathsf{o}}$. Let $\pi$ be an ${\rm R}$-representation of ${\rm G}$ of Whittaker type (i.e. of finite length with a one dimensional space of Whittaker functionals) with Whittaker model $\mathcal{W}(\pi,\psi)$. We refer to [17, Section 2] for more details as well as basic facts about Whittaker functions and their analytic behaviour. For ${\rm W}\in\mathcal{W}(\pi,\psi)$ and $\Phi\in\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$ and $l\in\mathbb{Z}$ define the local Asai coefficient to be

[TABLE]

where $\eta_{n}$ denotes the row vector $\left(\begin{smallmatrix}0&\cdots&0&1\end{smallmatrix}\right)$ and $dg$ denotes a right invariant measure on ${\rm N}_{\mathsf{o}}\backslash{\rm G}_{\mathsf{o}}$ with values in ${\rm R}$. We refer the reader to [17, Section 2.2] for details on ${\rm R}$-valued equivariant measures on homogeneous spaces and their properties. The integrand in the Asai coefficient has compact support so it is well-defined and it moreover vanishes for $l<<0$. We define the Asai integral of ${\rm W}\in\mathcal{W}(\pi,\psi)$ and $\Phi\in\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$ to be the formal Laurent series

[TABLE]

In exactly the same way as in [17, Theorem 3.5], we deduce the following lemma:

Lemma 7.3**.**

For ${\rm W}\in\mathcal{W}(\pi,\psi)$ and $\Phi\in\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$, ${\rm I}_{\mathrm{As}}({\rm X},\Phi,{\rm W})\in{\rm R}({\rm X})$ is a rational function. Moreover, as ${\rm W}$ varies in $\mathcal{W}(\pi,\psi)$ and $\Phi$ varies in $\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$ these functions generate a ${\rm R}[{\rm X}^{\pm 1}]$-fractional ideal of ${\rm R}({\rm X})$ independent of the choice of $\psi$.

In the setting of the lemma, it follows that there is a unique generator ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ which is an Euler factor and that it is independent of the character $\psi$. We call ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ the Asai ${\rm L}$-factor of $\pi$.

For an element $s({\rm X})\in\overline{\mathbb{Z}_{\ell}}({\rm X})$ of the form $1/{\rm P}({\rm X})$ with ${\rm P}({\rm X})\in\overline{\mathbb{Z}_{\ell}}[{\rm X}]$ we write $r_{\ell}(s({\rm X}))=1/r_{\ell}{({\rm P}({\rm X}))}$, and if $s^{\prime}({\rm X})\in\overline{\mathbb{Z}_{\ell}}({\rm X})$ is of the form $1/{\rm P}^{\prime}({\rm X})$ with ${\rm P}^{\prime}({\rm X})\in\overline{\mathbb{Z}_{\ell}}[{\rm X}]$ we write $s({\rm X})\mid s^{\prime}({\rm X})$ if ${\rm P}({\rm X})\mid{\rm P}^{\prime}({\rm X})$.

Lemma 7.4**.**

Let $\pi$ be an integral cuspidal $\overline{\mathbb{Q}_{\ell}}$-representations of ${\rm G}$ and $\overline{\pi}$ its reduction modulo $\ell$.

1. (i)

Then ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ is the inverse of a polynomial in $\overline{\mathbb{Z}_{\ell}}[{\rm X}]$. 2. (ii)

Moreover,

[TABLE]

Proof.

The first part (i) is in fact true more general for not-necessarily integral representations of Whittaker type, and follows from the asymptotic expansion of Whittaker functions as in [17, Corollary 3.6]. The second part (ii) follows by imitating the proof of [17, Theorem 3.13], we recall the argument here: By definition, we can write the ${\rm L}$-factor ${\rm L}_{\mathrm{As}}({\rm X},\overline{\pi})$ as a finite sum of Asai integrals: For $i\in\{1,\ldots,r\}$, there are $\Phi_{i}\in\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$ and ${\rm W}_{i}\in\mathcal{W}(\overline{\pi},\overline{\psi})$ such that

[TABLE]

By [17, Lemma 2.23], there are Whittaker functions ${\rm W}_{i,e}\in\mathcal{W}(\pi,\psi)$ which take values in $\overline{\mathbb{Z}_{\ell}}$ such that ${\rm W}_{i}=r_{\ell}({\rm W}_{i,e})$, and clearly there are Schwartz functions $\Phi_{i,e}\in\mathscr{C}_{c}^{\infty}({\rm F}_{\mathsf{o}}^{n})$ which take values in $\overline{\mathbb{Z}_{\ell}}$ such that $\Phi_{i}=r_{\ell}(\Phi_{i,e})$. Moreover,

[TABLE]

hence ${\rm L}_{\mathrm{As}}({\rm X},\overline{\pi})=\sum_{i=1}^{r}{\rm I}_{\mathrm{As}}({\rm X},\Phi_{i},{\rm W}_{i})\in r_{\ell}({\rm L}_{\mathrm{As}}({\rm X},\pi))\overline{\mathbb{F}_{\ell}}[{\rm X}^{\pm 1}]$. ∎

As we shall see later, strict divisions do occur.

7.2. Test vectors

In [3], test vectors for the Asai integral of a distinguished supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation were given with the Asai integral computed explicitly. The pro-order of a compact open subgroup of ${\rm G}_{\mathsf{o}}$ may be zero in $\overline{\mathbb{F}_{\ell}}$, and so one cannot normalise a right Haar measure with values in $\overline{\mathbb{F}_{\ell}}$ arbitrarily. So, for compatibility with reduction modulo $\ell$, we need to be more careful with normalisation of measures over $\overline{\mathbb{Q}_{\ell}}$. We set ${\rm K}=\operatorname{GL}_{n}(\mathscr{O})$ and ${\rm K}_{\mathsf{o}}={\rm K}^{\sigma}$, ${\rm K}_{\mathsf{o}}^{1}={\rm I}_{n}+\mathcal{M}_{n}(\mathscr{O}_{\mathsf{o}})$, and ${\rm P}$ the $\sigma$-stable mirabolic subgroup of ${\rm G}$ of all elements with final row $(\begin{smallmatrix}0&\cdots&0&1\end{smallmatrix})$ and.

Definition 7.5**.**

A triple $(\mathbf{J},\boldsymbol{\lambda},\psi)$ with $(\mathbf{J},\boldsymbol{\lambda})$ a $\sigma$-self-dual ${\rm R}$-type, and $\psi$ a distinguished nondegenerate character of ${\rm N}$ satisfying conditions (i) and (ii) of [3, Lemma 6.8] will be called an adapted type.

Remark 7.6**.**

1. (i)

In particular if $(\mathbf{J},\boldsymbol{\lambda},\psi)$ is an adapted type, the type $(\mathbf{J},\boldsymbol{\lambda})$ is a $\sigma$-self-dual $\psi$-generic type (in particular $\psi$ is distinguished). 2. (ii)

By the proof of [3, Lemma 6.8], if $\pi$ is a $\sigma$-self-dual cuspidal ${\rm R}$-representation, it contains an adapted type, the point of the remark being that the ${\rm N}$ of [ibid.] can be chosen to be our ${\rm N}$: the group of unipotent upper triangular matrices in ${\rm G}$.

Now let $\pi$ be a $\sigma$-self-dual cuspidal ${\rm R}$-representation, and $(\mathbf{J},\boldsymbol{\lambda},\psi)$ be an adapted type of $\pi$. We associate to $(\mathbf{J},\boldsymbol{\lambda},\psi)$ the Paskunas–Stevens Whittaker function ${\rm W}_{\boldsymbol{\lambda}}\in\mathcal{W}(\pi,\psi)$ defined in [3, (6.3)]. Note that, ${\rm W}_{\boldsymbol{\lambda}}$ takes values in in $\overline{\mathbb{Z}_{\ell}}$ as soon as $\pi$ is integral (see for example [18, Lemma 10.2]). One of the main results of [3] is that this Whittaker function is a test vector for the Asai ${\rm L}$-factor:

Proposition 7.7** ([3, Theorem 7.14]).**

Let $\pi$ be a distinguished supercuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of ${\rm G}$, and ${\rm W}_{\boldsymbol{\lambda}}$ be the explicit Whittaker function defined above. There is a unique nomalisation of the invariant measure on ${\rm N}_{\mathsf{o}}\backslash{\rm G}_{\mathsf{o}}$ such that

[TABLE]

The volume of ${\rm N}_{\mathsf{o}}\cap{\rm K}_{\mathsf{o}}^{1}\backslash{\rm K}_{\mathsf{o}}^{1}$ is of the form $p^{l}$ for $l\in\mathbb{Z}$ with this normalisation.

Proof.

We start with Haar measures $dg$ and $dn$ on ${\rm G}_{\mathsf{o}}$ with values in $\overline{\mathbb{Q}_{\ell}}$ normalised by $dg({\rm K}_{\mathsf{o}}^{1})=1$ and $dg({\rm N}_{\mathsf{o}}\cap{\rm K}_{\mathsf{o}}^{1})=1$, which in turns normalises the measure (still denoted $dg$) on the quotient ${\rm N}_{\mathsf{o}}\backslash{\rm G}_{\mathsf{o}}$.

With this normalisation, which is the exact parallel of the normalisation used in [18] for the analogue Rankin-Selberg computation, first of all we get an extra factor of $(q_{\mathsf{o}}-1)$ on the top of the Tate factor defined before [3, Lemma 7.11]. Then there is a factor $dk(({\rm P}^{\sigma}\cap{\rm K}^{\sigma})\backslash{\rm J}^{\sigma})$ which appears in [3, Lemma 6.11], and we have

[TABLE]

As $dk(({\rm P}^{\sigma}\cap({\rm J}^{1})^{\sigma})\backslash({\rm J}^{1})^{\sigma})$ is a (possibly negative) power of $p$ we can renormalise our measure to remove it. The image of ${\rm P}\cap{\rm J}$ modulo ${\rm J}^{1}$ is a $\sigma$-stable mirabolic $\overline{{\rm P}}_{m}(\mathrm{k}_{\rm E})$ of ${\rm J}/{\rm J}^{1}$, and we thus have

[TABLE]

thanks to Lemma 6.5. ∎

Corollary 7.8**.**

Suppose that $\pi$ is an unramified twist of a relatively banal distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representation, and let $\widetilde{\pi}$ be a supercuspidal lift of $\pi$. Then

[TABLE]

Proof.

Let $\widetilde{\pi}$ be such a lift, thanks to Theorem 6.11 there is an unramified character $\widetilde{\chi}$ of ${\rm F}^{\times}$ such that $\widetilde{\pi}_{0}=(\widetilde{\chi}\circ\mathrm{det})^{-1}\otimes\widetilde{\pi}$ is distinguished. Let $(\mathbf{J},\widetilde{\boldsymbol{\lambda}},\widetilde{\psi})$ be an adapted type of $\widetilde{\pi}_{\mathsf{o}}$. Proposition 7.7 then implies that

[TABLE]

Then setting $\pi_{\mathsf{o}}=r_{\ell}(\widetilde{\pi}_{\mathsf{o}})$, we deduce that

[TABLE]

in the exact same way that [18, Corollary 10.1] follows from [18, Proposition 9.3]. We obtain the statement of the corollary by twisting $\widetilde{\pi}_{\mathsf{o}}$ by $\widetilde{\chi}\circ\mathrm{det}$ in this equality as it sends ${\rm X}$ to $\chi(\varpi_{\mathsf{o}}){\rm X}$ on the left hand side and to $\widetilde{\chi}(\varpi_{\mathsf{o}}){\rm X}$ on the right hand side. ∎

7.3. Asai ${\rm L}$-factors of cuspidal representations

We first recall the computation of the Asai ${\rm L}$-function of a cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation:

Proposition 7.9** ([3, Corollary 7.6] and Remark 7.7).**

Let $\pi$ be a cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation. If no unramified twist of $\pi$ is distinguished then ${\rm L}_{\mathrm{As}}({\rm X},\pi)=1$. If $\pi$ is distinguished then

[TABLE]

This gives a complete description in the cuspidal case, as for an unramified character $\chi:{\rm F}^{\times}\rightarrow{\rm K}^{\times}$ we have

[TABLE]

Theorem 7.10**.**

Let $\pi$ be a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of $\operatorname{GL}_{n}({\rm F})$.

1. (i)

If $\pi$ is an unramified twist $(\chi\circ\mathrm{det})\otimes\pi_{0}$ of a relatively banal distinguished representation $\pi_{0}$, then

[TABLE] 2. (ii)

If $\pi$ is not an unramified twist of a relatively banal distinguished representation, then

[TABLE]

Proof.

If $\pi$ is an unramified twist of a relatively banal distinguished representation the statement follows for example from Corollary 7.8 and Proposition 7.9.

If $\pi$ is not an unramified twist of a relatively banal distinguished representation, it has a supercuspidal lift $\widetilde{\pi}$ which is not an unramified twist of a distinguished representation thanks to Theorem 6.11. By Proposition 7.9 we have ${\rm L}_{\mathrm{As}}({\rm X},\widetilde{\pi})=1$, hence ${\rm L}_{\mathrm{As}}({\rm X},\pi)=1$ as ${\rm L}_{\mathrm{As}}({\rm X},\pi)\mid r_{\ell}({\rm L}_{\mathrm{As}}({\rm X},\widetilde{\pi}))$ by Lemma 7.4. ∎

Remark 7.11**.**

Note that when $\ell=2$, then we are in case (ii) of Theorem 7.10 and ${\rm L}_{\mathrm{As}}({\rm X},\pi)=1$. This can also be seen directly from the asymptotics of Whittaker functions. Without entering the details as we don’t need this, the asymptotic expansion of Whittaker functions on the diagonal torus allow one to express the Asai integrals in terms of Tate integrals for ${\rm F}_{\mathsf{o}}^{\times}$, and these Tate integrals are all $1$ because $q=1[2]$ as shown in [22].

8. Distinction and poles of the Asai ${\rm L}$-factor

8.1. Characterisation of the poles of the Asai ${\rm L}$-factor

We are now in position to prove the main results of this paper:

Theorem 8.1**.**

Let $\pi$ be a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation of ${\rm G}$. Then ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ has a pole at ${\rm X}=1$ if and only if $\pi$ is relatively banal distinguished. In this case, the pole is of order $\ell^{r}$ where $n/e_{\mathsf{o}}(\pi)=a\ell^{r}$ with $a$ prime to $\ell$.

Proof.

If ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ has a pole at ${\rm X}=1$, in particular ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ is not equal to $1$, hence the representation $\pi$ is an unramified twist of a relatively banal distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representation $\pi_{0}$, say $\pi=(\chi\circ\mathrm{det})\otimes\pi_{0}$ with $\chi$ an unramified characer of ${\rm F}^{\times}$. Denote by $\chi_{\mathsf{o}}$ the restriction of $\chi$ to ${\rm F}_{\mathsf{o}}^{\times}$, then by Theorem 7.10,

[TABLE]

which has a pole at ${\rm X}=1$ if and only if $\chi_{\mathsf{o}}(\varpi_{\mathsf{o}})^{n/e_{\mathsf{o}}(\pi)}=1$. By Theorem 5.16 this implies that $\chi_{\mathsf{o}}\circ\mathrm{det}$ belongs to ${\rm X}_{\mathsf{o}}(\pi)$, i.e. that $\pi=(\chi\circ\mathrm{det})\otimes\pi_{0}$ is distinguished. The converse is just Theorem 7.10. ∎

Remark 8.2**.**

Note that our proof of Theorem 8.1 is very different from the proof over the field of complex numbers. In the proof above, the direction $\pi$ relatively banal distinguished implies ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ having a pole at $1$ is an immediate consequence of Theorem 7.10, and works for complex representations as well (in which case we consider all cuspidal distinguished $\mathbb{C}$-representations as “relatively banal”) thanks to [3, Corollary 7.6]. Saying this is not enough to claim a proof in the case of complex cuspidal representations different from the original one given in [1, Theorem 1.4], as the way the equality of [3, Corollary 7.6] is obtained is a consequence of [20, Proposition 6.3], which itself follows either from [20, Theorem 3.1] or from [1, Theorem 1.4] and [14, Theorem 4], together with the fact that the poles of the Asai ${\rm L}$-factor are simple in the cuspidal case. However, the first equality in [3, Theorem 7.14] is independant of the results cited above, and it in particular implies that if $\pi$ is a cuspidal distinguished $\mathbb{C}$-representation, its Asai ${\rm L}$-factor has a pole at ${\rm X}=1$. The proof of the other implication that we give also works in the complex case, and is again different from the original proof given in [14, Theorem 4]. Kable shows that if ${\rm L}_{\mathrm{As}}({\rm X},\pi)$ has a pole at ${\rm X}=1$, the rational function $(1-{\rm X}){\rm I}_{\mathrm{As}}(X,{\rm W},\Phi)$ is regular at ${\rm X}=1$ and that up to a nonzero constant independant of ${\rm W}$ and $\Phi$ that its value at ${\rm X}=1$ is given by

[TABLE]

As by assumption the Asai ${\rm L}$-factor has a pole at ${\rm X}=1$ the ${\rm G}^{\sigma}$-invariant linear form

[TABLE]

is nonzero. Note that to adapt this proof to the modular setting with ${\rm R}=\overline{\mathbb{F}_{\ell}}$ we would need to take $1-{\rm X}^{n/e_{\mathsf{o}}(\pi)}$ where he takes $1-{\rm X}$ (this does not matter over $\mathbb{C}$ as both polynomials have a simple zero at ${\rm X}=1$) to get the correct order of the pole, though from Kable’s proof one sees that the natural choice is in fact $1-{\rm X}^{n}$. But we claim that this can’t be done in general, as we shall now see that the local period $\mathcal{L}_{\pi}$, though well defined for cuspidal $\overline{\mathbb{F}_{\ell}}$-representations might vanish even for relatively banal distinguished cuspidal $\overline{\mathbb{F}_{\ell}}$-representations.

8.2. The ${\rm G}_{\mathsf{o}}$-period of cuspidal distinguished representations

Let $\pi$ be a cuspidal distinguished ${\rm R}$-representation, and we still denote by $\psi$ a distinguished nondegenerate character of ${\rm N}$. There are two natural ${\rm G}^{\sigma}$-invariant linear forms on $\mathcal{W}(\pi,\psi)$. The first is

[TABLE]

which is well-defined and nonzero thanks to [28, Chapter III, Theorem 1.1]. Though it does not look ${\rm G}^{\sigma}$-invariant it is by [3, Proposition B.23]. Let $(\mathbf{J},\boldsymbol{\lambda},\psi)$ be an adapted type in $\pi$. A natural test vector for this linear form is the Paskunas-Stevens Whittaker function ${\rm W}_{\boldsymbol{\lambda}}$: we have $\mathcal{P}_{\pi}({\rm W}_{\boldsymbol{\lambda}})\neq 0$ according to the proof of [3, Proposition 6.5].

The second is

[TABLE]

It is ${\rm G}^{\sigma}$-invariant by definition, and well defined, as all ${\rm W}\in\mathcal{W}(\pi,\psi)$ have compact support on ${\rm N}\backslash{\rm P}$, they have compact support of ${\rm Z}{\rm N}\backslash{\rm G}$ thanks to the Iwasawa decomposition ${\rm G}={\rm P}{\rm Z}{\rm K}$. By cuspidal multiplicity one for the pair $({\rm P},{\rm P}^{\sigma})$ ([3, Proposition B.23]), $\mathcal{L}_{\pi}$ is a multiple of $\mathcal{P}_{\pi}$ and the proportionality constant between them turns out to be a very interesting quantity; this scalar is related to the formal degrees of complex discrete series representations of unitary groups (see [4] and Remark 8.4). When ${\rm R}=\overline{\mathbb{Q}_{\ell}}$ the linear form $\mathcal{L}_{\pi}$ is nonzero (Remark 8.2); here, we solve the problem of understanding when $\mathcal{L}_{\pi}$ is nonzero when ${\rm R}=\overline{\mathbb{F}_{\ell}}$:

Theorem 8.3**.**

Let $\pi$ be a cuspidal distinguished $\overline{\mathbb{F}_{\ell}}$-representation of ${\rm G}$, then $\mathcal{L}_{\pi}$ is nonzero if and only if:

1. (i)

$\pi$* is relatively banal.* 2. (ii)

$\ell$* does not divide $e_{\mathsf{o}}(\pi)$.*

Proof.

Thanks to the Iwasawa decomposition ${\rm G}={\rm P}{\rm Z}{\rm K}$, we have the equality:

[TABLE]

We introduce the power series

[TABLE]

where ${{\rm P}^{\sigma}}^{(l)}=\{p\in{\rm P}^{\sigma},\ \operatorname{val}_{{\rm F}_{\mathsf{o}}}(\mathrm{det}(p))=l\}$ which is in fact a Laurent polynomial as $\pi$ is cuspidal, so that

[TABLE]

Now suppose that $\pi$ is not relatively banal, then $\pi$ is $|\mathrm{det}(~{}\cdot~{})|_{\mathsf{o}}$-distinguished, and appealing to [3, Proposition B.23], it means that the linear form

[TABLE]

is $|\mathrm{det}(~{}\cdot~{})|_{\mathsf{o}}$-equivariant under the action of ${\rm G}_{\mathsf{o}}$. So in particular, up to possible renormalisation of the invariant measure,

[TABLE]

as $q_{\mathsf{o}}^{n/e_{\mathsf{o}}(\pi)}=1$.

So it remains to understand what happens when $\pi$ is relatively banal. As we said by multiplicity one we know that $\mathcal{L}_{\pi}=\lambda\mathcal{P}_{\pi}$ and we noticed that $\mathcal{P}_{\pi}({\rm W}_{\boldsymbol{\lambda}})\neq 0$. Hence $\mathcal{L}_{\pi}=0$ if and only if $\mathcal{L}_{\pi}({\rm W}_{\boldsymbol{\lambda}})=0$. However, following the proof of [18, Theorem 9.1] at the end of p.19 of [ibid.] or the proof of [3, Theorem 7.14], one gets up to a possible renormalisation of invariant measures:

[TABLE]

Now the value at ${\rm X}=1$ of $\mathcal{L}_{\pi,{\rm X}}$ is $\mathcal{L}_{\pi}$, so $\mathcal{L}_{\pi}$ vanishes if and only if $\frac{1-{\rm X}^{n}}{1-{\rm X}^{n/e_{\mathsf{o}}(\pi)}}$ vanishes at ${\rm X}=1$. However the order of the zero of $1-{\rm X}^{n}$ is the $\ell^{\mathrm{val}_{\ell}(n)}$ whereas that of of the zero of $1-{\rm X}^{n/e_{\mathsf{o}}(\pi)}$ is $\ell^{\mathrm{val}_{\ell}(n/e_{\mathsf{o}}(\pi))}$. This means that if $\pi$ is relatively banal, $\mathcal{L}_{\pi}$ is nonzero if and only if $\mathrm{val}_{\ell}(n)=\mathrm{val}_{\ell}(n/e_{\mathsf{o}}(\pi))$, i.e. if and only if $\ell$ does not divide $e_{\mathsf{o}}(\pi)$. ∎

Remark 8.4**.**

Here we explain how this vanishing result modulo $\ell$ is related to the vanishing of the $\ell$-adic proportionality constant between $\mathcal{L}$ and $\mathcal{P}$. For an algebraically closed field $\mathbf{C}$, write $\mathrm{Cusp}_{\mathbf{C},\text{dist}}({\rm G})$ for the set of isomorphism classes of distinguished cuspidal $\mathbf{C}$-representations. Fix an isomorphism $\mathbb{C}\simeq\overline{\mathbb{Q}_{\ell}}$, this induces a bijection $\mathrm{Cusp}_{\mathbb{C},\text{dist}}({\rm G})\rightarrow\mathrm{Cusp}_{\overline{\mathbb{Q}_{\ell}},\text{dist}}({\rm G})$, independent of the choice of isomorphism.

Let $\pi\in\mathrm{Cusp}_{\overline{\mathbb{Q}_{\ell}},\text{dist}}({\rm G})$ and $\psi:{\rm N}\rightarrow\overline{\mathbb{Z}_{\ell}}^{\times}$ be an ${\rm N}^{\sigma}$-distinguished nondegenerate character of ${\rm N}$. Then by [1, Corollary 1.2], there exists $\mu\in\overline{\mathbb{Q}_{\ell}}$ such that

[TABLE]

Let $c_{\pi}$ denote the central character of $\pi$ and $\operatorname{Res}_{\rm P}$ denote the restriction of Whittaker functions to ${\rm P}$. Then

[TABLE]

the first fact being a consequence of the second, which has been known since [5]. Now let $\mathcal{W}(\pi,\psi)_{e}$ denote the $\overline{\mathbb{Z}_{\ell}}$-submodule of $\mathcal{W}(\pi,\psi)$ consisting of Whittaker functions with values in $\overline{\mathbb{Z}_{\ell}}$, it follows from [30, Theorem 2] and [29, Theorem 2] that $\mathcal{W}(\pi,\psi)_{e}$ is a lattice in $\mathcal{W}(\pi,\psi)$ and $\operatorname{Res}_{{\rm P}}(\mathcal{W}(\pi,\psi)_{e})=\operatorname{ind}_{{\rm N}}^{{\rm P}}(\psi,\overline{\mathbb{Z}_{\ell}})$ is a lattice in $\operatorname{Res}_{{\rm P}}(\mathcal{W}(\pi,\psi))$, reducing to $\mathcal{W}(\pi,\psi)$ and $\operatorname{Res}_{{\rm P}}(\mathcal{W}(\pi,\psi))$ respectively.

Finally from [17, Section 2.2], there are appropriate $\ell$-adic and $\ell$-modular invariant measures on ${\rm Z}^{\sigma}{\rm N}^{\sigma}\backslash{\rm G}^{\sigma}$ and ${\rm N}^{\sigma}\backslash{\rm P}^{\sigma}$ such that

[TABLE]

for all ${\rm W}_{e}\in{\rm W}(\pi,\psi)_{e}$ and $\mathcal{P}_{\pi}(\operatorname{Res}_{{\rm P}}(\mathcal{W}(\pi,\psi)_{e}))=\overline{\mathbb{Z}_{\ell}}$.

Evaluating Equation 8.5 on an element ${\rm W}_{e}\in\mathcal{W}(\pi,\psi)_{e}$ such that $\mathcal{P}_{\pi}({\rm W}_{e})=1$ we deduce that $\mu\in\overline{\mathbb{Z}_{\ell}}$. Now Theorem 8.3 and Equations 8.6 and 8.7 imply that $\ell$ divides $\mu$ if and only if either $r_{\ell}(\pi)$ is not relatively banal or $\ell\mid e_{\mathsf{o}}(\pi)$. Running over all $\ell\neq p$, we recover the radical of the $p$-regular part of $\mu$, (explicitly using the type theoretic definition of relatively banal, Definition 6.2).

As mentioned already, this scalar $\mu$ is a very interesting and subtle quantity: by [4, Theorem 7.1], we have

[TABLE]

where $\lambda$ is a constant independent of $\pi$, $\rho$ is the cuspidal $\overline{\mathbb{Q}_{\ell}}$-representation of the quasi-split unitary group in $n$-variables defined over ${\rm F}_{\mathsf{o}}$ which base changes to $\pi$ (stably or unstably depending on the parity of $n$), and $d(\rho)$ and $d(\pi)$ denote the formal degrees of $\rho$ and $\pi$ respectively under the normalisation of invariant measures of [12]. One could check that the formal degrees are rational for our well chosen measures (and $\lambda$ as well), and preserved under the bijection $\mathrm{Cusp}_{\mathbb{C},\text{dist}}({\rm G})\rightarrow\mathrm{Cusp}_{\overline{\mathbb{Q}_{\ell}},\text{dist}}({\rm G})$.

While we have explained how Theorem 8.3 tells us exactly when $\mu$ vanishes modulo $\ell$, we could also go in the other direction: By [12] and [6], the constant $\mu$ could be computed explicitly and its explicit description would give a different proof of Theorem 8.3. It should be clear to the reader that the amount of work required for such a proof is much more considerable than that of the proof given above.

8.3. Comparison of banal and relatively banal

Finally, we compare our notion of relatively banal distinguished with the notion of banal representation introduced in [24] for cuspidal representations.

By [24, Remarque 8.15], a cuspidal $\overline{\mathbb{F}_{\ell}}$-representation $\pi$ of ${\rm G}_{\mathsf{o}}$ is banal if and only if

[TABLE]

However the map

[TABLE]

is a bijection between the set of (isomorphism classes of) irreducible representations of ${\rm G}_{\mathsf{o}}$ and the set of $\Delta({\rm G}_{\mathsf{o}})$-distinguished irreducible representations of ${\rm G}^{\prime}={\rm G}_{\mathsf{o}}\times{\rm G}_{\mathsf{o}}$, where $\Delta$ is the diagonal embedding of ${\rm G}_{\mathsf{o}}$ into ${\rm G}^{\prime}$. In particular, $\pi$ (seen as the distinguished representation $b(\pi)$ of ${\rm G}^{\prime}$) is banal if and only if $|\mathrm{det}(\ )|_{\mathsf{o}}\otimes b(\pi)=(|\mathrm{det}(\ )|_{\mathsf{o}}\otimes\pi)\otimes\pi^{\vee}$ is not distinguished. Note that $|\ |_{\mathsf{o}}$ plays the same role for the split quadratic algebra $({\rm F}_{\mathsf{o}}\times{\rm F}_{\mathsf{o}})/{\rm F}_{\mathsf{o}}$ that $|\ |_{\mathsf{o}}$ plays for ${\rm F}/{\rm F}_{\mathsf{o}}$, i.e. it is a square root of the absolute value on the bigger algebra. So this proves the exact analogy of banal cuspidal representations of ${\rm G}_{\mathsf{o}}$ and relatively banal distinguished cuspidal representations of ${\rm G}$ according to Corollary 6.3.

The analogy can also be seen at the ${\rm L}$-factor level: it follows from [18, Theorem 4.9] that if $\pi\otimes\pi^{\prime}$ is a cuspidal representation of ${\rm G}^{\prime}$, then the Rankin–Selberg ${\rm L}$-factor ${\rm L}({\rm X},\pi,\pi^{\prime})$ (which can be thought of as the Asai ${\rm L}$-factor of $\pi\otimes\pi^{\prime}$) has a pole at ${\rm X}=1$ if and only if $\pi\otimes\pi^{\prime}$ is $\Delta({\rm G}_{\mathsf{o}})$-distinguished and $\pi$ is banal, which is the exact analogue of Theorem 8.1 replacing banal with relatively banal.

Finally, in terms of the type theory definition, a cuspidal representation $\pi$ of ${\rm G}_{\mathsf{o}}$ is banal if and only if $q_{\mathsf{o}}^{n/e(\pi)}\not\simeq 1[\ell]$, but tracking down how $e(\pi)$ is defined with respect to $\pi$ in terms of type theory (more precisely lattice periods) shows that it plays the same role for the $\Delta({\rm G}_{\mathsf{o}})$-distinguished representation $b(\pi)$ of ${\rm G}^{\prime}$ that $e_{\mathsf{o}}(\tau)$ plays for a distinguished cuspidal representation $\tau$ of ${\rm G}$.