Unramified Whittaker functions for certain Brylinski-Deligne covering groups

TL;DR

This paper computes specific values of unramified Whittaker functions for certain theta-like representations of Brylinski-Deligne covering groups of general linear groups.

Contribution

It provides explicit calculations of Whittaker functions for a class of representations in Brylinski-Deligne covering groups, extending understanding of their harmonic analysis.

Findings

Explicit formulas for unramified Whittaker functions.

Identification of representations analogous to theta representations.

Enhanced understanding of harmonic analysis on covering groups.

Abstract

For a Brylinski-Deligne covering group of a general linear group, we calculate some values of unramified Whittaker functions for certain representations that are analogous to the theta representations.

Equations

$$\{X,\Phi ,\Delta ;Y,{\Phi }^{\vee },{\Delta }^{\vee }\}$$

$$\{e_{\alpha }:{\mathbb{G}}_a\to {\mathbb{U}}_{\alpha }{\}}_{\alpha \in \Phi },$$

$$D:Y\times Y\to \mathbb{Z}$$

$$B_Q(y_1,y_2)=Q(y_1+y_2)-Q(y_1)-Q(y_2),$$

$$D(y_1,y_2)+D(y_2,y_1)=B_Q(y_1,y_2).$$

$${\mu }_n\hookrightarrow \overline{G}\twoheadrightarrow G.$$

$$\mathbf{s}(y_1(a))\cdot \mathbf{s}(y_2(b))=(a,b{)}_n^{D(y_1,y_2)}\cdot \mathbf{s}(y_1(a)\cdot y_2(b)).$$

$${\overline{h}}_{\alpha }(a)=(\eta ({\alpha }^{\vee }),a{)}_n\cdot \mathbf{s}(h_{\alpha }(a))\in \overline{T}.$$

$${\mathbf{\sigma }}_{\alpha }\cdot \overline{y(a)}\cdot {\mathbf{\sigma }}_{\alpha }^{-1}=\overline{y(a)}\cdot {\overline{h}}_{\alpha }(a^{-⟨y,\alpha ⟩}),$$

$$w\cdot {\mathbf{s}}_y\cdot w^{-1}={\mathbf{s}}_{\mathbf{w}(y)}.$$

$$Y_{Q,n}:=\{y\in Y:B_Q(y,y^{\prime })\in n\mathbb{Z}\}$$

$$n_{\alpha }:=n/\gcd (n,Q({\alpha }^{\vee })).$$

$$(Y_{Q,n},\{{\alpha }_{Q,n}^{\vee }\},\mathrm{Hom}(Y_{Q,n},\mathbb{Z}),\{{\alpha }_{Q,n}\}).$$

$$\mu (O_F^{\times })=1-1/q.$$

$$G_{\psi }(a,b)={\int }_{O_F^{\times }}(u,\varpi {)}_n^a\cdot \psi ({\varpi }^{b}u)\mu (u),a,b\in \mathbb{Z}.$$

$$G_{\psi }(a,b)=\{\begin{array}{ll}0,& \text{mbox}ifb<-1\\ 1-1/q,& \text{mbox}ifn\mid a,b\ge 0\\ 0,& \text{mbox}ifn\nmid a,b\ge 0\\ -1/q,& \text{mbox}ifn\mid a,b=-1\\ G_{\psi }(a,-1)\text{ with }|G_{\psi }(a,-1)|=q^{-1/2},& \text{mbox}ifn\nmid a,b=-1.\end{array}$$

$${\mathbf{g}}_{\psi }(k):=G_{\psi }(k,-1).$$

$$\mathbf{w}[y]:=\mathbf{w}(y-\rho )+\rho .$$

$${\psi }_a:x\mapsto \psi (ax).$$

$${\mu }_n\hookrightarrow {\overline{Y}}_{Q,n}\twoheadrightarrow Y_{Q,n}$$

$$i(\chi {)}^{\overline{T}\cap K}\simeq I(\overline{\chi }{)}^K.$$

$$(T_{\mathbf{w},\overline{\chi }}f)(\overline{g})={\int }_{U_w}f(w^{-1}u\overline{g})du$$

$$T_{{\mathbf{\sigma }}_{\alpha },\overline{\chi }}({\varphi }_K)=c({\mathbf{\sigma }}_{\alpha },\overline{\chi })\cdot {\varphi }_K^{{\mathbf{\sigma }}_{\alpha }}$$

$$c({\mathbf{\sigma }}_{\alpha },\overline{\chi })=\frac{1-q^{-1}\overline{\chi }({\overline{h}}_{\alpha }({\varpi }^{n_{\alpha }}))}{1-\overline{\chi }({\overline{h}}_{\alpha }({\varpi }^{n_{\alpha }}))}.$$

$$\Phi (\mathbf{w}):=\{\alpha \in \Phi :\alpha >0\text{ and }\mathbf{w}(\alpha )<0\}.$$

$$c(\mathbf{w},\chi )=\prod _{\alpha \in \Phi (\mathbf{w})}c({\mathbf{\sigma }}_{\alpha },\chi )$$

$$\mathbf{c}(\overline{t}\cdot \overline{z})=\mathbf{c}(\overline{t})\cdot \overline{\chi }(\overline{z}),\overline{t}\in \overline{T}\text{ and }\overline{z}\in \overline{A}.$$

$${\lambda }_{{\gamma }_i}^{\overline{\chi }}(f_{{\gamma }_j})={\delta }_{ij},$$

$$f_{{\gamma }_j}=i(\overline{\chi })({\gamma }_j){\varphi }_K.$$

$$\mathbf{c}\mapsto {\lambda }_{\mathbf{c}}^{\overline{\chi }}:=\sum _{{\gamma }_i\in \overline{T}\backslash \overline{A}}\mathbf{c}({\gamma }_i){\lambda }_{{\gamma }_i}^{\overline{\chi }}.$$

Full text

This research was supported by the ERC, StG grant number 637912.

1. 1 Introduction
2. 2 Preliminaries
3. 3 Unramified principal series representations
4. 4 An inductive formula
5. 5 Relative theta representations
6. 6 The case of general linear groups
7. 7 Calculation of certain local matrix coefficients
8. 8 Main result

1. Introduction

The unramified Whittaker functions and their analogues play an important role in modern number theory, arising naturally as terms in the Fourier coefficients of automorphic forms. It is generally desirable to calculate explicit values for these functions, as the information proves useful in many aspects of study related to the automorphic form (for example, in the construction of associated $L$-functions). When an automorphic representation possesses a Whittaker model or another suitable unique model, the method described in [CS80] may be used to compute an explicit formula (the Casselman-Shalika formula) for the values of the unramified Whittaker function (or the analogous function).

In this paper, we consider representations of Brylinski-Deligne covering groups. For these groups, the uniqueness of Whittaker models fails in general. This causes obstructions to some advancement of the theory. Nevertheless, in the past decades, it is discovered that Fourier coefficients of Eisenstein series on covering groups are closely tied to the Weyl group multiple Dirichlet series. This leads to several generalizations of the Casselman-Shalika formula to the covering group setup. One is to interpret the value of an unramified Whittaker function as a weighted sum over a crystal graph. In this vein, this beautiful idea is realized in [BBF11, McN11, FZ15] for root systems of type $A$ and $C$. The other description is to express the value as the average of a Weyl group action. This approach is closer to the one of Casselman-Shalika and is successful for all types of root systems (see [CO13, McN16, CG10]). In the linear case, the equivalence of these two descriptions is a formula of Tokuyama.

However, the formulas mentioned above are not explicit to work with. To seek applications towards the theory of automorphic forms on covering groups, we would like to have a formula analogous to the original Casselman-Shalika formula. At the moment, we believe that this is impossible in general. Thus, in this paper, we would like to consider the following weaker question:

• •

For representations on covering groups with additional features (for example, theta representations), is it possible to give a simple formula for some values of the unramified Whittaker functions?

In this paper, we address this question for Brylinski-Deligne covering groups of general linear groups. We give an answer to this question for a family of representations, that can be viewed as analogues of the theta representations. Such representations were also studied in [Suz97, Suz98], and a formula was successfully obtained in some cases. Our results generalize part of Suzuki’s results.

Let $\mathbb{G}=\mathbb{GL}_{r}$ over a local non-Archimedean field $F$ and $\overline{G}$ be the degree $n$ Brylinski-Deligne covering group arising from a $\mathbb{K}_{2}$-extension $\overline{\mathbb{G}}$ of $\mathbb{G}$. Let $\mathbb{M}$ be a Levi subgroup of $\mathbb{G}$. Let $I(\overline{\chi})$ be an unramified principle series representation of $\overline{G}$. Suppose that $\overline{\chi}$ is an “anti-exceptional character in $\mathbb{M}$” (Definition 5.2). Let $\mathbf{w}_{M}$ be the long element in the Weyl group $W(\mathbb{M})$. Define $\Theta(\overline{G}/\overline{M},\overline{\chi})$ as the image of the intertwining operator $I(\overline{\chi})\to I({}^{\mathbf{w}_{M}}\overline{\chi})$ (Sect. 5). Let $\mathcal{W}_{0}(\overline{g},\overline{\chi})$ be an unramified Whittaker function in a certain Whittaker model of $\Theta(\overline{G}/\overline{M},\overline{\chi})$. Let $e$ be the identity element in $\overline{G}$.

Theorem 1.1** (Theorem 8.1).**

With the above notations and certain assumptions on the rank of $\mathbb{M}$ and the degree of $\overline{G}$, $\mathcal{W}_{0}(e,\overline{\chi})$ is a product of a certain Gauss sum and a polynomial in terms of ‘Satake parameters’ of $I(\overline{\chi})$.

When $\mathbb{M}=\mathbb{G}$, then $\Theta(\overline{G}/\overline{M},\overline{\chi})$ is the theta representation studied in [KP84, Gao17]. When $\mathbb{M}$ has up to two factors, such results are obtained in [Suz97, Suz98]. Our proof uses ideas in these two papers.

To generalize the results in Suzuki’s papers to our setup, another idea is required. That is to utilize the crystal graph description as a key input. This idea was already used in [Kap19] Theorem 43. Here we extend it to a slightly more general setup.

For small rank symplectic groups, similar formulas were obtained in [Gao18b]. It will be interesting to see whether the method in this paper can be extended to other groups.

We now give an outline of this paper. Sect. 2 gives preliminary results on the Brylinski-Deligne covering groups. We introduce the unramified principal series representations and the Casselman-Shalika formula in Sect. 3. We then prove an inductive formula for unramified Whittaker functions in Sect. 4. Such results were obtained by Suzuki in type $A$ and here we extend it to all types. We then introduce the representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$, which we call the relative theta representation (Sect. 5). In Sect. 6, we specialize our results to the case of general linear groups. We calculate a crucial local matrix coefficient in Sect. 7. This is where the ideas of Suzuki are used. In Sect. 8, we state our main results and give a proof. We also add simple examples to help the reader understand the ideas. As the area of covering groups is of deep nature, we either give reliable references or reproduce the necessary proofs here. We also try to fill gaps in past literatures as much as possible.

Acknowledgments

The author would like to thank Solomon Friedberg and Eyal Kaplan for explaining to him that his original approach did not work. The author would also like to thank the Institute for Mathematical Sciences at the National University of Singapore, where part of this work was done during a visit from December 2018 to January 2019.

2. Preliminaries

We first recall some structural facts on the Brylinski-Deligne covering groups [BD01, GG18]. In this paper, we concentrate exclusively on unramified Brylinski-Deligne covering groups. We use [Gao17] as our main reference.

2.1. $\mathbb{K}_{2}$-extensions

Let $F$ be a non-Archimedean field of characteristic [math], with residual characteristic $p$. Let $O_{F}$ be the ring of integers. Fix a uniformizer $\varpi$ of $F$. Let $\mathbb{G}$ be a split connected linear algebraic group over $F$ with maximal split torus $\mathbb{T}$. Let

[TABLE]

be the based root datum of $\mathbb{G}$. Here $X$ (resp. $Y$) is the character lattice (resp. cocharacter lattice) for $(\mathbb{G},\mathbb{T})$. Choose a set $\Delta\subset\Phi$ of simple roots from the set of roots $\Phi$, and $\Delta^{\vee}$ the corresponding simple coroots from $\Phi^{\vee}$. Write $Y^{sc}\subset Y$ for the sublattice generated by $\Phi^{\vee}$. Let $\mathbb{B}=\mathbb{T}\mathbb{U}$ be the Borel subgroup associated with $\Delta$. Denote by $\mathbb{U}^{-}\subset\mathbb{G}$ the unipotent subgroup opposite to $\mathbb{U}$.

Fix a Chevalley system of pinnings for $(\mathbb{G},\mathbb{T})$, that is, we fix a set of compatible isomorphisms

[TABLE]

where $\mathbb{U}_{\alpha}\subset\mathbb{G}$ is the root subgroup associated with $\alpha$. In particular, for each $\alpha\in\Phi$, there is a unique morphism $\varphi_{\alpha}:\mathbb{SL}_{2}\to\mathbb{G}$ which restricts to $e_{\pm\alpha}$ on the upper and lower triangular subgroup of unipotent matrixes of $\mathbb{SL}_{2}$.

Denote by $W=W(\mathbb{G})$ the Weyl group of $(\mathbb{G},\mathbb{T})$, which we identify with the Weyl group of the coroot system. In particular, $W$ is generated by simple reflections $\{\boldsymbol{\sigma}_{\alpha}:\alpha^{\vee}\in\Delta^{\vee}\}$ for $Y\otimes\mathbb{Q}$. Let $\ell:W\to\mathbb{N}$ be the length function. Let $\mathbf{w}_{G}$ be the longest element in $W$.

Consider the algebro-geometric $\mathbb{K}_{2}$-extension $\overline{\mathbb{G}}$ of $\mathbb{G}$, which is categorically equivalent to the pairs $\{(D,\eta)\}$ (see [GG18] Section 2.6). Here $\eta:Y^{sc}\to F^{\times}$ is a homomorphism. On the other hand,

[TABLE]

is a bisector associated to a Weyl-invariant quadratic form $Q:Y\to\mathbb{Z}$. That is, let $B_{Q}$ be the Weyl-invariant bilinear form associated to $Q$ such that

[TABLE]

then $D$ is a bilinear form on $Y$ satisfying

[TABLE]

The bisection $D$ is not necessarily symmetric. Any $\overline{\mathbb{G}}$ is, up to isomorphism, incarnated by (i.e. categorically associated to) a pair $(D,\eta)$ for a bisector $D$ and $\eta$.

2.2. Topological covering

Let $n\geq 1$ be a natural number. Assume that $F^{\times}$ contains the full group $\mu_{2n}$ of $2n$-th roots of unity and $p\nmid n$. With this assumption, $(\varpi,\varpi)_{n}=1$ for the Hilbert symbol $(\cdot,\cdot)_{n}$. This fact is crucial for several results later.

Let $\overline{\mathbb{G}}$ be incarnated by $(D,\eta)$. One naturally obtains degree $n$ topological covering groups $\overline{G},\overline{T},\overline{B}$ of rational points $G:=\mathbb{G}(F),T:=\mathbb{T}(F),B:=\mathbb{B}(F)$, such as

[TABLE]

We may write $\overline{G}^{(n)}$ for $\overline{G}$ to emphasize the degree of covering. For any subset $H\subset G$, we write $\overline{H}\subset\overline{G}$ for the preimage of $H$ with respect to the quotient map $\overline{G}\to G$. The Bruhat-Tits theory gives a maximal compact subgroup $K\subset G$, which depends on the fixed pinnings. We assume that $\overline{G}$ splits over $K$ and fixes such a splitting; the group $\overline{G}$ is called an unramified Brylinski-Deligne covering group in this case. We remark that if the derived group of $\mathbb{G}$ is simply connected, then $\overline{G}$ splits over $K$ (see [GG18] Theorem 4.2). On the other hand, there is a certain double cover of $\mathrm{PGL}_{2}$ where the splitting does not exist (see [GG18], Sect. 4.6).

The data $(D,\eta)$ play the following role for the structural fact on $\overline{G}$:

• •

The group $\overline{G}$ splits canonically over any unipotent element of $G$. In particular, we write $\overline{e}_{\alpha}(u)\in\overline{G},\ \alpha\in\Phi,\ u\in F$ for the canonical lifting of $e_{\alpha}(u)\in G$. For any $\alpha\in\Phi$, there is a natural representative $\sigma_{\alpha}:=e_{\alpha}(1)e_{-\alpha}(-1)e_{\alpha}(1)\in K$ (and therefore $\overline{\sigma}_{\alpha}\in\overline{G}$ by the splitting of $K$) of the Weyl element $\boldsymbol{\sigma}_{\alpha}\in W$. For a general Weyl group element $\mathbf{w}$, one can find a lift $w\in\overline{G}$ based on a reduced decomposition of $\mathbf{w}$. This lift does not depend on the choice of reduced decomposition. We refer to [Gao18c] Sect. 6.1 for a detailed discussion on this matter. Moreover, for $h_{\alpha}(a):=\alpha^{\vee}(a)\in G,\alpha\in\Phi,a\in F^{\times}$, there is a natural lifting $\overline{h}_{\alpha}(a)\in\overline{G}$ of $h_{\alpha}(a)$, which depends only on the pinning and the canonical unipotent splitting ([GG18] Sect. 4.6).

• •

There is a section $\mathbf{s}$ of $\overline{T}$ over $T$ such that the group law on $\overline{T}$ is given by

[TABLE]

Moreover, for the natural lifting $\overline{h}_{\alpha}(a)$, one has

[TABLE]

• •

Let $\boldsymbol{\sigma}_{\alpha}\in G$ be the natural representative of $\boldsymbol{\sigma}_{\alpha}\in W$. For any $\overline{y(a)}\in\overline{T}$,

[TABLE]

where $\langle\ ,\ \rangle$ is the pairing between $Y$ and $X$.

We recall the following lemma.

Lemma 2.1** ([Gao18a] Lemma 2.1).**

For all $y\in Y$,

[TABLE]

Define the sublattice

[TABLE]

of $Y$. For every $\alpha^{\vee}\in\Phi^{\vee}$, define

[TABLE]

Write $\alpha_{Q,n}^{\vee}:=n_{\alpha}\alpha^{\vee},\alpha_{Q,n}:=n_{\alpha}^{-1}\alpha$. Let $Y_{Q,n}^{sc}\subset Y$ be the sublattice generated by $\{\alpha_{Q,n}^{\vee}\}_{\alpha\in\Phi}$. The complex dual group $\overline{G}^{\vee}$ for $\overline{G}$ as given in [FL10, McN12, Rei12] has root data

[TABLE]

In particular, $Y_{Q,n}^{sc}$ is the root lattice for $\overline{G}^{\vee}$.

2.3. Gauss sum

Consider the Haar measure $\mu$ of $F$ such that $\mu(O_{F})=1$. Thus,

[TABLE]

The Gauss sum is given by

[TABLE]

It is known that

[TABLE]

Let $\varepsilon:=(-1,\varpi)_{n}\in\mathbb{C}^{\times}$. One has $\overline{G_{\psi}(a,b)}=\varepsilon^{a}\cdot G_{\psi}(-a,b)$. For any $k\in\mathbb{Z}$, we write

[TABLE]

2.4. Actions

Let $\rho=\frac{1}{2}\sum_{\alpha\in\Phi^{+}}\alpha^{\vee}$. We define an action of $W$ on $Y\otimes\mathbb{Q}$, which we denote by $\mathbf{w}[y]$ by

[TABLE]

If we write $y_{\rho}:=y-\rho$ for any $y\in Y$, then $\mathbf{w}[y]-y=\mathbf{w}(y_{\rho})-y_{\rho}$. From now on, by Weyl orbits in $Y$ or $Y\otimes\mathbb{Q}$ we always refer to the ones with respect to the action $\mathbf{w}[y]$. Note that here $0\in Y$ is a vector. The size of this vector is always clear in the context, and we hope that this does not arise any confusion.

We now list some other notations which appear frequently in the text:

• •

$\psi$: a fixed additive character of $F\to\mathbb{C}^{\times}$ with conductor $O_{F}$. For any $a\in F^{\times}$, the twisted character $\psi_{a}$ is given by

[TABLE]

• •

$\mathbf{s}_{y}:$ for any $y\in Y$, we write $\mathbf{s}_{y}:=\mathbf{s}(\varpi^{y})\in\overline{T}$.

• •

$\lceil x\rceil$: the minimum integer such that $\lceil x\rceil\geq x$ for a real number $x$.

• •

$\lfloor x\rfloor$: the maxmial integer such that $\lfloor x\rfloor\leq x$ for a real number $x$.

• •

$\overline{\chi}_{\alpha}$: for an unramified character $\overline{\chi}$, we sometimes write $\overline{\chi}_{\alpha}=\overline{\chi}(\overline{h}_{\alpha}(\varpi^{n_{\alpha}}))$.

• •

$y\sim y^{\prime}$: if $y,y^{\prime}\in Y$, we write $y\sim y^{\prime}$ if there exists $\mathbf{w}\in W$ such that $y^{\prime}=\mathbf{w}[y]$.

• •

$\mathrm{id}\in W$: the identity element in $W$.

• •

$\overline{\chi}\sim(\overline{\chi}_{1},\cdots,\overline{\chi}_{k})$: see Sect. 4.3.

3. Unramified principal series representations

Fix an embedding $\iota:\mu_{n}\hookrightarrow\mathbb{C}^{\times}$. A representation of $\overline{G}$ is called $\iota$-genuine if $\mu_{n}$ acts via $\iota$. We consider throughout the paper $\iota$-genuine (or simply genuine) representations of $\overline{G}$.

Let $U$ be the unipotent subgroup of $B=TU$. As $U$ splits canonically in $\overline{G}$, we have $\overline{B}=\overline{T}U$. The covering torus $\overline{T}$ is a Heisenberg group with center $Z(\overline{T})$. The image of $Z(\overline{T})$ in $T$ is equal to the image of the isogeny $Y_{Q,n}\otimes F^{\times}\to T$ induced from $Y_{Q,n}\to Y$.

Let $\overline{\chi}\in\operatorname{Hom}_{\iota}(Z(\overline{T}),\mathbb{C}^{\times})$ be a genuine character of $Z(\overline{T})$. Write $i(\overline{\chi}):=\operatorname{Ind}_{\overline{A}}^{\overline{T}}\overline{\chi}^{\prime}$ for the induced representation on $\overline{T}$, where $\overline{A}$ is any maximal abelian subgroup of $\overline{T}$, and $\overline{\chi}^{\prime}$ is any extension of $\overline{\chi}$. By the Stone-von Neumann theorem (see [Wei09] Theorem 3.1, [McN12] Theorem 3), the construction $\overline{\chi}\mapsto i(\overline{\chi})$ gives a bijection between isomorphism classes of genuine representations of $Z(\overline{T})$ and $\overline{T}$. Since we consider an unramified covering group $\overline{G}$ in this paper, we take $\overline{A}$ to be $Z(\overline{T})\cdot(K\cap T)$ from now on.

The choice of this maximal abelian group here is crucial for our calculation in Sect. 8.

3.1. Definition

View $i(\overline{\chi})$ as a genuine representation of $\overline{B}$ by inflation from the quotient map $\overline{B}\to\overline{T}$. We now define the unramified principal series representation $I(\chi):=\operatorname{Ind}_{\overline{B}}^{\overline{G}}i(\chi)$. The induction is normalized. One knows that $I(\overline{\chi})$ is unramified (i.e. $I(\overline{\chi})^{K}\neq 0$) if and only if $\overline{\chi}$ is unramified (i.e. $\overline{\chi}$ is trivial on $Z(\overline{T})\cap K$). We only consider unramified genuine representations in this paper. One has the natually arising abelian extension

[TABLE]

such that unramified genuine characters $\overline{\chi}$ of $Z(\overline{T})$ correspond to genuine characters of $\overline{Y}_{Q,n}$. Here $\overline{Y}_{Q,n}=Z(\overline{T})/Z(\overline{T})\cap K$. Since $\overline{A}/(T\cap K)\simeq\overline{Y}_{Q,n}$ as well, there is a canonical extension (also denoted by $\overline{\chi}$) of an unramified character $\overline{\chi}$ of $Z(\overline{T})$ to $\overline{A}$, by composing $\overline{\chi}$ with $\overline{A}\twoheadrightarrow\overline{Y}_{Q,n}$. Therefore, we will identity $i(\overline{\chi})$ as $\operatorname{Ind}_{\overline{A}}^{\overline{T}}\overline{\chi}$ with this $\overline{\chi}$.

The following result appears in the proof of [McN12] Lemma 2.

Lemma 3.1**.**

An unramified principal series representation $I(\chi)$ has a one-dimensional space of $K$-fixed vectors. There is an isomorphism

[TABLE]

Given $f\in i(\chi)^{\overline{T}\cap K}$, the support of $f$ is in $\overline{A}$.

For any $\mathbf{w}\in W$, the intertwining operator $T_{\mathbf{w},\overline{\chi}}:I(\overline{\chi})\to I({}^{\mathbf{w}}\overline{\chi})$ is defined by

[TABLE]

when it is absolutely convergent. Here, $U_{w}=U\cap wU^{-}w^{-1}$. Moreover, it can be meromorphically continued for all $\overline{\chi}$ ([McN12] Sect. 7). For $I(\overline{\chi})$ unramified and $\mathbf{w}=\boldsymbol{\sigma}_{\alpha}$ with $\alpha\in\Delta$, $T_{\boldsymbol{\sigma}_{\alpha},\overline{\chi}}$ is determined by

[TABLE]

where

[TABLE]

Here $\phi_{K}\in I(\overline{\chi})$ and $\phi_{K}^{\boldsymbol{\sigma}_{\alpha}}\in I({}^{\boldsymbol{\sigma}}\overline{\chi})$ are the normalized unramified vectors ([McN12, Gao18c]).

For a general $\mathbf{w}\in W$, denote

[TABLE]

Then the Gindikin-Karpelevich coefficient $c(\mathbf{w},\chi)$ associated with $T_{\mathbf{w},\chi}$ is

[TABLE]

such that $T_{\mathbf{w},\chi}(\phi_{K})=c(\mathbf{w},\overline{\chi})\phi^{\prime}_{K}$.

3.2. Whittaker functional

Let $\mathrm{Ftn}(i(\overline{\chi}))$ be the vector space of functions $\mathbf{c}$ on $\overline{T}$ satisfying

[TABLE]

The support of any $\mathbf{c}\in\mathrm{Ftn}(i(\overline{\chi}))$ is a disjoint union of cosets in $\overline{T}/\overline{A}$. Moreover $\dim(\mathrm{Ftn}(i(\overline{\chi})))=|Y/Y_{Q,n}|$ since $\overline{T}/\overline{A}$ has the same size as $Y/Y_{Q,n}$.

There is a natural isomorphism of vector spaces $\mathrm{Ftn}(i(\overline{\chi}))\simeq i(\overline{\chi})^{\vee}$, where $i(\overline{\chi})^{\vee}$ is the complex dual space of functionals of $i(\overline{\chi})$. Explicitly, let $\{\gamma_{i}\}\subset\overline{T}$ be a set of representatives of $\overline{T}/\overline{A}$. Consider $\mathbf{c}_{\gamma_{i}}\in\mathrm{Ftn}(i(\overline{\chi}))$ which has support $\gamma_{i}\cdot\overline{A}$ and $\mathbf{c}_{\gamma_{i}}(\gamma_{i})=1$. It gives rise to a linear functional $\lambda_{\gamma_{i}}^{\overline{\chi}}\in i(\overline{\chi})^{\vee}$ such that

[TABLE]

where $f_{\gamma_{j}}\in i(\overline{\chi})$ is the unique element such that $\mathrm{supp}(f_{\gamma_{j}})=\overline{A}\cdot\gamma_{j}^{-1}$ and $f_{\gamma_{j}}(\gamma_{j}^{-1})=1$. That is,

[TABLE]

The isomorphism $\mathrm{Ftn}(i(\overline{\chi}))\simeq i(\overline{\chi})^{\vee}$ is given explicitly by

[TABLE]

Consider the principal series $I(\overline{\chi}):=I(i(\overline{\chi}))$ for an unramified character $\overline{\chi}\in\operatorname{Hom}(Z(\overline{T}),\mathbb{C}^{\times})$. We define a space of Whittaker functionals on $I(\overline{\chi})$.

Let $\psi_{U}:U\to\mathbb{C}^{\times}$ be the character on $U$ such that its restriction to every $U_{\alpha},\alpha\in\Delta$ is given by $\psi\circ e_{\alpha}^{-1}$. We may write $\psi$ for $\psi_{U}$ if no confusion arises.

Definition 3.2**.**

For any genuine representation $(\overline{\sigma},V_{\overline{\sigma}})$ of $\overline{G}$, a linear functional $\lambda:V_{\overline{\sigma}}\to\mathbb{C}$ is called a $\psi$-Whittaker functional if $\lambda(\overline{\sigma}(u)v)=\psi(u)\cdot v$ for all $u\in U$ and $v\in V_{\overline{\sigma}}$. Write $\mathrm{Wh}_{\psi}(\overline{\sigma})$ for the space of $\psi$-Whittaker functionals for $\overline{\sigma}$.

Consider the following integral

[TABLE]

for $f\in I(\overline{\chi})$. This is a $i(\overline{\chi})$-valued functional. To obtain a Whittaker functional, we need to apply an element in $i(\overline{\chi})^{\vee}$. By [McN16] Sect. 6, there is an isomorphism between $i(\overline{\chi})^{\vee}$ and the space $\mathrm{Wh}_{\psi}(I(\overline{\chi}))$ of $\psi$-Whittaker functionals on $I(\overline{\chi})$, given by $\lambda\mapsto W_{\lambda}$ with

[TABLE]

where $f\in I(\overline{\chi})$ is an $i(\overline{\chi})$-valued function on $\overline{G}$; $w_{G}\in K$ is a representative of $\mathbf{w}_{G}$.

For $\mathbf{c}\in\mathrm{Ftn}(i(\overline{\chi}))$, by abuse of notation, we will write $\lambda_{\mathbf{c}}^{\overline{\chi}}\in\mathrm{Wh}_{\psi}(I(\overline{\chi}))$ for the resulting $\psi$-Whittaker functional of $I(\overline{\chi})$ from the isomorphism $\mathrm{Ftn}(i(\overline{\chi}))\simeq i(\overline{\chi})^{\vee}\simeq\mathrm{Wh}_{\psi}(I(\overline{\chi}))$. As a consequence, $\dim\mathrm{Wh}_{\psi}(I(\overline{\chi}))\simeq|Y/Y_{Q,n}|$.

3.3. Local coefficient matrix

Let $J(\mathbf{w},\overline{\chi})$ be the image of $T_{\mathbf{w},\overline{\chi}}$. The operator $T_{\mathbf{w},\overline{\chi}}$ induces a homomorphism $T_{\mathbf{w},\overline{\chi}}^{\ast}$ of vector spaces with image $\mathrm{Wh}_{\psi}(J(\mathbf{w},\chi))$:

[TABLE]

which is given by

[TABLE]

for any $\mathbf{c}\in\mathrm{Ftn}(i({}^{\mathbf{w}}\overline{\chi}))$. Let $\{\lambda_{\gamma}^{{}^{\mathbf{w}}\overline{\chi}}\}_{\gamma\in\overline{T}/\overline{A}}$ be a basis for $\mathrm{Wh}_{\psi}(I({}^{\mathbf{w}}\overline{\chi}))$, and $\lambda_{\gamma^{\prime}}^{\overline{\chi}}$ a basis for $\mathrm{Wh}_{\psi}(I(\overline{\chi}))$. The map $T_{\mathbf{w},\overline{\chi}}^{\ast}$ is then determined by the square matrix $[\tau(\overline{\chi},\boldsymbol{\sigma},\gamma,\gamma^{\prime})]_{\gamma,\gamma^{\prime}\in\overline{T}/\overline{A}}$ of size $|Y/Y_{Q,n}|$ such that

[TABLE]

The local coefficient matrix satisfies the following properties.

Lemma 3.3** ([KP84, McN16, Gao17]).**

For $\mathbf{w}\in W$ and $\bar{z},\bar{z}^{\prime}\in\bar{A}$, the following identity holds:

[TABLE]

Moreover, for $\mathbf{w}_{1},\mathbf{w}_{2}\in W$ such that $\ell(\mathbf{w}_{1}\mathbf{w}_{2})=\ell(\mathbf{w}_{1})+\ell(\mathbf{w}_{2})$, one has

[TABLE]

which is referred to as the cocycle relation.

Proof.

This fact is standard. For example, it follows from [Gao17] Lemma 3.2. ∎

Thus the calculation of the local coefficient matrix $[\tau(\overline{\chi},\mathbf{w},\gamma,\gamma^{\prime})]_{\gamma,\gamma^{\prime}}$ is reduced to the case when $\mathbf{w}$ is a simple reflection.

We now would like to compute the matrix $[\tau(\overline{\chi},\boldsymbol{\sigma}_{\alpha},\gamma,\gamma^{\prime})]_{\gamma,\gamma^{\prime}}$ for any unramified character $\overline{\chi}$ and simple reflection $\boldsymbol{\sigma}_{\alpha},\alpha\in\Delta$.

Theorem 3.4** ([KP84] Lemma I.3.3 ,[McN16] Theorem 13.1, [Gao17] Theorem 3.6.).**

Suppose that $\gamma=\mathbf{s}_{y_{1}}$ and $\gamma^{\prime}=\mathbf{s}_{y}$ by $y$. Then we can write

[TABLE]

with the following properties:

• •

$\tau^{i}(\boldsymbol{\sigma}_{\alpha},\overline{\chi},\gamma\cdot\bar{z},\gamma^{\prime}\cdot\bar{z}^{\prime})=({}^{\boldsymbol{\sigma}_{\alpha}}\overline{\chi})^{-1}(\bar{z})\cdot\tau^{i}(\boldsymbol{\sigma}_{\alpha},\overline{\chi},\gamma,\gamma^{\prime})\cdot\overline{\chi}(\bar{z}^{\prime}),\qquad\bar{z},\bar{z}^{\prime}\in\bar{A}$;

• •

$\tau^{1}(\boldsymbol{\sigma}_{\alpha},\overline{\chi},\gamma,\gamma^{\prime})=0$* unless $y_{1}\equiv y\mod Y_{Q,n}$;*

• •

$\tau^{2}(\boldsymbol{\sigma}_{\alpha},\overline{\chi},\gamma,\gamma^{\prime})$* unless $y_{1}\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$.*

Moreover,

• •

If $y_{1}=y$, then

[TABLE]

• •

If $y_{1}=\boldsymbol{\sigma}_{\alpha}[y]$, then

[TABLE]

3.4. Explicit calculation of the local coefficient matrix

Lemma 3.3 and Theorem 3.4 determine the local coefficient matrix completely. However, it is too complicated to obtain a general formula as one has to analyze the sum over $\overline{T}/\overline{A}$ inductively. In this section, we highlight some observations that will be useful for our calculation.

Notations: for $y,y^{\prime}\in Y$, we write

[TABLE]

Let $\mathbf{w}=\mathbf{w}_{1}\cdots\mathbf{w}_{k}$ be a reduced decompositioin of $\mathbf{w}$ by simple reflections.

Lemma 3.5**.**

The coefficient $\tau(\mathbf{w},\overline{\chi},y,y^{\prime})=0$ unless $y^{\prime}\equiv\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]\mod Y_{Q,n}$ for some $a_{1},\cdots,a_{k}\in\{0,1\}$.

Proof.

We can prove this by induction on $k$. When $k=1$, this follows from Theorem 3.4. We now assume that the result is true for $k-1$. Then

[TABLE]

If this is nonzero, then $\tau(\mathbf{w}_{1}\cdots\mathbf{w}_{k-1},{}^{\mathbf{w}_{k}}\overline{\chi},y,y^{\prime\prime})\neq 0$ and $\tau(\mathbf{w}_{k},\overline{\chi},y^{\prime\prime},y^{\prime})\neq 0$ for some $y^{\prime\prime}$. This implies that

[TABLE]

and $y^{\prime}\equiv\mathbf{w}_{k}^{a_{k}}[y^{\prime\prime}]\mod Y_{Q,n}$ for some $a_{k}\in\{0,1\}$. This proves the result. ∎

We have an immediate corollary.

Corollary 3.6**.**

The coefficient $\tau(\mathbf{w}_{G},\overline{\chi},y,y^{\prime})=0$ unless $\mathbf{w}[y]\equiv y^{\prime}\mod Y_{Q,n}$ for some $\mathbf{w}\in W$.

The next result is very useful for calculation.

Lemma 3.7**.**

Assume that $\mathbf{w}=\mathbf{w}_{1}\cdots\mathbf{w}_{k}$ is a reduced decomposition of $\mathbf{w}$, and for any two subexpressions $\mathbf{w}_{1}^{a_{1}}\cdots\mathbf{w}_{k}^{a_{k}}=\mathbf{w}_{1}^{a^{\prime}_{1}}\cdots\mathbf{w}_{k}^{a^{\prime}_{k}}$, $a_{1},\cdots,a_{k},a^{\prime}_{1},\cdots,a^{\prime}_{k}\in\{0,1\}$, we have $a_{i}=a_{i}^{\prime}$ for $i=1,\cdots,k$. If the orbit of $y$ is free, then

[TABLE]

In other words, only one term in the summation is nonzero.

Proof.

The assumption implies that $\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]$ are all distinct in $Y/Y_{Q,n}$ for $a_{1},\cdots,a_{k}\in\{0,1\}$.

We prove it by induction on $k$. If $k=1$, there is nothing to prove. Assume the result is true for $\mathbf{w}_{1}\cdots\mathbf{w}_{k-1}$. Then

[TABLE]

For a nonzero term in the summation, we have

[TABLE]

and $\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]\equiv\mathbf{w}_{k}^{a^{\prime}_{k}}[y^{\prime\prime}]\mod Y_{Q,n}$ for some $a_{k}\in\{0,1\}$. As the orbit of $y$ is free, this implies that

[TABLE]

and therefore $a_{i}=a^{\prime}_{i}$ for $i=1,\cdots,k$. We now conclude that only the term $y^{\prime\prime}=\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]$ has nonzero contribution in the summation and therefore

[TABLE]

By induction we obtain the desired formula. ∎

Remark 3.8*.*

The conditions in the lemma are satisfied in the following example: $\mathbb{G}=\mathbb{GL}_{r}$ and $\mathbf{w}=\boldsymbol{\sigma}_{\alpha_{1}}\cdots\boldsymbol{\sigma}_{\alpha_{r}}$. We will use it later.

Notice that $Y_{Q,n}$ is not well-behaved with respect to Levi subgroup so it is better to work with the lattice $Y_{Q,n}^{sc}$. Observe that $Y_{Q,n}\cap Y^{sc}=Y_{Q,n}^{sc}$.

Lemma 3.9**.**

If $y\equiv\mathbf{w}[y]\mod Y_{Q,n}$ for some $\mathbf{w}\in W$, then $y\equiv\mathbf{w}[y]\mod Y_{Q,n}^{sc}$.

Proof.

By [BBF08] Lemma 2, $y-\mathbf{w}[y]\in Y^{sc}$. If $y-\mathbf{w}[y]\in Y_{Q,n}$, then it is in $Y_{Q,n}^{sc}$. ∎

Let $\mathbb{T}_{Q,n}^{sc}$ be the split torus with cocharacter group $Y_{Q,n}^{sc}$, and $T_{Q,n}^{sc}:=\mathbb{T}_{Q,n}^{sc}(F)$.

Lemma 3.10**.**

The coefficient $\tau(\mathbf{w},\overline{\chi},y,\mathbf{w}^{\prime}[y])$ depends only on $\overline{\chi}|_{\overline{T}_{Q,n}^{sc}}$ for $\mathbf{w},\mathbf{w}^{\prime}\in W$.

Proof.

If $\mathbf{w}=\mathrm{id}$, this result follows from Lemma 3.9 and Lemma 3.3.

We now consider a nontrivial Weyl group element with reduced decomposition $\mathbf{w}=\mathbf{w}_{1}\cdots\mathbf{w}_{k}$. If $\tau(\mathbf{w},\overline{\chi},y,\mathbf{w}^{\prime}[y])\neq 0$, then by Lemma 3.5, $\mathbf{w}^{\prime}[y]\equiv\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]\mod Y_{Q,n}$ for some $a_{1},\cdots,a_{k}\in\{0,1\}$. By Lemma 3.9, $\mathbf{w}^{\prime}[y]\equiv\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}[y]\mod Y_{Q,n}^{sc}$. So it suffices to prove the result for elements of the form $\mathbf{w}^{\prime}=\mathbf{w}_{k}^{a_{k}}\cdots\mathbf{w}_{1}^{a_{1}}$.

We now argue by induction on the length of $\mathbf{w}$. If $\mathbf{w}=\boldsymbol{\sigma}_{\alpha}$, then the result is straightforward when $y\not\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$. If $y\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$, then $y\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}^{sc}$. The same argument above applies. The same argument again applies in the induction argument. This proves the result. ∎

3.5. Unramified Whittaker functions

For an unramified principal series representation $I(\overline{\chi})$, let $\mathcal{W}$ be the image of $\phi_{K}$ in the Whittaker model defined by (1). In other words,

[TABLE]

Note that our definition here is slightly different from [Gao18a]. We divide by the modular quasi-character $\delta_{B}^{-1/2}$ to make our calculation slightly easier. If $\lambda$ is defined by $\gamma$, we write $\mathcal{W}_{\gamma}=\mathcal{W}_{\lambda_{\gamma}}$. We also define $\mathcal{W}_{y}(y^{\prime},\overline{\chi})=\mathcal{W}_{\mathbf{s}_{y}}(\mathbf{s}_{y^{\prime}},\overline{\chi})$.

An element $\overline{t}\in\overline{T}$ is called dominant if $\overline{t}\cdot(U\cap K)\cdot\overline{t}^{-1}\subset K$.

Theorem 3.11**.**

Let $I(\overline{\chi})$ be an unramified principal series of $\overline{G}$ and $\gamma\in\overline{T}$. Let $\mathcal{W}_{\gamma}$ be the unramified Whittaker function associated to $\phi_{K}$. Then, $\mathcal{W}_{\gamma}(\overline{t})=0$ unless $\overline{t}\in\overline{T}$ is dominant. Moreover, for dominant $\overline{t}$, one has

[TABLE]

Proof.

The proof in [Gao18a] Proposition 3.3 works without essential change. ∎

4. An inductive formula

As a consequence of Theorem 3.11, we now prove an inductive formula for unramified Whittaker function. The main result in this section is a generalization of the material presented in [Suz98] Section 7.1.

For certain types of root systems, our formula might admit simplification – we discuss this in Sect. 6. See also [Suz97] Lemma 4.1 and [Suz98] Section 7.1. Note that there are some typos in the proofs of these two papers. We give full details here.

4.1. Basic setup

Let $\Delta^{\prime}$ be a subset of $\Delta$. Let $\mathbb{P}=\mathbb{M}\mathbb{N}$ be the parabolic subgroup of $\mathbb{G}$ associated with $\Delta^{\prime}$. We write

[TABLE]

for the root datum of $M$. Since $\mathbb{T}\subset\mathbb{M}$, the character and cocharacter lattices $X$ and $Y$ respectively are unchanged. However, we have $\Delta_{M}=\Delta^{\prime}$ and $\Delta_{M}^{\vee}=\{\beta^{\vee}:\beta\in\Delta^{\prime}\}$. Let $\mathbb{B}_{M}=\mathbb{T}\mathbb{U}_{M}$ be the Borel subgroup of $\mathbb{M}$ corresponding to $\Delta_{M}$. Denote by $W(\mathbb{M})\subset W(\mathbb{G})$ the Weyl group of $(\mathbb{M},\mathbb{T})$.

The functorial properties with respect to restriction is studied in [GG18] Sect. 5.5. The cover $\overline{M}$ is associated to the pair $(D,\eta|_{Y_{M}^{\mathrm{sc}}})$, where the quadratic form $Q(x)=D(x,x)$ carries only the $W(\mathbb{M})$-invariance by applying the “forgetful” functor from $W$-invariance.

Given a genuine character $\overline{\chi}:\overline{T}\to\mathbb{C}^{\times}$, one can define an unramified principal series representation $I_{\overline{M}}(\overline{\chi})$ on $\overline{M}$. By induction in stages, $I(\overline{\chi})=\operatorname{Ind}_{\overline{P}}^{\overline{G}}I_{\overline{M}}(\chi)$. Here $I_{\overline{M}}(\overline{\chi})$ is inflated to a representation on $\overline{P}$ in the usual way. The study of Whittaker models and Whittaker functions applies to representations on $\overline{M}$. We add subscript $\overline{M}$ to indicate the ambient group.

We have the following observations:

• •

The section $\mathbf{s}_{\overline{M},y}=\mathbf{s}_{\overline{G},y}$ for $y\in Y$. So the notation $\mathbf{s}_{y}$ does not arise any confusion.

• •

For $\mathbf{w}\in W(\mathbb{M})$, one can calculate the local coefficient matrix $\tau_{\overline{M}}(\mathbf{w},\overline{\chi},y,y^{\prime})$. It is easy to check that $\tau_{\overline{M}}(\mathbf{w},\overline{\chi},y,y^{\prime})=\tau_{\overline{G}}(\mathbf{w},\overline{\chi},y,y^{\prime})$. Thus we can safely drop the subscript.

Let $W^{M}$ be the set of minimal representatives in $W(\mathbb{M})\backslash W$. A element $\mathbf{w}\in W$ can be uniquely written as $\mathbf{w}=\mathbf{w}_{1}\mathbf{w}_{2}$, where $\mathbf{w}_{1}\in W(\mathbb{M})$ and $\mathbf{w}_{2}\in W^{M}$. The long element $\mathbf{w}_{G}$ is written as $\mathbf{w}_{M}\mathbf{w}^{M}$.

Lemma 4.1**.**

We have

[TABLE]

Proof.

Observe that

[TABLE]

and any element in this set satisfies $\mathbf{w}_{2}(\alpha)>0$. We have

[TABLE]

We now show that the first set is $\Phi(\mathbf{w}^{M,-1}\mathbf{w}_{2})$ and the second set is $\mathbf{w}_{2}^{-1}(\Phi_{M}(\mathbf{w}_{M}\mathbf{w}_{1}))$.

Note that $\Phi(\mathbf{w}^{M,-1})=\Phi^{+}-\Phi_{M}^{+}$. Thus

[TABLE]

Note that if $\mathbf{w}_{2}(\alpha)\in\Phi^{+}-\Phi_{M}^{+}$, then $\mathbf{w}(\alpha)>0$. Thus (2) is the first set.

Let $\beta=\mathbf{w}_{2}(\alpha)$. Then the second set is

[TABLE]

Now the result follows. ∎

4.2. The inductive formula

We now give the inductive formula.

Proposition 4.2**.**

We have

[TABLE]

Proof.

Recall that

[TABLE]

Given $\mathbf{w}\in W$, it can be uniquely written as $\mathbf{w}=\mathbf{w}_{1}\mathbf{w}_{2}$ as above. By the cocycle relation in Lemma 3.3, we deduce that

[TABLE]

On the other hand,

[TABLE]

Here, we use the following fact: $\overline{\chi}_{\mathbf{w}^{-1}(\alpha)}=({}^{\mathbf{w}}\overline{\chi})_{\alpha}$. This can be seen from the following identity: $\langle x,\mathbf{w}^{-1}(\alpha^{\vee})\rangle=\langle\mathbf{w}(x),\alpha^{\vee}\rangle$ for any $x\in X$.

From this we deduce that

[TABLE]

Note that

[TABLE]

Thus we deduce that $\mathcal{W}_{\gamma}(\overline{t},\overline{\chi})$ equals

[TABLE]

∎

4.3. Local coefficient matrix

We end this section with a useful result on the local coefficient matrix. We now write $\mathbb{M}=\mathbb{M}_{1}\times\cdots\times\mathbb{M}_{k}$. Let $\mathbb{T}_{i}=\mathbb{T}\cap\mathbb{M}_{i}$. Let $Y_{i}$ be the cocharacter lattice of $\mathbb{T}_{i}$. Let $W(\mathbb{M}_{i})$ be the Weyl group of $(\mathbb{M}_{i},\mathbb{T}_{i})$.

Let $\mathbf{w}=(\mathbf{w}_{1},\cdots,\mathbf{w}_{k})\in W(\mathbb{M})$ with $\mathbf{w}_{i}\in W(\mathbb{M}_{i})$. Let $\mathbf{w}^{\prime}=(\mathbf{w}^{\prime}_{1},\cdots,\mathbf{w}^{\prime}_{k})\in W(\mathbb{M})$ with $\mathbf{w}^{\prime}_{i}\in W(\mathbb{M}_{i})$. Let $y=(y_{1},\cdots,y_{k})$ where $y_{i}\in Y_{i}$. Let $y^{\prime}_{i}=\mathbf{w}_{i}^{\prime}[y_{i}]$.

We now consider $\overline{\chi}|_{\overline{T}_{Q,n}^{sc}}$. Let $\overline{\chi}_{i}$ be a character of $Z(\overline{T}_{i})$ so that its restriction to $\overline{T}_{i,Q,n}^{sc}$ agrees with $\overline{\chi}_{i}|_{\overline{T}_{i,Q,n}^{sc}}$. In such situations, we write $\overline{\chi}\sim(\overline{\chi}_{1},\cdots,\overline{\chi}_{k})$. Recall from Lemma 3.10 that $\tau(\overline{\chi}_{i},\mathbf{w}_{i},y,\mathbf{w}_{i}^{\prime}[y^{\prime}])$ only depends on the choice of $\overline{\chi}_{i}|_{\overline{T}_{i,Q,n}^{sc}}$ but not on the choice of $\overline{\chi}_{i}$.

Lemma 4.3**.**

With notations as above,

[TABLE]

Proof.

By induction, it suffices to prove the case $k=2$. So we assume $k=2$ from now on.

For the case $k=2$, we prove it by induction on the length of $\mathbf{w}$. If $\mathbf{w}=\mathrm{id}$, the result is trivial.

We now assume the result is true for $\mathbf{w}$ and prove it for $\boldsymbol{\sigma}_{\alpha}\mathbf{w}$ where $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})=\ell(\mathbf{w})+1$ and $\boldsymbol{\sigma}_{\alpha}$ is in either $W(\mathbb{M}_{1})$ or $W(\mathbb{M}_{2})$. We assume that $\boldsymbol{\sigma}_{\alpha}\in W(\mathbb{M}_{1})$ without loss of generality. We have

[TABLE]

The first term is nonzero only when $y^{\prime\prime}=y$ or $\sigma_{\alpha}[y]$. We write $y^{\prime\prime}=(y_{1}^{\prime\prime},y_{2}^{\prime\prime})$. By induction, we have

[TABLE]

Note that $\boldsymbol{\sigma}_{\alpha}[y]=(\boldsymbol{\sigma}_{\alpha}[y_{1}],y_{2})$ and $\tau({}^{\boldsymbol{\sigma}_{\alpha}}\overline{\chi},\boldsymbol{\sigma}_{\alpha},y,\boldsymbol{\sigma}_{\alpha}[y])=\tau({}^{\boldsymbol{\sigma}_{\alpha}}\overline{\chi}_{1},\boldsymbol{\sigma}_{\alpha},y_{1},\boldsymbol{\sigma}_{\alpha}[y_{1}])$. By Lemma 3.9, it is easy to verify that $y\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$ if and only if $y_{1}\equiv\boldsymbol{\sigma}_{\alpha}[y_{1}]\mod Y_{1,Q,n}$. If $y\not\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$, then

[TABLE]

If $y\equiv\boldsymbol{\sigma}_{\alpha}[y]\mod Y_{Q,n}$,

[TABLE]

∎

5. Relative theta representations

We first recall the definition of theta representations and discuss its generalization given in [Suz98] and [Gao18b].

5.1. Definition

We start with the following definition.

Definition 5.1**.**

An unramified genuine character $\overline{\chi}$ of $Z(\overline{T})$ is called exceptional if

[TABLE]

The theta representation $\Theta(\overline{G},\overline{\chi})$ associated to an exceptional character $\overline{\chi}$ is the unique Langlands quotient (see [BJ13]) of $I(\overline{\chi})$, which is also equal to the image of the intertwining operator $T_{\mathbf{w}_{G},\overline{\chi}}:I(\overline{\chi})\to I({}^{\mathbf{w}_{G}}\overline{\chi})$.

To make our discussion more flexible, we introduce the following definition. It can be viewed as a generalization of [Suz98] and [Gao18b].

Definition 5.2**.**

For any subset $\Delta^{\prime}\subset\Delta$, a genuine character $\chi$ is called $\Delta^{\prime}$-exceptional (resp. $\Delta^{\prime}$-anti-exceptional) if $\overline{\chi}(\overline{h}_{\alpha}(\varpi^{n_{\alpha}}))=q^{-1}$ (resp. $\overline{\chi}(\overline{h}_{\alpha}(\varpi^{n_{\alpha}}))=q$) for every $\alpha\in\Delta^{\prime}$. In the case $\Delta^{\prime}=\Delta$, it is simply called exceptional or anti-exceptional, respectively.

Let $\mathbb{M}$ be the Levi subgroup corresponding to $\Delta^{\prime}$. Then a $\Delta^{\prime}$-exceptional character can be viewed as an exceptional character for $\overline{M}$. In other words, we obtain a representation $\Theta(\overline{M},\overline{\chi})$ of $\overline{M}$ as the image of the intertwining operator

[TABLE]

Here $\mathbf{w}_{M}$ is the longest element in the Weyl group of $M$. We also add subscript ‘$\overline{M}$’ to indicate the ambient group. We will do so in the rest of this section.

We can now define a representation on $\overline{G}$ by normalized induction:

[TABLE]

We call it a relative Theta representation. The representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ can also be defined as the image of the intertwining operator

[TABLE]

Note that $\Theta(\overline{G}/\overline{M},\overline{\chi})$ might be reducible.

5.2. Some properties

We discuss some properties of $\Theta(\overline{G}/\overline{M},\overline{\chi})$. The intertwining operator $T_{\mathbf{w}_{M},\overline{\chi}}:I(\overline{\chi})\to I({}^{\mathbf{w}_{M}}\overline{\chi})$ induces a map on the space of Whittaker functional

[TABLE]

The matrix is defined by

[TABLE]

Proposition 5.3**.**

A function $\mathbf{c}\in\mathrm{Ftn}(i(\overline{\chi}))$ gives rise to a functional in $\mathrm{Wh}_{\psi}(\Theta(\overline{G}/\overline{M},\overline{\chi}))$ if and only if for all $\alpha\in\Delta^{\prime}$,

[TABLE]

The left-hand side is independent of the choice of representatives for $\overline{T}/\overline{A}$.

Proof.

The same proof in [KP84] page 76 works here as well. ∎

Proposition 5.4**.**

Let $\overline{\chi}$ be an unramified $\Delta^{\prime}$-exceptional character. Let $\lambda_{\mathbf{c}}^{\overline{\chi}}\in\mathrm{Wh}_{\psi}(I(\overline{\chi}))$ be the $\psi$-Whittaker functional of $I(\overline{\chi})$ associated to some $\mathbf{c}\in\mathrm{Ftn}(i(\overline{\chi}))$. Then, $\lambda_{\mathbf{c}}^{\overline{\chi}}$ lies in $\mathrm{Wh}_{\psi}(\Theta(\overline{G}/\overline{M},\overline{\chi}))$ if and only for any simple root $\alpha\in\Delta^{\prime}$ one has

[TABLE]

Proof.

The proof in [Gao17] Corollary 3.7 works the same here. ∎

We now state some basic properties of these coefficients. See also [Suz97] Sect. 3.6.

Proposition 5.5**.**

Let $\overline{\chi}$ be an unramified $\Delta^{\prime}$-exceptional character.

1. (1)

$\tau(\mathbf{w}_{M},\overline{\chi},y,y^{\prime})=0$* unless $y^{\prime}=\mathbf{w}[y]$ for some $\mathbf{w}\in W(\mathbb{M})$.* 2. (2)

$\tau(\mathbf{w}_{M},\overline{\chi},y,\mathbf{w}[y^{\prime}])=R(\mathbf{w},y)\tau(\mathbf{w}_{M},\overline{\chi},y,y^{\prime})$* for $\mathbf{w}\in W(\mathbb{M})$, where $R(\mathbf{w},y)$ is some function of $W$ and $Y$.*

Proof.

The first one is obvious. The second one follows from the above lemma. In fact, $\tau(\overline{\chi},\mathbf{w}_{M},\gamma,-)\in\mathrm{Ftn}(i(\overline{\chi}))$ gives rise to a functional in $\mathrm{Wh}_{\psi}(\Theta(\overline{G}/\overline{M},\overline{\chi}))$. When $\mathbf{w}=\boldsymbol{\sigma}_{\alpha}$ is a simple reflection, then

[TABLE]

The rest follows by induction. ∎

Corollary 5.6**.**

Let $\mathbf{w}\in W(\mathbb{M})$. Then $T_{\mathbf{w}_{M},\overline{\chi}}^{\ast}(\lambda_{y}^{{}^{\mathbf{w}_{M}}\overline{\chi}})$ and $T_{\mathbf{w}_{M},\overline{\chi}}^{\ast}(\lambda_{\mathbf{w}[y]}^{{}^{\mathbf{w}_{M}}\overline{\chi}})$ are proportional on $I(\overline{\chi})$, and

[TABLE]

Proof.

This is an immediate consequence of Proposition 5.5. In fact,

[TABLE]

and similarly

[TABLE]

This gives the desired result. ∎

5.3. Rodier’s lemma

We end this section with a generalization of a lemma of Rodier. This will be useful later. Recall that Rodier’s result says that when the inducing data is generic, then so is the induced representation.

Proposition 5.7**.**

The representation $\operatorname{Ind}_{\overline{P}}^{\overline{G}}(\pi)$ is generic if and only if $\pi$ is generic. Moreover,

[TABLE]

Proof.

This follows from [BZ77] Theorem 5.2 and [CS80] Lemma 1.5. ∎

6. The case of general linear groups

From now on, we focus on the case of $\mathbb{G}=\mathbb{GL}_{r+1}$. We now introduce some notations in this setup. Write $\Delta=\{\alpha_{1},\cdots,\alpha_{r}\}$ with the standard enumeration and the Weyl group is generated by $\{\boldsymbol{\sigma}_{1},\cdots,\boldsymbol{\sigma}_{r}\}$. The root system is simply-laced, and we write $n_{Q}=n_{\alpha}$ for any $\alpha\in\Delta$. For $\alpha=\alpha_{i}+\cdots+\alpha_{j-1}$, write $\overline{\chi}_{ij}=\overline{\chi}_{\alpha}$.

6.1. Inductive formula

The inductive formula in Proposition 4.2 admits a refinement in the case of $\mathbb{M}=\mathbb{GL}_{r}\times\mathbb{GL}_{1}$ in $\mathbb{GL}_{r+1}$.. This is similar to [Suz97] Lemma 4.1. In this case, the $W^{M}$ is $\{\boldsymbol{\sigma}_{r},\ \boldsymbol{\sigma}_{r}\boldsymbol{\sigma}_{r-1},\ \cdots,\ \boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}\}$.

Recall that $\mathcal{W}_{y}(y^{\prime},\overline{\chi})\neq 0$ if and only if $y$ and $\mathbf{w}_{G}(y^{\prime})$ lies in the same orbit under the Weyl group action. (Note that this is not $\mathbf{w}_{G}[y^{\prime}]$.) We now assume that $\mathbf{w}_{G}(y^{\prime})=\mathbf{w}^{\prime}[y]$ for some $\mathbf{w}^{\prime}\in W$. Note that this is true identity instead of $\mod Y_{Q,n}$. Any $\mathbf{w}^{\prime}\in W$ can be uniquely written as $\mathbf{w}^{\prime}=\mathbf{w}\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{r_{0}}$ for an integer $r_{0}$ and $\mathbf{w}\in W(\mathbb{M})$. We have arrived at

[TABLE]

with $\mathbf{w}\in W(\mathbb{M})$.

Proposition 6.1**.**

Assume that the orbit of $y$ under $W$ is free. We have

[TABLE]

where the second sum is over the set

[TABLE]

This result is probably true in general. But we only prove what we need here.

Proof.

Note $\mathbf{w}^{M}=\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}$. If $\mathbf{w}_{2}=\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{i}$, then $\mathbf{w}^{M,-1}\mathbf{w}_{2}=\boldsymbol{\sigma}_{1}\cdots\boldsymbol{\sigma}_{i-1}$. Thus,

[TABLE]

In this case,

[TABLE]

We may rewrite the formula as

[TABLE]

We now analyze when both $\tau(\boldsymbol{\sigma}_{i}\cdots\boldsymbol{\sigma}_{r},{}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{i}}\overline{\chi},y,y^{\prime\prime})$ and $\mathcal{W}_{M,y^{\prime\prime}}(\mathbf{w}^{M}(y^{\prime}),{}^{\mathbf{w}_{2}}\overline{\chi})$ are nonzero.

We know that $\mathbf{w}_{G}(y^{\prime})=\mathbf{w}\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{r_{0}}[y]$ with $\mathbf{w}\in W(\mathbb{M})$. If $\mathcal{W}_{M,y^{\prime\prime}}(\mathbf{w}^{M}(y^{\prime}),{}^{\mathbf{w}_{2}}\overline{\chi})\neq 0$, then for some $y^{\prime\prime}\in Y$ and $\mathbf{w}^{\prime}\in W$,

[TABLE]

This condition implies that

[TABLE]

for some $\mathbf{w}^{\prime\prime}\in W(\mathbb{M})$.

If $\tau(\boldsymbol{\sigma}_{i}\cdots\boldsymbol{\sigma}_{r},{}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{i}}\overline{\chi},y,y^{\prime\prime})\neq 0$, then

[TABLE]

for some $a_{i},\cdots,a_{r}\in\{0,1\}$.

We now use the assumption that the orbit of $y$ is free. This implies that

[TABLE]

for some $a_{i},\cdots,a_{r}\in\{0,1\}$ and $\mathbf{w}^{\prime\prime}\in W(\mathbb{M})$. By considering the images of both sides in $W(\mathbb{M})\backslash W$, we know that this is possible only when $1\leq i\leq r_{0}$. The same argument shows that we must have $a_{r_{0}-1}=0$. Thus we conclude that elements in (3) is of the form

[TABLE]

So finally, we have arrived at

[TABLE]

where the second sum is over the set

[TABLE]

∎

We now discuss the covering group obtained by

[TABLE]

Write $Y=Y_{1}\oplus Y_{2}$ where $Y_{1}$ (resp. $Y_{2}$) be the cocharacter lattice of $\mathbb{GL}_{r}$ (resp. $\mathbb{GL}_{1}$). Then we have embeddings $\iota:Y_{1}\hookrightarrow Y$ and $\iota:Y_{1}^{sc}\hookrightarrow Y^{sc}$. The cover of $\overline{\operatorname{GL}}_{r}$ is associated with $(D|_{Y^{\prime}},\eta|_{Y_{1}^{sc}})$. We also observe that for $y\in Y_{1}$, $\mathbf{s}_{\mathbb{GL}_{r},\iota(y)}=\mathbf{s}_{\mathbb{GL}_{r-1},y}$. Thus, the notation $\mathbf{s}_{y}$ as this does not arise any confusion.

Lemma 6.2**.**

Let

• •

$y=(y_{1},y_{2}),y^{\prime}=(y^{\prime}_{1},y^{\prime}_{2})$* with $y_{1},y_{1}^{\prime}\in Y_{1}$ and $y_{2},y_{2}^{\prime}\in Y_{2}$,*

• •

$\overline{\chi}_{1}$* be an unramified character for $\overline{\operatorname{GL}_{r}}$ such that $\overline{\chi}_{1}|_{Y_{1,Q,n}^{sc}}=\overline{\chi}|_{Y_{Q,n}^{sc}}$.*

If $y_{2}=y^{\prime}_{2}$, then $\mathcal{W}_{M,y}(y^{\prime},\overline{\chi})=\mathcal{W}_{\operatorname{GL}_{r},y_{1}}(y^{\prime}_{1},\overline{\chi}_{1})$.

Proof.

Recall that

[TABLE]

It is straightforward to see that $c_{M}(\mathbf{w}_{M}\mathbf{w},\overline{\chi})=c_{\operatorname{GL}_{r}}(\mathbf{w}_{M}\mathbf{w},\overline{\chi}_{1})$. By Lemma 4.3,

[TABLE]

(Note that the condition $y_{2}=y_{2}^{\prime}$ does not appear explicitly in the proof but must be satisfied.) Therefore, $\mathcal{W}_{M,y}(y^{\prime},\overline{\chi})=\mathcal{W}_{\operatorname{GL}_{r},y_{1}}(y^{\prime}_{1},\overline{\chi}_{1})$. ∎

6.2. Relative theta representations

We now discuss the Whittaker models for the relative theta representations. In particular, we determine when these representations are non-generic and possess a unique Whittaker model. The main ingredient here is [Gao17] Theorem 1.1, which is a generalization of [KP84] Theorem I.3.5.

The root system spanned by $\Delta^{\prime}$ is of type $A_{r_{1}-1}\times\cdots\times A_{r_{k}-1}$, where $r_{1}+\cdots+r_{k}=r$. In this way, we obtain a bijection between subsets of $\Delta$ and ordered partitions $(r_{1}\cdots r_{k})$ of $r$. The following result can be proved along the same line as in [Gao17].

Theorem 6.3**.**

1. (1)

If $r_{i}>n_{Q}$ for some $i$, then the representation $\Theta(\overline{M},\overline{\chi})$ is non-generic. 2. (2)

If $r_{i}\leq n_{Q}$ for all $i$, then the representation $\Theta(\overline{M},\overline{\chi})$ is generic. 3. (3)

If $r_{i}=n_{Q}$ for all $i$, then the representation $\Theta(\overline{M},\overline{\chi})$ has a unique Whittaker model.

Proof.

The proof in [Gao17] Example 3.16 and [KP84] Corollary I.3.6 applies without essential change. ∎

By combining this result with Proposition 5.7, we deduce the following result, in analogy with [Suz98] Corollary 3.3.

Theorem 6.4**.**

1. (1)

If $r_{i}>n_{Q}$ for some $i$, then the representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ is non-generic. 2. (2)

If $r_{i}\leq n_{Q}$ for all $i$, then the representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ is generic. 3. (3)

If $r_{i}=n_{Q}$ for all $i$, then the representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ has a unique Whittaker model.

In the rest of this paper, we would like find a formula for some values of the unramified Whittaker functions in some special instances.

6.3. Some calculation of $c(\mathbf{w},\overline{\chi})$

In this section, we carry out some calculation of $c(\mathbf{w},\overline{\chi})$ for exceptional and anti-exceptional characters. In particular, we show that $\Theta(\overline{G}/\overline{M},\overline{\chi})$ contains a spherical vector.

Lemma 6.5**.**

Let $\overline{\chi}$ be an exceptional character. Then for $1\leq i\leq r-1$

[TABLE]

Proof.

By direct calculation,

[TABLE]

∎

Lemma 6.6**.**

Let $\overline{\chi}$ be an exceptional character. Then

[TABLE]

Proof.

This follows the above lemma and induction. ∎

Corollary 6.7**.**

The representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ contains a spherical vector.

Proof.

The representation $\Theta(\overline{G}/\overline{M},\overline{\chi})$ is defined as the image of $T_{\mathbf{w}_{M},\overline{\chi}}:I(\overline{\chi})\to I({}^{\mathbf{w}_{M}}\overline{\chi})$. The image contains the vector $c(\mathbf{w}_{G},\overline{\chi})\phi_{K}^{{}^{\mathbf{w}}\overline{\chi}}$, which is nonzero. ∎

Corollary 6.8**.**

If $r_{i}>n_{Q}$ for some $i$, then

[TABLE]

for any $\gamma,\gamma^{\prime}$.

Proof.

If $\mathcal{W}_{\gamma}(\gamma^{\prime},{}^{\mathbf{w}_{M}}\overline{\chi})\neq 0$, then the Whittaker functional is nonzero on a spherical vector $\phi_{K}$.

We already know that $\phi_{K}\in\Theta(\overline{G}/\overline{M},\overline{\chi})$. Thus this implies that $\Theta(\overline{G}/\overline{M},\overline{\chi})$ has a nonzero Whittaker functional. This contradicts with our assumption and Theorem 6.4. ∎

Lemma 6.9**.**

Let $\overline{\chi}$ be an anti-exceptional character. Then

[TABLE]

Proof.

The proof is the same as Lemma 6.5. We have

[TABLE]

∎

Lemma 6.10**.**

Let $\overline{\chi}$ be an anti-exceptional character. Let

[TABLE]

Then $c(\mathbf{w}^{-1},{}^{\mathbf{w}}\overline{\chi})\neq 0$.

Proof.

This follows from induction and Lemma 6.10. ∎

7. Calculation of certain local matrix coefficients

Assume that $n_{Q}\geq r$. The goal in this section is to calculation $\tau(\mathbf{w}_{G},{}^{\mathbf{w}_{G}}\overline{\chi},y,0)$ where $\overline{\chi}$ is an anti-exceptional character for $\overline{\operatorname{GL}}_{r}$. Recall that $\tau(\mathbf{w}_{G},{}^{\mathbf{w}_{G}}\overline{\chi},y,0)\neq 0$ unless $y=\mathbf{w}[0]$ for some $\mathbf{w}\in W$. As $n_{Q}\geq r$, the orbit of [math] under the action of $W$ is free. The theta representation $\Theta(\overline{G},{}^{\mathbf{w}_{G}}\overline{\chi})$ is realized as a subrepresentation of $I(\overline{\chi})$. Recall that $\phi_{K}\in\Theta(\overline{G},{}^{\mathbf{w}_{G}}\overline{\chi})$.

Lemma 7.1**.**

For $\mathbf{w}\in W$,

[TABLE]

Proof.

This follows from Theorem 3.11. Note that $c(\mathbf{w}_{G}\mathbf{w},\overline{\chi})=0$ unless $\mathbf{w}=\mathbf{w}_{G}$. ∎

Let $y\in Y$. We define the Gauss sum $g(\mathbf{w},y)$ for $\mathbf{w}\in W$ as follows:

1. (1)

$g(\mathrm{id},y)=1$; 2. (2)

For a simple reflection $\boldsymbol{\sigma}_{\alpha}$,

[TABLE] 3. (3)

If $\mathbf{w}_{1},\mathbf{w}_{2}\in W$ such that $\ell(\mathbf{w}_{1}\mathbf{w}_{2})=\ell(\mathbf{w}_{1})+\ell(\mathbf{w}_{2})$, then

[TABLE]

We have to verify that this is well-defined.

Lemma 7.2**.**

We have

[TABLE]

Therefore, $g(\mathbf{w},y)$ is well-defined.

Proof.

Recall that $\mathbf{w}[y]_{\rho}=\mathbf{w}(y-\rho)$ for any $\mathbf{w}\in W$. Fix a reduced decomposition $\mathbf{w}=\boldsymbol{\sigma}_{i_{1}}\cdots\boldsymbol{\sigma}_{i_{k}}$. Then

[TABLE]

The last equality follows from [Bum13] Proposition 20.10. ∎

We can now state the main result of this section.

Proposition 7.3**.**

For $\mathbf{w}\in W$,

[TABLE]

The rest of this section is devoted to proving this result. Before the proof, we need some preparation.

7.1. Two lemmas

Lemma 7.4**.**

We have $\tau(\boldsymbol{\sigma}_{\alpha},\overline{\chi},y,\boldsymbol{\sigma}_{\alpha}[y])=\mathbf{g}_{\psi^{-1}}(-\langle y_{\rho},\alpha\rangle Q(\alpha^{\vee}))$.

Proof.

This is done by direct calculation. Recall $\mathbf{w}[y]=\mathbf{w}(y-\rho)+\rho$. The left-hand side is

[TABLE]

∎

Lemma 7.5**.**

If $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})=\ell(\mathbf{w})+1$, then $\langle\mathbf{w}[0],\alpha\rangle\leq 0$; if $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})=\ell(\mathbf{w})-1$, then $\langle\mathbf{w}[0],\alpha\rangle>0$.

Proof.

If $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})=\ell(\mathbf{w})+1$, then $\mathbf{w}^{-1}(\alpha)$ is a positive root ([Bum13] Proposition 20.2). This implies that $\langle\mathbf{w}(\rho),\alpha\rangle=\langle\rho,\mathbf{w}^{-1}(\alpha)\rangle\geq 1$. Note that $\langle\rho,\alpha^{\vee}\rangle=1$. Thus

[TABLE]

We now consider the other case. Note

[TABLE]

If $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})=\ell(\mathbf{w})-1$ or $\ell(\boldsymbol{\sigma}_{\alpha}\mathbf{w})+1=\ell(\mathbf{w})$, then $\langle(\boldsymbol{\sigma}_{\alpha}\mathbf{w})[0],\alpha\rangle\leq 0$. But

[TABLE]

Here we use $\langle\boldsymbol{\sigma}_{\alpha}[0],\alpha\rangle=-2$. This gives the desired result. ∎

7.2. Proof of Proposition 7.3

We first check some small rank cases. If $r=1$, then both sides are $1$. If $r=2$, we only have two Weyl group elements two consider. If $\mathbf{w}=\mathrm{id}$, then

[TABLE]

if $\mathbf{w}=\mathbf{w}_{G}$, then clearly

[TABLE]

Our proof is a simplified version of the proof of [Suz97] Lemma 4.2. We now assume that the result is true for $r$ and prove it for $r+1$. We first apply the inductive formula in Proposition 6.1. Observe the following:

• •

Recall that $\mathbf{w}^{-1}[\mathbf{w}[0]]=0$ and we write $\mathbf{w}^{-1}=\mathbf{w}^{{}^{\prime}-1}\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{r_{0}}$ for a unique integer $r_{0}$ and $\mathbf{w}^{\prime}\in W(\mathbb{M})$.

• •

We are working with the exceptional representation. If $i\neq 1$, then $\prod_{j=1}^{i-1}\frac{1-q^{-1}\overline{\chi}_{ji}}{1-\overline{\chi}_{ji}}=0$ since $1-q^{-1}\overline{\chi}_{i-1,i}=0$. Thus only one term $(i=1)$ in the outer summation is nonzero.

We obtain

[TABLE]

where the sum is over the set

[TABLE]

Since the orbit of [math] (hence $y$) is free, this set has no repetition for different $(a_{1},\cdots,a_{r_{0}-2})$. Thus we now obtain

[TABLE]

where

[TABLE]

For $y^{\prime}=\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}^{\prime}[0]$, using Lemma 3.7, we see that $\tau(\boldsymbol{\sigma}_{1}\cdots\boldsymbol{\sigma}_{r},{}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi},y,y^{\prime})$ is the product of the following three terms:

• •

$\tau(\boldsymbol{\sigma}_{1}\cdots\boldsymbol{\sigma}_{r_{0}-2},~{}{}^{\boldsymbol{\sigma}_{r_{0}-2}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi},~{}y,~{}\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}[0])$,

• •

$\tau(\boldsymbol{\sigma}_{r_{0}-1},~{}{}^{\boldsymbol{\sigma}_{r_{0}-1}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi},~{}\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}[0],~{}\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}[0])$,

• •

$\tau(\boldsymbol{\sigma}_{r_{0}}\cdots\boldsymbol{\sigma}_{r},~{}{}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi},~{}\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}[0],~{}y^{\prime})$.

We now analyze each term. We start with $\mathcal{W}_{M,y^{\prime}}(0,{}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi})$.

Lemma 7.6**.**

We have

[TABLE]

Proof.

Here we apply Lemma 6.2. We observe that the character ${}^{\boldsymbol{\sigma}_{r}\cdots\boldsymbol{\sigma}_{1}}\overline{\chi}$ restricted to $\overline{T}_{\operatorname{GL}_{r},Q,n}^{sc}$ is again an anti-exceptional character. So we can apply induction to calculate the value. It is $g(\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}^{\prime},0)$. ∎

Lemma 7.7**.**

We have

[TABLE]

Proof.

Note $\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}$ is of the form $\boldsymbol{\sigma}_{r_{0}}\cdots\boldsymbol{\sigma}_{r}\mathbf{w}^{\prime\prime}$ with $\mathbf{w}^{\prime\prime}\in W(\mathbb{GL}_{r})$. Lemma 7.5 shows that $\langle\boldsymbol{\sigma}_{r_{0}}\cdots\boldsymbol{\sigma}_{r-1}\mathbf{w}^{\prime\prime}[0],\alpha_{r_{0}-1}\rangle<0$. Therefore $k_{\boldsymbol{\sigma}_{r_{0}}\cdots\boldsymbol{\sigma}_{r-1}\mathbf{w}^{\prime\prime}[0],\alpha_{r_{0}-1}}=0$ and

[TABLE]

∎

Lemma 7.8**.**

We have

[TABLE]

Proof.

Since the action of $\boldsymbol{\sigma}_{r_{0}}\cdots\boldsymbol{\sigma}_{r}$ and $\boldsymbol{\sigma}_{r_{0}-2}^{a_{r_{0}-2}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}$ are disjoint, we have

[TABLE]

We can now use Lemma 3.7 to calculate

[TABLE]

∎

Let us summarize what we have done so far. Let us rewrite

[TABLE]

and use

[TABLE]

By the above results, we deduce that

[TABLE]

Thus, it remains to the summation in the second line. Notice that

[TABLE]

We now rewrite the summation as

[TABLE]

We first calculate the inner sum.

Lemma 7.9**.**

The inner sum in (5) is equal to $\dfrac{1-q^{-(r_{0}-1)}}{1-q^{-(r_{0}-2)}}$.

Proof.

To calculate the inner sum, there are two cases to consider. (Note that this discussion does not appear in [Suz97] Sect. 4.2.)

For ease of notation, we write $\tilde{y}=\boldsymbol{\sigma}_{r_{0}-3}^{i_{r_{0}-3}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w}[0]$. Clearly, $k_{\tilde{y},\alpha_{r_{0}-2}}$ is either [math] or $1$. We have two cases to consider.

Case 1: $\ell(\boldsymbol{\sigma}_{r_{0}-2}\boldsymbol{\sigma}_{r_{0}-3}^{i_{r_{0}-3}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w})=\ell(\boldsymbol{\sigma}_{r_{0}-3}^{i_{r_{0}-3}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w})+1$.

When $a_{r_{0}-2}=0$,

[TABLE]

By Lemma 7.5, $\langle\tilde{y},\alpha_{r_{0}-2}\rangle\leq 0$ and $k_{\tilde{y},\alpha_{r_{0}-2}}=0$. Thus,

[TABLE]

When $a_{r_{0}-2}=1$,

[TABLE]

and

[TABLE]

Thus the inner sum in (5) is

[TABLE]

Case 2: $\ell(\boldsymbol{\sigma}_{r_{0}-2}\boldsymbol{\sigma}_{r_{0}-3}^{i_{r_{0}-3}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w})=\ell(\boldsymbol{\sigma}_{r_{0}-3}^{i_{r_{0}-3}}\cdots\boldsymbol{\sigma}_{1}^{a_{1}}\mathbf{w})-1$. When $a_{r_{0}-2}=0$,

[TABLE]

By Lemma 7.5, $\langle\tilde{y},\alpha_{r_{0}-2}\rangle>0$ and $k_{\tilde{y},\alpha_{r_{0}-2}}=1$. Thus,

[TABLE]

When $a_{r_{0}-2}=1$,

[TABLE]

and

[TABLE]

Then the inner sum in (5) is

[TABLE]

∎

This finishes the calculation of the inner sum in (5). We can now proceed for the other summations and deduce that (5) is

[TABLE]

By (4), this implies that $\mathcal{W}_{y}(0,\overline{\chi})=g(\mathbf{w},0)$. The proof of Proposition 7.3 is complete.

8. Main result

8.1. Statement

We now state our main result. We work with the group $\overline{\operatorname{GL}}_{r}$. Let $\Delta^{\prime}\subset\Delta$ so that the corresponding Levi subgroup is $\mathbb{M}=\mathbb{GL}_{r_{1}}\times\cdots\times\mathbb{GL}_{r_{k}}$. Let $\overline{\chi}$ be an $\Delta^{\prime}$-anti-exceptional character for $\overline{\operatorname{GL}}_{r}$. Define $e_{i}=r_{1}+\cdots+r_{i-1}$ and $x_{ij}=\overline{\chi}_{e_{i}+1,e_{j}+1}$.

Theorem 8.1**.**

Assume that

• •

for all $1\leq i\leq k$, $r_{i}\leq n_{Q}$,

• •

for all $1\leq i,j\leq k$, $r_{i}+r_{j}>n_{Q}$

• •

for $i=1,\cdots,k-1$,

[TABLE]

Then

[TABLE]

for $\mathbf{w}\in W(\mathbb{M})$.

We will first prove the result for $\mathbf{w}=\mathrm{id}$ and then for the general case. We begin with some remarks in the case of $\mathbf{w}=\mathrm{id}$.

Remark 8.2*.*

1. (1)

A result of [McN11] says that $\mathcal{W}_{0}(0,\overline{\chi})$ is a weighted sum over a finite crystal graph and is therefore a polynomial in $x_{12},\cdots,x_{k-1,k}$. Note that everything stated here is done in $\overline{\operatorname{SL}}_{r}$ so McNamara’s result does apply. 2. (2)

When $\mathbf{w}=\mathrm{id}$, we can rewrite the right-hand side as a polynomial in $x_{12},\cdots,x_{k-1,k}$. Let $f(x_{12},\cdots,x_{k-1,k})$ be this polynomial. The monomial with highest total degree is

[TABLE]

where

[TABLE]

is the right-hand side of (6). 3. (3)

The condition in (6) does seem strange and this is not satisfied for all tuples. However, it is easy to check that (6) holds when $(r_{1},\cdots,r_{k})=(n_{Q},\cdots,n_{Q},n^{\prime})$ where $1\leq n^{\prime}\leq n_{Q}$. 4. (4)

We expect the result to be true without the condition in (6). But we do not know how to extend it at the moment.

8.2. Proof of Theorem 8.1: the base case

For the base case, the proof presented here is adapted from [Kap19] Theorem 43. We will give also examples to explain some ideas and give the reader some flavor of the proof.

We now give an outline of the proof. We first observe that, by the results in [McN11], $\mathcal{W}_{0}(0,\overline{\chi})$ is weight sum over certain Gelfand-Tsetlin patterns and is therefore a polynomial in $x_{12},\cdots,x_{k-1,k}$.

It is sufficient to prove the following three things:

1. (1)

Every factor of $f(x_{12},\cdots,x_{k-1,k})$ divides $\mathcal{W}_{0}(0,\overline{\chi})$. 2. (2)

The monomial of the highest total degree of $\mathcal{W}_{0}(0,\overline{\chi})$ is the same as $f(x_{12},\cdots,x_{k-1,k})$, up to a scalar. 3. (3)

The constant coefficient of $f(x_{12},\cdots,x_{k-1,k})$ is $1$. So it is enough to prove that the constant coefficient of $\mathcal{W}_{0}(0,\overline{\chi})$ is $1$.

The first one is proved by a representation-theoretic argument. The last two are proved using the formula of [McN11] Sect. 8, which is based on the Gelfand-Tsetlin description of [BBF11] Sect. 8. Note that the proof in [Kap19] Theorem 43 does not use uniqueness of Whittaker models.

We start with the representation-theoretic argument.

Example 8.3*.*

We assume that $n=3$, $r=8$, $(r_{1},r_{2},r_{3})=(3,3,2)$, $Q(\alpha^{\vee})=1$. Let $\Delta^{\prime}=\{\alpha_{1},\alpha_{2},\alpha_{4},\alpha_{5},\alpha_{7}\}$ and $\overline{\chi}$ be a $\Delta^{\prime}$-anti-exceptional character. Therefore, $\overline{\chi}_{\alpha}=q$ for $\alpha\in\Delta^{\prime}$. Let $x_{1}=\overline{\chi}_{\alpha_{1}+\alpha_{2}+\alpha_{3}}$ and $x_{2}=\overline{\chi}_{\alpha_{4}+\alpha_{5}+\alpha_{6}}$. It is easy to check, for instance, $\overline{\chi}_{\alpha_{3}}=q^{-2}x_{1}$.

Clearly if $x_{1}=q^{3}$, then $\overline{\chi}$ is a $\Delta^{\prime}\cup\{\alpha_{3}\}$-anti-exceptional character and $\mathcal{W}_{0}(0,\overline{\chi})=0$ by Corollary 6.8. In other words, as a function of $x_{1},x_{2}$, $\mathcal{W}_{0}(0,\overline{\chi})$ is zero along the hyperplane $1-q^{-3}x_{1}=0$.

Now let us consider the following question: is $\mathcal{W}_{0}(0,\overline{\chi})$ along other hyperplanes? A quick examination shows that $1-q^{-2}x_{1}=0$ does the job. In fact, under this assumption, $\overline{\chi}_{\alpha_{3}+\alpha_{4}}=q$. Thus ${}^{\boldsymbol{\sigma}_{4}}\overline{\chi}$ is an $\{\alpha_{1},\alpha_{2},\alpha_{3}\}$-anti-exceptional character. We consider the following intertwining operator $T_{\boldsymbol{\sigma}_{4},{}^{\boldsymbol{\sigma}_{4}}\overline{\chi}}:I({}^{\boldsymbol{\sigma}_{4}}\overline{\chi})\to I(\overline{\chi})$. Using Lemma 6.10, it is easy to check

[TABLE]

If $\mathcal{W}_{0}(0,\overline{\chi})\neq 0$, then by composing this Whittaker functional with $T_{\boldsymbol{\sigma}_{4},{}^{\boldsymbol{\sigma}_{4}}\overline{\chi}}$, we obtain a nonzero Whittaker functional on $I({}^{\boldsymbol{\sigma}_{4}}\overline{\chi})$. However, this contradicts with Corollary 6.8 as ${}^{\boldsymbol{\sigma}_{4}}\overline{\chi}$ is an $\{\alpha_{1},\alpha_{2},\alpha_{3}\}$-anti-exceptional character.

The same argument shows that $\mathcal{W}_{0}(0,\overline{\chi})=0$ if $1-q^{-1}x_{1}=0$. The same argument can be applied for $x_{2}$.

We now consider the hyperplane $1-\overline{\chi}_{\alpha_{1}+\cdots+\alpha_{6}}q^{-2}=1-x_{1}x_{2}q^{-2}=0$. With this assumption, $\overline{\chi}_{\alpha_{3}+\cdots+\alpha_{7}}=q$. Therefore, ${}^{\boldsymbol{\sigma}_{4}\cdots\boldsymbol{\sigma}_{7}}\overline{\chi}$ is $\{\alpha_{1},\alpha_{2},\alpha_{3}\}$-anti-exceptional. The intertwining operator

[TABLE]

could have zeros. But $c(\boldsymbol{\sigma}_{7}\cdots\boldsymbol{\sigma}_{4},{}^{\boldsymbol{\sigma}_{4}\cdots\boldsymbol{\sigma}_{7}}\overline{\chi})$ has two types of factors: the first of the form $1-x_{2}q^{-l}$ for some integer $l$ and the factor as in the statement of Lemma 6.10. In any case, (7) is nonzero on spherical vectors along $1-x_{1}x_{2}q^{-2}=0$. Now the same argument as above shows that $\mathcal{W}_{0}(0,\overline{\chi})=0$.

By repeating this argument, one can find $3+2+2$ factors of $\mathcal{W}_{0}(0,\overline{\chi})$. They are exactly the factors appearing in the statement of Theorem 8.1.

Lemma 8.4**.**

If $1-x_{ij}q^{-l}=0$ for $i<j$ and $n_{Q}-r_{j}+1\leq l\leq r_{i}$, then $\mathcal{W}_{0}(0,\overline{\chi})=0$.

Proof.

We write $\overline{\chi}\sim(\overline{\chi}_{1},\cdots,\overline{\chi}_{k})$ where $\overline{\chi}_{m}$ is anti-exceptional for the group $\overline{\operatorname{GL}}_{r_{m}}$. We further write $\overline{\chi}_{j}\sim(\overline{\chi}_{j}^{\dagger},\overline{\chi}_{j}^{\ddagger})$ where the size of $\overline{\chi}_{j}^{\dagger}$ is $r_{i}-l$ (which could be [math]). Let $\mathbf{w}$ be the Weyl group element so that

[TABLE]

Observe that since $1-x_{ij}q^{-1}=0$, $(\overline{\chi}_{i},\overline{\chi}_{j}^{\ddagger})$ is an anti-exceptional character of size $r_{i}+r_{j}-r_{i}+l=r_{j}+l\geq n_{Q}+1$. Thus ${}^{\mathbf{w}}\overline{\chi}$ is an anti-exceptional character that satisfies the condition in Corollary 6.8.

We now check that the intertwining operator

[TABLE]

is nonzero on spherical vectors along $1-x_{ij}q^{-l}=0$. Indeed, it is enough check that $c(\mathbf{w}^{-1},{}^{\mathbf{w}}\overline{\chi})\neq 0$ along $1-x_{ij}q^{-l}=0$. A quick calculation shows that the denominator of $c(\mathbf{w}^{-1},{}^{\mathbf{w}}\overline{\chi})$ is either of the form $1-x_{jj^{\prime}}q^{-l^{\prime}}$ for $j^{\prime}\neq i,j$, or a factor of the form as in Lemma 6.10. In either case, this is nonzero when $1-x_{ij}q^{-l}=0$.

Suppose now that $\mathcal{W}_{0}(0,\overline{\chi})\neq 0$ along $1-x_{ij}q^{-l}=0$. We then have a Whittaker functional on $I({}^{\mathbf{w}}\overline{\chi})$ via

[TABLE]

This is nonzero since it is nonzero on the spherical vector $\phi_{K}$. However, by our discussion above, this contradicts Corollary 6.8. ∎

Thus we know that $(1-\chi_{ij}q^{-l})\mid\mathcal{W}_{0}(0,\overline{\chi})$. As the factors of $f(x_{12},\cdots,x_{k-1,k})$ are distinct and $\mathbb{C}[x_{12},\cdots,x_{k-1,k}]$ is a unique factorization domain, $f(x_{12},\cdots,x_{k-1,k})$ divides $\mathcal{W}_{0}(0,\overline{\chi})$.

We now use the formula in [McN11] to estimate the degree of $\mathcal{W}_{0}(0,\overline{\chi})$. We briefly recall how this is derived. In [McN11], with a choice of a reduced decomposition of the longest element in the Weyl group, McNamara introduces an algorithm, called explicit Iwasawa decomposition. This is to write an element $u\in U^{-}$ as $u=tnk$ where $t\in T$, $n\in U$ and $k\in K$. Equivalently, this is to write $U^{-}$ as a cell decomposition $U^{-}=\bigsqcup_{\mathbf{m}}C_{\mathbf{m}}$, where $\mathbf{m}$ is a tuple of integers. Thus one can write

[TABLE]

and this yields a combinatorial sum of these integrals. The main result in [McN11] says that unramified Whittaker functions can be calculated in this way, and the tuples $\mathbf{m}$ with nonzero contributions are in bijection with a set of Gelfand-Tsetlin patterns. The contribution can be calculated in terms of Gauss sums.

Recall the a strict Gelfand-Tsetlin pattern is a triangular array $\{a_{i,j}\}$ of non-negative integers, such that each row is strictly decreasing and $a_{i,j}\geq a_{i+1,j}\geq a_{i_{j+1}}$ for all $i,j$ such that all entries exist. For each $i$, define

[TABLE]

Here $\mathcal{W}_{0}(0,\overline{\chi})$ is expressed as a sum over the set of Gelfand-Tsetlin patterns with the first row $a_{1,j}=r-j,1\leq j\leq r$. The resulting monomial for such a pattern is of the form

[TABLE]

where $C$ is a certain product of powers of $q$ and Gauss sums.

In this paper, we choose a particular maximal abelian subgroup $\overline{A}$. This imposes another condition on the patterns we need to consider. With this choice of torus, by Lemma 3.1, the torus elements that lies in the support of $\phi_{K}$ are in $\overline{A}$. Also the torus elements appearing in the calculation are in $\overline{\operatorname{SL}}_{r}$. Recall that $Y_{Q,n}\cap Y^{sc}=Y_{Q,n}^{sc}=n_{Q}Y^{sc}$. As a consequence, the only patterns to consider are those where $d_{i}\equiv 0\mod n_{Q}$. (See also [Kap19] Theorem 43.)

Lemma 8.5**.**

The monomial of the highest total degree in $\mathcal{W}_{0}(0,\overline{\chi})$ is at most the same as in $f(x_{12},\cdots,x_{k-1,k})$, up to a scalar.

Example 8.6* (Continuation of Example 8.3).*

We continue with the set up in Example 8.3. A quick calculation shows that the monomial of the highest total degree should be $x_{1}^{5}x_{2}^{4}$.

Now let us check it from the Gelfand-Tsetlin description. We require that the $4$th and $7$th row to be as large as possible. So the maximal possible $4$th row is $(7~{}6~{}5~{}4~{}3)$ and the maximal possible $7$th row is $(7~{}6)$. This gives $d_{4}\leq 3\times 5=15$ and $d_{7}\leq 6\times 2=12$. Therefore, the monomial with the highest total degree is again $x_{1}^{5}x_{2}^{4}$.

[TABLE]

Proof of Lemma 8.5.

We now seek the monomial of highest total degree. We consider patterns with the maximal entries $a_{e_{l+1}+1,j}$ for $1\leq l\leq k-1$ and $1\leq j\leq r-(e_{l+1}+1)$ possible. We now fix $l$. Note that $a_{e_{l+1}+1,1}\leq r-1$ as $a_{1,1}=r-1$. The maximal possible choice of row $e_{l+1}+1$ is

[TABLE]

Therefore, $d_{e_{l+1}+1}\leq e_{l+1}(\sum_{j=l+1}^{k}r_{j})=(\sum_{j=1}^{l}r_{j})(\sum_{j=l+1}^{k}r_{j})$. As $d_{e_{l+1}+1}/n_{Q}$ must be an integer, this implies that

[TABLE]

The result now follows from (6). ∎

By the above two results, we know that $\mathcal{W}_{0}(0,\overline{\chi})=cf(x_{12},\cdots,x_{k-1,k})$ for some constant $c$. It remains to compute a single coefficient of $f$. In [Kap19], the highest monomial is used for this purpose. Here, we calculate the constant coefficient. We claim that only the lowest pattern contributes to the constant coefficient and the contribution is therefore $1$.

Example 8.7* (Continuation of Example 8.6).*

We are again in the setup of Example 8.3. We would like to show that only the lowest pattern contributes to the constant coefficient. First of all, we have $d_{4}=d_{7}=0$. This determines the $4$th and $7$th rows. These entries are as small as possible. Thus some entries in the other rows are determined. So far we have:

[TABLE]

We next show that $d_{i}=0$ for all $i$. This determines the pattern completely. For instance,

[TABLE]

As $d_{2}\equiv 0\mod 3$, we must have $d_{2}=0$. The other cases can be proved similarly.

Lemma 8.8**.**

Only the lowest pattern contributes to the constant coefficient of $\mathcal{W}_{0}(0,\overline{\chi})$.

Proof.

To find the term contributing to the constant coefficient, we must have

[TABLE]

Given a fixed $l$, this determines row $e_{l+1}+1$, which is

[TABLE]

The last $r-e_{l+1}$ entries from row $e_{l}+2$ to row $e_{l+1}$ are also determined. They are (8) as well. We now determine the remaining coefficients.

We now fix $1\leq l\leq k$. We argue by induction to show $d_{e_{l}+i}=0$ for $i=1,\cdots,r_{l}$. The case $i=1$ follows from our discussion above. We now assume that $d_{e_{l}+i}=0$. Then row $e_{l}+i$ is

[TABLE]

In other words, $a_{e_{l}+i,j}=r-e_{l}-i-j+1$. Thus,

[TABLE]

As $d_{e_{l}+i+1}\equiv 0\mod n_{Q}$, we deduce that $d_{e_{l}+i+1}=0$.

We now conclude that $d_{i}=0$ for $i$ and the pattern must be the lowest pattern. This completes the proof. ∎

It is straightforward to see that the contribution of the lowest pattern is $1$. The proof of the base case is now complete.

8.3. Proof of Theorem 8.1: the general case

We now prove the general case of Theorem 8.1. It remains to show that for $\mathbf{w}\in W(\mathbb{M})$,

[TABLE]

By Corollary 5.6, it suffices to show that

[TABLE]

We now use Lemma 4.3 to calculate the left-hand side. Suppose $\overline{\chi}\sim(\overline{\chi}_{1},\cdots,\overline{\chi}_{k})$ for some anti-exceptional characters $\overline{\chi}_{1},\cdots,\overline{\chi}_{k}$. We have

[TABLE]

This finishes the proof.