A reciprocal branching problem for automorphic representations and global Vogan packets

TL;DR

This paper explores the reciprocal branching problem for automorphic representations of special orthogonal groups, using the twisted automorphic descent method to find representations with prescribed restrictions, extending classical branching concepts.

Contribution

It introduces a novel approach to the reciprocal branching problem for automorphic representations, applying the twisted automorphic descent method to special orthogonal groups.

Findings

Established a framework for the reciprocal branching problem in automorphic representations.

Demonstrated the applicability of the twisted automorphic descent method to classical groups.

Extended the understanding of automorphic representation restrictions in the context of orthogonal groups.

Abstract

Let $G$ be a group and $H$ be a subgroup of $G$. The classical branching rule (or symmetry breaking) asks: For an irreducible representation $\pi$ of $G$, determine the occurrence of an irreducible representation $\sigma$ of $H$ in the restriction of $\pi$ to $H$. The reciprocal branching problem of this classical branching problem is to ask: For an irreducible representation $\sigma$ of $H$, find an irreducible representation $\pi$ of $G$ such that $\sigma$ occurs in the restriction of $\pi$ to $H$. For automorphic representations of classical groups, the branching problem has been addressed by the well-known global Gan-Gross-Prasad conjecture. In this paper, we investigate the reciprocal branching problem for automorphic representations of special orthogonal groups using the twisted automorphic descent method as developed in [JZ15]. The method may be applied to other classical groups as well.

Equations

$$\varphi =({\tau }_1,1)⊞\cdots ⊞({\tau }_r,1),$$

$${\tilde{\Pi }}_{\varphi }[G_n^{\ast }]:={\cup }_{G_n}{\tilde{\Pi }}_{\varphi }(G_n),$$

$${\varphi }^{\prime }=({\tau }_1^{\prime },1)⊞\cdots ⊞({\tau }_{r^{\prime }}^{\prime },1),$$

$${\tilde{\Pi }}_{\varphi \times {\varphi }^{\prime }}[G_n^{\ast }\times H_m^{\ast }]:={\cup }_{G_n\times H_m}{\tilde{\Pi }}_{\varphi \times {\varphi }^{\prime }}(G_n\times H_m),$$

$$L(s,\varphi \times {\varphi }^{\prime }):=\prod _{i=1}^r\prod _{j=1}^{r^{\prime }}L(s,{\tau }_i\times {\tau }_j^{\prime }).$$

$$\tau ={\tau }_1⊞{\tau }_2⊞\cdots ⊞{\tau }_r$$

$$\tilde{J}=\left(\begin{array}{ccc}& & w_{2n-\ell -1}\\ & J_{V_{0,\beta }}& \\ w_{2n-\ell -1}& & \end{array}\right),$$

$$V=V^{+}\oplus W\oplus V^{-},$$

$$\mathcal{F}:\{0\}\subset V_1^{+}\subset V_2^{+}\subset \cdots \subset V_{\tilde{m}}^{+}=V^{+}$$

$$⟨e_i,e_{-j}⟩={\delta }_{i,j},\text{for}1\le i,j\le \tilde{m}.$$

$$0\subset V_{\ell }^{+}\subset V^{+}.$$

$$V=V_{\ell }^{+}\oplus V^{(\ell )}\oplus V_{\ell }^{-}.$$

$${\mathcal{F}}_{\ell }:0\subset V_1^{+}\subset V_2^{+}\subset \cdots \subset V_{\ell }^{+}.$$

$${\varphi }_{\tau \otimes \sigma }\in \mathcal{A}(M(F)U(\mathbb{A})\backslash H(\mathbb{A}){)}_{\tau \otimes \sigma },$$

$$E(s,h,{\varphi }_{\tau \otimes \sigma })=\sum _{\gamma \in P(F)\backslash H(F)}{\lambda }_s{\varphi }_{\tau \otimes \sigma }(\gamma h),$$

$${\varphi }_{\tau \otimes \sigma }\in \mathcal{A}(M(F)U(\mathbb{A})\backslash H(\mathbb{A}){)}_{\tau \otimes \sigma }.$$

$$u=u_{\ell }(z,x,y)=\left(\begin{array}{ccc}z& z\cdot x& y\\ & I_{4n+3-2\ell }& x^{\prime }\\ & & z^{\ast }\end{array}\right),$$

$${\chi }_{\ell ,w_0}(u)=\sum _{i=2}^{\ell }⟨u\cdot e_i,e_{-(i-1)}⟩+⟨u\cdot w_0,e_{-\ell }⟩.$$

$${\psi }_{\ell ,w_0}={\psi }_F\circ {\chi }_{\ell ,w_0}:N_{\ell }(\mathbb{A})⟶{\mathbb{C}}^{\times },$$

$$f^{{\psi }_{\ell ,w_0}}(h)={\int }_{[N_{\ell }]}f(vh){\psi }_{\ell ,w_0}^{-1}(v)\mathrm{d}v,$$

$$w_0=y_{\beta }=e_{2n}+\frac{\beta }{2}{e}_{-2n}$$

$${\psi }_{\ell ,\beta }(u_{\ell }(z,x,y))$$

$$={\psi }_{Z_{\ell }}(z)\psi (x_{\ell ,2n-\ell }+\frac{\beta }{2}{x}_{\ell ,2n+4-\ell }).$$

$$\tilde{J}=\left(\begin{array}{ccc}& & w_{2n-\ell -1}\\ & J_{V_{0,\beta }}& \\ w_{2n-\ell -1}& & \end{array}\right),$$

$${\pi }_{\ell ,\beta }={\mathcal{D}}_{{\psi }_{\ell ,\beta }}({\mathcal{E}}_{\tau \otimes \sigma }).$$

$$J_{{\psi }_{\ell ,w_0}}(\Pi ):=\Pi /\mathrm{Span}\{\pi (u)\xi -{\psi }_{\ell ,w_0}(u)\xi |u\in N_{\ell }(F),\xi \in \Pi \}.$$

$$\tilde{J}=\{\begin{array}{ll}\left(\begin{array}{ccc}& & w_{2n-\ell -1}\\ & J_{W,\beta }& \\ w_{2n-\ell -1}& & \end{array}\right),& \text{if m=2n},\\ & \\ \left(\begin{array}{ccc}& & w_{2n-\ell }\\ & J_{\beta }& \\ w_{2n-\ell }& & \end{array}\right),& \text{if m=2n+1},\end{array}$$

$$\tau ={\mu }_1\times \cdots \times {\mu }_n\times {\mu }_n^{-1}\times \cdots \times {\mu }_1^{-1},$$

$$\left(\underset{\lambda }{⨁}{\mathrm{Ind}}_{B_{{\mathrm{SO}}_{2n+2}}(F)}^{{\mathrm{SO}}_{2n+2}(F)}{\mu }_1\otimes \cdots \otimes {\mu }_n\otimes \lambda \right)$$

$$\oplus \left(\underset{t=1}{\overset{n}{⨁}}{\mathrm{Ind}}_{P_{{\mathrm{SO}}_{2n+2}}^{1,\cdots ,1,2,1,\cdots ,1}(F)}^{{\mathrm{SO}}_{2n+2}(F)}{\mu }_1\otimes \cdots \otimes {\mu }_{t-1}\otimes {\mu }_t(\det {}_{{\mathrm{GL}}_2})\otimes {\mu }_{t+1}\otimes \cdots \otimes {\mu }_n\right),$$

$${\tau }^{\prime }={\mathrm{Ind}}_{P_{2,\cdots ,2}(F)}^{{\mathrm{GL}}_{2n}(F)}{\mu }_1(\det {}_{{\mathrm{GL}}_2})\otimes \cdots \otimes {\mu }_n(\det {}_{{\mathrm{GL}}_2}).$$

Full text

The first named author is partially supported by NSF grant DMS-1600685. The second named author is partially supported by NSF grant DMS-1702218 and by start-up funds from the Department of Mathematics at Purdue University. The third named author is partially supported by NSFC grant No.11501382 and by the Fundamental Research Funds for the Central Universities

1. Introduction

The classical branching rule or the so called symmetry breaking in representation theory is to ask a question that can be formulated in an over-simplified way as follows: Let $G$ be a group and $H$ be a subgroup. For an irreducible representation $\pi$ of $G$, the problem is to ask which irreducible representation $\sigma$ of $H$ occurs in the restriction of $\pi$ to $H$. A refinement of this classical problem is to ask if an irreducible representation $\sigma$ of $H$ with extra condition can appear in the decomposition when $\pi$ is restricted to $H$. Such a classical problem has been successfully studied in many different contents. For automorphic representations of classical groups, the global Gan-Gross-Prasad conjecture addresses this classical problem in a certain format, which will be described with some details below.

The objective of this paper is to consider the automorphic version of the problem reciprocal to the refined classical branching rule problem. The reciprocal branching problem could be formulated as follows: For an irreducible representation $\sigma$ of $H$, with a certain extra property, find an irreducible representation $\pi$ (possibly with a certain extra property) of $G$, which contains $H$ as a subgroup, such that $\sigma$ occurs in the restriction of $\pi$ to $H$. It is clear that without those extra conditions on $\pi$ or $\sigma$, the usual Frobenius reciprocity law suggests that one may take $\pi$ to be the induced representation from $\sigma$. However, with those extra conditions, the usual induced representation may not be enough for such refined problems. This paper is to understand this reciprocal problem in the theory of automorphic representations of special orthogonal groups in terms of global Vogan packets and with connection to the global Gan-Gross-Prasad conjecture. We believe that the method used in this paper should be applicable to other classical groups as well.

1.1. The branching problem and the global Gan-Gross-Prasad conjecture

Before introducing the precise problem we will consider, we recall from [8] the global Gan-Gross-Prasad conjecture for the case closely relevant to the topics discussed in this paper.

Let $F$ be a number field and ${\mathbb{A}}={\mathbb{A}}_{F}$ be the ring of adeles of $F$. Following [3], we denote by $G_{n}^{*}:={\mathrm{SO}}_{2n}^{\epsilon}$ an $F$-quasi-split special even orthogonal group, and denote by $H_{m}^{*}:={\mathrm{SO}}_{2m+1}^{*}$ the $F$-split special odd orthogonal group. Here ${\mathrm{SO}}_{2n}^{\epsilon}$ ($\epsilon\in F^{\times}/(F^{\times})^{2}$) is the $F$-quasi-split special orthogonal group determined by $(n-1)$ hyperbolic planes and $E_{\epsilon}=F[X]/(X^{2}-\epsilon)$, and is $F$-split if $\epsilon\in(F^{\times})^{2}$. As in [3] and [8], we denote by $G_{n}$ and $H_{m}$ a pure inner form of $G_{n}^{*}$ and $H_{m}^{*}$, respectively. Note that $G_{n}$ and $G_{n}^{*}$ share the same Langlands $L$-group, and so do $H_{m}$ and $H_{m}^{*}$. We recall the global Arthur parameters from [3], and consider the generic ones mostly. A generic global Arthur parameter for $G_{n}^{*}$ is given as a formal sum

[TABLE]

where $\tau_{i}$ with $i=1,2,\cdots,r$ is an irreducible unitary, self-dual cuspidal automorphic representation of ${\mathrm{GL}}_{a_{i}}({\mathbb{A}})$, and $\tau_{i}\not\cong\tau_{j}$ if $i\neq j$. Moreover, each $\tau_{i}$ is of orthogonal type in the sense that the symmetric square $L$-function $L(s,\tau_{i},{\mathrm{Sym}}^{2})$ has a pole at $s=1$. Note that one must have $2n=\sum_{i=1}^{r}a_{i}$. The global Arthur parameter $\phi$ as in (1.1) can be realized as an irreducible automorphic representation of ${\mathrm{GL}}_{2n}({\mathbb{A}})$. The set of generic global Arthur parameters of $G_{n}^{*}$ is denoted by $\widetilde{\Phi}_{2}(G_{n}^{*})$. For each Arthur parameter $\phi\in\widetilde{\Phi}_{2}(G_{n}^{*})$, the associated global Arthur packet is denoted by $\widetilde{\Pi}_{\phi}(G_{n}^{*})$. Since $G_{n}$ shares the Langlands $L$-group with $G_{n}^{*}$, an Arthur parameter $\phi\in\widetilde{\Phi}_{2}(G_{n}^{*})$ may be regarded as an Arthur parameter of $G_{n}$. However, the associated global Arthur packet $\widetilde{\Pi}_{\phi}(G_{n})$ could be empty. If the global Arthur packet $\widetilde{\Pi}_{\phi}(G_{n})$ is not empty, we call the global Arthur parameter $\phi$ of $G_{n}^{*}$ is $G_{n}$-relevant. The set of all $G_{n}$-relevant, generic global Arthur parameters of $G_{n}^{*}$ is denoted by $\widetilde{\Phi}_{2}(G_{n}^{*})_{G_{n}}$. Now the global Vogan packet associated to $\phi\in\widetilde{\Phi}_{2}(G_{n}^{*})$ is given by

[TABLE]

where $G_{n}$ runs over all pure inner forms of $G_{n}^{*}$ over $F$. Similarly, we have global Arthur parameters $\phi^{\prime}$ for $H_{m}^{*}$ as in (1.1),

[TABLE]

with only difference that $\tau^{\prime}_{1},\cdots,\tau^{\prime}_{r^{\prime}}$ are now of symplectic type. The set of generic global Arthur parameters of $H_{m}^{*}$ is denoted by $\widetilde{\Phi}_{2}(H_{m}^{*})$. Accordingly, we have the global Arthur packet $\widetilde{\Pi}_{\phi^{\prime}}(H_{m}^{*})$ and the global Vogan packet $\widetilde{\Pi}_{\phi^{\prime}}[H_{m}^{*}]$.

Assume that $(G_{n}^{*},H_{m}^{*})$ $(m<n)$ is a relevant pair in the sense of the Gan-Gross-Prasad conjecture in [8], and $G_{n}\times H_{m}$ is a relevant pure inner form of $G_{n}^{*}\times H_{m}^{*}$ over $F$. The global Vogan packet for $\phi\times\phi^{\prime}$ is given by

[TABLE]

where $G_{n}\times H_{m}$ runs through all relevant pure inner forms of $G_{n}^{*}\times H_{m}^{*}$ over $F$. The tensor product $L$-function associated to the pair $\phi$ and $\phi^{\prime}$ is defined to be

[TABLE]

The global Gan-Gross-Prasad conjecture (or GGP conjecture for short) ([8]) asserts that the central value $L(\frac{1}{2},\phi\times\phi^{\prime})$ is non-zero if and only if there exists a pair $(\pi_{0},\sigma_{0})$ in the global Vogan packet $\widetilde{\Pi}_{\phi\times\phi^{\prime}}[G_{n}^{*}\times H_{m}^{*}]$ with a non-zero Bessel period for $(\pi_{0},\sigma_{0})$. The uniqueness of such a pair $(\pi_{0},\sigma_{0})$ follows from the local GGP conjecture ([8]). When such a pair exists, we call it a the Gan-Gross-Prasad pair or GGP pair for short.

The branching problem for this case is to ask: For any $\pi$ in the global Vogan packet $\widetilde{\Pi}_{\phi}[G_{n}^{*}]$, is there any $\sigma$ in the global Vogan packet $\widetilde{\Pi}_{\phi^{\prime}}[H_{m}^{*}]$ such that $\pi\otimes\sigma$ belongs to the global Vogan packet $\widetilde{\Pi}_{\phi\times\phi^{\prime}}[G_{n}^{*}\times H_{m}^{*}]$ and $(\pi,\sigma)$ forms a GGP pair, i.e. $(\pi,\sigma)$ has a non-zero Bessel period. The global Gan-Gross-Prasad conjecture is to characterize the occurrence of $\sigma$ in the branching decomposition of $\pi$ with respect to relevant pair $(G_{n},H_{m})$ in terms of the central value of the $L$-function, $L(\frac{1}{2},\phi\times\phi^{\prime})$ and the local symplectic root numbers associated to the pair $(\pi,\sigma)$. Because of the uniqueness of the pair $(\pi_{0},\sigma_{0})$, it makes sense to ask the following question:

Reciprocal Branching Problem: For $\sigma\in\widetilde{\Pi}_{\phi^{\prime}}(H_{m})$, with $\sigma\not\cong\sigma_{0}$, how to find some group $G^{\prime}$ and an irreducible cuspidal automorphic representation $\pi^{\prime}$ of $G^{\prime}({\mathbb{A}})$ with a generic global Arthur parameter $\phi^{\prime}$, such that $G^{\prime}$ and $H_{m}$ form a relevant pair and $\pi^{\prime}$ and $\sigma$ have a non-zero Bessel period?

1.2. The reciprocal branching problem and the twisted automorphic descent

We are going to study this reciprocal branching problem for automorphic representations of orthogonal groups within the general framework of global Vogan packets and the global Gan-Gross-Prasad conjecture, by means of the twisted automorphic descents.

Assume that $\sigma$ belongs to the global Arthur packet $\widetilde{\Pi}_{\phi^{\prime}}(H_{m})$ for some pure inner form $H_{m}$ of $H_{m}^{*}$ over $F$. To present an answer to the reciprocal branching problem in this situation, one may take a generic global Arthur parameter $\phi\in\widetilde{\Phi}[G_{n}^{*}]$ for some quasi-split form $G_{n}^{*}$ at first. By the implication of the global GGP conjecture, we may also assume $L(\frac{1}{2},\phi\times\phi^{\prime})\neq 0$. Assume that $(\pi_{0},\sigma_{0})$ is the unique GGP pair in the global Vogan packet $\widetilde{\Pi}_{\phi\times\phi^{\prime}}[G_{n}^{*}\times H_{m}^{*}]$, which has a non-zero Bessel period, as given by the global GGP conjecture. If $\sigma\simeq\sigma_{0}$, then the global GGP conjecture predicts that the member $\pi_{0}$ in the global Vogan packet $\widetilde{\Pi}_{\phi}[G_{n}^{*}]$ gives an answer to the reciprocal branching problem. The twisted automorphic descent of [18] provides an explicit construction of this $\pi_{0}$ in terms of the generic global Arthur parameter $\phi$ and $\sigma$.

In this paper we consider the reciprocal branching problem for any $\sigma$ belonging to the global Vogan packet $\widetilde{\Pi}_{\phi^{\prime}}[H_{m}^{*}]$ but not equivalent to $\sigma_{0}$. The interesting part of this situation is that, by the uniqueness property in the GGP conjecture, for the fixed parameter $\phi$ of dimension $2n$, one does not expect that there exist $\pi\in\widetilde{\Pi}_{\phi}[G_{n}^{*}]$ such that $\pi$ and $\sigma$ have a non-zero Bessel period. The idea is to find an (specific) even special orthogonal group $G_{n+k}$ for some $k\geq 1$, which is a pure inner form of some quasi-split form $G_{n+k}^{*}$ and is relevant to $H_{m}$, and construct explicitly an irreducible cuspidal automorphic representation $\pi_{n+k}$ of $G_{n+k}({\mathbb{A}})$ with the properties that $\pi_{n+k}$ has a generic global Arthur parameter and has a non-zero Bessel period with the given $\sigma$. We expect that the integer $k$ should be determined by the first occurrence index of $\sigma$ (see [18] or §2.3 for the definition) in the tower of the Bessel descents from $\sigma$. Hence the construction of $G_{n+k}$ and $\pi_{n+k}$ should depend on the structure of the Bessel-Fourier coefficients of $\sigma$, which reflects the natural relation between the Bessel-Fourier coefficients and the twisted automorphic descents.

We will give an answer to this reciprocal branching problem when $m=1$. We fix a non-trivial additive character $\psi:F\backslash{\mathbb{A}}\longrightarrow{\mathbb{C}}^{\times}$ and fix an positive integer $n$. Let

[TABLE]

be the isobaric sum automorphic representation of ${\mathrm{GL}}_{2n}({\mathbb{A}})$. Here $\tau_{i}$ is a unitary irreducible cuspidal automorphic representation of ${\mathrm{GL}}_{n_{i}}({\mathbb{A}})$ such that $\sum_{i=1}^{r}n_{i}=2n$. Assume that $\tau$ corresponds to the generic Arthur parameter $\phi_{\tau}=\phi_{\tau_{1}}\boxplus\cdots\boxplus\phi_{\tau_{r}}$. Moreover, We assume that each $\tau_{i}$ is of orthogonal type, i.e. the $L$-function $L(s,\tau_{i},{\mathrm{Sym}}^{2})$ has a pole at $s=1$. Hence $\phi_{\tau}$ is a generic global Arthur parameter of some $G_{n}^{*}$.

On the other hand, we assume that $\tau_{0}$ is an irreducible unitary cuspidal representation of ${\mathrm{GL}}_{2}({\mathbb{A}})$ of symplectic type and has the property that $L(\frac{1}{2},\tau\times\tau_{0})\neq 0$. Let $V_{0}$ be a quadratic space over $F$ of dimension $3$ and $H_{1}^{V_{0}}={\mathrm{SO}}(V_{0})$ be the corresponding special orthogonal group. In any case, the $F$-split group is $H_{1}^{*}={\mathrm{SO}}_{3}$. We also denote by $J_{V_{0}}$ the quadratic form of $V_{0}$. Let $\sigma$ be an irreducible cuspidal representation of $H_{1}^{V_{0}}({\mathbb{A}})$ parametrized by $\phi_{\tau_{0}}$. For simplicity, we say a quasi-split form $G_{n}^{*}$ is relevant to $H_{1}^{V_{0}}$ if one of its pure inner form $G_{n}$ is relevant to $H_{1}^{V_{0}}$. In this paper, we assume

Assumption 1.1**.**

The representation $\sigma$ does not occur in the Gan-Gross-Prasad pair $(\pi_{0},\sigma_{0})$ in the global Vogan $L$-packet $\widetilde{\Pi}_{\phi_{\tau}\times\phi_{\tau_{0}}}[G^{*}_{n}\times H_{1}^{*}]$ for the fixed positive integer $n$ and any quasi-split form $G_{n}^{*}$ relevant to $H_{1}^{V_{0}}$.

The goal of this paper is, under Assumption 1.1, to construct an even special orthogonal group $G_{n+1}$ such that $G_{n+1}\times H^{V_{0}}_{1}$ is a relevant pure inner form of $G_{n+1}^{*}\times H_{1}^{*}$, and construct an irreducible cuspidal automorphic representation $\pi_{n+1}$ of $G_{n+1}({\mathbb{A}})$ such that $\pi_{n+1}$ has a generic Arthur parameter and has a non-zero Bessel period with respect to $\sigma$. The more precise explanation is in order.

For the given representations $(\tau,\sigma)$, we construct in §2 a square-integrable residual automorphic representation ${\mathcal{E}}_{\tau\otimes\sigma}$ on ${\mathrm{SO}}^{V_{0}}_{4n+3}({\mathbb{A}})$ with support $(P,\tau\otimes\sigma)$, where $P=MN$ is the standard parabolic subgroup of ${\mathrm{SO}}^{V_{0}}_{4n+3}$ such that $M\simeq{\mathrm{GL}}_{2n}\times H_{1}^{V_{0}}.$ A tower of automorphic descent of $\tau$ twisted by $\sigma$, which we denoted by $\pi_{\ell,\beta}={\mathcal{D}}_{\psi_{\ell,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ with $\beta\in F^{\times}$, can be constructed by taking the Bessel-Fourier coefficients of certain depth $\ell$ (see §2) of the residual representation ${\mathcal{E}}_{\tau\otimes\sigma}$. For each $\ell$, if the twisted descent $\pi_{\ell,\beta}={\mathcal{D}}_{\psi_{\ell,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero, then it consists of certain automorphic functions on $G_{2n-\ell+1,\beta}({\mathbb{A}})={\mathrm{SO}}_{4n-2\ell+2,\beta}({\mathbb{A}})$, with moderate growth. After choosing a suitable basis, the group $G_{2n-\ell+1,\beta}$ can be arranged to associate with the symmetric matrix

[TABLE]

where $w_{i}$ is an ($i\times i$)-matrix with only $1$’s on its anti-diagonal. Note that it could be either $F$-split, quasi-split or non-quasi-split, where $J_{V_{0,\beta}}=\begin{pmatrix}J_{V_{0}}&\\ &-\beta\end{pmatrix}$ is a $(4\times 4)$-symmetric matrix that defines the $4$-dimensional quadratic space $V_{0,\beta}$. We also denote by $\eta_{V_{0,\beta}}:F^{\times}\backslash{\mathbb{A}}^{\times}\longrightarrow\{\pm 1\}$ the quadratic character associated to the quadratic space $V_{0,\beta}$. We usually call $G_{2n-\ell+1,\beta}$ the target group of the descent module ${\mathcal{D}}_{\psi_{\ell},\beta}({\mathcal{E}}_{\tau\otimes\sigma})$. The point here is that under the Assumption 1.1, the first occurrence in this tower is at the depth $\ell^{*}=n$. In this case, we denote the resulting representation by $\pi_{\beta}={\mathcal{D}}_{\psi_{n},\beta}({\mathcal{E}}_{\tau\otimes\sigma})$, which consists of certain automorphic functions on $G_{n+1,\beta}({\mathbb{A}})={\mathrm{SO}}_{2n+2,\beta}({\mathbb{A}})$, with moderate growth. The main results of this paper can be summarized as follows.

Theorem 1.2**.**

Let $\tau$ and $\sigma$ be as given above.

• (a)

There exists $\beta\in F^{\times}$ such that the twisted descent $\pi_{\beta}={\mathcal{D}}_{\psi_{n},\beta}({\mathcal{E}}_{\tau\otimes\sigma})\neq 0$.

• (b)

If $\sigma$ satisfies Assumption 1.1, then $\pi_{\beta}=\bigoplus_{i}\pi_{\beta}^{(i)}$ is a multiplicity free direct sum of irreducible cuspidal automorphic representations $\pi_{\beta}^{(i)}$ of ${\mathrm{SO}}_{2n+2,\beta}({\mathbb{A}})$. Moreover, if $\omega_{\tau}=1$ and the $\beta\in F^{\times}$ in Part (a) is not a square, or if $\omega_{\tau}\neq 1$ and the $\beta\in F^{\times}$ in Part (a) is a square, then the same result holds without the above assumption.

• (c)

Assume $\omega_{\tau}=1$, and the $\beta\in F^{\times}$ in Part (a) is not a square. Then each irreducible component $\pi_{\beta}^{(i)}$ of $\pi_{\beta}$ belongs to a global Arthur packet with a generic global Arthur parameter $\phi^{(i)}$. The parameter $\phi^{(i)}$ has the central character $\eta_{V_{0,\beta}}$, and has the property that $L(\frac{1}{2},\phi^{(i)}\times\phi_{\tau_{0}})\neq 0$ and $(\pi_{\beta}^{(i)},\sigma)$ has a non-zero Bessel period.

Theorem 1.2 gives an answer to the reciprocal branching problem for $\sigma$, and one may refer to Theorem 6.3 at the end of this paper.

Part (a) of Theorem 1.2 provides a base for establishing the main results in this paper and will be proved in §5. It is always the hardest part in the theory of global automorphc descents to prove the global non-vanishing of the construction. In this paper, we find a new argument to do so. It is a combination of two methods recently developed. One of them is the result in [12] on relations between degenerate Whittaker models and generalized Whittaker models of representations (see §5.1 for definitions). Another one is the result in [14] on raising nilpotent orbits in the wave front set of representations. They are indispensable for this new way to establish the global non-vanishing of the construction of the twisted automorphic descents. More explicitly, first, using results in [12], we show that ${\mathcal{E}}_{\tau\otimes\sigma}$ has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n)^{2}1^{3}]$, which is not special. Then, using results in [14], we show that ${\mathcal{E}}_{\tau\otimes\sigma}$ has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n+1)(2n-1)1^{3}]$, which is the smallest orthogonal special partition bigger than $[(2n)^{2}1^{3}]$. This eventually implies that there exists $\beta\in F^{\times}$, such that ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero. We note that the method we use here is different from that previously used in the theory of descents (see, for example, [11, 15]), and is more conceptual and valid for more general situation (see Remark 5.5).

Part (c) of Theorem 1.2 is a connection of the descent construction to the reciprocal branching problem we are considering. Briefly the argument is as follows. Let $\pi_{\beta}^{(i)}$ be an irreducible component of the descent $\pi_{\beta}$, which is cuspidal. By the unramified structure and the endoscopic classification of Arthur ([3]), $\pi_{\beta}^{(i)}$ has a generic Arthur parameter $\phi^{(i)}$. By unfolding of Eisenstein series as used in [18], we can show that the Bessel period for $(\pi_{\beta}^{(i)},\sigma)$ is non-zero. Note that the main result of [18] also tells that the central value $L(\frac{1}{2},\phi^{(i)}\times\phi_{\tau_{0}})$ is also non-zero (see §6).

The content of this paper is organized as following. In §2, we introduce the construction of the automorphic descent in the case considered in this paper, and study some of its local structures in §3. In §4 – §5, we study some global aspects of the descent. In particular, we show the cuspidality of the descent in §4, and the non-vanishing of the descent in §5. Hence we obtain Part (a) and Part (b) of Theorem 1.2. Finally in §6, we prove Part (c) of Theorem 1.2.

Acknowledgement

Parts of this paper were written in the Spring of 2016 when the third named author visited the School of Mathematics, University of Minnesota. He appreciates very much the hospitality and comfortable working condition provided by the School of Mathematics. We would like to thank Lei Zhang for helpful comments.

2. Residual representations and the descent construction

2.1. Notation

We set up some general notation that will be used throughout this paper. Let $F$ be a field of characteristic [math], and $V$ be a quadratic space of dimension $4n+3$ defined over $F$, with quadratic form denoted by $\langle{\ ,\ }\rangle$. Let $\widetilde{m}$ be the Witt index of $V$. Then $V$ has a polar decomposition

[TABLE]

where $V^{+}$ has dimension $\widetilde{m}$ and is a maximal totally isotropic subspace of $V$, and $W$ is anisotropic of dimension $4n+3-2\widetilde{m}$. Fix a maximal flag

[TABLE]

in $V^{+}$, and choose a basis $\{e_{1},\cdots,e_{\widetilde{m}}\}$ of $V^{+}$ over $F$ such that $V_{i}^{+}={\mathrm{Span}}\{e_{1},\cdots,e_{i}\}$, for $1\leq i\leq\widetilde{m}$. Let $\{e_{-1},\cdots,e_{-\widetilde{m}}\}$ be a basis for $V^{-}$, which is dual to $\{e_{1},\cdots,e_{\widetilde{m}}\}$, that is,

[TABLE]

We have two cases to consider: $\widetilde{m}=2n$ or $\widetilde{m}=2n+1$. If $\widetilde{m}=2n$, take a basis $\{e_{0}^{(1)},e_{0}^{(2)},e_{0}^{(3)}\}$ of $W$ such that the anisotropic quadratic form on $W$ is associated with $\displaystyle{J_{W}=\begin{pmatrix}1&&\\ &\delta&\\ &&\alpha\end{pmatrix}}$, where $\delta,\alpha\in F^{\times}$. In this case, the ternary quadratic form $x^{2}+\delta y^{2}+\alpha z^{2}$ does not represent [math] in $F$, and the form of $V$ is associated with $J=\begin{pmatrix}&&w_{2n}\\ &J_{W}&\\ w_{2n}&&\end{pmatrix}$, where $w_{i}$ is a $(i\times i)$-matrix with $1$’s on the anti-diagonal and zero’s elsewhere. If $\widetilde{m}=2n+1$, take a basis $\{e_{0}\}$ of $W$ with $\langle{e_{0},e_{0}}\rangle=1$. Then the form of $V$ is associated with $J=w_{4n+3}$. For each case, let $H={\mathrm{SO}}(V)$, which could be $F$-split or $F$-non-split, according to the above two cases.

For $1\leq\ell\leq\widetilde{m}$, let $Q_{\ell}$ be the (maximal) parabolic subgroup fixing the flag

[TABLE]

Then $Q_{\ell}$ has a Levi decomposition $M^{\prime}_{\ell}\cdot U_{\ell}$ with Levi subgroup $M^{\prime}_{\ell}\simeq{\mathrm{GL}}_{\ell}\times{\mathrm{SO}}(V^{(\ell)})$, where $V^{(\ell)}$ is the subspace which sits into the decomposition

[TABLE]

For simplicity, when $\ell=2n$, let $P=Q_{2n}$, $M=M^{\prime}_{2n}$, $U=U_{2n}$, and $V_{0}=V^{(2n)}$. Then $M\simeq{\mathrm{GL}}_{2n}\times{\mathrm{SO}}(V_{0})$, and ${\mathrm{SO}}(V_{0})$ is an $F$- split or $F$-non-split special orthogonal group with $\mathrm{dim}\,V_{0}=3$. Accordingly, we may sometimes denote $H={\mathrm{SO}}(V)$ by ${\mathrm{SO}}_{4n+3}^{V_{0}}$ if we want to indicate the structure of the group. Note that if $\widetilde{m}=2n$, we have $V_{0}=W$, the anisotropic kernel of $V$ (and hence $J_{V_{0}}=J_{W}$).

For $1\leq\ell\leq\widetilde{m}$, we also let $P_{\ell}$ be the parabolic subgroup of $H$ which stabilizes the partial flag

[TABLE]

It has a Levi decomposition $P_{\ell}=M_{\ell}\cdot N_{\ell}$ with Levi subgroup $M_{\ell}\simeq{\mathrm{GL}}_{1}^{\ell}\times{\mathrm{SO}}(V^{(\ell)})$.

2.2. The residual representations

Now we take $F$ to be a number field and denote ${\mathbb{A}}={\mathbb{A}}_{F}$ to be its ring of adeles. Let $\tau=\tau_{1}\boxplus\tau_{2}\boxplus\cdots\boxplus\tau_{r}$ be an isobaric sum automorphic representation of ${\mathrm{GL}}_{2n}({\mathbb{A}})$, and $\sigma$ an irreducible unitary cuspidal representation of ${\mathrm{SO}}(V_{0})({\mathbb{A}})$. Here $\tau_{i}$’s are unitary irreducible cuspidal automorphic representations of ${\mathrm{GL}}_{n_{i}}({\mathbb{A}})$, satisfying $\sum_{i=1}^{r}n_{i}=2n$. Take $H={\mathrm{SO}}_{4n+3}^{V_{0}}$ with $\widetilde{m}=2n$ as before. For $s\in{\mathbb{C}}$ and an automorphic function $H({\mathbb{A}})$

[TABLE]

following [20, §II.1], one defines $\lambda_{s}\phi_{\tau\otimes\sigma}$ to be $(\lambda_{s}\circ m_{P})\phi_{\tau\otimes\sigma}$, where $\lambda_{s}\in X_{M}^{H}\simeq{\mathbb{C}}$ (see [20, §I.1] for the definition of $X_{M}^{H}$ and the map $m_{P}$), and defines the corresponding Eisenstein series

[TABLE]

which converges absolutely for ${\mathrm{Re}}(s)\gg 0$ and has meromorphic continuation to the whole complex plane ([20, §IV]). Under the normalization of Shahidi ([22]), one may take $\lambda_{s}=s\widetilde{\alpha}$, where $\alpha$ is the unique reduced root of the maximal $F$-split torus of $H$ in $U$.

When $L(s,\tau_{i},{\mathrm{Sym}}^{2})$ has a pole at $s=1$ for all $i=1,\cdots,r$ (i.e. $\tau$ is of orthogonal type), and $L(\frac{1}{2},\tau\times\sigma)\neq 0$, the Eisenstein series $E(s,h,\phi_{\tau\otimes\sigma})$ has a pole at $s=\frac{1}{2}$ of order $r$ (see [18, Proposition 5.3]). Let ${\mathcal{E}}_{\tau\otimes\sigma}$ denote the automorphic representation of $H({\mathbb{A}})$ generated by the iterated residues ${\mathrm{Res}}_{s=\frac{1}{2}}E(s,h,\phi_{\tau\otimes\sigma})$ for all

[TABLE]

It is square integrable by the $L^{2}$-criterion in [20]. Moreover, the residual representation ${\mathcal{E}}_{\tau\otimes\sigma}$ is irreducible (see [19, Theroem A]). Note that the global Arthur parameter ([3]) for ${\mathcal{E}}_{\tau\otimes\sigma}$ is $(\tau_{1},2)\boxplus(\tau_{2},2)\boxplus\cdots\boxplus(\tau_{r},2)\boxplus\phi_{\sigma}$ ([18, §6]).

2.3. The twisted automorphic descent

The twisted automorphic descents was introduced in [18], which extends the automorphic descent of Ginzburg-Rallis-Soudry ([11]) to much more general situation. Following [18] and [11], we introduce a family of Bessel-Fourier coefficients that defines the descent.

For $1\leq\ell<2n$, we consider the parabolic subgroup $P_{\ell}=M_{\ell}N_{\ell}$ of $H$ defined in §2.1. The nilpotent subgroup $N_{\ell}(F)$ consists of elements of the form

[TABLE]

where $z\in Z_{\ell}(F)$, $x\in\mathrm{Mat}_{\ell\times(4n+3-2\ell)}(F)$, and $y\in\mathrm{Mat}_{\ell\times\ell}(F)$. Here $Z_{\ell}$ is the maximal upper-triangular unipotent subgroup of ${\mathrm{GL}}_{\ell}$, and $x^{\prime}=J^{(\ell)}\,\!{}^{t}\!xw_{\ell}$, where $J^{(\ell)}=\begin{pmatrix}&&w_{2n-\ell}\\ &J_{V_{0}}&\\ w_{2n-\ell}&&\end{pmatrix}$. Take an anisotropic vector $w_{0}\in W_{\ell}$ with $\langle{w_{0},w_{0}}\rangle$ in a given square class of $F^{\times}$, and define a homomorphism $\chi_{\ell,w_{0}}:N_{\ell}\longrightarrow{\mathbb{G}}_{a}$ by

[TABLE]

Define also a character

[TABLE]

with $\psi_{F}:F\backslash{\mathbb{A}}\longrightarrow{\mathbb{C}}^{\times}$ being a fixed non-trivial additive character. It is clear that the character $\psi_{\ell,w_{0}}$ is trivial on $N_{\ell}(F)$. Now the adjoint action of $M_{\ell}$ on $N_{\ell}$ induces an action of ${\mathrm{SO}}(V^{(\ell)})$ on the set of all such characters $\psi_{\ell,w_{0}}$. The stabilizer $L_{\ell,w_{0}}$ of $\chi_{\ell,w_{0}}$ in ${\mathrm{SO}}(V^{(\ell)})$ is equal to ${\mathrm{SO}}(w_{0}^{\perp}\cap V^{(\ell)})$.

Let $\Pi$ be an automorphic representation of $H({\mathbb{A}})$. For $f\in V_{\Pi}$ and $h\in H({\mathbb{A}})$, we define the $\psi_{\ell,w_{0}}$-Fourier coefficients of $f$ by

[TABLE]

where $[N_{\ell}]$ denotes the quotient $N_{\ell}(F)\backslash N_{\ell}({\mathbb{A}})$. This is one of the Fourier coefficients of $f$ associated to the partition $[(2\ell+1)1^{4n+2-2\ell}]$. As $\ell$ varies, it produces a family of Bessel-Fourier coefficients of $f$ needed in this paper. It is clear that $f^{\psi_{\ell,w_{0}}}(h)$ is left $L_{\ell,w_{0}}(F)$-invariant, and is of moderate growth on the Siegel domain of $L_{\ell,w_{0}}(F)\backslash L_{\ell,w_{0}}({\mathbb{A}})$.

As explained in [15, §2] and [18, §2.4], for $1\leq\ell<2n$, we may take

[TABLE]

for some $\beta\in F^{\times}$ as a precise choice of $w_{0}$. We have $\langle{y_{\beta},y_{\beta}}\rangle=\beta$, and also denote $\psi_{\ell,\beta}=\psi_{\ell,y_{\beta}}$ for simplicity. Then in matrix form, we have

[TABLE]

It follows that the stabilizer of the character is $L_{\ell,\beta}:=L_{\ell,y_{\beta}}={\mathrm{SO}}_{4n+2-2\ell,\beta}$, which is an even special orthogonal group associated to the symmetric matrix

[TABLE]

where $\displaystyle{J_{V_{0,\beta}}=\begin{pmatrix}J_{V_{0}}&\\ &-\beta\end{pmatrix}}$. This group can be split, quasi-split or non-quasi-split over $F$, depending on the choice of $V_{0}$ and $\beta$ (Proposition 2.5 of [18]). We also denote by $V_{0,\beta}$ the $4$-dimensional quadratic space over $F$ associated to $J_{V_{0,\beta}}$. Define ${\mathcal{D}}_{\psi_{\ell,\beta}}(\Pi)$ to be the space of $L_{\ell,\beta}({\mathbb{A}})$-span of the automorphic functions f^{\psi_{\ell,w_{0}}}\big{|}_{L_{\ell,\beta}({\mathbb{A}})} with all $f\in V_{\Pi}$, where the group $L_{\ell,\beta}({\mathbb{A}})$ acts by right translation.

Following [18], the twisted automorphic descent construction in this paper is to take $\Pi={\mathcal{E}}_{\tau\otimes\sigma}$ in the above construction, here ${\mathcal{E}}_{\tau\otimes\sigma}$ is the residual representation we have introduced before. In this case, we have a family of automorphic $L_{\ell,\beta}({\mathbb{A}})$-modules:

[TABLE]

It has been seen in many previous works (see [11, 15, 18]) that $L_{\ell,\beta}({\mathbb{A}})$-modules $\pi_{\ell,\beta}$ satisfy the so-called tower property when the depth $\ell$ varies. That is, there exists an $\ell^{*}$ such that $\pi_{\ell^{*},\beta}\neq 0$ for some choice of data, and $\pi_{\ell,\beta}=0$ for all $\ell^{*}<\ell<2n$. We call $\ell^{*}$ the first occurrence index (of $\sigma$) for the tower $\{\pi_{\ell,\beta}\}_{\ell}$. In particular, at the first occurrence index $\ell^{*}=\ell^{*}({\mathcal{E}}_{\tau\otimes\sigma})$, the $L_{\ell^{*},\beta}({\mathbb{A}})$-module $\pi_{\ell^{*},\beta}={\mathcal{D}}_{\psi_{\ell^{*},\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ consists of cuspidal automorphic functions $f^{\psi_{\ell^{*},\beta}}(\cdot)$ (see [11, 15, 18]). In this case we denote $\pi_{\beta}=\pi_{\ell^{*},\beta}$ for simplicity, and call it the automorphic descent of $\tau$ to $L_{\ell^{*},\beta}({\mathbb{A}})$, twisted by $\sigma$ or $\sigma$-twisted automorphic descent of $\tau$. To be compatible with the notation before, we denote $G_{2n-\ell+1,\beta}=L_{\ell,\beta}$, and call $G_{2n-\ell^{*}+1,\beta}({\mathbb{A}})$ the target group of this descent construction.

In this paper, we will restrict ourselves to the pair $(\tau,\sigma)$ that satisfies Assumption 1.1 in §1. In this situation, the first occurrence index is $\ell^{*}=n$ (see §4–§5), and the target group is $G_{n+1,\beta}({\mathbb{A}})={\mathrm{SO}}_{2n+2,\beta}({\mathbb{A}})$, as stated in Theorem 1.2.

3. Local aspects of the descent

In this section, we study the local analogue of the automorphic descent discussed in Section 2.3, which is certain twisted Jacquet module. We will calculate these Jacquet modules at unramified places.

3.1. The twisted Jacquet modules

We define the twisted Jacquet modules in both cases we are considering. Let $F$ be a $p$-adic field of characteristic [math], and fix a non-trivial additive character $\psi:F\longrightarrow{\mathbb{C}}^{\times}$. Let $H={\mathrm{SO}}(V)$ be a special orthogonal group over $F$ with $\dim(V)=4n+3$, and suppose that the Witt index $\widetilde{m}=2n$ or $2n+1$. Let $1\leq\ell\leq\widetilde{m}$ and take an anisotropic vector $w_{0}\in V^{(\ell)}$. We define the character $\psi_{\ell,w_{0}}$ on $N_{\ell}(F)$ similar to (2.1) in the global setting. Note that if $\ell=\widetilde{m}$, we take $w_{0}\in W=V^{(\widetilde{m})}$, the anisotropic kernel of $V$ (see §2.1).

For an irreducible admissible representation $\Pi$ of $H(F)$, one defines the Jacquet module of Bessel type (of depth $\ell$) to be

[TABLE]

Embed ${\mathrm{SO}}(V^{(\ell)})$ into $H$ via the Levi subgroup $M_{\ell}$, and set $L_{\ell,w_{0}}={\mathrm{Stab}}_{\psi_{\ell,w_{0}}}({\mathrm{SO}}(V^{(\ell)}))$ as before. By definition, $J_{\psi_{\ell,w_{0}}}(\Pi)$ is a $L_{\ell,w_{0}}(F)$-module. As before, when $\ell<\widetilde{m}$, we may take $w_{0}=y_{\beta}=e_{2n}+\frac{\beta}{2}e_{-2n}\in V^{(\ell)}$ with $\beta\in F^{\times}$. Then we set $J_{\psi_{\ell,y_{\beta}}}(\Pi)=J_{\psi_{\ell,\beta}}(\Pi)$, and $L_{\ell,y_{\beta}}=L_{\ell,\beta}$. With a suitable choice of basis, $L_{\ell,\beta}(F)$ is determined by the symmetric matrix

[TABLE]

where $J_{\beta}=\begin{pmatrix}1&\\ &-\beta\end{pmatrix}$. Recall that we have $V_{0}=W$ if $\widetilde{m}=2n$ (see §2.1).

3.2. The local unramified calculation of Jacquet modules

We keep some notation which are used in [11, Chapter 5].

Case I: trivial central character. Let $\tau$ be an irreducible, generic, admissible representation of ${\mathrm{GL}}_{2n}(F)$, which is of orthogonal type. We assume moreover that $\omega_{\tau}=1$. To consider the unramified cases, we write $\tau$ as a fully induced representation from the Borel subgroup:

[TABLE]

where $\mu_{i}$’s are unramified characters of $F^{\times}$. First we consider the case that $\widetilde{m}=2n+1$, and hence $H={\mathrm{SO}}_{4n+3}(F)$ is $F$-split. Let $\sigma={\mathrm{Ind}}_{B_{{\mathrm{SO}}_{3}}(F)}^{{\mathrm{SO}}_{3}(F)}\xi$, here $\xi$ is also an unramified character of $F^{\times}$. Let $\pi_{\tau\otimes\sigma}$ be the unramified constituent of the induced representation ${\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}_{4n+3}(F)}(\tau\lvert\cdot\rvert^{1/2}\otimes\sigma)$. We want to study the unramified constituents of the twisted Jacquet module $J_{\psi_{\ell,\beta}}(\pi_{\tau\otimes\sigma})$.

Proposition 3.1**.**

Assume that $\widetilde{m}=2n+1$. Then the following hold:

1. (1)

*Suppose that $\beta\in F^{\times}-{(F^{\times})}^{2}$. Then the twisted Jacquet module $J_{\psi_{\ell,\beta}}(\pi_{\tau\otimes\sigma})=0$ for all $n+1\leq\ell\leq 2n$. And when $\ell=n$, any unramified constituent of $J_{\psi_{n,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of ${\mathrm{Ind}}^{{\mathrm{SO}}_{2n+2,\beta}(F)}_{B_{{\mathrm{SO}}_{2n+2,\beta}}(F)}\mu_{1}\otimes\cdots\otimes\mu_{n}\otimes 1\,.$ * 2. (2)

Suppose that $\beta\in{(F^{\times})}^{2}$. Then the twisted Jacquet module $J_{\psi_{\ell,\beta}}(\pi_{\tau\otimes\sigma})=0$ for all $n+2\leq\ell\leq 2n$. When $\ell=n+1$, any unramified constituent of $J_{\psi_{n+1,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{2n}}(F)}^{{\mathrm{SO}}_{2n}(F)}\mu_{1}\otimes\cdots\otimes\mu_{n}\,,$ and when $\ell=n$, the unramified constituent of $J_{\psi_{n,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of

[TABLE]

where $\lambda$ runs over all unramified representations (characters) of occurring in the restriction of $\sigma$ to the $F$-split torus ${\mathrm{SO}}_{2}(F)$.

Proof.

We will apply [11, Theorem 5.1] to calculate $J_{\psi_{\ell,\beta}}(\pi_{\tau\otimes\sigma})$, and we also follow the same notation there. By conjugation of some Weyl element, it suffices to consider the unramified constituent of the induced representation ${\mathrm{Ind}}_{Q_{2n}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau^{\prime}\otimes\sigma$ instead of $\pi_{\tau\otimes\sigma}$, here

[TABLE]

Note that the derivative (see [4] and [11, §5]) $\tau^{\prime(\ell)}$ of $\tau^{\prime}$ vanishes for $\ell\geq n+1$. Now applying [11, Theorem 5.1 (i)] with $j=2n<\widetilde{m}$, one can see that the corresponding twisted Jacquet module vanishes for all $n+2\leq\ell\leq 2n+1$.

When $\ell=n+1$, by the formula in [11, Theorem 5.1 (1)] we have

[TABLE]

here $\delta_{\beta}=1$ if $\beta\in{(F^{\times})}^{2}$, and is zero otherwise, and $Q^{\prime}_{n,\pm}$ are defined as in [11, Remark 5.1]. Note that $|\cdot|^{\frac{1-n}{2}}\tau^{\prime(n)}=\mu_{1}\times\cdots\times\mu_{n}$, we obtain all the statements for $n+1\leq\ell\leq 2n$ in part (1) and (2) of the proposition.

Finally we take $\ell=n$. Also by [11, Theorem 5.1 (1)], we have

[TABLE]

where $J_{\mathrm{Wh},\pm}(\sigma)$ are Jacquet modules with respect to certain Whittaker characters (see [11, Proposition 5.5]). Here we have used the fact that $\tau^{\prime}_{(n)}=0$ (see, for example, [11, page 99]), and $Q^{\prime}_{n}$ is defined in [11, (5.16)]. Note also that here $Q_{n}^{\prime}$ contains $B_{{\mathrm{SO}}_{2n+2,\beta}}$. Moreover, we have $J_{\mathrm{Wh},\pm}(\sigma)\simeq{\mathbb{C}}$ since $\sigma$ is generic. Since $\delta_{\beta}=0$ if $\beta\notin{(F^{\times})}^{2}$, we can conclude (1) from the above formula. And if $\beta\in{(F^{\times})}^{2}$, we take $(n-1)$-th derivative of $\tau^{\prime}$ (as in [4, §4]), and then the last expression stated in the proposition follows as desired. The $\lambda$’s in Part (2) of the proposition come from the Jacquet module $J_{\psi^{\prime}_{0,\beta}}(\sigma)$, which is exactly the restriction of $\sigma$ to the $F$-split torus ${\mathrm{SO}}_{2}(F)$. ∎

Now suppose that $\widetilde{m}=2n$, so that $V^{(2n)}=V_{0}$ is an anisotropic quadratic space of dimension $3$. Let $\sigma$ be an irreducible admissible (finite dimensional) representation of ${\mathrm{SO}}_{3}^{V_{0}}(F)$. Given $\tau$ be as in (3.2), we also want to study the possible unramified constituents of the twisted Jacquet module $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)\,.$ Since we are interested in the unramified constituents, we only need to consider $\beta\in F^{\times}$ such that the target group $G_{2n-\ell+1,\beta}$ is $F$-quasi-split.

Proposition 3.2**.**

Let $\tau$ and $\sigma$ be as above, and take $\beta\in F^{\times}$ such that $G_{2n-\ell+1,\beta}$ is quasi-split over $F$. Then any unramified constituent of $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)$ is zero for $n+1\leq\ell\leq 2n-1$. When $\ell=n$, any unramified constituent of $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)$ is a subquotient of ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{2n+2,\beta}}(F)}^{{\mathrm{SO}}_{2n+2,\beta}(F)}\mu_{1}\otimes\cdots\otimes\mu_{n}\otimes 1\,.$

Proof.

We will also use [11, Theorem 5.1]. By conjugation of some Weyl element, it suffices to consider the unramified constituent of $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau^{\prime}\otimes\sigma)$, where

[TABLE]

Applying [11, Theorem 5.1 (2)] with $j=2n=\widetilde{m}$, we can see that $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau^{\prime}\otimes\sigma)=0$ for $n+1\leq\ell\leq 2n-1$. When $\ell=n$, we have

[TABLE]

Here $\delta^{1}_{{\mathrm{SO}}^{V_{0}}_{4n+3},\beta}=1$ (see [11, Theorem 5.1]) only if $\beta\in F^{\times}$ is taken to be such that $G_{2n-\ell+1,\beta}$ is quasi-split over $F$, and $Q_{\beta}^{\prime}$ is the same as in [11, (5.30)]. Moreover, the Jacquet module $J_{\psi_{0}^{\prime},v_{\beta}}(\sigma)$ is just the restriction to an $F$-quasi-split group ${\mathrm{SO}}^{V_{0}}_{2,\beta}(F)$. Then we obtain the last statement the proposition by taking $n$-th derivative of $\tau^{\prime}$. ∎

Case II: non-trivial central character. Let $\lambda_{0}$ be the unique non-trivial unramified quadratic character of $F^{\times}$, and let

[TABLE]

where $\mu_{i}$’s are unramified characters of $F^{\times}$ (recall that $\tau$ is of orthogonal type). We also first consider the case $\widetilde{m}=2n+1$. Let $\sigma={\mathrm{Ind}}_{B_{{\mathrm{SO}}_{3}}(F)}^{{\mathrm{SO}}_{3}(F)}\xi$, where $\xi$ is also an unramified character of $F^{\times}$. In this case, we have

Proposition 3.3**.**

The twisted Jacquet module $J_{\psi_{\ell,\beta}}(\pi_{\tau\otimes\sigma})=0$ for all $n+2\leq\ell\leq 2n$. Moreover, define that $\delta_{\beta}=1$ if $\beta\in{(F^{\times})}^{2}$ and $\delta_{\beta}=0$ otherwise, and define that $\delta^{0}_{\beta}=1$ if $\beta\notin{(F^{\times})}^{2}$ and $\delta^{0}_{\beta}=0$ otherwise. Then

1. (1)

When $\ell=n+1$, any unramified constituent of $J_{\psi_{n+1,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of

[TABLE] 2. (2)

When $\ell=n$, any unramified constituent of $J_{\psi_{n,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of

[TABLE]

Proof.

By conjugation of some Weyl element, it suffices to consider the unramified constituent of the induced representation ${\mathrm{Ind}}_{Q_{2n-2}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1}$, here

[TABLE]

and $\sigma_{1}={\mathrm{Ind}}_{Q_{2}^{{\mathrm{SO}}_{7}}(F)}^{{\mathrm{SO}}_{7}(F)}\widetilde{\tau}_{1}\otimes\mathbf{1}_{{\mathrm{SO}}_{3}}$, with $\widetilde{\tau}_{1}=\lambda_{0}|\cdot|^{1/2}\times\xi$ being a representation of ${\mathrm{GL}}_{2}(F)$. Note that $\sigma_{1}$ is non-generic and $\tau^{\prime(\ell)}=0$ for $\ell\geq n$. Applying [11, Theorem 5.1 (1)] with $j=2n-2<\widetilde{m}=2n+1$, we see that if $n+2\leq\ell\leq 2n$, this twisted Jacquet module is always zero.

Now we consider the case $\ell=n+1$. Applying [11, Theorem 5.1 (1)] with $j=2n-2<\widetilde{m}=2n+1$, we have

[TABLE]

To calculate the Jacquet module $J_{\psi^{\prime}_{2,\beta}}(\sigma_{1})$, applying [11, Theorem 5.1 (1)] again with $j=2$ and $\ell=2$, we have

[TABLE]

Hence by taking derivatives $\tau_{1}^{\prime(n-1)}$ and $\widetilde{\tau}_{1}^{(2)}$, we can see that any unramified constituent of $J_{\psi_{n+1,\beta}}({\mathrm{Ind}}_{Q_{2n-2}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1})$ is a subquotient of

[TABLE]

On the other hand, we may also write $\sigma_{1}={\mathrm{Ind}}_{Q_{2}^{{\mathrm{SO}}_{7}}(F)}^{{\mathrm{SO}}_{7}(F)}\widetilde{\tau}_{2}\otimes\sigma_{\lambda_{0}}$, with $\widetilde{\tau}_{2}=|\cdot|^{1/2}\times\xi$ and $\sigma_{\lambda_{0}}$ is the unique irreducible quotient in ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{3}}(F)}^{{\mathrm{SO}}_{3}(F)}\lambda_{0}|\cdot|^{1/2}$. Recall that $\lambda_{0}$ is a quadratic character, then $\sigma_{\lambda_{0}}$ can be viewed as a character of ${\mathrm{PGL}}_{2}(F)$ by composing it with the determinant, modulo squares. If $\beta\notin(F^{\times})^{2}$, similarly we can see that any unramified constituent of $J_{\psi_{n+1,\beta}}({\mathrm{Ind}}_{Q_{2n-2}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1})$ also lies in (3.4). If $\beta\in(F^{\times})^{2}$, applying [11, Theorem 5.1 (2)] with $j=1$ and $\ell=0$, we can see that

[TABLE]

and hence any unramified constituent of $J_{\psi_{n+1,\beta}}({\mathrm{Ind}}_{Q_{2n-2}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1})$ is a subquotient of

[TABLE]

Hence when $\beta\in(F^{\times})^{2}$, any unramified constituent of $J_{\psi_{n+2,\beta}}(\pi_{\tau\otimes\sigma})$ is a subquotient of both (3.4) and (3.5). If it is non-zero, then the two sets $\{\mu_{1}^{\pm},\mu_{2}^{\pm},\cdots,\mu_{n-1}^{\pm},1\}$ and $\{\mu_{1}^{\pm},\mu_{2}^{\pm},\cdots,\mu_{n-1}^{\pm},\lambda_{0}\}$ will be equal, which is impossible. This shows that any irreducible constituent must be [math], and this finishes the proof of Part (1) of the proposition.

Finally we take $\ell=n$. Applying [11, Theorem 5.1 (1)] with $j=2n-2<\widetilde{m}=2n+1$, we have

[TABLE]

Now, arguing as in the case of $\ell=n+1$, we can see that if an unramified constituent of $J_{\psi_{n,\beta}}({\mathrm{Ind}}_{Q_{2n-1}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1})$ comes from the first summand of (3.6), then it is a subquotient of

[TABLE]

To compute the Jacquet module $J_{\psi^{\prime}_{1,\beta}}(\sigma_{1})$ in the second summand of (3.6), we apply [11, Theorem 5.1 (1)] with $j=2$ and $\ell=1$, and get

[TABLE]

Moreover, one may also write $\sigma_{1}={\mathrm{Ind}}_{Q_{2}^{{\mathrm{SO}}_{7}}(F)}^{{\mathrm{SO}}_{7}(F)}\widetilde{\tau}_{2}\otimes\sigma_{\lambda_{0}}$ and get

[TABLE]

Recall that we have $J_{\psi^{\prime}_{0,\beta}}(\sigma_{\lambda_{0}})\equiv\lambda_{0}$ if $\beta\in(F^{\times})^{2}$. Then we obtain that, if an unramified constituent of $J_{\psi_{n,\beta}}({\mathrm{Ind}}_{Q_{2n-1}(F)}^{{\mathrm{SO}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1})$ comes from the second summand of (3.6), then it is a subquotient of

[TABLE]

as desired. ∎

We still need to consider the case that $\widetilde{m}=2n$, where $\sigma$ is an irreducible admissible representation of a non-split group ${\mathrm{SO}}_{3}^{V_{0}}(F)$ over $F$.

Proposition 3.4**.**

Let $\tau$ be as in (3.3), $\sigma$ be as above, and take $\beta\in F^{\times}$ such that $G_{2n-\ell+1,\beta}$ is quasi-split over $F$. Then any unramified constituent of the twisted Jacquet module $J_{\psi_{\ell,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)$ is zero for $n+2\leq\ell\leq 2n-1$. Moreover, the following hold.

1. (1)

When $\ell=n+1$, any unramified constituent of $J_{\psi_{n+1,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)$ is a subquotient of ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{2n,\beta}}(F)}^{{\mathrm{SO}}_{2n,\beta}(F)}\mu_{1}\otimes\cdots\otimes\mu_{n-1}\otimes 1\,.$ 2. (2)

When $\ell=n$, any unramified constituent of $J_{\psi_{n,\beta}}({\mathrm{Ind}}_{P(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau|\cdot|^{1/2}\otimes\sigma)$ is a subquotient of

[TABLE]

Proof.

As before, it suffices to consider the unramified constituent of the induced representation ${\mathrm{Ind}}_{Q_{2n-2}(F)}^{{\mathrm{SO}}^{V_{0}}_{4n+3}(F)}\tau_{1}^{\prime}\otimes\sigma_{1}$, here

[TABLE]

and $\sigma_{1}={\mathrm{Ind}}_{Q_{2}^{{\mathrm{SO}}^{V_{0}}_{7}}(F)}^{{\mathrm{SO}}^{V_{0}}_{7}(F)}\tau_{2}\otimes\sigma$ with $\tau_{2}=\lambda_{0}|\cdot|^{1/2}\times|\cdot|^{1/2}$. Note that $\sigma_{1}$ is non-generic. Applying [11, Theorem 5.1 (2)] with $j=2n-2$, we can see that this twisted Jacquet module is zero if $n+2\leq\ell\leq 2n-1$, since ${\tau_{1}^{\prime}}^{(\ell)}=0$ for $\ell\geq n$.

When $\ell=n+1$, also by [11, Theorem 5.1 (1)], we have

[TABLE]

Note that we have assumed that $G_{2n-\ell+1,\beta}$ is $F$-quasi-split. Consider the Jacquet module $J_{\psi^{\prime}_{2,v_{\beta}}}(\sigma_{1})$. Applying [11, Theorem 5.1 (2)] with $\ell=2$ and $j=2$, we have

[TABLE]

The last Jacquet module is just a restriction to the anisotropic ${\mathrm{SO}}_{2,\beta}(F)$. This proves Part (1) of the proposition.

Finally we take $\ell=n$. Applying [11, Theorem 5.1 (1)] with $j=2n-2$, we have

[TABLE]

We still need to calculate the Jacquet module $J_{\psi^{\prime}_{1,\beta}}(\sigma_{1})$. Applying [11, Theorem 5.1 (2)] again with $j=2$ and $\ell=1$, we have

[TABLE]

Then, granting the above calculation of $J_{\psi^{\prime}_{2,v_{\beta}}}(\sigma_{1})$, we obtain Part (2) of the proposition by considering the unramified constituents. ∎

We consider moreover the special case that $\ell=\widetilde{m}$, and $w_{0}\in W$. The result is the following:

Proposition 3.5**.**

Let $\tau$ and $\sigma$ be the same as in any case of Propositions 3.1–3.4, and let $w_{0}\in W$. Then we have $J_{\psi_{\widetilde{m},w_{0}}}(\pi_{\tau\otimes\sigma})=0\,.$

Proof.

As before, conjugating by some Weyl element, it suffices to consider the unramified constituent of $\displaystyle{J_{\psi_{\widetilde{m},w_{0}}}({\mathrm{Ind}}_{P(F)}^{H(F)}(\tau^{\prime}\otimes\sigma))}$, where

[TABLE]

or

[TABLE]

By [11, Theorem 5.1 (3)], we have

[TABLE]

where $d_{\tau^{\prime}}$ is the dimension of the space of $\psi$-Whittaker functionals on $\tau^{\prime}$. By the construction of $\tau^{\prime}$, we can see that $d_{\tau^{\prime}}=0$, and the proposition follows. ∎

At the end of this section, we consider an additional case which will be used in the proof of Proposition 4.3 later. In this case, we consider the induced representation

[TABLE]

of the $F$-split group ${\mathrm{SO}}_{4n+1}(F)$. Here $R_{2n-1}\subset{\mathrm{SO}}_{4n+1}$ is the maximal parabolic subgroup whose Levi subgroup is isomorphic to ${\mathrm{GL}}_{2n-1}\times{\mathrm{SO}}_{3}$, $\sigma={\mathrm{Ind}}_{B_{{\mathrm{SO}}_{3}}(F)}^{{\mathrm{SO}}_{3}(F)}\xi$ with $\xi$ being an unramified character of $F^{\times}$, and

[TABLE]

with $\mu_{i}$’s and $\lambda$ being unramified characters of $F^{\times}$ and in addition $\lambda$ being quadratic. For $1\leq\ell\leq 2n$, let $N_{\ell}^{\sharp}\subset{\mathrm{SO}}_{4n+1}$ and $\psi_{\ell,\beta}^{\sharp}:N_{\ell}^{\sharp}(F)\longrightarrow{\mathbb{C}}^{\times}$ be similar to $N_{\ell}\subset H$ and $\psi_{\ell,\beta}:N_{\ell}(F)\longrightarrow{\mathbb{C}}^{\times}$ in previous parts of this section. Now let $\pi_{\tau_{1}\otimes\sigma}$ be the unramified constituent of ${\mathrm{Ind}}_{R_{2n-1}}^{{\mathrm{SO}}_{4n+1}(F)}\tau_{1}|\cdot|^{1/2}\otimes\sigma$, we are going to consider the twisted Jacquet module $J_{\psi^{\sharp}_{\ell,\beta}}(\pi_{\tau_{1}\otimes\sigma})\,.$

Proposition 3.6**.**

The following hold:

1. (1)

Suppose that $\beta\in F^{\times}-{(F^{\times})}^{2}$. Then any twisted Jacquet module $J_{\psi^{\sharp}_{\ell,\beta}}(\pi_{\tau_{1}\otimes\sigma})=0$ for all $n\leq\ell\leq 2n$. 2. (2)

Suppose that $\beta\in{(F^{\times})}^{2}$. Then any twisted Jacquet module $J_{\psi^{\sharp}_{\ell,\beta}}(\pi_{\tau_{1}\otimes\sigma})=0$ for all $n+1\leq\ell\leq 2n$. When $\ell=n$, any unramified constituent of $J_{\psi^{\sharp}_{n+1,\beta}}(\pi_{\tau_{1}\otimes\sigma})$ is a subquotient of ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{2n}}(F)}^{{\mathrm{SO}}_{2n}(F)}\mu_{1}\otimes\cdots\otimes\mu_{n-1}\otimes\lambda\,.$

Proof.

The calculation is similar to those in other cases. By conjugation of some Weyl element, it suffices to consider the unramified constituent of the induced representation ${\mathrm{Ind}}_{R_{2n-2}(F)}^{{\mathrm{SO}}_{4n+1}(F)}\tau_{1}^{\prime}\otimes\sigma^{\prime}$ instead of $\pi_{\tau\otimes\sigma}$, here

[TABLE]

and $\sigma^{\prime}={\mathrm{Ind}}_{P_{{\mathrm{SO}}_{5}}^{1}(F)}^{{\mathrm{SO}}_{5}(F)}\xi\otimes\sigma_{\lambda}$. As before, we have deonted by $\sigma_{\lambda}$ the unique irreducible quotient of ${\mathrm{Ind}}_{B_{{\mathrm{SO}}_{3}}(F)}^{{\mathrm{SO}}_{3}(F)}\lambda|\cdot|^{1/2}$, which can be viewed as a character of ${\mathrm{PGL}}_{2}(F)$ by composing it with the determinant, modulo squares. Note that $\sigma^{\prime}$ is non-generic and the derivative $\tau_{1}^{\prime(\ell)}$ of $\tau_{1}^{\prime}$ vanishes for $\ell\geq n$. Now applying [11, Theorem 5.1 (i)] with $j=2n-2$, we can see that the corresponding twisted Jacquet module vanishes for all $n+1\leq\ell\leq 2n$.

When $\ell=n$, by the formula in [11, Theorem 5.1 (i)] we have

[TABLE]

Now we consider $J_{{\psi^{\sharp\,\prime}_{1,\beta}}}(\sigma^{\prime})$. Applying [11, Theorem 5.1 (i)] again with $\ell=1$ and $j=1$, we get

[TABLE]

which is zero unless $\beta\in(F^{\times})^{2}$, in which case we have $J_{{\psi^{\sharp\,\prime}_{0,\beta}}}(\sigma_{\lambda})\equiv\lambda$. Then the desired results follow. ∎

4. The cuspidality of the descent

In the section, we show the cuspidality of the descent representation. We come back to the global settings and start with the following lemma, which is a direct consequence of the local calculations in the previous section (Propositions 3.1 and 3.5). Throughout this section, we let $\tau$ and $\sigma$ be as in §2.2.

Lemma 4.1**.**

The Bessel-Fourier coefficients $f^{\psi_{\ell,w_{0}}}$ vanish for any anisotropic vector $w_{0}\in V^{(\ell)}$ and $f\in{\mathcal{E}}_{\tau\otimes\sigma}$ while $n+2\leq\ell\leq\widetilde{m}$.

Proof.

Write the residual representation as ${\mathcal{E}}_{\tau\otimes\sigma}=\otimes_{v}^{\prime}\Pi_{v}$. Note that $H(F_{v})$ is $F_{v}$-split at almost all places $v$. We fix a finite place $v$ such that $H(F_{v})={\mathrm{SO}}_{4n+3}(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, and $\omega_{\tau_{v}}$ is trivial. By a suitable conjugation we may assume that $w_{0}=y_{\beta}$ for some $\beta\in F^{\times}$ (see [15] or [18, Lemma 2.4]), and consider the corresponding twisted Jacquet module $J_{\psi_{\ell,\beta}}(\Pi_{v})$. Write

[TABLE]

where $\mu_{i,v}$’s are unramified characters of $F_{v}^{\times}$. By Propositions 3.1 Part (2) and Proposition 3.5, the Jacquet module $J_{\psi_{\ell,\beta}}(\Pi_{v})$ vanishes for any $n+2\leq\ell\leq\widetilde{m}$, hence the Bessel-Fourier coefficient $f^{\psi_{\ell,w_{0}}}$ vanishes for any $n+2\leq\ell\leq\widetilde{m}$. This completes the proof of the lemma. ∎

To prove the cuspidality of the descent, we also need the vanishing property for the depth $\ell=n+1$.

Proposition 4.2**.**

Suppose that $(\tau,\sigma)$ satisfies Assumption 1.1. Then the Bessel-Fourier coefficients $f^{\psi_{n+1,w_{0}}}$ vanish for any anisotropic vector $w_{0}\in V^{(\ell)}$ and $f\in{\mathcal{E}}_{\tau\otimes\sigma}$.

Proof.

Assume that the Bessel-Fourier coefficient $f^{\psi_{n+1,\beta}}\neq 0$ for some choice of data. Let ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ be the ${\mathrm{SO}}_{2n,\beta}({\mathbb{A}})$-span of the coefficients $f^{\psi_{n+1,\beta}}|_{{\mathrm{SO}}_{2n,\beta}({\mathbb{A}})}$. Note that ${\mathrm{SO}}_{2n,\beta}({\mathbb{A}})$ is relevant to ${\mathrm{SO}}(V_{0})({\mathbb{A}})$. Then by a formula for constant terms in [11, Theorem 7.2] and a similar argument to that in the proof of [11, Theorem 7.6] (see also the remarks in [11, §7.4] and the proof of [18, Proposition 6.3]), the automorphic module ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is cuspidal and has a direct sum decomposition of irreducible cuspidal automorphic representations.

Let $\pi$ be any irreducible summand of ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$. We claim that $\pi$ has the generic global Arthur parameter $\phi_{\tau}$, and lies in the global Vogan packet $\widetilde{\Pi}_{\phi_{\tau}}[{\mathrm{SO}}^{*}_{2n}]$.

We prove the claim case by case. Frist we assume that $\omega_{\tau}=1$. If $\beta\in(F^{\times})^{2}$, then for almost all places $v$ of $F$, we have $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, $\omega_{\tau_{v}}=1$, and $\beta\in(F_{v}^{\times})^{2}$. Fix any such place $v$, and write

[TABLE]

where $\mu_{i,v}$’s are unramified characters of $F_{v}^{\times}$. By Proposition 3.1 Part (2), the unramified constituent of the Jacquet module $J_{\psi_{n+1,\beta}}(\Pi_{v})$ is a subquotient of

[TABLE]

which shows that the Satake parameter of $\pi_{v}$ matches the Satake parameter of $\tau_{v}$, for almost all places. This shows that the global Arthur parameter $\phi_{\pi}=\phi_{\tau}$. If $\beta\notin(F^{\times})^{2}$, then there exists a finite place $v$ such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, $\omega_{\tau_{v}}=1$, and $\beta\notin(F_{v}^{\times})^{2}$. Then by Proposition 3.1 Part (1), one sees that ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})=0$, which is not the case we are considering.

Next we assume that $\omega_{\tau}\neq 1$. If $\beta\in(F^{\times})^{2}$, then there exists a finite place $v$ such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, $\omega_{\tau_{v}}\neq 1$, and $\beta\in(F_{v}^{\times})^{2}$. Then by Proposition 3.3 Part (1), one sees that ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})=0$, which is also not the case we are considering. If $\beta\notin(F^{\times})^{2}$, since we just consider the case that ${\mathcal{D}}_{\psi_{n+1,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})\neq 0$, then by Proposition 3.1 Part (2) and Proposition 3.3 Part (1), for the finite places $v$ such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, we have either $\omega_{\tau_{v}}=1$ and $\beta\in(F_{v}^{\times})^{2}$ or $\omega_{\tau_{v}}\neq 1$ and $\beta\notin(F_{v}^{\times})^{2}$. Now we consider the parameter of $\pi$. For the places such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, $\omega_{\tau_{v}}=1$, and $\beta\in(F_{v}^{\times})^{2}$, the argument for the first case ($\omega_{\tau}=1$ and $\beta\in(F^{\times})^{2}$) already shows that the Satake parameters of $\tau_{v}$ and $\pi_{v}$ match each other. And for the places such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$, $\psi_{v}$ are all unramified, $\omega_{\tau_{v}}\neq 1$, and $\beta\notin(F_{v}^{\times})^{2}$, we write

[TABLE]

where $\lambda_{0,v}$ is an unramified quadratic character of $F_{v}^{\times}$. Then by Proposition 3.3 Part (1), the unramified constituent of the Jacquet module $J_{\psi_{n+1,\beta}}(\Pi_{v})$ is a subquotient of

[TABLE]

which also shows that the Satake parameters of $\tau_{v}$ and $\pi_{v}$ match each other. Therefore, the Satake parameter of $\pi_{v}$ matches the Satake parameter of $\tau_{v}$ for almost all places, and hence $\phi_{\pi}=\phi_{\tau}$ by strong multiplicity one theorem for general linear groups.

We continue the proof of the proposition. By construction, there exists an automorphic form $\varphi_{\pi}\in\pi$ such that the inner product

[TABLE]

for some choice of data. By [18, Corollary 4.4], the pair $(\pi,\sigma)$ has a non-zero Bessel period, and hence is the GGP pair in the given global Vogan packet $\widetilde{\Pi}_{\phi_{\pi}\times\phi_{\tau_{0}}}[{\mathrm{SO}}^{*}_{2n}\times{\mathrm{SO}}^{*}_{3}]$ in Assumption 1.1. But this contradicts to Assumption 1.1. Hence the proposition is proved. ∎

As a corollary, we have the cuspidality of the descent ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$.

Proposition 4.3**.**

Suppose that $(\tau,\sigma)$ satisfies Assumption 1.1. Then the twisted automorphic descent ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is a cuspidal automorphic $G_{n+1,\beta}({\mathbb{A}})$-module.

Proof.

The proposition can be proved by considering all the constant terms of the Bessel-Fourier coefficient $f^{\psi_{n,\beta}}$ with $f\in{\mathcal{E}}_{\tau\otimes\sigma}$. The proof is similar to that of [11, Theorem 7.6], see also [15, Proposition 2.6] and [18, Proposition 6.3]. We briefly introduce it as follows.

Let $\widetilde{m}_{n,\beta}$ be the Witt index of $y_{\beta}^{\perp}\cap V^{(n)}$. Using [11, Theorem 7.3], and also the remarks in [11, §7.4], the constant terms $c_{p}(f^{\psi_{n,\beta}})$ $(1\leq p\leq\widetilde{m}_{n,\beta})$, along maximal parabolic subgroup of $G_{n+1,\beta}$ with Levi subgroup isomorphic to ${\mathrm{GL}}_{p}\times G_{n+1-p,\beta}$, can be expressed as a summation whose summands are expressed as integrals of following terms

[TABLE]

where $f^{U_{p-i}}$ is the constant term of $f$ along the maximal parabolic subgroup $Q_{p-i}\subset H$ with Levi subgroup isomorphic to ${\mathrm{GL}}_{p-i}\times{\mathrm{SO}}(V^{(p-i)})$. Here $(f^{U_{p-i}})^{\psi_{n+i,\beta}}$ denotes the $\psi_{n+i,\beta}$-Bessel-Fourier coefficient of $f^{U_{p-i}}|_{{\mathrm{SO}}(V^{(p-i)})({\mathbb{A}})}$. By Lemma 4.1 and Proposition 4.2, $f^{\psi_{n+p,\beta}}$ vanishes since $p\geq 1$. By the cuspidal support of ${\mathcal{E}}_{\tau\otimes\sigma}$, the constant term $f^{U_{p-i}}$ vanishes unless the parabolic $Q_{p-i}$ contains the cuspidal support $\tau_{1}\otimes\cdots\otimes\tau_{r}\otimes\sigma$. It is easy to see that $f^{U_{p-i}}$ belongs to the representation

[TABLE]

where $\{i_{1},\ldots,i_{k}\}\cup\{j_{1},\ldots,j_{r-k}\}=\{1,2,\ldots,r\}$, and ${\mathcal{E}}_{(\tau_{j_{1}}\boxplus\cdots\boxplus\tau_{j_{r-k}})\otimes\sigma}$ is the residual representation of ${\mathrm{SO}}(V^{(p-i)})({\mathbb{A}})$ constructed in the same way as ${\mathcal{E}}_{\tau\otimes\sigma}$. Note that $f^{U_{p-i}}|_{{\mathrm{SO}}(V^{(p-i)})({\mathbb{A}})}\in{\mathcal{E}}_{(\tau_{j_{1}}\boxplus\cdots\boxplus\tau_{j_{r-k}})\otimes\sigma}$. Using a similar argument to that in Lemma 4.1 (note that here $n+i$ is large enough by Assumption 1.1, Propositions 3.1 and 3.6), by Propositions 3.1 and 3.6 again, all $\psi_{n+i,\beta}$-Bessel-Fourier coefficients of ${\mathcal{E}}_{(\tau_{j_{1}}\boxplus\cdots\boxplus\tau_{j_{r-k}})\otimes\sigma}$ vanish, hence $(f^{U_{p-i}})^{\psi_{n+i,\beta}}$ also vanishes for any $0\leq i\leq p-1$. Therefore, all the constant terms of $f^{\psi_{n,\beta}}$ along various unipotent subgroups of $G_{n+1,\beta}$ vanish, which means that the Bessel-Fourier coefficient $f^{\psi_{n,\beta}}$ is cuspidal, and the twisted automorphic descent ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is a cuspidal automorphic $G_{n+1,\beta}({\mathbb{A}})$-module.

This completes the proof of the proposition. ∎

In following proposition, we show that in certain cases, to obtain the cuspidality of the descent, we do not need to assume Assumption 1.1.

Proposition 4.4**.**

If we take $w_{0}=y_{\beta}$ with $\beta\notin(F^{\times})^{2}$ when $\omega_{\tau}=1$, or take $w_{0}=y_{\beta}$ with $\beta\in(F^{\times})^{2}$ when $\omega_{\tau}\neq 1$, then the Bessel-Fourier coefficient $f^{\psi_{\ell,\beta}}$ vanishes for any $f\in{\mathcal{E}}_{\tau\otimes\sigma}$ and any $n+1\leq\ell\leq\widetilde{m}$, and the descent ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is cuspidal.

Proof.

We have already known from Lemma 4.1 that $f^{\psi_{\ell,\beta}}$ vanishes for any $n+2\leq\ell\leq\widetilde{m}$. Hence, for the first statement, we only need to consider the case of $\ell=n+1$. As we have seen in the proof of Proposition 4.2, if $\beta\notin(F^{\times})^{2}$ and $\omega_{\tau}=1$, there exists a finite place $v$ of $F$ such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$ and $\psi_{v}$ are are all unramified, $\beta\notin(F_{v}^{\times})^{2}$ and $\omega_{\tau_{v}}=1$. And if $\beta\in(F^{\times})^{2}$ and $\omega_{\tau}\neq 1$, there exists a finite place $v$ of $F$ such that $H(F_{v})$ is $F_{v}$-split, $\tau_{v}$, $\sigma_{v}$, $\Pi_{v}$ and $\psi_{v}$ are all unramified, $\beta\in(F_{v}^{\times})^{2}$ and $\omega_{\tau_{v}}\neq 1$. Then by Proposition 3.1 Part (1) and Proposition 3.3 Part (1), the global Fourier coefficient $f^{\psi_{n+1,\beta}}$ vanishes, as desired.

The cuspidality of the descent follows similarly along the same lines of the proof of Proposition 4.3. ∎

5. The non-vanishing of the descent construction

In this section, we show that the descent ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-vanishing.

5.1. Generalized and degenerate Whittaker-Fourier coefficients

First, we recall the generalized and degenerate Whittaker-Fourier coefficients attached to nilpotent orbits, following the formulation in [12]. Let ${\mathrm{G}}$ be a reductive group defined over a number field $F$. Fix a nontrivial additive character $\psi:F\backslash{\mathbb{A}}\rightarrow{\mathbb{C}}^{\times}$. Let $\mathfrak{g}$ be the Lie algebra of ${\mathrm{G}}(F)$ and $u$ be a nilpotent element in $\mathfrak{g}$. The element $u$ defines a function on $\mathfrak{g}$:

[TABLE]

by $\psi_{u}(x)=\psi(\kappa(u,x))$, where $\kappa$ is the Killing form on $\mathfrak{g}$.

Given any semi-simple element $s\in\mathfrak{g}$, under the adjoint action, $\mathfrak{g}$ is decomposed to a direct sum of eigenspaces $\mathfrak{g}^{s}_{i}$ corresponding to eigenvalues $i$. The element $s$ is called rational semi-simple if all its eigenvalues are in ${\mathbb{Q}}$. Given a nilpotent element $u$, a Whittaker pair is a pair $(s,u)$ with $s\in\mathfrak{g}$ being a rational semi-simple element, and $u\in\mathfrak{g}^{s}_{-2}$. The element $s$ in a Whittaker pair $(s,u)$ is called a neutral element for $u$ if there is a nilpotent element $v\in\mathfrak{g}$ such that $(v,s,u)$ is an $\mathfrak{sl}_{2}$-triple. A Whittaker pair $(s,u)$ with $s$ being a neutral element for $u$ is called a neutral pair.

Given any Whittaker pair $(s,u)$, define an anti-symmetric form $\omega_{u}$ on $\mathfrak{g}$ by

[TABLE]

For any $X\in\mathfrak{g}$, let $\mathfrak{g}_{X}$ be the centralizer of $X$ in $\mathfrak{g}$. For any rational number $r\in{\mathbb{Q}}$, let $\mathfrak{g}^{s}_{\geq r}=\oplus_{r^{\prime}\geq r}\mathfrak{g}^{s}_{r^{\prime}}$. Let $\mathfrak{u}_{s}=\mathfrak{g}^{s}_{\geq 1}$ and let $\mathfrak{n}_{s,u}$ be the radical of $\omega_{u}|_{\mathfrak{u}_{s}}$. Then it is clear that $[\mathfrak{u}_{s},\mathfrak{u}_{s}]\subset\mathfrak{g}^{s}_{\geq 2}\subset\mathfrak{n}_{s,u}$. By [12, Lemma 3.2.6], $\mathfrak{n}_{s,u}=\mathfrak{g}^{s}_{\geq 2}+\mathfrak{g}^{s}_{1}\cap\mathfrak{g}_{u}$. Note that if the Whittaker pair $(s,u)$ comes from an $\mathfrak{sl}_{2}$-triple $(v,s,u)$, then $\mathfrak{n}_{s,u}=\mathfrak{g}^{s}_{\geq 2}$. Let $U_{s}=\exp(\mathfrak{u}_{s})$ and $N_{s,u}=\exp(\mathfrak{n}_{s,u})$ be the corresponding unipotent subgroups of ${\mathrm{G}}$. Define a character of $N_{s,u}$ by

[TABLE]

Let $N_{s,u}^{\prime}=N_{s,u}\cap\ker(\psi_{u})$. Then $U_{s}/N_{s,u}^{\prime}$ is a Heisenberg group with center $N_{s,u}/N_{s,u}^{\prime}$, here $U_{s}=\exp({\mathfrak{u}}_{s})$.

Let $\pi$ be an irreducible automorphic representation of ${\mathrm{G}}({\mathbb{A}})$. For any $\phi\in\pi$, the degenerate Whittaker-Fourier coefficient of $\phi$ attached to $(s,u)$ is defined to be

[TABLE]

If $s$ is a neutral element for $u$, then ${\mathcal{F}}_{s,u}(\phi)$ is also called a generalized Whittaker-Fourier coefficient of $\phi$. Let ${\mathcal{F}}_{s,u}(\pi)=\{{\mathcal{F}}_{s,u}(\phi)|\phi\in\pi\}$. The wave-front set $\mathfrak{n}(\pi)$ of $\pi$ is defined to the set of nilpotent orbits ${\mathcal{O}}$ such that ${\mathcal{F}}_{s,u}(\pi)$ is non-zero, for some neutral pair $(s,u)$ with $u\in{\mathcal{O}}$. Note that if ${\mathcal{F}}_{s,u}(\pi)$ is non-zero for some neutral pair $(s,u)$ with $u\in{\mathcal{O}}$, then it is non-zero for any such neutral pair $(s,u)$, since the non-vanishing property of such generalized Whittaker-Fourier coefficients does not depend on the choices of representatives of ${\mathcal{O}}$. Let $\mathfrak{n}^{m}(\pi)$ be the set of maximal elements in $\mathfrak{n}(\pi)$ under the natural order of nilpotent orbits. We recall a theorem from [12] in the following.

Theorem 5.1** (Theorem C, [12]).**

Let $\pi$ be an irreducible automorphic representation of ${\mathrm{G}}({\mathbb{A}})$. Given a Whittaker pair $(s^{\prime},u)$ and a neutral pair $(s,u)$, if ${\mathcal{F}}_{s^{\prime},u}(\pi)$ is non-zero, then ${\mathcal{F}}_{s,u}(\pi)$ is non-zero.

When ${\mathrm{G}}$ is a quasi-split classical group, it is known that the nilpotent orbits are parametrized by pairs $(\underline{p},\underline{q})$, where $\underline{p}$ is a partition and $\underline{q}$ is a set of non-degenerate quadratic forms (see [27, Section I.6]). When ${\mathrm{G}}={\mathrm{Sp}}_{2n}$, $\underline{p}$ is symplectic partition, namely, odd parts occur with even multiplicities. When ${\mathrm{G}}={\mathrm{SO}}^{\alpha}_{2n},{\mathrm{SO}}_{2n+1}$, $\underline{p}$ is orthogonal partition, namely, even parts occur with even multiplicities. In these cases, let $\mathfrak{p}^{m}(\pi)$ be the set of partitions corresponding to nilpotent orbits in $\mathfrak{n}^{m}(\pi)$. A well-known folklore conjecture is that $\mathfrak{p}^{m}(\pi)$ is a singleton. In this section, for any symplectic or orthogonal partition $\underline{p}$, by a generalized Whittaker-Fourier coefficient of $\pi$ attached to $\underline{p}$, we mean a generalized Whittaker-Fourier coefficient ${\mathcal{F}}_{s,u}(\phi)$ attached to a nilpotent orbit ${\mathcal{O}}$ parametrized by a pair $(\underline{p},\underline{q})$ for some $\underline{q}$, with $\phi\in\pi$, $u\in{\mathcal{O}}$ and $(s,u)$ being a neutral pair. For convenience, sometimes we also write a generalized Whittaker-Fourier coefficient attached to $\underline{p}$ as ${\mathcal{F}}^{\psi_{\underline{p}}}(\phi)$, without specifying the $F$-rational nilpotent orbit ${\mathcal{O}}$ and neutral pairs.

For ${\mathrm{G}}={\mathrm{SO}}_{2n+1}$, an orthogonal partition $\underline{p}$ is called special if it has an even number of odd parts between two consecutive even parts and an odd number of odd parts greater than the largest even part (see [7, Section 6.3]). By the main results of [14], any $\underline{p}\in\mathfrak{p}^{m}(\pi)$ is special. This will play an important role in the following.

5.2. Non-vanishing of the descent

Now we come back to the global situation where the groups and representations are the same as in §2. First we prove the following proposition.

Proposition 5.2**.**

${\mathcal{E}}_{\tau\otimes\sigma}$* has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n)^{2}1^{3}]$.*

Proof.

Let $\alpha_{i}=e_{i}-e_{i+1}$ $(1\leq i\leq 2n-1)$ be a subset of simple roots for ${\mathrm{SO}}^{V_{0}}_{4n+3}$. For $1\leq i\leq 2n$, let $x_{\alpha_{i}}$ be the one-dimensional root subgroup in $\mathfrak{g}$ corresponding to $\alpha_{i}$. By [27, Section I.6], there is only one nilpotent orbit ${\mathcal{O}}$ corresponding to the partition $[(2n)^{2}1^{3}]$. A representative of the nilpotent orbit ${\mathcal{O}}$ can be taken to be $u=\sum_{i=1}^{2n-1}x_{-\alpha_{i}}(1)\,.$ Let $s$ be the following semi-simple element

[TABLE]

Then it is clear that $(s,u)$ is a neutral pair.

We want to show that ${\mathcal{F}}_{s,u}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero. To this end, we take another semisimple element

[TABLE]

It is clear that $(s^{\prime},u)$ is a Whittaker pair. We consider ${\mathcal{F}}_{s^{\prime},u}({\mathcal{E}}_{\tau\otimes\sigma})$. Recall that $Q_{2n}$ is the parabolic subgroup of ${\mathrm{SO}}^{V_{0}}_{4n+3}$ with Levi subgroup isomorphic to ${\mathrm{GL}}_{2n}\times{\mathrm{SO}}_{3}(V_{0})$ and unipotent radical subgroup $U_{2n}$. Then, by definition, for any $\phi\in{\mathcal{E}}_{\tau\otimes\sigma}$, ${\mathcal{F}}_{s^{\prime},u}(\phi)$ is the constant term integral over $U_{2n}(F)\backslash U_{2n}({\mathbb{A}})$ combined with a non-degenerate Whittaker-Fourier coefficient of $\tau$. Since ${\mathcal{E}}_{\tau\otimes\sigma}$ is constructed from data $\tau\otimes\sigma$ on the Levi subgroup ${\mathrm{GL}}_{2n}({\mathbb{A}})\times{\mathrm{SO}}_{3}(V_{0})({\mathbb{A}})$ with $\tau$ generic, the constant term integral over $U_{2n}(F)\backslash U_{2n}({\mathbb{A}})$ is non-zero and non-degenerate Whittaker-Fourier coefficients of $\tau$ are also non-zero. Hence, ${\mathcal{F}}_{s^{\prime},u}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero. Then, by Theorem 5.1, ${\mathcal{F}}_{s,u}({\mathcal{E}}_{\tau\otimes\sigma})$ is also non-zero. This completes the proof of the proposition. ∎

Next we prove the following.

Proposition 5.3**.**

${\mathcal{E}}_{\tau\otimes\sigma}$* has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n+1)(2n-1)1^{3}]$.*

Proof.

By Proposition 5.2, we know that ${\mathcal{E}}_{\tau\otimes\sigma}$ has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n)^{2}1^{3}]$. It is clear that as an orthogonal partition, $[(2n)^{2}1^{3}]$ is not special, and the smallest special partition which is greater than it is $[(2n+1)(2n-1)1^{3}]$, which is called the special expansion of the partition $[(2n)^{2}1^{3}]$. By [14, Theorem 11.2], we must have that ${\mathcal{E}}_{\tau\otimes\sigma}$ has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n+1)(2n-1)1^{3}]$. ∎

Now we are ready to prove Part (a) of Theorem 1.2. In the following, given $\beta\in F^{\times}$, we do not distinguish $\beta$ with its square class or the quadratic form corresponding to it.

Theorem 5.4**.**

There exists $\beta\in F^{\times}$, such that ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero.

Proof.

By Proposition 5.3, we know that ${\mathcal{E}}_{\tau\otimes\sigma}$ has a non-zero generalized Whittaker-Fourier coefficient attached to the partition $[(2n+1)(2n-1)1^{3}]$. By [27, Section I.6], nilpotent orbits corresponding to the partition $[(2n+1)(2n-1)1^{3}]$ are parametrized by certain quadratic forms $\{\beta_{2n+1},\beta_{2n-1},q_{V_{0}}\}$, corresponding to the parts $(2n+1),(2n-1)$ and $1^{3}$, where $\beta_{2n+1}$ and $\beta_{2n-1}$ are square classes, and $q_{V_{0}}$ is the quadratic form in $3$ variables on $V_{0}$ (see Section 2.1). This parametrization can be refined according to [14, Proposition 8.1], that is, ${\mathcal{E}}_{\tau\otimes\sigma}$ actually has a non-zero generalized Whittaker-Fourier coefficient attached to the nilpotent ${\mathcal{O}}$, corresponding to the partition $[(2n+1)(2n-1)1^{3}]$ and parametrized by quadratic forms $\{\beta,-\beta,q_{V_{0}}\}$ for some $\beta\in F^{\times}$. Note that the normalization of the bilinear form for the irreducible representation of $\mathfrak{sl}_{2}(\mathbb{C})$ of dimension $i$ in [14] differs from the one in [27] by the factor $(-1)^{[(i-1)/2]}$ (See [14, Section 7]). In the following, we show that such a $\beta$ will suffice for the theorem.

For the nilpotent orbit ${\mathcal{O}}$ above which is parametrized by quadratic forms $\{\beta,-\beta,q_{V_{0}}\}$, one can take a representative $u=u_{1}+u_{2}$, where

[TABLE]

$u_{2}$ is the embedding into ${\mathrm{SO}}_{4n+3}^{V_{0}}$ of any representative of a nilpotent orbit in the Levi part of the stabilizer of $u_{1}$ which is ${\mathrm{SO}}_{2n+2,\beta}$. This nilpotent orbit in ${\mathrm{SO}}_{2n+2,\beta}$ corresponds to the partition $[(2n-1)1^{3}]$ and is parametrized by quadratic forms $\{-\beta,q_{V_{0}}\}$. Let $s_{1}$ be the following semisimple element of the Lie algebra of ${\mathrm{SO}}_{4n+3}^{V_{0}}$:

[TABLE]

Then it is clear that $(s_{1},u_{1})$ is a neutral pair.

We make $u_{2}$ explicit as follows. First, we give a representative of the nilpotent orbit in ${\mathrm{SO}}_{2n+2,\beta}$ corresponding to the partition $[(2n-1)1^{3}]$ and parametrized by the quadratic forms $\{-\beta,q_{V_{0}}\}$. Let $s_{2}$ be the following semisimple element of the Lie algebra of ${\mathrm{SO}}_{2n+2,\beta}$:

[TABLE]

Let $N_{s_{2}}=\exp(\mathfrak{g}^{s_{2}}_{\geq 2})$, which is the unipotent radical of the parabolic subgroup of ${\mathrm{SO}}_{2n+2,\beta}$ with Levi subgroup isomorphic to ${\mathrm{GL}}_{1}^{n-1}\times{\mathrm{SO}}_{4,\beta}$, here ${\mathrm{SO}}_{4,\beta}={\mathrm{SO}}(V_{0,\beta})$ (see §2.3). Write elements of $N_{s_{2}}$ as $n=\begin{pmatrix}z&zx&y\\ 0&I_{4}&x^{\prime}\\ 0&0&z^{*}\end{pmatrix},$ where $z$ is an upper triangular matrix in ${\mathrm{GL}}_{n-1}$, $x\in\mathrm{Mat}_{(n-1)\times 4}$, and $x^{\prime}$ is defined in Section 2.3. Let

[TABLE]

such that $(a,b,c,d)\in(F^{\times})^{4}$ is an anisotropic vector with respect the quadratic form of ${\mathrm{SO}}_{4,\beta}$. Then $(s_{2},u_{2})$ is a neutral pair and $u_{2}$ is a representative of the nilpotent orbit in ${\mathrm{SO}}_{2n+2,\beta}$ corresponding to the partition $[(2n-1)1^{3}]$ and parametrized by quadratic forms $\{-\beta,q_{V_{0}}\}$, and $N_{s_{2}}=N_{s_{2},u_{2}}$ (see §5.1). Then, we embed $s_{2}$ into the Lie algebra of ${\mathrm{SO}}_{4n+3}^{V_{0}}$ as follows:

[TABLE]

which is still denoted by $s_{2}$. We embed elements of $N_{s_{2},u_{2}}$ into ${\mathrm{SO}}_{4n+3}^{V_{0}}$ as follows:

[TABLE]

where $\overline{x}=\{x_{4},x^{(4)},-\frac{2x_{4}}{\beta}\}$ when we write $x=\{x^{(4)},2x_{4}\}$ with $2x_{4}$ being the last column of $x$. Still denote the image subgroup by $N_{s_{2},u_{2}}$. Similarly, we can embed $u_{2}$ into the Lie algebra of ${\mathrm{SO}}_{4n+3}^{V_{0}}$, and still denote the image by $u_{2}$.

Let $s=s_{1}+s_{2}$. Then it is clear that $(s,u)$ is a neutral pair. From the above discussion, there is a $\varphi\in{\mathcal{E}}_{\tau\otimes\sigma}$ such that the generalized Whittaker-Fourier coefficient ${\mathcal{F}}_{s,u}(\varphi)\neq 0$, for some choice of $u_{2}$. We fix such a choice.

Recall that

[TABLE]

Note that the elements of $N_{s,u}$ have the form:

[TABLE]

where $z$ is an upper triangular matrix in ${\mathrm{GL}}_{n}$, $\mathrm{diag}(I_{n},\widetilde{n},I_{n})$ is in $N_{s_{2},u_{2}}$, and $x\in\mathrm{Mat}_{n\times(2n+3)}$ and $w\in\mathrm{Mat}_{(2n+3)\times n}$ with some entries being zero.

To proceed, we define some unipotent subgroups. For $k=2,3,\ldots,n-1$, let $R_{k}$ be the unipotent subgroup consists of elements of the form $n=\begin{pmatrix}I_{n}&r&0\\ 0&I_{2n+3}&r^{\prime}\\ 0&0&I_{n}\end{pmatrix},$ such that $r_{i,j}$ are all zero except possibly when $i=k$ and $1\leq j\leq k-1$. Similarly, for $k=2,3,\ldots,n-1$, let $C_{k}$ be the unipotent subgroup consists of elements of the form $n=\begin{pmatrix}I_{n}&0&0\\ c^{\prime}&I_{2n+3}&0\\ 0&c&I_{n}\end{pmatrix},$ such that $c_{i,j}$ are all zero except possibly when $j=k+1$ and $1\leq i\leq k-1$. One can see that $\prod_{k=1}^{n-1}R_{k}\cap N_{s,u}=\{I_{4n+3}\}$ and $\prod_{k=1}^{n-1}C_{k}$ consists of all the elements in $N_{s,u}$ of the form $n(0,0,I_{n},w,I_{2n+3})$. Let $N_{s,u}^{\prime}$ be the subgroup of $N_{s,u}$ consisting of all the elements of the form $n(x,y,z,0,\widetilde{n})$.

Now we apply [13, Lemma 6.4] (It is clear that all the assumptions there hold,) to the quadruple

[TABLE]

and obtain that the following integral

[TABLE]

where elements in $N^{\prime}_{s,u}\prod_{k=1}^{n-1}R_{k}$ have the form $n(x,y,z,\widetilde{n}):=\begin{pmatrix}z&zx&y\\ 0&\widetilde{n}&\widetilde{n}x^{\prime}\\ 0&0&z^{*}\end{pmatrix},$ here $z$ is an upper triangular matrix in ${\mathrm{GL}}_{n}$, $\mathrm{diag}(I_{n},\widetilde{n},I_{n})$ is in $N_{s_{2},u_{2}}$, and $x\in\mathrm{Mat}_{n\times(2n+3)}$ with only $x_{n,j}=0$, for $1\leq j\leq n-1$.

Then one finds that, as an inner integral of (5.3), the integral

[TABLE]

is non-vanishing, where $N_{0}$ consists of all elements in $N^{\prime}_{s,u}\prod_{k=1}^{n-1}R_{k}$ having the form $n(x,y,z,I_{2n+3})$. Note that here $x\in\mathrm{Mat}_{n\times(2n+3)}$ with only $x_{n,j}=0$, for $1\leq j\leq n-1$.

Let $X$ be the unipotent subgroup consisting of elements of the form

[TABLE]

such that $x_{i,j}$ are all zero except possibly when $i=n$ and $1\leq j\leq n-1$. Then $X$ is an abelian subgroup, preserving $N_{0}$ and the character $\psi_{u_{1}}$. Taking Fourier expansion of the integral (5.4) along $[X]$, there exists a non-zero Fourier coefficient. Assume that one such non-zero Fourier coefficient is given by a Lie algebra element $u_{1}^{\prime}=\begin{pmatrix}0_{n}&0&0\\ y^{\prime}&0_{2n+3}&0\\ 0&y&0_{n}\end{pmatrix}$, where $y^{\prime}_{i,j}$ are all zero except possibly when $j=n$ and $1\leq i\leq n-1$. Then we can rewrite this Fourier coefficient as

[TABLE]

We claim that $u_{1}$ and $u_{1}+u_{1}^{\prime}$ are in the same nilpotent orbit, that is, one can find a element $g\in{\mathrm{SO}}_{4n+3}^{V_{0}}$ such that $u_{1}=g(u_{1}+u_{1}^{\prime})g^{-1}$. Indeed, write $u_{1}=\begin{pmatrix}x&0&0\\ y^{\prime}&0_{2n+3}&0\\ 0&y&x^{*}\end{pmatrix}$ and $u_{1}+u_{1}^{\prime}=\begin{pmatrix}w&0&0\\ z^{\prime}&0_{2n+3}&0\\ 0&z&w^{*}\end{pmatrix}$, then $x=w\in\mathrm{Mat}_{(2n+3)\times n}$ and all columns of $y^{\prime}$ and $z^{\prime}$ are zero except the last columns $y^{\prime}_{n}$ and $z^{\prime}_{n}$. From the form of $u_{1}^{\prime}$ above, regarding our choice of basis, one can see that as two vectors in an orthogonal space with action of ${\mathrm{SO}}_{2n+3}^{V_{0}}$, $y^{\prime}_{n}$ and $z^{\prime}_{n}$ have the same length, which equals $\beta\in F^{\times}$. Hence, one can find $h\in{\mathrm{SO}}_{2n+3}^{V_{0}}$ such that $y^{\prime}_{n}=z^{\prime}_{n}h^{-1}$. Let $g=\begin{pmatrix}I_{n}&0&0\\ 0&h&0\\ 0&0&I_{n}\end{pmatrix}$, then we have $u_{1}=g(u_{1}+u_{1}^{\prime})g^{-1}$. Therefore, $u_{1}+u_{1}^{\prime}$ is also a representative of the nilpotent orbit corresponding to the partition $[(2n+1)1^{2n+2}]$ and parametrized by quadratic forms $\{\beta,q\}$, where $q$ is a quadratic form in $(2n+2)$-variables such that $\beta\oplus q$ is isomorphic to $q_{V_{0}}$ composing with $n$ hyperplanes. In other words, this nilpotent orbit is the same as the one corresponding to the partition $[(2n+1)1^{2n+2}]$ and parametrized by quadratic forms $\{\beta,q_{V_{0}}\}$. It follows that $(s_{1},u_{1}+u_{1}^{\prime})$ is a neutral pair. Therefore, we obtain that the generalized Whittaker-Fourier coefficient ${\mathcal{F}}_{s_{1},u_{1}}$ is non-zero, that is, ${\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ is non-zero.

This completes the proof of the theorem. ∎

Remark 5.5**.**

It is clear that the argument above can be easily modified and applied to the case that $\sigma$ is a cuspidal representation of ${\mathrm{SO}}(V_{0})({\mathbb{A}})$ with $\dim_{F}(V_{0})=2r+1$, $r\in{\mathbb{Z}}_{>0}$, and the residual representation ${\mathcal{E}}_{\tau\otimes\sigma}$ for ${\mathrm{SO}}^{V_{0}}_{4n+2r+1}({\mathbb{A}})$.

6. On the reciprocal branching problem

In this section, we consider the reciprocal branching problem and prove Parts (b) and (c) of Theorem 1.2.

For simplicity, write $\pi_{\beta}={\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$. By Theorem 5.4, there exists $\beta\in F^{\times}$ such that $\pi_{\beta}\neq 0$. Moreover, if the representation $\sigma$ satisfies Assumption 1.1, or the $\beta\in F^{\times}$ in Theorem 5.4 and $\omega_{\tau}$ satisfy the condition of Proposition 4.4, then $\pi_{\beta}$ is cuspidal by Propositions 4.3, 4.4. By the uniqueness of local Bessel models (see [2, 8, 17, 25]), one has a multiplicity free direct sum decomposition:

[TABLE]

where $\pi_{\beta}^{(i)}$’s are non-zero mutual-inequivalent irreducible cuspidal automorphic representations of the group $G_{n+1,\beta}({\mathbb{A}})={\mathrm{SO}}_{2n+2,\beta}({\mathbb{A}})$. This proves Part (b) of Theorem 1.2.

Let $\pi=\pi_{\beta}^{(i)}$ be any irreducible summand of $\pi_{\beta}$. By construction, there exists an automorphic form $\varphi_{\pi}\in\pi$ such that the inner product

[TABLE]

for some choice of data. By Corollary 4.4 of [18], the pair $(\pi,\sigma)$ has a non-zero Bessel period. In particular, we have

Proposition 6.1**.**

The automorphic descent $\pi_{\beta}={\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ has a non-zero Fourier coefficient attached to the nilpotent orbit corresponding to the partition $[(2n-1)1^{3}]$ and parameterized by quadratic forms $\{-\beta,q_{V_{0}}\}$.

To get a connection with the reciprocal branching problem we have introduced in §1, we need to study the Arthur parameter $\phi_{\pi}$ of each irreducible summand $\pi$ of $\pi_{\beta}$. In particular, we hope that $\phi_{\pi}$ is generic. Combining with the local results we have obtained in §3, we have the following proposition, which is Part (c) of Theorem 1.2.

Proposition 6.2**.**

Assume that $\omega_{\tau}=1$, and the $\beta\in F^{\times}$ in Theorem 5.4 is not a square. Then each irreducible summand $\pi$ of the descent $\pi_{\beta}$ belongs to a global Arthur packet corresponding to a generic global Arthur parameter $\phi_{\pi}$. The parameter $\phi_{\pi}$ has the central character $\eta_{V_{0,\beta}}$, and satisfies the property that $L(\frac{1}{2},\phi_{\pi}\times\phi_{\tau_{0}})\neq 0$ and $(\pi,\sigma)$ has a non-zero Bessel period.

Proof.

By Part (b) of Theorem 1.2, under the assumption of the proposition, $\pi_{\beta}$ is cuspidal, so is $\pi$. Write $\pi=\otimes_{v}^{\prime}\pi_{v}$, and consider all the finite places $v$ (infinitely many) such that both ${\mathcal{E}}_{\tau\otimes\sigma,v}$ and $\psi_{v}$ are unramified and $\beta\in F_{v}^{\times}-{(F_{v}^{\times})}^{2}$. Note that $\tau_{v}$ is a self-dual irreducible generic unitary unramified representation, which has the following form (see [26])

[TABLE]

where $\chi_{i}$ is a unitary unramified character of $F_{v}^{\times}$ and $0\leq s_{i}<\frac{1}{2}$, for each $1\leq i\leq n$. By Proposition 3.1, Part (1), $\pi_{v}$ is the fully-induced irreducible generic unitary unramified representation

[TABLE]

By the endoscopic classification theory of Arthur [3], if $\pi$ has a non-generic global Arthur parameter, then $\pi_{v}$ is non-generic for almost all finite places. Therefore, $\pi$ must have a generic global Arthur parameter $\phi_{\pi}$. This proves the first statement.

The central character of the parameter $\phi_{\pi}$ is determined by the form of the group $G_{n+1,\beta}$, and hence is $\eta_{V_{0,\beta}}$. From the discussion right before Proposition 6.1, the pair $(\pi,\sigma)$ has a non-zero Bessel period, and hence by Theorem 5.7 of [18], one has that $L(\frac{1}{2},\phi_{\pi}\times\phi_{\tau_{0}})\neq 0$. ∎

To end this section, we state a theorem on the reciprocal branching problem, which completes the proof of Theorem 1.2.

Theorem 6.3**.**

Let $\tau=\tau_{1}\boxplus\cdots\boxplus\tau_{r}$ be an irreducible isobaric sum automorphic representation of ${\mathrm{GL}}_{2n}({\mathbb{A}})$ with trivial central character, such that each $\tau_{i}$ is a unitary irreducible cuspidal representation of ${\mathrm{GL}}_{n_{i}}({\mathbb{A}})$ of orthogonal type. Let $V_{0}$ be a quadratic space of dimension $3$ over $F$, and $\sigma$ be an irreducible cuspidal automorphic representation of ${\mathrm{SO}}(V_{0})({\mathbb{A}})$ which lies in the global Vogan packet $\widetilde{\Pi}_{\phi_{\tau_{0}}}[H_{1}^{*}]$ (here $H_{1}^{*}={\mathrm{SO}}_{3}^{*}$, $F$-split) for some irreducible cuspidal automorphic representation $\tau_{0}$ of ${\mathrm{GL}}_{2}({\mathbb{A}})$ of symplectic type. Assume that

• (i)

$L(\frac{1}{2},\tau\times\tau_{0})\neq 0$,

• (ii)

the representation $\sigma$ satisfies Assumption 1.1.

Then there exists a $\beta\in F^{\times}$, such that the twisted descent $\pi_{\beta}={\mathcal{D}}_{\psi_{n,\beta}}({\mathcal{E}}_{\tau\otimes\sigma})$ has all of its irreducible summands $\pi_{\beta}^{(i)}$, as cuspidal automorphic representations of $G_{n+1,\beta}({\mathbb{A}})$, enjoying the property that each $(\pi_{\beta}^{(i)},\sigma)$ has a non-trivial Bessel period. Moreover, if the $\beta\in F^{\times}$ taken in Theorem 5.4 is not a square, then each $\pi_{\beta}^{(i)}$ has a generic global Arthur parameter $\phi^{(i)}$, and gives an answer to the reciprocal branching problem with respect to $\sigma\in\widetilde{\Pi}_{\phi_{\tau_{0}}}[H_{1}^{*}]$.

Remark 6.4**.**

A more precise description of those parameters $\phi^{(i)}$ can be deduced from the refined local theory of the global zeta integrals in [16], we will not discuss them here.