Some applications of theta correspondence to branching laws

TL;DR

This paper explores how local theta correspondences for the dual pair (Sp(W),O(V)) can be applied to analyze branching law problems in representation theory.

Contribution

It introduces new methods leveraging theta correspondence to address branching laws, providing novel insights into their structure and applications.

Findings

Established connections between theta correspondence and branching laws

Derived new results on the decomposition of representations

Enhanced understanding of dual pair interactions in representation theory

Abstract

In this paper, we will use the local theta correspondences for the dual pair (Sp(W),O(V)) to investigate some branching law problems.

Tables

$\mathrm{SO}\left(V\right)$	${\mathrm{SO}}_{3,2}\left(F\right)$	${\mathrm{SO}}_{4,1}\left(F\right)$	${\mathrm{SO}}_{4,1}\left(F\right)$	${\mathrm{SO}}_{4,2}\left(F\right)$	${\mathrm{SO}}_{5,1}\left(F\right)$	${\mathrm{SO}}_{5,2}\left(F\right)$
$\mathrm{SO}\left(V^{\prime }\right)$	${\mathrm{SO}}_{3,1}\left(F\right)$	${\mathrm{SO}}_{4,0}\left(F\right)$	${\mathrm{SO}}_{3,1}\left(F\right)$	${\mathrm{SO}}_{4,1}\left(F\right)$	${\mathrm{SO}}_{4,1}\left(F\right)$	${\mathrm{SO}}_{5,1}\left(F\right)$

Equations

$$\dim {\mathrm{Hom}}_{H(F)}(\pi ,\tau )\le 1$$

$${\mathrm{Hom}}_{H(F)}(\pi ,\mathbb{C})\ne 0,$$

$${\psi }_0(e)={\psi }_E(\delta e)=\psi (t{r}_{E/F}(\delta e)).$$

$$\dim {\mathrm{Hom}}_{{\mathrm{GSpin}}_4^{1,1}(F)}(\pi ,\tau )\le 1.$$

$$\pi ⊠{\Theta }_{\psi }(\pi )$$

$$\Theta (\tau ){|}_{\mathrm{O}(V)}={\oplus }_i{\Theta }_{\psi }({\tau }_i).$$

$$\{\begin{array}{l}{r}^{+}(\pi )=\inf \{2r:{\theta }_{\psi }^{r,r}(\pi )\ne 0\};\\ r^{-}(\pi )=\inf \{2r:{\theta }_{\psi }^{r+2,r-2}(\pi )\ne 0\}.\end{array}$$

$$r^{+}(\pi )+r^{-}(\pi )=4n+4=4+2\dim W_n.$$

$${\mathrm{Hom}}_{H_1}({\Theta }_{\psi }({\pi }_2),{\pi }_1)\cong {\mathrm{Hom}}_{H_2}({\Theta }_{\psi }({\pi }_1),{\pi }_2).$$

$${\mathrm{GSpin}}_4(F)=\{(g_1,g_2)\in {\mathrm{GL}}_2(F)\times {\mathrm{GL}}_2(F):\det (g_1)=\det (g_2)\}$$

$$\varphi :W{D}_F⟶{}^L{\mathrm{GSpin}}_4={\mathrm{GO}}_4(\mathbb{C}),$$

$$|{\Pi }_{\varphi }|=|\{(\chi ,\chi ):(\chi ,\chi )\otimes \tilde{\tau }=\tilde{\tau }\}|$$

$$1\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$${\Pi }_{\varphi }({\mathrm{GSpin}}_4^{i,j})\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$${\mathrm{GSpin}}_4^{1,2}(F)=\{(g_1,g_2)\in {\mathrm{GL}}_1(D)\times {\mathrm{GL}}_2(F):\det (g_1)=\det (g_2)\}$$

$${\Pi }_{\varphi }({\mathrm{GSpin}}_4)=\{\begin{array}{ll}1,& \text{mbox}if{\varphi }_1\text{mbox}isprimitiveornontrivialon{\mathrm{SL}}_2(\mathbb{C});\\ 2,& \text{mbox}if{\varphi }_1\text{mbox}isdihedralwithrepecttoonequadraticcharacter;\\ 4,& \text{mbox}if{\varphi }_1\text{mbox}isdihedralwithrespecttothreequadraticcharacters.\end{array}$$

$${\mathrm{GSO}}_{2,2}(F)=\frac{{\mathrm{GL}}_2(F)\times {\mathrm{GL}}_2(F)}{\{(t,t),t\in F^{\times }\}}\text{mbox}and{\mathrm{SO}}_{2,2}(F)\cong \frac{{\mathrm{GSpin}}_4(F)}{\{(t,t),t\in F^{\times }\}}.$$

$${\theta }_{\psi }(\rho )=JL(\rho )⊠(JL(\rho ){)}^{\vee }$$

$${\mathrm{GSO}}_{4,0}(F)\cong \frac{{\mathrm{GL}}_1(D)\times {\mathrm{GL}}_1(D)}{\{(t,t):t\in F^{\times }\}}.$$

$$JL({\pi }_1){|}_{{\mathrm{SL}}_2(F)}={\tau }_1\oplus {\tau }_2\oplus {\tau }_3\oplus {\tau }_4$$

$$\pi {|}_{{\mathrm{GSpin}}_4^{1,1}(F)}=\underset{i}{⨁}{p}^{\ast }({\theta }_{\psi }({\tau }_i))\otimes (1,{\chi }^{-1}\det {\varphi }_1){|}_{{\mathrm{GSpin}}_4^{1,1}(F)}.$$

$$|{\Pi }_{\varphi }({\mathrm{GSpin}}_4^{1,2}(F))|=1$$

$$\varphi =\chi ({\varphi }_2\otimes {\varphi }_2)=\chi \oplus \chi {\chi }_1\oplus \chi {\chi }_2\oplus \chi {\chi }_1{\chi }_2:W{D}_F⟶{\mathrm{GO}}_4(\mathbb{C}),$$

$$|\mathrm{Irr}({\tilde{S}}_{\varphi },{\xi }_4^{1,1})|=4.$$

$${\mathrm{GSpin}}_4^{1,1}(F)=\{(g_1,g_2)\in {\mathrm{GL}}_1(D)\times {\mathrm{GL}}_1(D)|\det g_1=\det g_2\}$$

$${\mathrm{End}}_{{\mathrm{SL}}_1(D)}(\pi )\cong {\mathrm{M}}_2(\mathbb{C}),$$

$${\mathrm{End}}_{{\mathrm{SL}}_1(D)\times {\mathrm{SL}}_1(D)}(\pi \otimes \pi )\cong {\mathrm{End}}_{{\mathrm{GL}}_1(D)}(\pi )\otimes {\mathrm{End}}_{{\mathrm{GL}}_1(D)}(\pi )\cong {\mathrm{M}}_2(\mathbb{C})\otimes {\mathrm{M}}_2(\mathbb{C}),$$

$${\mathrm{End}}_{{\mathrm{GSpin}}_4^{1,1}(F)}(\pi \otimes \pi )=\mathbb{C}+\mathbb{C}+\mathbb{C}+\mathbb{C},$$

$$q_F(v)=\frac{1}{2}t{r}_{E/F}\circ q_E(v).$$

$${\mathrm{Res}}_{E/F}(V_E\otimes (W_n{\otimes }_{F}E))=W_n\otimes V_F:=\mathbf{W}.$$

Full text

Key words

the local theta lift, branching laws, disctinction problems

MSC(2000)

11F27$\cdot$11F70$\cdot$22E50

Contents

1. 1 Introduction
2. 2 The local theta correspondence
3. 3 The $L$-packet for $\mathrm{GSpin}_{4}$
4. 4 The disctinction problems for $\mathrm{Sp}(W_{n})$ and $\mathrm{O}(V)$
5. 5 The $\mathrm{Sp}_{4}(F)$-distinguished representation of $\mathrm{SU}_{2,2}$
6. A The local theta correspondence of tempered representations and Langlands parameters

1. Introduction

Let $F$ be a nonarchimedean local field of characteristic zero. Let $G$ be a reductive group defined over $F$. Assume that $H$ is a closed subgroup of $G$ defined over $F$. Given an irreducible smooth representation $\pi$ of $G(F)$, there is a rich literature studying the restricted representation $\pi|_{H}$, called the branching law problems, such as [GGP12, Gan18, DP18, GS15, Wal12, SV17] and so on. If

[TABLE]

for any irreducible smooth representation $\pi$ (resp. $\tau$) of $G(F)$ (resp. $H(F)$), then the pair $(G(F),H(F))$ is called multiplicity-free, such as $(\mathrm{SO}_{n},\mathrm{SO}_{n-1})$. If $\dim\mathrm{Hom}_{H(F)}(\pi,\mathbb{C})\leq 1$ for any irreducible smooth representation $\pi$ of $G(F)$, then the pair $(G(F),H(F))$ is called the Gelfand pair. If

[TABLE]

then $\pi$ is called $H(F)$-distinguished. The relative trace formula is a very powerful tool that can be applied to deal with the disctinction problems, such as [Wal12]. In this paper, we will use the local theta correspondence to deal with the disctinction problems, following [GS15, Gan18].

Let $D$ be a $4$-dimensional division quaternion algebra over $F$. Let $E=F[\delta]$ be a quadratic field extension of $F$. Let $W_{F}$ (resp. $W_{E}$) be the Weil group of $F$ (resp. $E$). Let $WD_{F}$ (resp. $WD_{E}$) be the Weil-Deligne group of $F$ (resp. $E$). Fix a nontrivial additive character $\psi$ of $F$ and $\psi_{E}=\psi\circ tr_{E/F}$. Set

[TABLE]

Then $\psi_{0}|_{F}$ is trivial.

There are $3$ aspects in this paper:

• •

to show that $(\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D),\mathrm{GSpin}_{4}^{1,1}(F))$ is multiplicity-free (see §3);

• •

to compare the distinction problems for $(\mathrm{Sp}_{2n}(E),\mathrm{Sp}_{2n}(F))$ and $(\mathrm{O}(V\otimes_{F}E),\mathrm{O}(V))$(see §4) ;

• •

to classify all the tempered representations of $\mathrm{SO}(V)$, distinguished by $\mathrm{SO}(V^{\prime})$, where $V^{\prime}$ is a subspace of $V$ of codimension $1$.

Let $\mathrm{GSpin}_{4}^{1,1}(F)=\{(g_{1},g_{2})\in\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D):\det(g_{1})=\deg(g_{2})\}$, where $\det(g_{1})=N_{D/F}(g_{1})$.

Theorem 1.1**.**

Given an irreducible representation $\pi=(\pi_{1},\pi_{2})$ of $\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D)$ and any irreducible representation $\tau$ of $\mathrm{GSpin}_{4}^{1,1}(F)$, one has

[TABLE]

We would like to highlight the fact that Asgari-Choiy [AC17] have proved the local Langlands conjectures for $\mathrm{GSpin}_{4}=\{(g_{1},g_{2})\in\mathrm{GL}_{2}\times\mathrm{GL}_{2}:\det(g_{1})=\det(g_{2})\}$ and its inner forms. However, in [AC17, Remark 5.11], they claim that $|\Pi_{\phi}(\mathrm{GSpin}_{4}^{1,1}(F))|=1$ if and only if $\pi_{1}=\pi_{2}\otimes\chi$ are dihedral with respect to three nontrivial quadratic characters of $F^{\times}$, where $\chi$ is any character of $F^{\times}$. In this case, they obtain the conclusion that the multiplicity in restriction from $\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D)$ to $\mathrm{GSpin}_{4}^{1,1}(F)$ could be $2$, which is false. In §3, we will show that $|\Pi_{\phi}(\mathrm{GSpin}_{4}^{1,1}(F))|=4$ instead of $1$ if $\pi_{1}=\pi_{2}\otimes\chi$ are dihedral with respect to three nontrivial quadratic characters of $F^{\times}$. On the other hand, Dipendra Prasad gives an another proof of Theorem 1.1 involving pure algebras which will be presented at the end of §3.

Theorem 1.2**.**

Let $E=F[\delta]$ be a quadratic field extension over a nonarchimedean local field $F$ of characteristic zero. Let $W_{n}$ be a $2n$-dimensional symplectic vector space over $F$. Let $\tau$ be a tempered representation of $\mathrm{Sp}_{2n}(E)$. The local theta lift to $\mathrm{O}_{r,r}(E)$ of $\tau$ from $\mathrm{Sp}_{2n}(E)$ is denoted by $\theta_{\psi_{0}}^{r,r}(\tau)$.

1. (i)

Suppose that the local theta lift $\theta^{n,n}_{\psi_{0}}(\tau)=0$. Then $\tau$ is $\mathrm{Sp}(W_{n})$-distinguished if and only if $\theta_{\psi_{0}}^{n+1,n+1}(\tau)$ is nonzero and $\mathrm{O}_{n+1,n+1}(F)$-distinguished. 2. (ii)

Suppose that $\theta^{n,n}_{\psi_{0}}(\tau)$ is a nonzero representation of $\mathrm{O}_{n,n}(E)$ and that the local theta lift to $\mathrm{Sp}_{2n-2}(E)$ of $\theta^{n,n}_{\psi_{0}}(\tau)$ is zero. Then $\tau$ is $\mathrm{Sp}_{2n}(F)$-distinguished if and only $\theta^{n,n}_{\psi_{0}}(\tau)$ is $\mathrm{O}(V)$-distinguished for a $2n$-dimensional quadratic space $V$ over $F$ satisfying $\mathrm{O}(V\otimes_{F}E)=\mathrm{O}_{n,n}(E)$.

In [Zha18], assuming $p\neq 2$, Zhang showed that given a $\mathrm{Sp}_{2n}(F)$-distinguished regular supercuspidal representation $\tau$ of $\mathrm{Sp}_{2n}(E)$, there exists a $2n$-dimensional quadratic space $V$ over $F$ such that the theta lift $\theta_{\psi_{0}}(\tau)$ is $\mathrm{O}(V)$-distinguished. Moreover, $V\otimes_{F}E$ is unique. We extend Zhang’s result from the regular supercuspidal representations to the tempered representations of $\mathrm{O}_{n,n}(E)$, whose first occurence index (defined in §2) is $2n$ in the Witt tower $\{W_{n}\otimes_{F}E\}$, including the case when $p=2$.

Finally, we use the twisted Jacquet modules of the Weil representation to classify all the tempered representations of $\mathrm{SO}(V)$ distinguished by $\mathrm{SO}(V^{\prime})$ with $\dim(V/V^{\prime})=1$, which turns out to be the local theta lifts from $\mathrm{SL}_{2}(F)$ or $\mathrm{Mp}_{2}(F)$ to $\mathrm{SO}(V)$. The key ingredient is that the local theta lifts to $\mathrm{SO}(V)$ from $\mathrm{SL}_{2}(F)$ or $\mathrm{Mp}_{2}(F)$ must not be tempered when $\dim V\geq 8$. For a small group $\mathrm{SO}(V)$ when $\dim V\leq 7$, the local theta lift from $\mathrm{SL}_{2}(F)$ or $\mathrm{Mp}_{2}(F)$ can be written down explicitly in terms of the Langlands parameter due to [AG17, Theorem 4.1]. And we give a proof to [DP18, Conjecture 2] in the case of non-archimedean fields.

The paper is organized as follows. In §$2$, we set up the notation about the local theta correspondence. We will review the results of Asgari-Chioy and then use the local theta correspondence to prove Theorem 1.1 in §3. Another proof of Theorem 1.1 due to Dipendra Prasad will be given at the end of §3. In §4, we will show that $\tau$ is $\mathrm{Sp}_{2n}(F)$-distinguished if and only if the local theta lift is $\mathrm{O}(V)$-distinguished for a quadratic space $V$ over $F$. In §$5$, we will classify all the tempered representations of $\mathrm{SO}(V)$ distinguished by $\mathrm{SO}(V^{\prime})$ with $\dim(V/V^{\prime})=1$ and then prove [DP18, Conjecture] in the case of non-archimedean fields. In the appendix, we review the local theta correspondence of tempered representations and Langlands parameters.

Acknowledgments

The author is grateful to Wee Teck Gan and Dipendra Prasad for their guidance and numerous discussions. He also wants to thank Sandeep Varma for helpful discussions. He would like to thank the referee for useful comments as well. This research was done when he was a Visiting Fellow at Tata Institute of Fundamental Research, Mumbai and it was partially supported by ERC StG grant number 637912 under revision.

2. The local theta correspondence

In this section, we will briefly recall some results about the local theta correspondence, following [Kud96].

Let $F$ be a local field of characteristic zero. Consider the dual pair $\mathrm{O}(V)\times\mathrm{Sp}(W)$. For simplicity, we may assume that $\dim V$ is even. Fix a nontrivial additive character $\psi$ of $F.$ Let $\omega_{\psi}$ be the Weil representation for $\mathrm{O}(V)\times\mathrm{Sp}(W).$ If $\pi$ is an irreducible representation of $\mathrm{O}(V)$ (resp. $\mathrm{Sp}(W)$), the maximal $\pi$-isotypic quotient of the Weil representation $\omega_{\psi}$ has the form

[TABLE]

for some smooth representation of $\mathrm{Sp}(W)$ (resp. some smooth representation of $\mathrm{O}(V)$). We call $\Theta_{\psi}(\pi)$ the big theta lift of $\pi$. Let $\theta_{\psi}(\pi)$ be the maximal semisimple quotient of $\Theta_{\psi}(\pi),$ which is called the small theta lift of $\pi.$

Theorem 2.1**.**

[GT16a, GT16b]** One has

1. (i)

$\theta_{\psi}(\pi)$* is irreducible whenever $\Theta_{\psi}(\pi)$ is non-zero.* 2. (ii)

the map $\pi\mapsto\theta_{\psi}(\pi)$ is injective on its domain.

It is called the Howe duality conjecture which has been proven by Waldspurger [Wal90] when $p\neq 2$.

Then Roberts [Rob96] extends the Weil representation to the case of similitude groups $\mathrm{GO}(V)\times\mathrm{GSp}(W)$ (the similitude character $\lambda_{V}$ is surjective in this paper).

Lemma 2.2**.**

[GT11, Lemma 2.2]** Let $\tau$ be an irreducible representation of $\mathrm{GSp}(W)$ and $\tau|_{\mathrm{Sp}(W)}=\oplus_{i}\tau_{i}$. Then

[TABLE]

2.1. First occurence indices for pairs of orthogonal Witt towers

Let $W_{n}$ be the $2n$-dimensional symplectic vector space over $F$ with associated metaplectic group $\mathrm{Sp}(W_{n})$ and consider the two towers of orthogonal groups attached to the quadratic spaces with trivial discriminant. More precisely, let $V^{+}_{2r}$ (resp. $V_{2r}^{-}$) be the $2r$-dimensional quadratic $F$-vector space with trivial discriminant and Hasse invariant $+1$ (resp. $-1$). Denote the orthogonal groups by $\mathrm{O}(V_{2r}^{+})=\mathrm{O}_{r,r}(F)$ (resp. $\mathrm{O}(V_{2r}^{-})=\mathrm{O}_{r+2,r-2}(F)$). For an irreducible smooth representation $\pi$ of $\mathrm{Sp}(W_{n}),$ one may consider the theta lifts $\theta^{r,r}_{\psi}(\pi)$ and $\theta^{r+2,r-2}_{\psi}(\pi)$ to $\mathrm{O}(V^{+}_{2r})$ and $\mathrm{O}(V_{2r}^{-})$ respectively, with respect to a fixed non-trivial additive character $\psi$. Set

[TABLE]

Then Kudla-Rallis [KR05] and Sun-Zhu [SZ15] showed:

Theorem 2.3** (Conservation Relation).**

For any irreducible representation $\pi$ of $\mathrm{Mp}(W_{n}),$ we have

[TABLE]

In [AG17], Atobe and Gan use $m^{\mbox{down}}(\pi)$ (resp. $m^{\mbox{up}}(\pi)$) to denote $\min\{r^{+}(\pi),r^{-}(\pi)\}$ (resp. $\max\{r^{+}(\pi),r^{-}(\pi)\}$).

2.2. The see-saw diagrams

A pair $(G_{1},H_{1})$ and $(G_{2},H_{2})$ of reductive dual pairs in a symplectic group is called the see-saw pair if $H_{1}\subset G_{2}$ and $H_{2}\subset G_{1}$.

Lemma 2.4**.**

[Pra96, Page 6]** For a see-saw pair of reductive dual pairs $(G_{1},H_{1})$ and $(G_{2},H_{2})$, let $\pi_{1}$ (resp. $\pi_{2}$) be a representation of $H_{1}$ (resp. $H_{2}$). Then we have the following isomorphism

[TABLE]

Dipendra Prasad proved that the see-saw identity (2.1) holds for the similitude groups as well.

3. The $L$-packet for $\mathrm{GSpin}_{4}$

This section focuses on the proof of Theorem 1.1. Let

[TABLE]

be a subgroup of $\mathrm{GL}_{2}(F)\times\mathrm{GL}_{2}(F)$. Asgari-Choiy [AC17] used the restriction from $\mathrm{GL}_{2}\times\mathrm{GL}_{2}$ to $\mathrm{GSpin}_{4}$ to set up the Langlands correspondence for the generic representations of $\mathrm{GSpin}_{4}$. We summarize their results as follow.

Let $\tau$ be an irreducible smooth generic representation of $\mathrm{GSpin}_{4}(F)$. There exists a unique enhanced $L$-parameter $(\phi,\lambda)$ such that

[TABLE]

where $\lambda$ is a character of the component group $S_{\phi}=C(\phi)/C^{\circ}(\phi)$, where $C(\phi)$ is the centralizer of $\phi$ in $\mathrm{GO}_{4}(\mathbb{C})$. Let $\tilde{\tau}$ be a representation of $\mathrm{GL}_{2}(F)\times\mathrm{GL}_{2}(F)$ such that $\tilde{\tau}|_{\mathrm{GSpin}_{4}(F)}\supset\tau$. Then the size of the $L$-packet $\Pi_{\phi}$ is

[TABLE]

where $\chi:F^{\times}\longrightarrow\mathbb{C}^{\times}$ is a quadratic character of $F^{\times}$.

For the inner forms of $\mathrm{GSpin}_{4}$, we consider the extended component group $\tilde{S}_{\phi}$, where $\tilde{S}_{\phi}$ sits in the following exact sequence

[TABLE]

Let $\xi_{4},\xi_{4}^{i,j}$ be the characters of $\mu_{2}\times\mu_{2}$, where $(i,j)=(1,1),(1,2)$ or $(2,1)$. We may also regard $\xi_{4}$ as $\xi_{4}^{2,2}$ and $\mathrm{GSpin}_{4}=\mathrm{GSpin}_{4}^{2,2}$.

Theorem 3.1**.**

[AC17, Theorem 5.1]** Given an $L$-parameter $\phi$ of $\mathrm{GSpin}_{4}(F)$, there exists a bijection

[TABLE]

where $\mathrm{Irr}(\tilde{S}_{\phi},\xi_{4}^{i,j})$ are the set of irreducible representations of $\tilde{S}_{\phi}$ which extends $\xi_{4}^{i,j}$,

[TABLE]

and $\mathrm{GSpin}_{4}^{2,1}(F)=\{(g_{1},g_{2})\in\mathrm{GL}_{2}(F)\times\mathrm{GL}_{1}(D):\det(g_{1})=\det(g_{2})\}$.

Suppose that $\pi=(\pi_{1},\pi_{2})$ is a representation of $\mathrm{GSpin}_{4}^{1,1}(F)$ with $\phi=(\phi_{1},\phi_{2})$. If $\phi_{1}=\phi_{2}\otimes\chi$, then

[TABLE]

If $\phi_{1}\neq\phi_{2}\otimes\chi$ for any character $\chi$ of $F^{\times}$, then $|\Pi_{\phi}(\mathrm{GSpin}_{4})|\leq 2$.

Let

[TABLE]

Given a square-integrable representation $\rho$ of $\mathrm{GL}_{2}(F)$, then the theta lift $\theta_{\psi}(\rho)$ from $\mathrm{GL}_{2}(F)$ to $\mathrm{GSO}_{2,2}(F)$ is isomorphic to $\rho\boxtimes\rho^{\vee}$. Similarly, the theta lift to $\mathrm{GSO}_{4,0}(F)$ from $\mathrm{GL}_{2}(F)$ is

[TABLE]

where $JL(\rho)$ is the Jacquet-Langlands correspondence representation of $\mathrm{GL}_{1}(D)$ associated to $\rho$ and

[TABLE]

Now we start to prove Theorem 1.1.

Proof of Theorem 1.1.

Assume that $\pi=(\pi_{1},\pi_{2})$ is an irreducible representation of $\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D)$ with $L$-parameter $(\phi_{1},\phi_{2})$. We divide them into two cases:

• •

If $|\Pi_{\phi}(\mathrm{GSpin}_{4})|\leq 2$, then due to Schur orthogonal relation, $|\Pi_{\phi}(\mathrm{GSpin}_{4})|=|\Pi_{\phi}(\mathrm{GSpin}_{4}^{1,1})|$ and $\pi|_{\mathrm{GSpin}_{4}^{1,1}(F)}$ is multiplicity free.

• •

If $|\Pi_{\phi}(\mathrm{GSpin}_{4})|=4$, then $\phi_{1}=\phi_{2}\otimes\chi$ for a character $\chi$ of $W_{F}$. Without loss of generality, we assume that $\chi=\det\phi_{1}$. Then $\phi_{2}=\phi_{1}^{\vee}$. Thus $\pi$ corresponds to a representation of $\mathrm{GSO}_{4,0}(F)$ and $\pi=\theta_{\psi}(JL(\pi_{1}))$, where $JL(\pi_{1})$ is the Jacquet-Langlands lift of $\pi_{1}$. If $\mathrm{Hom}_{\mathrm{GSpin}_{4}^{1,1}(F)}(\pi,\tau)\neq 0$ for a representation $\tau$ of $\mathrm{GSpin}_{4}^{1,1}(F)$, then $\tau$ is a representation of $\mathrm{SO}_{4,0}(F)$. Thanks to [AP19, §6], the restriction from $\mathrm{GSO}(n)$ to $\mathrm{SO}(n)$ is multiplicity-free. Then $\dim\mathrm{Hom}_{\mathrm{GSpin}_{4}^{1,1}(F)}(\pi,\tau)=1$. In fact,

[TABLE]

and so $\pi|_{\mathrm{GSpin}_{4}^{1,1}(F)}=\oplus_{i}p^{\ast}(\theta_{\psi}(\tau_{i}))$, where $p:\mathrm{GSpin}_{4}^{1,1}(F)\longrightarrow\mathrm{SO}_{4,0}(F)$ is the natural projection and $p^{\ast}$ is the pull back. In general,

[TABLE]

Therefore $|\Pi_{\phi}(\mathrm{GSpin}_{4}^{1,1})|=4$ and $\pi|_{\mathrm{GSpin}_{4}^{1,1}(F)}$ is multiplicity-free.

∎

Remark 3.2*.*

It is known that $\pi_{1}|_{\mathrm{SL}_{1}(D)}=2\tau_{0}$ is of multiplicity two if $\pi_{1}$ is dihedral with respect to three quadratic character, where $\tau_{0}$ is an irreducible representation of $\mathrm{SL}_{1}(D)$. However, the representation of two copies of $\mathrm{GL}_{1}(D)$ restricts to the subgroup $\mathrm{GSpin}_{4}^{1,1}(F)$ is multiplicity-free.

Remark 3.3*.*

One can use the theta correspondence for the similitude quaternionic dual pair to show that

[TABLE]

when

[TABLE]

where $\chi_{1}$ and $\chi_{2}$ are quadratic characters of $W_{F}$. In this case, let $\pi^{\prime}=(\pi_{2}\otimes\chi,JL(\pi_{2}))$ be a representation of $\mathrm{GL}_{1}(D)\times\mathrm{GL}_{2}(F)$. Then the restricted representation $\pi^{\prime}|_{\mathrm{GSpin}_{4}^{1,2}(F)}$ is of multiplicity two.

Remark 3.4*.*

Let $\phi_{1}=\phi_{2}\otimes\chi$ and $\phi_{1}$ is dihedral with respect to three quadratic characters. On the parameter side, $S_{\phi}$ is a finite group of order $4$. The extended component group $\tilde{S}_{\phi}$ is isomorphic to $\langle a\rangle\times Q_{8}$ where $a^{2}=1$ and $Q_{8}=\{\pm 1,\pm i,\pm j,\pm ij\}$ is the quaternion group of order $8$. The center of $\tilde{S}_{\phi}$ is $\langle a\rangle\times\langle-1\rangle$. Note that $\xi_{4}^{1,1}(-1)=1$ and $\xi_{4}^{1,1}(a)=-1$. Then there are four ways to extend the character $\xi_{4}^{1,1}$ to $\tilde{S}_{\phi}$ and so

[TABLE]

We give an another proof of Theorem 1.1 due to Dipendra Prasad.

Another proof of Theorem 1.1.

Recall that the essential case of the theorem is to prove that if $\pi\otimes\pi$ is an irreducible representation of $\mathrm{GL}_{1}(D)\times\mathrm{GL}_{1}(D)$ where $\pi$ is an irreducible representation of $\mathrm{GL}_{1}(D)$ with two distinct quadratic self-twists, then the restriction of $\pi\otimes\pi$ to

[TABLE]

decomposes as a sum of 4 distinct irreducible representations of $\mathrm{GSpin}^{1,1}_{4}(F)$. The representation $\pi$ (with two distinct quadratic twists) will be fixed in what follows. We begin by recalling what the standard theory says about the restriction of $\pi$ from $\mathrm{GL}_{1}(D)$ to $\mathrm{SL}_{1}(D)$.

It is known that $\pi$ restricted to $\mathrm{SL}_{1}(D)$ is $2\pi^{\prime}$ for an irreducible representation $\pi^{\prime}$ of $\mathrm{SL}_{1}(D)$. It follows that

[TABLE]

and the natural action of $\mathrm{GL}_{1}(D)$ on ${\rm End}_{\mathrm{SL}_{1}(D)}(\pi)\cong{\rm M}_{2}(\mathbb{C}),$ is via algebra automorphisms of ${\rm M}_{2}(\mathbb{C})$, and thus gives a projective representation of $\mathrm{GL}_{1}(D)/\mathrm{SL}_{1}(D)\cong F^{\times}\rightarrow\mathrm{PGL}_{2}(\mathbb{C})$. This projective representation of $F^{\times}$ factors through the quotient of $F^{\times}$ by the common kernel of the quadratic self-twists of $\pi$, and thus $\mathrm{GL}_{1}(D)/\mathrm{SL}_{1}(D)$ acts through a quotient which is $\mathbb{Z}/2+\mathbb{Z}/2$, giving rise to a projective faithful representation of $\mathbb{Z}/2+\mathbb{Z}/2$ into $\mathrm{PGL}_{2}(\mathbb{C})$. Now there is a unique subgroup of $\mathrm{PGL}_{2}(\mathbb{C})$ isomorphic to $\mathbb{Z}/2+\mathbb{Z}/2$, consisting of the image of the diagonal matrix $(-1,1)$ in $\mathrm{GL}_{2}(\mathbb{C})$, and an element in the Weyl group of the diagonal torus. (The resulting map from $F^{\times}$ to $\mathrm{PGL}_{2}(\mathbb{C})$ is the Langlands parameter of $\pi$ with values in $\mathrm{PGL}_{2}(\mathbb{C})$, although this fact has no relevance for us here.)

The projective representation of $\mathrm{GL}_{1}(D)/\mathrm{SL}_{1}(D)\cong F^{\times}$ into $\mathrm{PGL}_{2}(\mathbb{C})$ could be considered as the projective representation attached to the unique 2-dimensional irreducible representation $V_{2}$ of $Q_{8}$, the quaternionic group of order 8.

Observe that

[TABLE]

and that, ${\rm End}_{\mathrm{SL}_{1}(D)\times\mathrm{SL}_{1}(D)}(\pi\otimes\pi)$ comes equipped with an action of $\mathrm{GL}_{1}(D)/\mathrm{SL}_{1}(D)\times\mathrm{GL}_{1}(D)/\mathrm{SL}_{1}(D)\cong F^{\times}\times F^{\times}$ by algebra automorphisms, which can be realized as the representation of $Q_{8}\times Q_{8}$ on the 4-dimensional irreducible representation $V_{2}\boxtimes V_{2}$ of it.

To prove that $\pi\otimes\pi$ restricted to $\mathrm{GSpin}^{1,1}_{4}(F)$ decomposes as a sum of 4 distinct irreducible representations of $\mathrm{GSpin}^{1,1}_{4}(F)$, we need to prove that

[TABLE]

as algebras. However, ${\rm End}_{\mathrm{GSpin}_{4}^{1,1}(F)}(\pi\otimes\pi)$ is simply $\Delta Q_{8}\subset Q_{8}\times Q_{8}$ invariants in ${\rm End}_{\mathrm{SL}_{1}(D)\times\mathrm{SL}_{1}(D)}(\pi\otimes\pi)$ which is now the commutant of the action of $\Delta Q_{8}$ on the 4-dimensional representation $V_{2}\otimes V_{2}$, which being the sum of 4 characters of $Q_{8}$, the commutant is simply $\mathbb{C}+\mathbb{C}+\mathbb{C}+\mathbb{C},$ as desired. ∎

4. The disctinction problems for $\mathrm{Sp}(W_{n})$ and $\mathrm{O}(V)$

In this section, we will study the disctinction problems for $\mathrm{Sp}(W_{n})$ and $\mathrm{O}(V)$ over a quadratic field extension $E/F$. We will use $V_{E}$ (resp. $V_{F}$ or $V$) to denote the quadratic space over $E$ (resp. $F$).

4.1. Local Siegel-Weil identity

Let $(V_{E},q_{E})$ be a quadratic vector space over $E$. Let $V_{F}=\mathrm{Res}_{E/F}V_{E}$ be the same vector space but now thought of as a vector space over $F$ with a quadratic form

[TABLE]

If $W_{n}$ is a symplectic vector space over $F$ with symplectic group $\mathrm{Sp}_{2n}(F)$, then $W_{n}\otimes_{F}E$ is a symplectic vector space over $E$ with symplectic group $\mathrm{Sp}_{2n}(E)$. Then we have the following isomorphism of symplectic spaces:

[TABLE]

The pair $(\mathrm{O}(V_{E}),\mathrm{Sp}_{2n}(E))$ and $(\mathrm{O}(V_{F}),\mathrm{Sp}_{2n}(F))$ of dual pairs in $\mathrm{Sp}(\mathbf{W})$ is a see-saw pair. Given an irreducible representation $\Sigma$ of $\mathrm{O}(V_{E})$, one has

[TABLE]

where $\Theta_{\psi}(\mathbf{1})$ (resp. $\Theta_{\psi_{E}}(\Sigma)$) is the big theta lift to $\mathrm{O}(V_{F})$ (resp. $\mathrm{Sp}_{2n}(E)$) of the trivial representation $\mathbb{C}$ (resp. the representation $\Sigma$) of $\mathrm{Sp}_{2n}(F)$ (resp. $\mathrm{O}(V_{E})$). It is called the local Siegel-Weil identity which will be used in the proof of Theorem 1.2 in the next subsection.

4.2. Proof of Theorem 1.2

Now we can give the proof to Theorem 1.2.

Proof of Theorem 1.2.

Assume that $\tau$ is a tempered representation of $\mathrm{Sp}_{2n}(E)$. Let $\tau^{\delta}$ be the conjugate representation of $\tau$, i.e.

[TABLE]

where $g_{\delta}\in\mathrm{GSp}_{2n}(E)$ with similitude $\lambda_{W}(g_{\delta})=\delta$.

1. (i)

If $\theta^{n+1,n+1}_{\psi_{0}}(\tau)$ is zero, then $\theta_{\psi_{0}}^{n+2,n-2}(\tau)\neq 0$ by the conservation relation. Suppose that $V_{E}$ is a $2n$-dimensional quadratic space over $E$ with non-quasi-split othogonal group $\mathrm{O}(V_{E})=\mathrm{O}_{n+2,n-2}(E)$. The theta lift to $\mathrm{O}(V_{E})$ of $\tau$ is nonzero. Due to [Lu17a, Lemma 4.2.2], the discriminant of the $4n$-dimensional $F$-vector space $V_{F}=\mathrm{Res}_{E/F}V_{E}$ is trivial and the Hasse invariant is $-1$. Consider the following see-saw diagram

[TABLE]

where $\mathrm{O}(V_{F})\cong\mathrm{O}_{2n+2,2n-2}(F)$. By the conservation relation, $\Theta^{2n+2,2n-2}_{\psi}(\mathbf{1})=0$. Then

[TABLE]

where $\Theta^{2n}_{\psi_{E}}(\Sigma)=\tau$ is the theta lift to $\mathrm{Sp}_{2n}(E)$ of $\Sigma$. Hence $\tau$ is not $\mathrm{Sp}(W_{n})$-distinguished.

Suppose that $\theta_{\psi_{0}}^{n+1,n+1}(\tau)$ is nonzero. Let us consider the following see-saw diagrams

[TABLE]

with $\Sigma=\theta_{\psi_{0}}^{n+1,n+1}(\tau)$. By the assumptions,

[TABLE]

Note that $\theta^{n+1,n+1}_{\psi_{E}}(\tau^{\delta})=\Sigma=\theta^{n+1,n+1}_{\psi_{0}}(\tau)$. There exists a short exact sequence

[TABLE]

of $\mathrm{Sp}_{4n}(F)$-modules, where $R^{i,j}(\mathbf{1})$ is the big theta lift of the trivial representation from $\mathrm{O}_{i,j}(F)$ to $\mathrm{Sp}_{4n}(F)$ and $I(s)$ is the degenerate principal series of $\mathrm{Sp}_{4n}(F)$ (see [GI14, Proposition 7.2]) induced from a character $|\det|^{s}$ on the Siegel parabolic subgroup $Q$. Then

[TABLE]

There exists a $\mathrm{Sp}_{2n}(E)$-equivariant filtration

[TABLE]

of $I(s)|_{\mathrm{Sp}_{2n}(E)}$ such that $I_{0}(s)\cong ind_{\mathrm{Sp}_{2n}(F)}^{\mathrm{Sp}_{2n}(E)}\mathbb{C}$ and $I_{i}(s)/I_{i-1}(s)\cong ind_{Q_{i}}^{\mathrm{Sp}_{2n}(E)}(|\det|_{E}^{s+i/2}\otimes\mathbf{1})$ for $i=1,2,\cdots,n$. Here $Q_{i}\cong(\mathrm{GL}_{i}(E)\times\mathrm{Sp}_{2n-2i}(F))\cdot(Mat_{2i,2n-2i}(F)\times Sym^{i}(E))$, where $Mat_{m,n}(F)$ is the matrix space consisting of all $m\times n$ matrices and $Sym^{i}(E)$ consists of symmetric matrices in $Mat_{i,i}(E)$. Since $\tau$ is tempered, $\tau^{\delta}$ is tempered. Observe that $\mathrm{Sp}_{2n}(E)$ is the fixed point of a involution on $\mathrm{Sp}_{4n}(F)$, which is given by the scalar matrix

[TABLE]

acting on $\mathrm{Sp}_{4n}(F)$ by conjugation. Due to [Ó87, Theorem 2.5], there exists a polynomial $f$ on $\mathrm{Sp}_{4n}(F)$ such that the complements of the open orbits in the double coset $Q\backslash\mathrm{Sp}_{4n}(F)/\mathrm{Sp}_{2n}(E)$ is the zero set of $f$. Thanks to [GSS18, Proposition 4.9], the multiplicity $\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(E)}(I(1/2),\tau^{\delta})$ is at least $\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(E)}(I_{0}(1/2),\tau^{\delta})$ where the submodule $I_{0}$ corresponds to the open orbits. More precisely,

[TABLE]

Then it suffices to show that $\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(F)}(\tau,\mathbb{C})\geq\dim\mathrm{Hom}_{\mathrm{O}_{n+1,n+1}(F)}(\Sigma,\mathbb{C})$.

Due to [GI14, Proposition 7.2], there are two exact sequences of $\mathrm{Sp}_{4n}(F)$-modules

[TABLE]

and

[TABLE]

Then $\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(E)}(I(-1/2),\tau^{\delta})=\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(E)}(R^{n+1,n+1}(\mathbf{1})\cap R^{n+3,n-1}(\mathbf{1}),\tau^{\delta})$ and

[TABLE]

Note that $\tau^{\delta}$ does not occur on the boundary of $I(-1/2)$. If not, assuming

[TABLE]

for some $i$ and $s_{0}=-1/2$, then

[TABLE]

with $s_{0}+i/2\geq 0$, where ${Q}^{\prime}_{i}$ is a maximal parabolic subgroup of $\mathrm{Sp}_{2n}(E)$ with Levi subgroup $\mathrm{GL}_{i}(E)\times\mathrm{Sp}_{2n-2i}(E)$ and $R_{\bar{Q}^{\prime}_{i}}$ is the Jaquect functor with respect to the opposite parabolic $\bar{Q}^{\prime}_{i}$. Due to Casselman’s criterion for the tempered representation $\tau^{\delta}$, $s_{0}+i/2=0$ and so $i=1$. Moreover, the $L$-parameter of $\tau$ is given by $2\mathbb{C}\oplus\phi^{\prime}$ with $\phi^{\prime}:WD_{E}\rightarrow\mathrm{SO}_{2n-1}(\mathbb{C})$, which implies that $\theta_{\psi_{0}}^{n,n}(\tau)\neq 0$ by [AG17, Theorem 4.3]. It contradict the assumption $\theta_{\psi_{0}}^{n,n}(\tau)=0$. Therefore

[TABLE]

and so $\dim\mathrm{Hom}_{\mathrm{Sp}_{2n}(F)}(\tau,\mathbb{C})\geq\dim\mathrm{Hom}_{\mathrm{O}_{n+1,n+1}(F)}(\Sigma,\mathbb{C})$. Thus

[TABLE] 2. (ii)

If $\theta^{n,n}_{\psi_{0}}(\tau)$ is nonzero, then one can consider

[TABLE]

with $\tau=\Theta^{2n}_{\psi_{0}}(\Sigma)=\Theta_{\psi_{E}}^{2n}(\Sigma^{\delta})$. There exists a short exact sequence of $\mathrm{O}_{2n,2n}(F)$-modules

[TABLE]

where $R^{2r}(\mathbf{1})$ is the big theta lift to $\mathrm{O}_{2n,2n}(F)$ of the trivial representation from $\mathrm{Sp}_{2r}(F)$, $\det$ is the determinant map of $\mathrm{O}_{2n,2n}(F)$ and $\mathcal{I}(s)$ is the Siegel degenerate principal series of $\mathrm{O}_{2n,2n}(F)$. (See [GI14, Proposition 7.2].) Note that

[TABLE]

Then there is an inequality

[TABLE]

where $\Theta^{2n-2}_{\psi_{0}}(\Sigma)$ is the big theta lift to $\mathrm{Sp}_{2n-2}(E)$ of $\Sigma$ of $\mathrm{O}_{n,n}(E)$.

By the assumption $\Theta_{\psi_{0}}^{2n-2}(\Sigma)=0$, we can use the same idea to show that

[TABLE]

which equals to the sum

[TABLE]

where $V$ runs over all $2n$-dimensional quadratic spaces over $F$ such that $\mathrm{O}(V\otimes_{F}E)=\mathrm{O}_{n,n}(E)$. Therefore, $\tau$ is $\mathrm{Sp}_{2n}(F)$-distinguished if and only if $\Sigma=\theta^{n,n}_{\psi_{0}}(\tau)$ is $\mathrm{O}(V)$-distinguished for some $V$.

This finishes the proof. ∎

Remark 4.1*.*

If $\Theta^{2n-2}_{\psi_{0}}(\Sigma)$ is nonzero, then $\Sigma$ is $\mathrm{O}(V)$-distinguished implies that $\tau$ is $\mathrm{Sp}(W_{n})$-distinguished. However, it is not obvious that whether $\tau$ is $\mathrm{Sp}_{2n}(F)$-distinguished implies that $\Sigma$ is $\mathrm{O}(V)$-distinguished for some certain quadratic space $V$ over $F$.

We give a conjecture to end this section.

Conjecture

Assume that $E$ is a quadratic field extension over $F$. Let $\tau$ be a supercuspidal representation of $\mathrm{Sp}_{2n}(E)$. Assume that $\theta(\tau)\neq 0$ for some $V_{E}$ with discriminant algebra a quadratic extension of $E$ which does not come from $F$, and that $\theta(\tau)=0$ for all $V_{E}$ with discriminant algebra a quadratic extension of $E$ which comes from $F$ (in particular if the normalized discriminant is trivial). Then $\tau$ is not $\mathrm{Sp}_{2n}(F)$-distinguished.

Remark 4.2*.*

Due to the fact that the Prasad conjectures for the L-parameters for theta liftings in [Pra93] have been proved by Gan-Ichino in [GI14, Appendix C], we can reformulate the above conjecture in terms of the $L$-parameter of $\tau$. Suppose that the $L$-parameter $\phi_{\tau}=\sum_{i=1}^{r}\phi_{i}$ where $\phi_{i}$ are distinct irreducible orthogonal representations of $WD_{E}$ and that $\phi_{\tau}$ contains at least one quadratic character $\chi$ of $E^{\times}$. Assume that $\chi_{i}|_{F^{\times}}\neq\mathbf{1}$ for each $1$-dimensional summand $\phi_{i}=\chi_{i}$ (the quadratic characters of $E^{\times}$) of $\phi_{\tau}$. Then $\tau$ is not $\mathrm{Sp}_{2n}(F)$-distinguished. On the Galois side, there does not exist any extension of $\phi_{\tau}$ from $WD_{E}$ to $WD_{F}$ which can be seen as part of the conjecture of Dipendra Prasad in [Pra15, Conjecture 2] for $\mathrm{Sp}_{2n}$.

Remark 4.3*.*

In fact, we have studied the case when $\mathrm{O}(\mathrm{Res}_{E/F}V_{E})\cong\mathrm{O}_{2n+2,2n-2}(F)$ in the proof of Theorem 1.2 and have verified the conjecture for $n\leq 2$ in [Lu18, Lu19].

5. The $\mathrm{Sp}_{4}(F)$-distinguished representation of $\mathrm{SU}_{2,2}$

In this section, we will show that there does not exist any tempered representation of $\mathrm{SU}_{2,2}(E/F)$ distinguished by $\mathrm{Sp}_{4}(F)$, where $E/F$ is a quadratic field extension.

Let $V$ be an $n$-dimensional quadratic space over $F$ with $n\geq 5$. Let

[TABLE]

We will consider the theta correspondence between the special orthogonal group $\mathrm{SO}(V)$ and $\mathrm{SL}_{2}(F)$ or $\mathrm{Mp}_{2}(F)$. Fix a unitary nontrivial additive character $\psi$ of $F$. Let $\omega_{\psi}$ be the Weil representation of $\mathrm{Mp}_{2n}(F)$. Given an irreducible representation $\Sigma$ of $\mathrm{SO}(V)$, there exists a unique (genuine) representation $\tau$ of $G$ such that the maximal $\Sigma$-isotropic quotient of $\omega_{\psi}$ is of the form

[TABLE]

where $\tau$ is called the big theta lift of $\Sigma$. Given a tempered representation $\Sigma$ of $\mathrm{SO}(V)$, assume that the small theta lift $\theta_{\psi}(\Sigma)$ is a nonzero representation of $G$. This means that the first occurence of $\Sigma$ is equal to $2$, i.e. $m^{\mbox{down}}(\Sigma)=2$ in the sense of [AG17]. We can use the theta correspondence for the pair $(\mathrm{SO}(V),G)$ instead of the dual pair $(\mathrm{O}(V),G)$ if $\dim V\geq 3$. (See [Pra93, §5].) Here $\dim V$ is very small, so the local theta correspondence for $(\mathrm{SO}(V),G)$ can be very explicit.

Theorem 5.1**.**

Let $V^{\prime}\subset V$ be a subspace of $V$ of codimension one and $\dim_{F}V\geq 5$. If there exists a tempered representation $\Sigma$ of $\mathrm{SO}(V)$ distinguished by $\mathrm{SO}(V^{\prime})$, then the rank of the subgroup $\mathrm{SO}(V^{\prime})$ is less or equal to $1$. Thus, the pair $(\mathrm{SO}(V),\mathrm{SO}(V^{\prime}))$ sits inside the following table

Proof.

Thanks to [DP18, Proposition 9], the assumption that $\Sigma$ is $\mathrm{SO}(V^{\prime})$-distinguished implies that the small theta lift $\theta_{\psi}(\Sigma)$ from $\mathrm{SO}(V)$ to $G$ is nonzero and that the big theta lift $\tau$ of $\Sigma$ is a $\psi_{a}$-generic representation of $G$, where $\psi_{a}(x)=\psi(ax)$ is a nontrivial additive character of $F$ for $a\in F^{\times}/{F^{\times}}^{2}$. Since $\theta_{\psi}(\Sigma)\neq 0$, we obtain $m^{\mbox{down}}(\Sigma)=2$. Due to the chain condition (defined in the Appendix) in [AG17, Theorem 4.1], there is an upper bound for the dimension of $V$, i.e. $\dim V=n\leq 7$. If $n=\dim V>5$ or the Langlands parameter $\phi_{\Sigma}\neq 2S_{2}$(where $S_{r}$ is the irreducible algebraic representation $\mathrm{Sym}^{r-1}(\mathbb{C})$ of $\mathrm{SL}_{2}(\mathbb{C})$), then the Langlands parameter of $\Sigma$ contains $S_{n-3}$ with multiplicity one. [AG17, Proposition 5.4] implies that the big theta lift $\tau=\Theta_{\psi}(\Sigma)$ is irreducible and tempered. So $\tau=\theta_{\psi}(\Sigma)$ and

[TABLE]

where $N\cong F$ is the unipotent subgroup in $G$ and $V^{a}$ is the subspace of $V$. If $n=5$ and $\phi_{\Sigma}=2S_{2}$, then $\theta_{\psi}(\Sigma)$ is a discrete series representation of $G$. Moreover, all irreducible subquotients of $\tau$ are tempered due to [AG17, Proposition 5.5] and so $\tau=\theta_{\psi}(\Sigma)$. Furthermore, $m^{\mbox{up}}(\tau)=\dim V$. According to $n$, there are a few cases:

1. (i).

The case $n=5$. There are two subcases.

• •

If $\mathrm{SO}(V)\cong\mathrm{PGSp}_{4}(F)$ is split, then the subgroup $\mathrm{SO}(V^{\prime})$ may be $\mathrm{SO}_{2,2}(F)$ or $\mathrm{SO}_{3,1}(F)\cong\mathrm{GL}_{2}(E)^{\natural}/F^{\times}$, where

[TABLE]

If $\mathrm{SO}(V^{\prime})\cong\mathrm{SO}_{2,2}(F)$ is split, then $a=1$ and

[TABLE]

Since $\tau$ participates in the theta correspondence between $PD^{\times}$ and $\mathrm{Mp}_{2}(F)$, it is not $\psi$-generic. Otherwise, if $\tau$ is $\psi$-generic, then the theta lift to $\mathrm{PGL}_{2}(F)$ of $\tau$ will be nonzero and tempered, which implies that $\Sigma$ is not tempered. Hence the tempered representation $\Sigma$ can never be $\mathrm{SO}_{2,2}(F)$-distinguished. However, (5.1) implies that $\Sigma$ is $\mathrm{SO}_{3,1}(F)$-distinguished if and only if $\tau$ is $\psi_{a}$-generic, where $a=\epsilon\in F^{\times}\setminus NE^{\times}$ and $\mathrm{SO}(V^{a})\cong\mathrm{SO}_{3,1}(F)$. Here the quadratic field extension $E/F$ depends on $\tau$.

• •

If $\mathrm{SO}(V)\cong\mathrm{PGSp}_{1,1}(F)$ (the pure inner form of $\mathrm{PGSp}_{4}(F)$) is not quasi-split, then $\tau$ could be a tempered principal series representation of $G$, which is both $\psi$-generic and $\psi_{\epsilon}$-generic. In this case, $\Sigma$ could be both $\mathrm{SO}_{4,0}(F)$-distinguished and $\mathrm{SO}_{3,1}(F)$-distinguished. 2. (ii).

The case $n=6$. The theta lift from $\mathrm{SL}_{2}(F)$ to the split special orthogonal group $\mathrm{SO}_{3,3}(F)$ can never be a tempered representation. (See [AG17, §4].) So we only study the cases $\mathrm{SO}(V)\cong\mathrm{SO}_{4,2}(F)$ or $\mathrm{SO}(V)\cong\mathrm{SO}_{5,1}(F)$, where $\mathrm{SO}_{4,2}(F)$ is the special orthogonal group (quasi-split) of a $6$-dimensional quadratic vector space with discriminant $E$ and Hasse invariant $+1$ and $\mathrm{SO}_{5,1}(F)$ is the special orthogonal group (non-split) of a $6$-dimensional quadratic vector space of trivial discriminant.

• •

If $\mathrm{SO}(V)\cong\mathrm{SO}_{5,1}(F)$, then $\tau$ could be an irreducible principal series $I(\chi)$ of $\mathrm{SL}_{2}(F)$. Then $\Sigma$ is $\mathrm{SO}_{4,1}(F)$-distinguished with Langlands parameter

[TABLE]

• •

If $\mathrm{SO}(V)\cong\mathrm{SO}_{4,2}(F)$, then the representation $\tau$ of $\mathrm{SL}_{2}(F)$ is not $\psi$-generic but $\psi_{\epsilon}$-generic. So $\Sigma$ can never be $\mathrm{SO}_{3,2}(F)$-distinguished. However, $\Sigma$ is $\mathrm{SO}_{4,1}(F)$-distinguished once $\theta_{\psi}(\Sigma)\neq 0$. 3. (iii).

The case $n=7$. The theta lift from $\mathrm{Mp}_{2}(F)$ to the split group $\mathrm{SO}_{4,3}(F)=\mathrm{SO}(V)$ must be nontempered, where the discriminant of $V$ is trivial. So we only need to consider the case $\mathrm{SO}(V)\cong\mathrm{SO}_{5,2}(F)$. There is only one case that $\phi_{\Sigma}=S_{2}\oplus S_{4}$ and $\tau=\omega_{\psi}^{-}$ is the $\psi$-generic odd Weil representation of $\mathrm{Mp}_{2}(F)$. Therefore, $\Sigma$ is $\mathrm{SO}_{5,1}(F)$-distinguished where $a=1$ and $\mathrm{SO}(V^{a})\cong\mathrm{SO}_{5,1}(F)$.

In a short summary, the rank of $\mathrm{SO}(V^{\prime})$ is less or equal to $1$. Therefore, we have finished the proof. ∎

Remark 5.2*.*

If $\dim_{F}V=4$ or $3$, then $\dim_{F}V^{\prime}$ is $3$ or $2$ respectively and so the rank of $\mathrm{SO}(V^{\prime})$ is less or equal to $1$ automatically. The case $\mathrm{SO}_{3}\hookrightarrow\mathrm{SO}_{4}$ has been investigated in [Lu17b]. Thus we have proved [DP18, Conjecture 2] in the case of non-archimedean fields.

Corollary 5.3**.**

Assume that $\dim V=6$ and $\mathrm{SO}(V)=\mathrm{SO}_{4,2}(F)$. There is an isogeny $\mathrm{SU}_{2,2}\rightarrow\mathrm{SO}_{4,2}$ with kernel $\{\pm 1\}$. (See [DP18, §6].) There are two isogeny diagrams

[TABLE]

where $\mathrm{Sp}_{1,1}$ is the unique inner form of $\mathrm{Sp}_{4}$ defined over $F$. Then

• •

There does not exist any tempered representation of $\mathrm{SU}_{2,2}(E/F)$ distinguished by $\mathrm{Sp}_{4}(F)$.

• •

There do exist supercuspidal representations of $\mathrm{SU}_{2,2}(E/F)$ distinguished by $\mathrm{Sp}_{1,1}(F)$.

Remark 5.4*.*

It generalizes the result of Dijols and Prasad [DP18] that there does not exist any supercuspidal representation of $\mathrm{SU}_{2,2}(F)$ distinguished by $\mathrm{Sp}_{4}(F)$.

Appendix A The local theta correspondence of tempered representations and Langlands parameters

The purpose of this appendix is to describe theta lifts of tempered representations in terms of the local Langlands correspondence. (See [AG17] for more details.)

Let $V$ be a $m$-dimensional quadratic space over $F$ with quadratic character $\chi_{V}$. Let $\pi$ be a tempered representation of $\mathrm{O}(V)$ with enhanced $L$-parameter $(\phi,\eta)$ and $\det\phi=\chi_{V}$. We can decompose

[TABLE]

where $\phi_{1},\cdots,\phi_{r}$ are distinct irreducible orthogonal representations of $WD_{F}$, $m_{i}=m_{\phi}(\phi_{i})$ is multiplicity of $\phi_{i}$ contained in $\phi$, and $\phi^{\prime}$ is a sum of irreducible representations of $WD_{F}$ which are not orthogonal. If $\tau$ is a square-integrable representation, then $m_{i}=1$ for all $i$ and $\phi^{\prime}=0$. Let $S_{\phi}$ be the component group of $\phi$. Let $W$ be a $2n$-dimensional symplectic vector space over $F$. Define $\ell=m-2n-1$ and

[TABLE]

Let $S_{r}$ be the irreducible algebraic representation $\mathrm{Sym}^{r-1}(\mathbb{C})$ of $\mathrm{SL}_{2}(\mathbb{C})$.

Theorem A.1** (Atobe-Gan).**

Let $\pi$ be a tempered representation of $\mathrm{O}(V)$ with enhanced $L$-parameter $(\phi,\eta)$.

1. (i)

Consider the set $\mathfrak{T}$ containing $\kappa-2$ and all integers $l>0$ with $\ell\equiv\kappa\pmod{2}$ satisfying the following conditions:

• •

(chain condition) $\phi$ contains $S_{r}$ for $r=\kappa,\kappa+2,\cdots,\ell$;

• •

(odd-ness condition) the multiplicity $m_{\phi}(S_{r})$ is odd for $r=\kappa,\kappa+2,\cdots,\ell-2$;

• •

(initial condition) if $\kappa=2$, then $\eta(e_{2})=-1$ if $m$ is odd;

• •

(alternationg condition) $\eta(e_{r})=\eta(e_{r+2})$ for $r=\kappa,\kappa+2,\cdots,\ell-2$.

Here, $e_{r}$ is the element in $S_{\phi}$ corresponding to $S_{r}$. Let $\ell(\pi)=\max\mathfrak{T}.$ Then

[TABLE] 2. (ii)

Assume that the theta lift $\theta(\pi)$ to $\mathrm{Sp}(W)$ or $\mathrm{Mp}(W)$ (depends on $n$) is nonzero and has an enhanced $L$-parameter $(\phi_{\theta(\pi)},\eta_{\theta(\pi)})$. Put $m_{1}=m^{\mbox{down}}(\pi)+1-\kappa$. If $m^{\mbox{down}}(\pi)\leq\dim W_{n}<m_{1}$, then

[TABLE]

where $\ell=m-2n-1$. If $\dim W_{n}=m_{1}$, then

[TABLE]

If $\dim W_{n}>m_{1}$, then $\phi_{\theta(\pi)}$ is not bounded. 3. (iii)

If $\dim W_{n}>m^{\mbox{up}}(\pi)$, then $\theta(\pi)$ is nonzero and $\phi_{\theta(\pi)}$ is not bounded. Therefore, $\theta(\pi)$ is not tempered.

The results of Atobe-Gan in [AG17] are very general. There are relations between the characters $\eta$ and $\eta_{\theta(\pi)}$ which are omitted here due to the purpose that this paper focuses on the relations between Langlands parameters $\phi$ and $\phi_{\theta(\pi)}$. There are analogous theorems if we switch $\mathrm{Sp}(W)$ and $\mathrm{O}(V)$.