Symmetry breaking operators for strongly spherical reductive pairs

TL;DR

This paper constructs explicit symmetry breaking operators for strongly spherical reductive pairs, providing new tools for understanding representation restrictions and applications to conjectures like Gross-Prasad.

Contribution

It explicitly constructs intertwining operators for all strongly spherical pairs, extending to vector-valued cases and deriving formulas for multiplicities in principal series.

Findings

Constructed symmetry breaking operators depending holomorphically on parameters.

Showed these operators generically span the Hom space for spherical principal series.

Applied results to prove cases of the Gross-Prasad conjecture and bounds for Shintani functions.

Abstract

A real reductive pair $(G,H)$ is called strongly spherical if the homogeneous space $(G\times H)/{\rm diag}(H)$ is real spherical. This geometric condition is equivalent to the representation theoretic property that ${\rm dim\,Hom}_H(\pi|_H,\tau)<\infty$ for all smooth admissible representations $\pi$ of $G$ and $\tau$ of $H$. In this paper we explicitly construct for all strongly spherical pairs $(G,H)$ intertwining operators in ${\rm Hom}_H(\pi|_H,\tau)$ for $\pi$ and $\tau$ spherical principal series representations of $G$ and $H$. These so-called symmetry breaking operators depend holomorphically on the induction parameters and we further show that they generically span the space ${\rm Hom}_H(\pi|_H,\tau)$. In the special case of multiplicity one pairs we extend our construction to vector-valued principal series representations and obtain generic formulas for the multiplicities between arbitrary principal series. As an application, we prove an early version of the Gross-Prasad conjecture for complex orthogonal groups, and also provide lower bounds for the dimension of the space of Shintani functions.

Equations

$$\dim {\mathrm{Hom}}_H(\pi {|}_H,\tau )\in \mathbb{N}\cup \{\infty \},$$

$$\dim {\mathrm{Hom}}_H(\pi {|}_H,\tau )<\infty$$

$$\begin{array}{c}(\mathrm{GL}(n+1,\mathbb{C}),\mathrm{GL}(n,\mathbb{C})),(\mathrm{GL}(n+1,\mathbb{R}),\mathrm{GL}(n,\mathbb{R})),(U(p,q+1),U(p,q)),\\ (\mathrm{SO}(n+1,\mathbb{C}),\mathrm{SO}(n,\mathbb{C})),(\mathrm{SO}(p,q+1),\mathrm{SO}(p,q)),\end{array}$$

$$\dim {\mathrm{Hom}}_H(\pi {|}_H,\tau )\le 1$$

$${\pi }_{\xi ,\lambda }={\mathrm{Ind}}_{P_G}^G(\xi \otimes e^{\lambda }\otimes 1),{\tau }_{\eta ,\nu }={\mathrm{Ind}}_{P_H}^H(\eta \otimes e^{\nu }\otimes 1).$$

$$\dim {\mathrm{Hom}}_H({\pi }_{\lambda }{|}_H,{\tau }_{\nu })\ge \#(P_H\backslash G/P_G{)}_{\mathrm{open}},$$

$$\dim {\mathrm{Hom}}_H({\pi }_{\lambda }{|}_H,{\tau }_{\nu })=\#(P_H\backslash G/P_G{)}_{\mathrm{open}}.$$

$$\dim {\mathrm{Hom}}_H({\pi }_{\xi ,\lambda }{|}_H,{\tau }_{\eta ,\nu })\ge 1\text{mbox}whenever{\mathrm{Hom}}_M(\xi {|}_M,\eta {|}_M)\ne \{0\},$$

$$\dim\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})=\begin{cases}1&\mbox{for $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$,}\\ 0&\mbox{for $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})=\{0\}$.}\end{cases}$$

$$\dim {\mathrm{Hom}}_H({\pi }_{\xi ,\lambda }{|}_H,{\tau }_{\eta ,\nu })=1.$$

$$\dim \mathrm{Sh}(\lambda ,\nu )\ge \dim {\mathrm{Sh}}_{\mathrm{mod}}(\lambda ,\nu )\ge \#(P_H\backslash G/P_G{)}_{\mathrm{open}},$$

$$\dim {\mathrm{Sh}}_{\mathrm{mod}}(\lambda ,\nu )=\#(P_H\backslash G/P_G{)}_{\mathrm{open}}.$$

$$\mathfrak{g}={\mathfrak{p}}_G+{\mathfrak{p}}_H.$$

$$\mathfrak{g}={\mathfrak{g}}_{\mathrm{n}}\oplus {\mathfrak{g}}_{\mathrm{el}},\mathfrak{h}={\mathfrak{h}}_{\mathrm{n}}\oplus {\mathfrak{h}}_{\mathrm{el}},$$

$$(\mathfrak{g},\mathfrak{h})=(\mathrm{sp}(p,q+1)+\cdots +\mathrm{sp}(p,q+1),\mathrm{sp}(p,q)+\cdots +\mathrm{sp}(p,q)+\mathrm{sp}(1)),$$

$$\sigma (X_1+\cdots +X_p)={\sigma }_1(X_1)+\cdots +{\sigma }_p(X_p),(X_j\in {\mathfrak{g}}_j),$$

$$K=G^{\theta }\subseteq G\text{mbox}andH\cap K=H^{\theta }\subseteq H$$

$$A_G=\exp ({\mathfrak{a}}_G),A_H=\exp ({\mathfrak{a}}_H).$$

$$\mathfrak{g}({\mathfrak{a}}_G;\alpha )$$

$$\mathfrak{g}({\mathfrak{a}}_H;\beta )$$

$$\overline{\alpha }\in {\Sigma }^{+}(\mathfrak{g},{\mathfrak{a}}_H)\cup \{0\}\forall \alpha \in {\Sigma }^{+}(\mathfrak{g},{\mathfrak{a}}_G).$$

$${\mathfrak{n}}_G=\underset{\alpha \in {\Sigma }^{+}(\mathfrak{g},{\mathfrak{a}}_G)}{⨁}\mathfrak{g}({\mathfrak{a}}_G;\alpha ),\mathfrak{n}=\underset{\begin{array}{c}\alpha \in {\Sigma }^{+}(\mathfrak{g},{\mathfrak{a}}_G)\\ \overline{\alpha }\ne 0\end{array}}{⨁}\mathfrak{g}({\mathfrak{a}}_G;\alpha )=\underset{\beta \in {\Sigma }^{+}(\mathfrak{g},{\mathfrak{a}}_H)}{⨁}\mathfrak{g}({\mathfrak{a}}_H;\beta ).$$

$$\mathfrak{n}={\mathfrak{n}}^{\sigma }\oplus {\mathfrak{n}}^{-\sigma }.$$

$$N_G=\exp ({\mathfrak{n}}_G),N=\exp (\mathfrak{n}),N_H=\exp ({\mathfrak{n}}_H).$$

$$M_G=Z_K({\mathfrak{a}}_G),L=Z_G({\mathfrak{a}}_H),M_H=Z_{H\cap K}({\mathfrak{a}}_H).$$

$$P_G\subseteq Q\supseteq P_H.$$

$$G=\bigcup _{w\in W_Q\backslash W}Q\tilde{w}{P}_G,$$

$$W_Q=Z_W(A_H)=\{w\in W:\tilde{w}\in L\cap K\}.$$

$$Q\cdot \tilde{w}{P}_G\simeq Q/(Q\cap \tilde{w}{P}_G{\tilde{w}}^{-1})$$

$$\mathfrak{l}\cap \mathfrak{k}=Z_{\mathfrak{k}}({\mathfrak{a}}_H)={\mathfrak{m}}_H+{\mathfrak{m}}_G\text{mbox}andL\cap K=Z_K({\mathfrak{a}}_H)=M_H{M}_G.$$

$$P_{H}g{P}_G=P_{H}p\exp (X){\tilde{w}}_0{P}_G.$$

Full text

As an application, we prove an early version of the Gross–Prasad conjecture for complex orthogonal groups, and also provide lower bounds for the dimension of the space of Shintani functions.

1. 1 The structure of strongly spherical reductive pairs
2. 2 Spherical matrix coefficients
3. 3 Lower multiplicity bounds – Construction of symmetry breaking operators
4. 4 Upper multiplicity bounds – Invariant distributions
5. 5 Application to Shintani functions
6. 6 Multiplicity one pairs
7. 7 An example: $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$
8. A Finite-dimensional branching rules for strong Gelfand pairs

Introduction

One major question in the representation theory of real reductive groups is how an irreducible representation of a group $G$ decomposes if restricted to a subgroup $H$. In the context of infinite-dimensional representations of non-compact Lie groups this leads to the study of the multiplicities

[TABLE]

where $\pi$ and $\tau$ are irreducible representations of $G$ and $H$, usually assumed to be smooth admissible Fréchet representations of moderate growth. In general these multiplicities might be infinite, so to find a good setting for studying them one is interested in pairs $(G,H)$ of real reductive groups where $\dim\operatorname{Hom}_{H}(\pi|_{H},\tau)$ is always finite. Elements of $\operatorname{Hom}_{H}(\pi|_{H},\tau)$ are also referred to as symmetry breaking operators, a term coined by Kobayashi [18].

Following [16] we call a pair $(G,H)$ consisting of a real reductive group $G$ and a reductive subgroup $H$ strongly spherical provided the homogeneous space $(G\times H)/\operatorname{diag}(H)$ is real spherical, i.e. a minimal parabolic subgroup $P_{G}\times P_{H}$ of $G\times H$ has an open orbit. We note that this is equivalent to the double coset space $P_{H}\backslash G/P_{G}$ being finite. The interest in strongly spherical pairs in the context of representation theory is due to the following result by Kobayashi–Oshima [21]:

Fact I** (see [21, Theorem C]).**

If $(G,H)$ is strongly spherical then

[TABLE]

for all smooth admissible representations $\pi$ of $G$ and $\tau$ of $H$. If additionally $G$ and $H$ are defined algebraically over $\mathbb{R}$, then also the converse statement holds.

Among the strongly spherical pairs, Kobayashi–Oshima [21] also characterized those with uniformly bounded multiplicities. The corresponding pairs of Lie algebras essentially form five families, and choosing the right Lie groups yields five families of groups $(G,H)$ whose multiplicities are uniformly bounded by one. This multiplicity one property is due to Sun–Zhu [41]:

Fact II** (see [41, Theorem B]).**

If $(G,H)$ is one of the pairs

[TABLE]

then

[TABLE]

for all irreducible smooth admissible Fréchet representations $\pi$ of $G$ and $\tau$ of $H$ of moderate growth.

Both Facts I and II lead to the natural problem of determining $\dim\operatorname{Hom}_{H}(\pi|_{H},\tau)$ for given irreducible representations $\pi$ and $\tau$ of $G$ and $H$, as advocated by Kobayashi [18] in his ABC program as Stages B and C. For the multiplicity one pairs of general linear groups, this question is linked to the famous Rankin–Selberg integrals, and for the pairs of orthogonal and unitary groups the Gross–Prasad and Gan–Gross–Prasad conjectures make predictions about when the multiplicities are non-zero.

Since every irreducible smooth admissible Fréchet representation of moderate growth is by the Harish-Chandra Subquotient Theorem a subquotient of a principal series representation, it is reasonable to study this problem for principal series representations $\pi$ and $\tau$. This has previously been done for special cases by Kobayashi–Speh [22, 23] and Clerc [2, 3]. In this paper, we construct for all strongly spherical reductive pairs $(G,H)$ explicit families of symmetry breaking operators between principal series representations that depend holomorphically on the principal series parameters. Our key new ingredient is the systematic use of matrix coefficients of finite-dimensional representations of $G$ and $H$ in the construction of the corresponding distribution kernels, and it provides a conceptual way to determine these kernels explicitly. This establishes lower bounds for the multiplicities in question and provides a first step to the full classification. Together with upper bounds (obtained using Bruhat’s theory of invariant distributions) we are able to compute $\dim\operatorname{Hom}_{H}(\pi|_{H},\tau)$, at least generically, for all strongly spherical pairs $(G,H)$ and spherical principal series representations, and for the multiplicity one pairs also for non-spherical principal series representations. In the case $(G,H)=(\operatorname{SO}(n+1,\mathbb{C}),\operatorname{SO}(n,\mathbb{C}))$ this proves the local Gross–Prasad conjecture at the Archimedean place $k=\mathbb{C}$ as initially stated in [8].

Statement of the main results

Let $G$ be a real reductive group in the Harish-Chandra class and $H\subseteq G$ be a reductive subgroup such that the pair $(G,H)$ is strongly spherical. Let $\mathfrak{g}$ and $\mathfrak{h}$ denote the corresponding Lie algebras and assume that the pair $(\mathfrak{g},\mathfrak{h})$ is indecomposable, i.e. it cannot be written as the direct sum of two non-trivial pairs of reductive Lie algebras. A classification of all such pairs $(\mathfrak{g},\mathfrak{h})$ (under a slightly stronger indecomposability assumption) was established by Kobayashi–Matsuki [20] and Knop–Krötz–Pecher–Schlichtkrull [15, 16], and we summarize the result in Theorem 1.6. We further assume that $(\mathfrak{g},\mathfrak{h})$ is non-trivial, i.e. $\mathfrak{g}\neq\mathfrak{h}$. (In fact, our results do not hold in the case $\mathfrak{g}=\mathfrak{h}$ since intertwining operators between principal series of a real reductive group $G$ can only exist if the induction parameters are related by an element of the Weyl group.)

Let $P_{G}=M_{G}A_{G}N_{G}\subseteq G$ and $P_{H}=M_{H}A_{H}N_{H}\subseteq H$ be minimal parabolic subgroups and write $\mathfrak{a}_{G}$ and $\mathfrak{a}_{H}$ for the Lie algebras of $A_{G}$ and $A_{H}$. For irreducible finite-dimensional representations $\xi$ of $M_{G}$, $\eta$ of $M_{H}$ and $\lambda\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}$, $\nu\in\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we consider the principal series representations (smooth normalized parabolic induction)

[TABLE]

In case $\xi={\rm\bf 1}$ and $\eta={\rm\bf 1}$ are the trivial representations of $M_{G}$ and $M_{H}$, we abbreviate $\pi_{\lambda}=\pi_{{\rm\bf 1},\lambda}$ and $\tau_{\nu}=\tau_{{\rm\bf 1},\nu}$. Our first main result relates $\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$ to $(P_{H}\backslash G/P_{G})_{\rm open}$, the set of open double cosets in $P_{H}\backslash G/P_{G}$.

Theorem A** (see Corollary 3.4 and 4.4).**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have the lower multiplicity bound

[TABLE]

and for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ (see (4.1) for the precise condition) we have

[TABLE]

Let us briefly explain the method of proof. First of all, the generic upper multiplicity bound $\leq\#(P_{H}\backslash G/P_{G})_{\rm open}$ is established by identifying intertwining operators with their distribution kernels which are certain $(P_{H}\times P_{G})$-invariant distributions on $G$ (see Section 4.2). To bound the dimension of the space of invariant distributions we use Bruhat’s theory. Here, the non-open $(P_{H}\times P_{G})$-orbits in $G$ are particularly important, and we prove a new structural result about them (see Theorem 1.3) which is the necessary technical ingredient in the proof of generic upper bounds (see Theorem 4.1). The lower multiplicity bounds are established by explicitly constructing a non-trivial holomorphic family $A_{\lambda,\nu}\in\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$ of symmetry breaking operators for every open double coset in $P_{H}\backslash G/P_{G}$. The construction of this family of operators is in terms of their distribution kernels which turn out to be products of complex powers of matrix coefficients belonging to finite-dimensional spherical representations of $G$. The technical ingredients in this part are the existence of enough such matrix coefficients (see Theorem 2.6) and the meromorphic/holomorphic extension of the complex powers (see Theorem 3.3). Then the lower bounds are established by regularizing the families $A_{\lambda,\nu}$ in the holomorphic parameters $(\lambda,\nu)$.

For the multiplicity one pairs in Fact II we also consider non-spherical principal series representations and show the following result:

Theorem B** (see Theorem 6.5).**

Assume that $(G,H)$ is a multiplicity one pair. Then for all $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$ and $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have the lower multiplicity bound

[TABLE]

and for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

Here $M\subseteq M_{G}\cap M_{H}$ denotes the stabilizer of the unique open $P_{H}$-orbit in $G/P_{G}$.

The passage from spherical principal series $\pi_{\lambda}$ as treated in Theorem A to general principal series $\pi_{\xi,\lambda}$ uses a variant of the Jantzen–Zuckerman translation principle. Here both $\pi_{\lambda}$ and $\tau_{\nu}$ are tensored with certain finite-dimensional representations of $G$ and $H$, and we show the existence of such representations case-by-case (see Theorem 6.4). We illustrate the construction of symmetry breaking operators for the multiplicity one pair $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$ by providing explicit formulas for the distribution kernels (see Section 7).

We remark that Theorems A and B leave open the question of determining the multiplicities $\dim\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})$ for all parameters. In general, this question is much more involved and has so far only been solved in special cases (see Clerc [2, 3] and Kobayashi–Speh [22]). In these cases, all operators in $\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})$ arise from a holomorphic family of operators, so that our explicit construction of meromorphic families provides a first major step for the full classification for general strongly spherical reductive pairs. We hope to return to this point later.

Applications

Let us present two interesting applications of the main results. Combining Theorem B with Fact II we immediately obtain the following corollary:

Corollary C** (see Corollary 6.8).**

Assume that $(G,H)$ is a multiplicity one pair, where we additionally assume $p=q$ or $p=q+1$ in the case of indefinite orthogonal or unitary groups. Then, if both $\pi_{\xi,\lambda}$ and $\tau_{\eta,\nu}$ are irreducible we have

[TABLE]

For $(G,H)=(\operatorname{SO}(n+1,\mathbb{C}),\operatorname{SO}(n,\mathbb{C}))$ this proves the local Gross–Prasad conjecture [8, Conjecture 11.5] at the Archimedean place $k=\mathbb{C}$ (see Conjecture 6.7). We remark that the Gross–Prasad conjecture has been extended by several people to the case of reducible principal series, for which we cannot make precise statements in the generality discussed in this paper. We expect that our results also provide some information towards the local Gross–Prasad conjecture at the Archimedean place $k=\mathbb{R}$ where one considers the pairs $(G,H)=(\operatorname{SO}(p,q+1),\operatorname{SO}(p,q))$. However, for real groups it will be necessary to also consider principal series representations induced from more general cuspidal parabolic subgroups. We hope to return to this topic in a future work.

Another application concerns the study of Shintani functions for the pair $(G,H)$ which was recently taken up by Kobayashi [17]. For $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we write $\operatorname{Sh}(\lambda,\nu)$ for the space of Shintani functions for $(G,H)$ of type $(\lambda,\nu)$ and $\operatorname{Sh}_{\rm mod}(\lambda,\nu)$ for its subspace of Shintani functions of moderate growth (see Section 5 for the precise definitions). Kobayashi [17] showed that $\dim\operatorname{Sh}(\lambda,\nu)<\infty$ for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ if and only if $(G,H)$ is strongly spherical. Combining results from [17] with Theorem A shows the following corollary:

Corollary D** (see Theorem 5.3).**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

and for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

Relation to other work

Let us first mention previous works in which holomorphic families of intertwining operators $A_{\lambda,\nu}\in\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$ appear. The construction of operators $A_{\lambda,\nu}$ for the pairs $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,n)+\operatorname{\mathfrak{so}}(1,n),\operatorname{diag}\operatorname{\mathfrak{so}}(1,n))$ can be found in the work of Oksak [36] for $n=3$ (the case of $\operatorname{\mathfrak{so}}(1,3)\simeq\operatorname{\mathfrak{sl}}(2,\mathbb{C})$), Bernstein–Reznikov [1] for $n=2$ (the case of $\operatorname{\mathfrak{so}}(1,2)\simeq\operatorname{\mathfrak{sl}}(2,\mathbb{R})$) and Clerc–Kobayashi–Ørsted–Pevzner [4] for arbitrary $n\geq 2$. Later Clerc [2, 3] gave a complete description of the space $\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$ for all parameters $(\lambda,\nu)$. For $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,n+1),\operatorname{\mathfrak{so}}(1,n))$ Kobayashi–Speh [22] obtained a full classification of all symmetry breaking operators in terms of the holomorphic family $A_{\lambda,\nu}$. In fact, some of the analytic arguments we use can also be found in [22, 23] (see e.g. [22, Lemma 11.10] which deduces lower multiplicity bounds from meromorphic families of intertwining operators). For some symmetric pairs of low rank, in particular for all symmetric pairs $(\mathfrak{g},\mathfrak{h})$ with $\mathfrak{g}$ of rank one, the work of Möllers–Ørsted–Oshima [29] yields holomorphic families of symmetry breaking operators. We also note that for $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{gl}}(n+1,\mathbb{R}),\operatorname{\mathfrak{gl}}(n,\mathbb{R}))$ the kernel functions given in Section 7 can be found in the work of Murase–Sugano [34] in the context of $p$-adic groups, and in a slightly different form also in the recent work of Neretin [35] in the context of finite-dimensional representations. The conceptual construction via finite-dimensional matrix-coefficients that we present in this paper seems to be new, and generalizes all previous constructions to the setting of strongly spherical pairs.

We remark that some of our statements can also be proven differently using the recent work of Gourevitch–Sahi–Sayag [7] on the extension of invariant distributions. In fact, their work uses a similar idea, namely the extension of an invariant distribution to a meromorphic family of distributions. However, their meromorphic families only depend on one complex parameter whereas our constructed families depend on $(\lambda,\nu)$ and hence contain more information. Moreover, we provide a method to explicitly determine the meromorphic families in terms of matrix coefficients, while in [7] the construction of the invariant distributions is more indirect.

Although our holomorphic families of symmetry breaking operators generically span the space of all intertwining operators, it is much more difficult to determine $\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})$ for singular parameters $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$. Theorems A and B provide lower bounds for the multiplicities, but it turns out that for singular parameters the multiplicities can be larger. A systematic study of multiplicities and symmetry breaking operators for $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,n+1),\operatorname{\mathfrak{so}}(1,n))$ was initiated by Kobayashi, and we refer the reader to the relevant articles by Kobayashi–Speh [22, 23] and Kobayashi–Kubo–Pevzner [19] (see also the work of Fischmann–Juhl–Somberg [5]). We expect that our holomorphic families play a major role in the full classification of symmetry breaking operators as is the case for $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,n+1),\operatorname{\mathfrak{so}}(1,n))$ (see [22]), and therefore view our general construction as a first step into this direction for the class of strongly spherical reductive pairs.

Outlook

The principal series representations considered in this paper are all induced from a minimal parabolic subgroup. However, by the Langlands classification every smooth admissible Fréchet representation of moderate growth is the unique irreducible quotient of a generalized principal series representation, induced from an arbitrary cuspidal parabolic subgroup. At least for the multiplicity one pairs a generalization of our construction to cuspidal parabolic subgroups is desirable.

Our holomorphic families of symmetry breaking operators also allow an interpretation as invariant distribution vectors. More precisely, one can view the distribution kernels as $\operatorname{diag}(H)$-invariant distribution vectors on principal series representations of $G\times H$. As such, they are expected to contribute to the most continuous part of the Plancherel formula for the real spherical homogeneous spaces $(G\times H)/\operatorname{diag}(H)$. It would be interesting to investigate this topic further, especially in connection with the recent advances in the harmonic analysis on real spherical spaces (see e.g. [25] and references therein).

Another possible application of symmetry breaking operators is the explicit construction of branching laws for unitary representations. This was successfully carried out for the pair $(G,H)=(\operatorname{O}(1,n+1),\operatorname{O}(1,m+1)\times\operatorname{O}(n-m))$ in the case of unitary principal series and complementary series representations by Möllers–Oshima [31]. For $(G,H)=(\operatorname{GL}(n+1,\mathbb{C}),\operatorname{GL}(n,\mathbb{C}))$ a similar suggestion was made by Neretin [35, Section 5].

Let us also mention connections to boundary value problems [30] and automorphic forms [1, 27] that were established for $(G,H)=(\operatorname{O}(1,n+1),\operatorname{O}(1,n))$ and might be of interest also for more general strongly spherical pairs.

Structure of the paper

In Section 1 we recall some structure theory and the classification of strongly spherical real reductive pairs. The main new result here is a characterization of the open double cosets in $P_{H}\backslash G/P_{G}$ (see Theorem 1.3). The construction of $(P_{H}\times P_{G})$-equivariant matrix coefficients on $G$ is the content of Section 2, and Theorem 2.6 ensures the existence of enough such matrix coefficients. Section 3 deals with the explicit construction of symmetry breaking operators between spherical principal series representations of strongly spherical pairs (see Theorem 3.3). This construction uses the results of Sections 1 and 2 in a crucial way, and implies the claimed lower bounds for multiplicities. The upper bounds are established in Section 4 using Bruhat’s theory of invariant distributions (see Theorem 4.1). The application of this technique depends heavily on the results of Section 1. In Section 5 the previous results are applied to obtain bounds for the space of Shintani functions (see Theorem 5.3), following Kobayashi’s recent approach via symmetry breaking operators. The topic of Section 6 is the construction of symmetry breaking operators between general principal series from symmetry breaking operators between spherical principal series for multiplicity one pairs. To prove the main statement Theorem 6.5 we employ a variant of the translation principle which we apply case-by-case to all multiplicity one pairs. Finally, Section 7 illustrates symmetry breaking operators between principal series for the pair $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$ by explicit formulas.

Acknowledgements

We thank Yoshiki Oshima for helpful and inspiring conversations on the topic of this paper. We further thank an anonymous referee for pointing out a classification-free proof of Theorem 2.6 using the local structure theorem for real spherical varieties.

Notation

$\mathbb{N}=\{0,1,2,\ldots,\}$, $V^{\vee}=\operatorname{Hom}_{\mathbb{C}}(V,\mathbb{C})$.

1. The structure of strongly spherical reductive pairs

We discuss strongly spherical reductive pairs $(G,H)$ and their structure theory following [16, 20]. First, using results from [16], we reduce the study of general strongly spherical reductive pairs to that of strongly spherical symmetric pairs. For the latter we recall some structure theory as developed in [20, Section 3]. This is used to derive some new results about the double coset space $P_{H}\backslash G/P_{G}$ for $P_{G}\subseteq G$ and $P_{H}\subseteq H$ minimal parabolic subgroups (see Theorem 1.3). These results are used both in Section 3 for the construction of symmetry breaking operators and in Section 4 for their uniqueness.

We remark that for complex groups the double coset space $P_{H}\backslash G/P_{G}$ was studied in [9] with similar techniques.

1.1. Strongly spherical reductive pairs

Consider a real reductive pair $(\mathfrak{g},\mathfrak{h})$ of Lie algebras, i.e. $\mathfrak{g}$ is a reductive Lie algebra and $\mathfrak{h}$ a reductive subalgebra of $\mathfrak{g}$. A pair of Lie groups $(G,H)$ with $H$ a closed subgroup of $G$ is called real reductive if the underlying pair $(\mathfrak{g},\mathfrak{h})$ of Lie algebras is real reductive. In this paper we will additionally assume that $G$ is of Harish-Chandra class (see e.g. [13, Chapter VII.2] for the precise definition).

Definition 1.1**.**

A real reductive pair $(\mathfrak{g},\mathfrak{h})$ of Lie algebras is called strongly spherical if there exist minimal parabolic subalgebras $\mathfrak{p}_{G}\subseteq\mathfrak{g}$ and $\mathfrak{p}_{H}\subseteq\mathfrak{h}$ such that

[TABLE]

A real reductive pair $(G,H)$ of Lie groups is called strongly spherical if the corresponding pair $(\mathfrak{g},\mathfrak{h})$ of Lie algebras is strongly spherical.

This property for a real reductive pair $(G,H)$ was introduced by Kobayashi–Oshima [21] as property (PP). In this paper we use the notion strongly spherical, following [16]. In view of Fact I, strongly spherical reductive pairs are sometimes also referred to as finite-multiplicity pairs (see e.g. [20, 21]). The following characterization of strongly spherical pairs follows e.g. from [21, Lemma 5.3 (1)] and [24, Theorem 1.1]:

Proposition 1.2**.**

Let $(G,H)$ be a real reductive pair of Lie groups and let $P_{G}\subseteq G$ and $P_{H}\subseteq H$ be minimal parabolic subalgebras. Then the following statements are equivalent:

1. (1)

$(G,H)$* is strongly spherical.* 2. (2)

The homogeneous space $(G\times H)/\operatorname{diag}(H)$ is real spherical. 3. (3)

There exists an open double coset in $P_{H}\backslash G/P_{G}$. 4. (4)

$\#(P_{H}\backslash G/P_{G})<\infty$.

A reductive pair $(\mathfrak{g},\mathfrak{h})$ is called non-trivial if $\mathfrak{g}\neq\mathfrak{h}$. We call $(\mathfrak{g},\mathfrak{h})$ indecomposable if there does not exist a non-trivial decomposition $\mathfrak{g}=\mathfrak{g}_{1}\oplus\mathfrak{g}_{2}$ with ideals $\mathfrak{g}_{i}\subseteq\mathfrak{g}$ such that $\mathfrak{h}=(\mathfrak{h}\cap\mathfrak{g}_{1})\oplus(\mathfrak{h}\cap\mathfrak{g}_{2})$. Every reductive pair $(\mathfrak{g},\mathfrak{h})$ of Lie algebras can be written as a direct sum of indecomposable pairs, and we therefore assume that $(\mathfrak{g},\mathfrak{h})$ is indecomposable.

The main result of this section is a statement about non-open double cosets in $P_{H}\backslash G/P_{G}$. Note that for $P_{H}gP_{G}\in P_{H}\backslash G/P_{G}$ the stabilizer of the $P_{H}$-orbit through $gP_{G}\in G/P_{G}$ is given by $P_{H}\cap gP_{G}g^{-1}$. Let $P_{G}=M_{G}A_{G}N_{G}$ and $P_{H}=M_{H}A_{H}N_{H}$ be Langlands decompositions of $P_{G}$ and $P_{H}$ and write $\mathfrak{m}_{G}$, $\mathfrak{a}_{G}$, $\mathfrak{n}_{G}$ and $\mathfrak{m}_{H}$, $\mathfrak{a}_{H}$, $\mathfrak{n}_{H}$ for the Lie algebras of $M_{G}$, $A_{G}$, $N_{G}$ and $M_{H}$, $A_{H}$, $N_{H}$.

Theorem 1.3**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for a double coset $P_{H}gP_{G}\in P_{H}\backslash G/P_{G}$ the following are equivalent:

1. (1)

$P_{H}gP_{G}$* is open in $G$.* 2. (2)

The projection of $\mathfrak{p}_{H}\cap\operatorname{Ad}(g)\mathfrak{p}_{G}$ to $\mathfrak{a}_{H}$ along $\mathfrak{m}_{H}\oplus\mathfrak{n}_{H}$ is trivial.

We first show that Theorem 1.3 can be reduced to the case of semisimple $\mathfrak{g}$. For this write

[TABLE]

where $\mathfrak{g}_{\rm n}$ resp. $\mathfrak{h}_{\rm n}$ is the direct sum of all simple non-compact ideals and $\mathfrak{g}_{\rm el}$ resp. $\mathfrak{h}_{\rm el}$ the sum of all simple compact and all abelian ideals. Denote by $p:\mathfrak{g}\to\mathfrak{g}_{\rm n}$ the projection map onto the $\mathfrak{g}_{\rm n}$ along $\mathfrak{g}_{\rm el}$.

Lemma 1.4**.**

Let $(\mathfrak{g},\mathfrak{h})$ be an indecomposable reductive pair.

1. (1)

$\ker p|_{\mathfrak{h}}$* does not contain any non-compact abelian ideals of $\mathfrak{h}$.* 2. (2)

If $(\mathfrak{g},\mathfrak{h})$ is strongly spherical then $(\mathfrak{g}_{\rm n},p(\mathfrak{h}))$ is strongly spherical.

Proof.

1. (1)

Let $\mathfrak{a}\subseteq\ker p|_{\mathfrak{h}}$ be a non-compact abelian ideal, then $\mathfrak{a}\subseteq\mathfrak{g}_{\rm el}$ and since $\mathfrak{a}$ is non-compact it has to be an ideal in $\mathfrak{g}$. Write $\mathfrak{g}=\mathfrak{a}\oplus\mathfrak{g}^{\prime}$ as sum of ideals, then $\mathfrak{h}=\mathfrak{a}\oplus(\mathfrak{g}^{\prime}\cap\mathfrak{h})$ so that the indecomposability of $(\mathfrak{g},\mathfrak{h})$ forces $\mathfrak{a}=0$ or $(\mathfrak{g},\mathfrak{h})=(\mathfrak{a},\mathfrak{a})$. Since $(\mathfrak{g},\mathfrak{h})$ is assumed to be non-trivial, the latter case cannot occur. 2. (2)

We have $\mathfrak{p}_{G}+\mathfrak{p}_{H}=\mathfrak{g}$ for some choice of minimal parabolic subalgebras $\mathfrak{p}_{G}\subseteq\mathfrak{g}$ and $\mathfrak{p}_{H}\subseteq\mathfrak{h}$. Since $\mathfrak{g}_{\rm el}\subseteq\mathfrak{p}_{G}$ we have $\mathfrak{p}_{G}=(\mathfrak{p}_{G}\cap\mathfrak{g}_{\rm n})\oplus\mathfrak{g}_{\rm el}$ and $\mathfrak{p}_{G}\cap\mathfrak{g}_{\rm n}\subseteq\mathfrak{g}_{\rm n}$ is a minimal parabolic subalgebra. This implies $p(\mathfrak{p}_{H})+(\mathfrak{p}_{G}\cap\mathfrak{g}_{\rm n})=\mathfrak{g}_{\rm n}$. Clearly $p(\mathfrak{p}_{H})$ is a minimal parabolic subalgebra of $p(\mathfrak{h})$ so the claim follows.∎

By the previous lemma, $p|_{\mathfrak{a}_{H}}:\mathfrak{a}_{H}\to\mathfrak{g}_{\rm n}$ is injective, and using $\operatorname{Ad}(g)\mathfrak{g}_{\rm n}=\mathfrak{g}_{\rm n}$ it is easy to see that Theorem 1.3 holds for $(\mathfrak{g},\mathfrak{h})$ if and only if it holds for $(\mathfrak{g}_{\rm n},p(\mathfrak{h}))$. Hence, it suffices to show Theorem 1.3 in the case where $\mathfrak{g}$ is semisimple.

In this section we prove the implication (2)$\Rightarrow$(1) by showing that for every non-open double coset $P_{H}gP_{G}$ there exists $Z=Z_{M}+Z_{A}+Z_{N}\in\mathfrak{p}_{H}\cap\operatorname{Ad}(g)\mathfrak{p}_{G}$ with $Z_{A}\neq 0$. The remaining implication (1)$\Rightarrow$(2) is proved in Section 2.3 (see Corollary 2.8). Note that Theorem 1.3 is not used in Section 2.

1.2. Classification of strongly spherical reductive pairs

To state an efficient classification of strongly spherical pairs, we need an additional assumption to avoid exotic situations as in the following example:

Example 1.5**.**

As observed in [16], there exist indecomposable strongly spherical pairs $(\mathfrak{g},\mathfrak{h})$ such that $\mathfrak{g}$ has arbitrarily many non-compact simple factors. For instance we have the indecomposable strongly spherical reductive pair

[TABLE]

where each of the $k$ factors $\operatorname{\mathfrak{sp}}(p,q)$ of $\mathfrak{h}$ is embedded into the corresponding factor $\operatorname{\mathfrak{sp}}(p,q+1)$ of $\mathfrak{g}$ in the standard way, and $\operatorname{\mathfrak{sp}}(1)$ is embedded diagonally into $\mathfrak{g}$ as the centralizer of $\operatorname{\mathfrak{sp}}(p,q)$ in each factor $\operatorname{\mathfrak{sp}}(p,q+1)$.

Following [16] we call a pair $(\mathfrak{g},\mathfrak{h})$ strictly indecomposable if both $(\mathfrak{g},\mathfrak{h})$ and $(\mathfrak{g}_{\rm n},\mathfrak{h}_{\rm n})$ are indecomposable. Note that $\mathfrak{h}_{\rm n}\subseteq\mathfrak{g}_{\rm n}$ is automatic from the definition of $\mathfrak{g}_{\rm n}$ and $\mathfrak{h}_{\rm n}$. This definition excludes exotic reductive pairs as in Example 1.5 but still allows certain central extensions of $\mathfrak{g}$ and $\mathfrak{h}$ as for the multiplicity one pairs $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{gl}}(n+1,\mathbb{F}),\operatorname{\mathfrak{gl}}(n,\mathbb{F}))$, $\mathbb{F}=\mathbb{R},\mathbb{C}$, and $(\mathfrak{u}(p,q+1),\mathfrak{u}(p,q))$.

By Lemma 1.4, the study of strictly indecomposable reductive pairs $(\mathfrak{g},\mathfrak{h})$ can be reduced to the case where $\mathfrak{g}$ is semisimple. In this case, a classification was obtained by Kobayashi–Matsuki [20] for symmetric pairs and by Knop–Krötz–Pecher–Schlichtkrull [15, 16] for arbitrary reductive pairs:

Theorem 1.6** (see [15, 16, 20]).**

Let $(\mathfrak{g},\mathfrak{h})$ be a strictly indecomposable reductive pair with $\mathfrak{g}$ semisimple. Then $(\mathfrak{g},\mathfrak{h})$ is strongly spherical if and only if it is isomorphic to one of the following pairs:

1. A)

Trivial case: $\mathfrak{g}=\mathfrak{h}$. 2. C)

Compact case: $\mathfrak{g}$ is the Lie algebra of a compact simple Lie group. 3. D)

Compact subgroup case: $\mathfrak{h}=\mathfrak{k}$ is the Lie algebra of a maximal compact subgroup $K$ of a non-compact simple Lie group $G$ with Lie algebra $\mathfrak{g}$, or $(\mathfrak{g},\mathfrak{h})$ is one of the following subpairs of $(\mathfrak{g},\mathfrak{k})$:

• •

$(\operatorname{\mathfrak{so}}(1,2n),\operatorname{\mathfrak{su}}(n)+\mathfrak{f})$* ($n\geq 2$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$.*

• •

$(\operatorname{\mathfrak{so}}(1,4n),\operatorname{\mathfrak{sp}}(n)+\mathfrak{f})$* ($n\geq 1$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{sp}}(1)$.*

• •

$(\operatorname{\mathfrak{so}}(1,16),\operatorname{\mathfrak{spin}}(9))$.

• •

$(\operatorname{\mathfrak{so}}(p,7),\operatorname{\mathfrak{so}}(p)+\mathfrak{g}_{2})$* ($p=1,2$).*

• •

$(\operatorname{\mathfrak{so}}(p,8),\operatorname{\mathfrak{so}}(p)+\operatorname{\mathfrak{spin}}(7))$* ($p=1,2,3$).*

• •

$(\operatorname{\mathfrak{su}}(1,2n),\operatorname{\mathfrak{sp}}(n)+\mathfrak{f})$* ($n\geq 1$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$.*

• •

$(\operatorname{\mathfrak{sp}}(1,n),\operatorname{\mathfrak{sp}}(n)+\mathfrak{f})$* ($n\geq 1$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{sp}}(1)$.*

• •

$(\operatorname{\mathfrak{su}}(p,q),\operatorname{\mathfrak{su}}(p)+\operatorname{\mathfrak{su}}(q))$* ($p,q\geq 1$, $p\neq q$).*

• •

$(\operatorname{\mathfrak{so}}^{*}(2n),\operatorname{\mathfrak{su}}(n))$* ($n\geq 3$ odd).*

• •

$(\mathfrak{e}_{6(-14)},\operatorname{\mathfrak{so}}(10))$. 4. E)

Split rank one case ($\operatorname{rank}_{\mathbb{R}}\mathfrak{g}=1$):

1. E1)

$(\operatorname{\mathfrak{so}}(1,p+q),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{so}}(q))$* ($p,q\geq 1$) or one of the following subpairs:*

• •

$(\operatorname{\mathfrak{so}}(1,p+2q),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{su}}(q)+\mathfrak{f})$* ($p\geq 1,q\geq 2$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$.*

• •

$(\operatorname{\mathfrak{so}}(1,p+4q),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{sp}}(q)+\mathfrak{f})$* ($p\geq 1,q\geq 2$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{sp}}(1)$.*

• •

$(\operatorname{\mathfrak{so}}(1,p+7),\operatorname{\mathfrak{so}}(1,p)+\mathfrak{g}_{2})$* ($p\geq 0$).*

• •

$(\operatorname{\mathfrak{so}}(1,p+8),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{spin}}(7))$* ($p\geq 0$).*

• •

$(\operatorname{\mathfrak{so}}(1,p+16),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{spin}}(9))$* ($p\geq 0$).* 2. E2)

$(\operatorname{\mathfrak{su}}(1,p+q),\operatorname{\mathfrak{su}}(1,p)+\operatorname{\mathfrak{su}}(q)+\mathfrak{u}(1))$* ($p,q\geq 1$) or one of the following subpairs:*

• •

$(\operatorname{\mathfrak{su}}(1,p+q),\operatorname{\mathfrak{su}}(1,p)+\operatorname{\mathfrak{su}}(q))$* ($p,q\geq 1$, $p+q\geq 3$).*

• •

$(\operatorname{\mathfrak{su}}(1,p+2q),\operatorname{\mathfrak{su}}(1,p)+\operatorname{\mathfrak{sp}}(q)+\mathfrak{f})$* ($p,q\geq 1$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$.* 3. E3)

$(\operatorname{\mathfrak{sp}}(1,p+q),\operatorname{\mathfrak{sp}}(1,p)+\operatorname{\mathfrak{sp}}(q))$* ($p,q\geq 1$) or the following subpair:*

• •

$(\operatorname{\mathfrak{sp}}(1,p+1),\operatorname{\mathfrak{sp}}(1,p))$* ($p\geq 1$).* 4. E4)

$(\mathfrak{f}_{4(-20)},\operatorname{\mathfrak{so}}(8,1))$. 5. F)

Strong Gelfand pairs and their real forms:

1. F1)

$(\operatorname{\mathfrak{sl}}(n+1,\mathbb{C}),\operatorname{\mathfrak{gl}}(n,\mathbb{C}))$* ($n\geq 2$).* 2. F2)

$(\operatorname{\mathfrak{so}}(n+1,\mathbb{C}),\operatorname{\mathfrak{so}}(n,\mathbb{C}))$* ($n\geq 2$).* 3. F3)

$(\operatorname{\mathfrak{sl}}(n+1,\mathbb{R}),\operatorname{\mathfrak{gl}}(n,\mathbb{R}))$* ($n\geq 1$).* 4. F4)

$(\operatorname{\mathfrak{su}}(p,q+1),\operatorname{\mathfrak{su}}(p,q)+\mathfrak{f})$* ($p,q+1\geq 1$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$ and $\mathfrak{f}=\mathfrak{u}(1)$ for $p=q,q+1$.* 5. F5)

$(\operatorname{\mathfrak{so}}(p,q+1),\operatorname{\mathfrak{so}}(p,q))$* ($p+q\geq 2$).* 6. G)

Group case: $(\mathfrak{g},\mathfrak{h})=(\mathfrak{g}^{\prime}+\mathfrak{g}^{\prime},\operatorname{diag}\mathfrak{g}^{\prime})$

1. G1)

$\mathfrak{g}^{\prime}$* is the Lie algebra of a compact simple Lie group.* 2. G2)

$\mathfrak{g}^{\prime}=\operatorname{\mathfrak{so}}(1,n)$* ($n\geq 2$).* 7. H)

Other cases:

1. H1)

$(\operatorname{\mathfrak{so}}(2,2n),\operatorname{\mathfrak{su}}(1,n)+\mathfrak{f})$* ($n\geq 1$) with $\mathfrak{f}\subseteq\mathfrak{u}(1)$.* 2. H2)

$(\operatorname{\mathfrak{su}}^{*}(2n+2),\operatorname{\mathfrak{su}}^{*}(2n)+\mathbb{R}+\mathfrak{f})$* ($n\geq 2$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{su}}(2)$.* 3. H3)

$(\operatorname{\mathfrak{so}}^{*}(2n+2),\operatorname{\mathfrak{so}}^{*}(2n)+\mathfrak{f})$* ($n\geq 1$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{so}}(2)$.* 4. H4)

$(\operatorname{\mathfrak{sp}}(p,q+1),\operatorname{\mathfrak{sp}}(p,q)+\mathfrak{f})$* ($p,q+1\geq 1$) with $\mathfrak{f}\subseteq\operatorname{\mathfrak{sp}}(1)$.* 5. H5)

$(\mathfrak{e}_{6(-26)},\operatorname{\mathfrak{so}}(9,1)+\mathbb{R})$.

The list and its enumeration is a copy of the list in [20, Theorem 1.3] to which we added the non-symmetric cases obtained in [16, Table 9] and the cases with $\mathfrak{h}$ compact which are listed in [15]. This is the reason why B) is missing since it was listed in [20] as the abelian case $(\mathfrak{g},\mathfrak{h})=(\mathbb{R},0)$, which we exclude by assuming that $\mathfrak{g}$ is semisimple.

It is immediate from the classification that every non-trivial strictly indecomposable strongly spherical reductive pair $(\mathfrak{g},\mathfrak{h})$ with $\mathfrak{g}$ semisimple is contained inside a non-trivial symmetric pair $(\mathfrak{g},\mathfrak{g}^{\sigma})$ (see also [15, Lemma 1.4]). This statement is still true if one replaces strictly indecomposable by indecomposable:

Corollary 1.7**.**

Let $(\mathfrak{g},\mathfrak{h})$ be a non-trivial indecomposable strongly spherical reductive pair with $\mathfrak{g}$ semisimple. Then there exists a non-trivial involution $\sigma$ of $\mathfrak{g}$ such that $\mathfrak{h}\subseteq\mathfrak{g}^{\sigma}$, $\mathfrak{h}_{\rm n}=(\mathfrak{g}^{\sigma})_{\rm n}$, and $\mathfrak{h}_{\rm el}$ and $(\mathfrak{g}^{\sigma})_{\rm el}$ only differ in compact factors.

Proof.

Write $(\mathfrak{g},\mathfrak{h}_{\rm n})=(\mathfrak{g}_{1},\mathfrak{h}_{1})\oplus\cdots(\mathfrak{g}_{p},\mathfrak{h}_{p})$ with each $(\mathfrak{g}_{j},\mathfrak{h}_{j})$ indecomposable. Let $p_{j}:\mathfrak{g}\to\mathfrak{g}_{j}$ denote the projection onto $\mathfrak{g}_{j}$ in the decomposition $\mathfrak{g}=\mathfrak{g}_{1}\oplus\cdots\oplus\mathfrak{g}_{p}$. Then each pair $(\mathfrak{g}_{j},\mathfrak{h}_{j}\oplus p_{j}(\mathfrak{h}_{\rm el})))$ is strongly spherical, strictly indecomposable and non-trivial, so by the classification in Theorem 1.6 there exists an involution $\sigma_{j}$ on $\mathfrak{g}_{j}$ such that $\mathfrak{h}_{j}\subseteq\mathfrak{g}_{j}^{\sigma_{j}}\subseteq\mathfrak{g}_{j}$, $(\mathfrak{h}_{j})_{\rm n}=(\mathfrak{g}_{j}^{\sigma_{j}})_{\rm n}$, and $p_{j}(\mathfrak{h}_{\rm el})$ and $(\mathfrak{g}_{j}^{\sigma_{j}})_{\rm el}$ only differ in compact factors. Define $\sigma$ on $\mathfrak{g}=\mathfrak{g}_{1}\oplus\cdots\oplus\mathfrak{g}_{p}$ by

[TABLE]

then clearly $\mathfrak{h}\subseteq\mathfrak{g}^{\sigma}\subseteq\mathfrak{g}$ and $\mathfrak{h}_{\rm n}=(\mathfrak{h}_{1})_{\rm n}\oplus\cdots\oplus(\mathfrak{h}_{p})_{\rm n}=(\mathfrak{g}_{1}^{\sigma_{1}})_{\rm n}\oplus\cdots\oplus(\mathfrak{g}_{p}^{\sigma_{p}})_{\rm n}=(\mathfrak{g}^{\sigma})_{\rm n}$. It remains to show that $(\mathfrak{g}^{\sigma})_{\rm el}=(\mathfrak{g}_{1}^{\sigma_{1}})_{\rm el}\oplus\cdots\oplus(\mathfrak{g}_{p}^{\sigma_{p}})_{\rm el}$ and $\mathfrak{h}_{\rm el}$ only differ in compact factors. Since for every $j$ the subalgebras $p_{j}(\mathfrak{h}_{\rm el})$ and $(\mathfrak{g}_{j}^{\sigma_{j}})_{\rm el}$ only differ in compact factors, it suffices to show that a non-compact abelian ideal $\mathfrak{a}\subseteq\mathfrak{h}_{\rm el}$ is already contained in one of the subalgebras $\mathfrak{g}_{j}^{\sigma_{j}}$. Let $\mathfrak{h}_{\rm el}=\mathfrak{a}\oplus\mathfrak{h}_{\rm el}^{\prime}$ with $\mathfrak{a}$ a non-compact abelian ideal and assume that $p_{j}(\mathfrak{a})\neq\{0\}$ for some $j$. Since $p_{j}(\mathfrak{a})$ is an ideal in $\mathfrak{h}_{j}\oplus p_{j}(\mathfrak{h}_{\rm el})$ and the pair $(\mathfrak{g}_{j},\mathfrak{h}_{j}\oplus p_{j}(\mathfrak{h}_{\rm el}))$ is in the list in Theorem 1.6, it follows from a case-by-case inspection that $p_{j}(\mathfrak{a})$ is one-dimensional and that $(\mathfrak{g}_{j},\mathfrak{h}_{j}\oplus p_{j}(\mathfrak{h}_{\rm el}^{\prime}))$ is not strongly spherical. Assume now that $p_{k}(\mathfrak{a})\neq\{0\}$ for some $k\neq j$; then by the same argument $p_{k}(\mathfrak{a})$ is one-dimensional and $(\mathfrak{g}_{k},\mathfrak{h}_{k}\oplus p_{k}(\mathfrak{h}_{\rm el}^{\prime}))$ is not strongly spherical. This contradicts the fact that $(\mathfrak{g},\mathfrak{h})$ is strongly spherical by a simple dimension count. Hence, $\mathfrak{a}\subseteq\mathfrak{g}_{j}$, and another look at the classification shows that $\mathfrak{a}\subseteq\mathfrak{g}_{j}^{\sigma_{j}}$. This finishes the proof. ∎

It is clear that in this case $(\mathfrak{g},\mathfrak{g}^{\sigma})$ is a strongly spherical symmetric pair. This observation will be used to reduce several statements to the case of symmetric pairs.

1.3. Structure of strongly spherical reductive pairs

We adapt the structure theory developed in [20] for strongly spherical symmetric pairs to the case of strongly spherical reductive pairs. For the rest of this section let $(G,H)$ be a strongly spherical reductive pair with $\mathfrak{g}$ semisimple such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. By Corollary 1.7 there exists an involution $\sigma$ of $G$ such that $\mathfrak{h}\subseteq\mathfrak{g}^{\sigma}$, $\mathfrak{h}_{\rm n}=(\mathfrak{g}^{\sigma})_{\rm n}$, and $\mathfrak{h}_{\rm el}$ and $(\mathfrak{g}^{\sigma})_{\rm el}$ differ only in compact factors. We first choose minimal parabolic subgroups $P_{G}\subseteq G$ and $P_{H}\subseteq H$ in a compatible way.

There exists a Cartan involution $\theta$ of $G$ which commutes with $\sigma$ and leaves $H$ invariant, and hence

[TABLE]

are maximal compact subgroups of $G$ and $H$. Fix a maximal abelian subspace $\mathfrak{a}_{H}\subseteq\mathfrak{h}^{-\theta}=\mathfrak{g}^{\sigma,-\theta}$ and extend it to a maximal abelian subspace $\mathfrak{a}_{G}$ in $\mathfrak{g}^{-\theta}$. Then $\mathfrak{a}_{G}$ is $\sigma$-stable and $\mathfrak{a}_{G}=\mathfrak{a}_{H}\oplus\mathfrak{a}_{G}^{-\sigma}$ with $\mathfrak{a}_{H}=\mathfrak{a}_{G}^{\sigma}$. We put

[TABLE]

For $\alpha\in\mathfrak{a}_{G}^{\vee}$ and $\beta\in\mathfrak{a}_{H}^{\vee}$ we write

[TABLE]

for the corresponding weight spaces. Let $\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ and $\Sigma(\mathfrak{g},\mathfrak{a}_{H})$ denote the respective non-zero weights with non-trivial weight spaces; then both sets form root systems. Denote by $\overline{\alpha}=\alpha|_{\mathfrak{a}_{H}}$ the restriction of a root $\alpha\in\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ to $\mathfrak{a}_{H}$, then $\overline{\alpha}\in\Sigma(\mathfrak{g},\mathfrak{a}_{H})\cup\{0\}$. We choose compatible positive systems $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ and $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{H})$ in the sense that

[TABLE]

As usual, for $\alpha\in\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ we write $\alpha>0$ if $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ and $\alpha<0$ if $-\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

Further, define the nilpotent subalgebras

[TABLE]

Then $\mathfrak{n}$ is $\sigma$-stable and therefore we have a direct sum decomposition

[TABLE]

Put $\mathfrak{n}_{H}=\mathfrak{n}^{\sigma}$ and

[TABLE]

Finally, we define

[TABLE]

Then $P_{G}=M_{G}A_{G}N_{G}$ is a minimal parabolic subgroup of $G$, $Q=LN$ is another parabolic subgroup of $G$, and $P_{H}=M_{H}A_{H}N_{H}$ is a minimal parabolic subgroup of $H$ such that

[TABLE]

1.4. The double coset space $P_{H}\backslash G/P_{G}$

To study the $P_{H}$-orbits in $G/P_{G}$ we use the Bruhat decomposition of $G$ with respect to the parabolic subgroups $P_{G}$ and $Q$. Let $W=W(\mathfrak{a}_{G})=N_{K}(A_{G})/A_{G}$ denote the Weyl group of $\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ and pick a representative $\tilde{w}\in N_{K}(A_{G})$ for every $w\in W$. Then, since $G$ is of Harish-Chandra class, we have the Bruhat decomposition (see e.g. [46, Proposition 1.2.1.10])

[TABLE]

where

[TABLE]

Since $P_{H}\subseteq Q$, every $P_{H}$-orbit $P_{H}\cdot gP_{G}$ is contained in a Bruhat cell $Q\cdot\tilde{w}P_{G}\subseteq G/P$. As homogeneous spaces we have

[TABLE]

whence we are led to study the $P_{H}$-orbits in $Q/(Q\cap\tilde{w}P_{G}\tilde{w}^{-1})$ for $w\in W$. The following result is shown in [20, Lemma 3.5 (2)] for strongly spherical symmetric pairs, and we extend it to the context of strongly spherical reductive pairs:

Lemma 1.8**.**

The extension of $\mathfrak{a}_{H}$ to $\mathfrak{a}_{G}$ can be chosen such that

[TABLE]

Proof.

Let $P_{G^{\sigma}}=Z_{K^{\sigma}}(\mathfrak{a}_{H})A_{H}N_{H}$, a minimal parabolic subgroup of $G^{\sigma}$. Since $(G,H)$ is strongly spherical, there exists $g\in G$ such that $P_{H}gP_{G}$ is open. Now $P_{H}\subseteq P_{G^{\sigma}}$ and hence $P_{H}gP_{G}$ is contained in an open double coset in $P_{G^{\sigma}}\backslash G/P_{G}$. By [20, Lemma 3.7] such open double cosets have representatives in $\exp(\mathfrak{n}^{-\sigma})\widetilde{w}_{0}$ so that $P_{H}gP_{G}\subseteq P_{G^{\sigma}}\exp(X)\widetilde{w}_{0}P_{G}$ for some $X\in\mathfrak{n}^{-\sigma}$. Hence, $g\in p\exp(X)\widetilde{w}_{0}P_{G}$ with $p\in P_{G^{\sigma}}$ so that

[TABLE]

Now $P_{H}p=P_{H}m$ with $m\in Z_{K^{\sigma}}(\mathfrak{a}_{H})$ and $(m^{-1}P_{H}m)\exp(X)\widetilde{w}_{0}P_{G}\subseteq G$ is open. This implies

[TABLE]

where $\overline{\mathfrak{p}}_{G}=\operatorname{Ad}(\widetilde{w}_{0})\mathfrak{p}_{G}$ denotes the opposite parabolic subalgebra. Note that for $Z=Z_{M}+Z_{A}+Z_{N}\in\operatorname{Ad}(m)^{-1}\mathfrak{m}_{H}+\mathfrak{a}_{H}+\mathfrak{n}_{H}=\operatorname{Ad}(m^{-1})\mathfrak{p}_{H}$ we have

[TABLE]

and further

[TABLE]

From the decomposition $\mathfrak{g}=\overline{\mathfrak{n}}\oplus\mathfrak{l}\oplus\mathfrak{n}$ it now follows that $\operatorname{Ad}(m^{-1})\mathfrak{m}_{H}+(\mathfrak{l}\cap\overline{\mathfrak{p}}_{G})=\mathfrak{l}$. Intersecting with $\mathfrak{k}$ we obtain $\operatorname{Ad}(m^{-1})\mathfrak{m}_{H}+\mathfrak{m}_{G}=\mathfrak{l}\cap\mathfrak{k}$, or equivalently $\mathfrak{m}_{H}+\operatorname{Ad}(m)\mathfrak{m}_{G}=\mathfrak{l}\cap\mathfrak{k}$. Replacing $\mathfrak{a}_{G}$ by $\operatorname{Ad}(m)\mathfrak{a}_{G}$ changes $\mathfrak{m}_{G}$ to $\operatorname{Ad}(m)\mathfrak{m}_{G}$ and hence we may assume $\mathfrak{m}_{H}+\mathfrak{m}_{G}=\mathfrak{l}\cap\mathfrak{k}$. The identity $L\cap K=M_{H}M_{G}$ then follows since $M_{G}A_{G}\subseteq L$ is the Levi factor of a minimal parabolic subgroup of $L$ and hence $M_{G}$ meets every connected component of $L$. ∎

For the rest of this section we choose the extension $\mathfrak{a}_{G}$ of $\mathfrak{a}_{H}$ as in Lemma 1.8. Then we obtain the following generalization of [20, Lemma 3.7] to not necessarily symmetric $(G,H)$:

Lemma 1.9**.**

For every $w\in W$ we have

[TABLE]

In particular, every $P_{H}$-orbit in $Q/(Q\cap\tilde{w}P_{G}\tilde{w}^{-1})$ has a representative of the form $\exp(X)$ for some $X\in\mathfrak{n}^{-\sigma}$, or equivalently every $P_{H}$-orbit in $G/P_{G}$ has a representative of the form $\exp(X)\tilde{w}$ for some $X\in\mathfrak{n}^{-\sigma}$, $w\in W$.

Proof.

First note that $\tilde{w}P_{G}\tilde{w}^{-1}$ is a minimal parabolic subgroup of $G$. Since $L$ is a reductive subgroup of $G$, the intersection $L\cap\tilde{w}P_{G}\tilde{w}^{-1}$ is parabolic in $L$. By the Iwasawa decomposition for $L$ we find that $L=(L\cap K)(L\cap\tilde{w}P_{G}\tilde{w}^{-1})$. Now, $L\cap K=M_{H}M_{G}$ by Lemma 1.8, whence

[TABLE]

Inserting this into the Langlands decomposition for $Q$ we find

[TABLE]

Finally, $N=N_{H}\exp(\mathfrak{n}^{-\sigma})$ by [20, Lemma 3.6] and the proof is complete. ∎

1.5. $P_{H}$-orbits in the open Bruhat cell

Let $w_{0}\in W$ denote the longest Weyl group element. Then the Bruhat cell $Q\tilde{w}_{0}P_{G}$ is open and dense in $G$. Hence, $Q\tilde{w}_{0}P_{G}$ is the unique open Bruhat cell.

Now let us consider the double cosets $P_{H}gP_{G}$ which are contained in the open Bruhat cell $Q\tilde{w}_{0}P_{H}$. These can be identified with the orbits of the adjoint action of $(M_{G}\cap M_{H})A_{H}$ on $\mathfrak{n}^{-\sigma}$:

Lemma 1.10**.**

The natural inclusion

[TABLE]

induces a bijection

[TABLE]

Proof.

The proof of [20, Lemma 3.7] for the case of symmetric pairs translates literally to our situation. ∎

Since $\mathfrak{a}_{H}$ preserves $\mathfrak{n}^{\pm\sigma}$ we can write

[TABLE]

where $\mathfrak{g}^{\pm\sigma}(\mathfrak{a}_{H};\beta)=\mathfrak{g}(\mathfrak{a}_{H};\beta)\cap\mathfrak{g}^{\pm\sigma}$ and

[TABLE]

Clearly $M_{H}$ preserves this decomposition. We can therefore endow $\mathfrak{n}^{-\sigma}$ with an $M_{H}$-invariant inner product such that the decomposition (1.2) is orthogonal. For each $\beta\in\Delta(\mathfrak{n}^{-\sigma})$, denote by $S_{\beta}\subseteq\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)$ the unit sphere with respect to this inner product. We then have the following version of [20, Proposition 3.11]:

Lemma 1.11**.**

If the pair $(G,H)$ is strongly spherical, then

1. (1)

the orbits of $(M_{G}\cap M_{H})$ on $\prod_{\beta\in\Delta(\mathfrak{n}^{-\sigma})}S_{\beta}$ are unions of connected components, in particular open and compact; 2. (2)

$\Delta(\mathfrak{n}^{-\sigma})$* is a basis of $\mathfrak{a}_{H}^{\vee}$.*

Proof.

The proof of [20, Proposition 3.11] implies (1) and linear independence of $\Delta(\mathfrak{n}^{-\sigma})$ in (2). Assume that $\Delta(\mathfrak{n}^{-\sigma})$ does not span $\mathfrak{a}_{H}^{\vee}$; then there exists $0\neq H\in\mathfrak{a}_{H}$ such that $\beta(H)=0$ for all $\beta\in\Delta(\mathfrak{n}^{-\sigma})$, or equivalently $\operatorname{ad}(H)|_{\mathfrak{n}^{-\sigma}}=0$. But this implies $H\in\mathfrak{p}_{H}\cap\operatorname{Ad}(e^{X}\tilde{w}_{0})\mathfrak{p}_{G}=\mathfrak{p}_{H}\cap\operatorname{Ad}(e^{X})\overline{\mathfrak{p}}_{G}$ for all $X\in\mathfrak{n}^{-\sigma}$. Since by Lemma 1.9 every open double coset in $P_{H}gP_{G}\in P_{H}\backslash G/P_{G}$ has a representative of the form $g=e^{X}\tilde{w}_{0}$, this shows that the projection of $\mathfrak{p}_{H}\cap\operatorname{Ad}(g)\mathfrak{p}_{G}$ to $\mathfrak{a}_{H}$ is non-trivial, which contradicts Theorem 1.3. ∎

1.6. Proof of Theorem 1.3

Let $P_{H}\cdot gP_{G}$ be a non-open $P_{H}$-orbit. By Lemma 1.9 there exists $p\in P_{H}$ such that $pgP_{G}=\exp(X)\tilde{w}P_{G}$ for some $X\in\mathfrak{n}^{-\sigma}$ and $w\in W$. Then the stabilizers of $gP_{G}$ and $\exp(X)\tilde{w}P_{G}$ are conjugate in $P_{H}$ via $p$. Since the projection of $\operatorname{Ad}(p)\mathfrak{a}_{H}\subseteq\mathfrak{p}_{H}$ to $\mathfrak{a}_{H}$ is equal to $\mathfrak{a}_{H}$, we may without loss of generality assume that $g=\exp(X)\tilde{w}$.

Write

[TABLE]

for its Lie algebra. Note that the Lie algebra $w\cdot\mathfrak{p}_{G}$ of $\tilde{w}P_{G}\tilde{w}^{-1}$ is given by

[TABLE]

In view of the decomposition (1.2) we can write

[TABLE]

with $X_{\beta}\in\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)$.

1.6.1. Reduction to generic $X$

Assume first that $X_{\beta}=0$ for some $\beta\in\Delta(\mathfrak{n}^{-\sigma})$. It follows from Lemma 1.11 (2) that the intersection

[TABLE]

is non-trivial, so there exists $Z_{A}\in\mathfrak{a}_{H}\setminus\{0\}$ with $\beta^{\prime}(Z_{A})=0$ for all $\beta^{\prime}\neq\beta$. Then $[Z_{A},X]=0$ since $X_{\beta}=0$ and $[Z_{A},X_{\beta^{\prime}}]=0$ for $\beta^{\prime}\neq\beta$, and hence $e^{tZ_{A}}e^{X}=e^{X}e^{tZ_{A}}$ for all $t\in\mathbb{R}$. Since $Z_{A}\in\mathfrak{a}_{H}\subseteq w\cdot\mathfrak{p}_{G}$ this implies $Z_{A}\in\mathfrak{s}(X,\tilde{w})$.

We may therefore assume $X_{\beta}\neq 0$ for all $\beta\in\Delta(\mathfrak{n}^{-\sigma})$ for the rest of the proof. By Lemma 1.11 this guarantees that the $(M_{G}\cap M_{H})A_{H}$-orbit of $X$ in $\mathfrak{n}^{-\sigma}$ is open.

1.6.2. Double cosets in the open Bruhat cell

Now suppose $Q\tilde{w}P_{G}=Q\tilde{w}_{0}P_{G}$, i.e. the double coset $P_{H}\exp(X)\tilde{w}P_{G}$ is contained in the open Bruhat cell $Q\tilde{w}_{0}P_{G}$. Without loss of generality we may assume $w=w_{0}$. If the double coset $P_{H}\exp(X)\tilde{w}_{0}P_{G}$ is not open, then by Lemma 1.10 the $(M_{G}\cap M_{H})A_{H}$-orbit of $X$ in $\mathfrak{n}^{-\sigma}$ is not open. By Lemma 1.11 this implies that one of the $X_{\beta}$ must vanish, the case we already treated in Section 1.6.1.

1.6.3. Double cosets in the non-open Bruhat cells

Now assume that $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$.

Lemma 1.12**.**

Let $w\in W$. If $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$, then there exists $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$.

Proof.

Assume $w^{-1}\alpha<0$ for all $\alpha>0$ with $\overline{\alpha}\neq 0$. The double coset $Q\tilde{w}P_{G}$ is the orbit of $\tilde{w}$ under the action of $Q\times P_{G}$ on $G$ given by $(q,p)\cdot g=qgp^{-1}$. Then the stabilizer of $\tilde{w}$ in $Q\times P_{G}$ is given by

[TABLE]

and hence its Lie algebra is isomorphic to $\mathfrak{q}\cap w\cdot\mathfrak{p}_{G}$. Since $\mathfrak{a}_{G}\subseteq\mathfrak{q}\cap w\cdot\mathfrak{p}_{G}$ we can decompose this Lie algebra into root spaces with respect to $\mathfrak{a}_{G}$:

[TABLE]

Now, if $\overline{\alpha}=0$ then either $w^{-1}\alpha>0$ or $w^{-1}(-\alpha)=-w^{-1}\alpha>0$, and therefore

[TABLE]

Hence,

[TABLE]

so that $Q\tilde{w}P_{G}$ must be open, whence equal to the unique open Bruhat cell $Q\tilde{w}_{0}P_{G}$. This contradicts the assumption $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$ and the proof is complete. ∎

Choose any root $\alpha>0$ with $\overline{\alpha}\neq 0$ and $w^{-1}\alpha>0$, then $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{q}\cap w\cdot\mathfrak{p}_{G}$. Let us fix $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ for now; later we will specify an appropriate choice of $Y$. Then $e^{tY}\in N$ and also $e^{X}e^{tY}\in N$ for all $t\in\mathbb{R}$. By [20, Lemma 3.6] we have $N=N^{\sigma}\exp(\mathfrak{n}^{-\sigma})$ so that we can uniquely write

[TABLE]

with $n_{t}\in N^{\sigma}$ and $X_{t}\in\mathfrak{n}^{-\sigma}$ depending differentiably on $t\in\mathbb{R}$. Recall that we may assume $X_{\beta}\neq 0$ for all $\beta\in\Delta(\mathfrak{n}^{-\sigma})$, so that $X$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit in $\mathfrak{n}^{-\sigma}$. Then there exists an interval $(-\varepsilon,\varepsilon)$ such that $X_{t}$ belongs to the open orbit $\operatorname{Ad}((M_{G}\cap M_{H})A_{H})X$, so there exists $m_{t}a_{t}\in(M_{G}\cap M_{H})A_{H}$ such that $\operatorname{Ad}(m_{t}a_{t})X=X_{t}$. Clearly, $m_{t}a_{t}$ can be chosen to depend differentiably on $t$ with $m_{0}=a_{0}={\rm\bf 1}$. Summarizing, we have

[TABLE]

Denoting $p_{t}=n_{t}a_{t}m_{t}\in N_{H}A_{H}(M_{G}\cap M_{H})\subseteq P_{G}\cap P_{H}$ we have

[TABLE]

whence $p_{t}\in S(X,\tilde{w})$. Now put

[TABLE]

and write $Z=Z_{M}+Z_{A}+Z_{N}\in(\mathfrak{m}_{G}\cap\mathfrak{m}_{H})\oplus\mathfrak{a}_{H}\oplus\mathfrak{n}_{H}$. It remains to show that $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ can be chosen such that $Z_{A}\neq 0$. For this we use the following identity which follows by taking the left logarithmic derivative of (1.3):

[TABLE]

Now, write

[TABLE]

with $Z_{N,\beta}\in\mathfrak{g}^{\sigma}(\mathfrak{a}_{H};\beta)$.

Lemma 1.13**.**

If $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ for $\alpha>0$ with $\beta=\overline{\alpha}\neq 0$ and $w^{-1}\alpha>0$, and $Z=Z_{M}+Z_{A}+Z_{N}\in(\mathfrak{m}_{G}\cap\mathfrak{m}_{H})\oplus\mathfrak{a}_{H}\oplus\mathfrak{n}_{H}$ satisfies (1.4), then

[TABLE]

Proof.

We can enumerate the positive $\mathfrak{a}_{H}$-roots as $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{H})=\{\beta_{1},\ldots,\beta_{p}\}$ so that $\beta_{j}\not<\beta_{i}$ whenever $i\leq j$. Form the nilpotent subalgebras

[TABLE]

then $\mathfrak{n}_{1}=\mathfrak{n}$ and $[\mathfrak{n},\mathfrak{n}_{i}]\subseteq\mathfrak{n}_{i+1}$. Since $\alpha>0$ with $\overline{\alpha}\neq 0$ there exists $1\leq i\leq p$ with $\overline{\alpha}=\beta_{i}$. We first prove by induction that

[TABLE]

For $j=1<p$, the $\beta_{1}$-component of $Y$ is trivial, and therefore, taking the $\beta_{1}$-component of (1.4) yields

[TABLE]

Since $\operatorname{ad}(Z_{M}+Z_{A})X_{\beta_{1}}\in\mathfrak{n}^{-\sigma}$ and $Z_{N,\beta_{1}}\in\mathfrak{n}^{\sigma}$, this implies $\operatorname{ad}(Z_{M}+Z_{A})X_{\beta_{1}}=0=Z_{N,\beta_{1}}$. For the induction step assume that $\operatorname{ad}(Z_{M}+Z_{A})X_{\beta_{k}}=0=Z_{N,\beta_{k}}$ for $1\leq k\leq j-1$. If $j<i$ then the $\beta_{j}$-component of $Y$ is trivial, and we can again take the $\beta_{j}$-component of (1.4) and find

[TABLE]

The same argument as above shows $\operatorname{ad}(Z_{M}+Z_{A})X_{\beta_{j}}=0=Z_{N,\beta_{j}}$. Finally, taking the $\beta_{i}$-component in (1.4) gives the desired identity. ∎

To choose $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ such that $Z_{A}\neq 0$, we need to relate $\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ and $\mathfrak{g}(\mathfrak{a}_{H};\beta)$.

Lemma 1.14**.**

Let $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$. Then precisely one of the following three statements holds:

1. (1)

$\sigma\alpha\neq\alpha$* and $\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\overline{\alpha})=\{Y-\sigma Y:Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)\}\neq\{0\}$,* 2. (2)

$\sigma\alpha=\alpha$* and $\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\overline{\alpha})=\mathfrak{g}(\mathfrak{a}_{G};\alpha)\cap\mathfrak{n}^{-\sigma}\neq\{0\}$,* 3. (3)

$\sigma\alpha=\alpha$* and $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{n}_{H}$.*

Proof.

Assume first that $\sigma\alpha\neq\alpha$; then $\sigma(\mathfrak{g}(\mathfrak{a}_{G};\alpha))=\mathfrak{g}(\mathfrak{a}_{G};\sigma\alpha)$ and the map

[TABLE]

is non-trivial and $(M_{G}\cap M_{H})$-equivariant. Since $\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\overline{\alpha})$ is irreducible under the action of $M_{G}\cap M_{H}$ by Lemma 1.11 (1), the map must be a linear isomorphism, so (1) holds.

Now assume, $\sigma\alpha=\alpha$, then $\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ is $\sigma$-stable and we have a decomposition into eigenspaces

[TABLE]

If $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\cap\mathfrak{n}^{-\sigma}\neq\{0\}$, then this is a non-trivial $(M_{G}\cap M_{H})$-invariant subspace of $\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\overline{\alpha})$ and the latter is irreducible under the action of $M_{G}\cap M_{H}$. This implies (2). The remaining possibility is $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\cap\mathfrak{n}^{-\sigma}=\{0\}$, which clearly implies (3). ∎

Before we can finish the proof we need to study case (3) in Lemma 1.14 in more detail. For this, the following two results will be used:

Lemma 1.15** ([37, Lemma 2.9]).**

Let $\alpha_{1},\alpha_{2}\in\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ and assume that $(\alpha_{1},\alpha_{2})<0$; then for any $X_{1}\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{1})$ and $X_{2}\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{2})$, $X_{1},X_{2}\neq 0$, we have $[X_{1},X_{2}]\neq 0$.

Lemma 1.16**.**

Let $w\in W$, then for any $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ there exists a sequence $\alpha=\alpha_{1},\ldots,\alpha_{r}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ such that $\langle\alpha_{i},\alpha_{i+1}\rangle\neq 0$ for $i=1,\ldots,r-1$ and $\alpha_{r}$ is simple in $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

This result and its proof were communicated to us by Yoshiki Oshima.

Proof.

Let $ww_{0}=s_{\beta_{1}}\cdots s_{\beta_{n}}$ be a reduced expression for $ww_{0}\in W$, i.e. $\beta_{1},\ldots,\beta_{n}$ are simple roots. It is known that

[TABLE]

and we can write $\alpha_{1}=\alpha=s_{\beta_{1}}\cdots s_{\beta_{k-1}}\beta_{k}$. If $\alpha_{1}$ is not a simple root, there exists $1\leq i<k$ such that $s_{\beta_{i+1}}\cdots s_{\beta_{k-1}}\beta_{k}$ is simple, but $s_{\beta_{i}}\cdots s_{\beta_{k-1}}\beta_{k}$ is not. In particular, $s_{\beta_{i+1}}\cdots s_{\beta_{k-1}}\beta_{k}\neq s_{\beta_{i}}\cdots s_{\beta_{k-1}}\beta_{k}$ and hence $\langle\beta_{i},s_{\beta_{i+1}}\cdots s_{\beta_{k-1}}\beta_{k}\rangle\neq 0$, which implies $\langle s_{\beta_{1}}\cdots s_{\beta_{i-1}}\beta_{i},s_{\beta_{1}}\cdots s_{\beta_{k-1}}\beta_{k}\rangle\neq 0$. Put $\alpha_{2}:=s_{\beta_{1}}\cdots s_{\beta_{i-1}}\beta_{i}$, then $\langle\alpha_{1},\alpha_{2}\rangle\neq 0$. Repeating this argument yields the desired sequence. ∎

The next lemma provides more information about case (3) in Lemma 1.14:

Lemma 1.17**.**

Let $w\in W$ with $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$. If for all $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$ we have $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{n}_{H}$, then for one of those $\alpha$ we must have $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{n}^{-\sigma}]\neq\{0\}$.

Proof.

Let $w\in W$ with $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$ and assume that $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{n}_{H}$ for all $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$. We explicitly construct a root $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$ such that $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{n}^{-\sigma}]\neq\{0\}$.

Step 1. If $Q\tilde{w}P_{G}\neq Q\tilde{w}_{0}P_{G}$, then there exists $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$ by Lemma 1.12. We claim that there exists a simple root $\alpha$ with this property. In fact, by Lemma 1.16 there exists a sequence $\alpha=\alpha_{1},\ldots,\alpha_{r}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\langle\alpha_{i},\alpha_{i+1}\rangle\neq 0$ for $i=1,\ldots,r-1$ and $\alpha_{r}$ simple. By our assumption, every $\beta\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ satisfies either $\overline{\beta}=0$ or $\mathfrak{g}(\mathfrak{a}_{G};\beta)\subseteq\mathfrak{n}_{H}$. By Lemma 1.14 this implies $\sigma\beta=-\beta$ or $\sigma\beta=\beta$. These two types of roots are obviously orthogonal to each other. Hence, $\sigma\alpha=\alpha$ implies $\sigma\alpha_{i}=\alpha_{i}$ for all roots $\alpha_{i}$ in the sequence. In particular, $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{r})\subseteq\mathfrak{n}_{H}$ by Lemma 1.14, and we can replace $\alpha$ by the simple root $\alpha_{r}$.

Step 2. Next, we claim that there exists a simple root $\alpha^{\prime}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha^{\prime}}\neq 0$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime})\not\subseteq\mathfrak{n}_{H}$. Assume that such a simple root does not exist. Then for every simple root $\alpha^{\prime}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ we have either $\overline{\alpha^{\prime}}=0$ or $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime})\subseteq\mathfrak{n}_{H}$. This implies that either $\sigma\alpha^{\prime}=-\alpha^{\prime}$ or $\sigma\alpha^{\prime}=\alpha^{\prime}$, so that the set of simple roots is the disjoint union of the two mutually orthogonal subsets $\{\alpha^{\prime}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\mbox{ simple, }\sigma\alpha^{\prime}=\pm\alpha^{\prime}\}$. First note that this cannot occur in the case $(\mathfrak{g},\mathfrak{h})=(\mathfrak{g}^{\prime}+\mathfrak{g}^{\prime},\operatorname{diag}\mathfrak{g}^{\prime})$, so that we may assume $\mathfrak{g}$ to be simple. Then the Dynkin diagram of $\Sigma(\mathfrak{g},\mathfrak{a}_{G})$ is connected and we must have $\sigma\alpha^{\prime}=\alpha^{\prime}$ for all simple roots, whence $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime})\subseteq\mathfrak{n}_{H}$ for all simple roots. This implies $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime})\subseteq\mathfrak{n}_{H}$ for all positive roots and therefore $\mathfrak{n}_{G}=\mathfrak{n}_{H}$ and also $\overline{\mathfrak{n}}_{G}=\overline{\mathfrak{n}}_{H}$. But $\mathfrak{n}_{G}$ and $\overline{\mathfrak{n}}_{G}$ generate $\mathfrak{g}$, whence $\mathfrak{g}=\mathfrak{h}$ which contradicts our assumption that $(\mathfrak{g},\mathfrak{h})$ is non-trivial.

Step 3. By Step 1 we find a simple root $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})\cap w\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha}\neq 0$ and hence $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{n}_{H}$. We claim that $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{n}^{-\sigma}]\neq\{0\}$. To see this, we use Step 2 to find another simple root $\alpha^{\prime}\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ with $\overline{\alpha^{\prime}}\neq 0$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime})\not\subseteq\mathfrak{n}_{H}$. Connecting $\alpha$ and $\alpha^{\prime}$ in the Dynkin diagram for $\Sigma(\mathfrak{g},\mathfrak{a}_{G})$, we obtain a sequence of simple roots $\alpha=\alpha_{1},\alpha_{2},\ldots,\alpha_{p-1},\alpha_{p}=\alpha^{\prime}$ such that $(\alpha_{i},\alpha_{j})\neq 0$ if and only if $|i-j|\leq 1$. By possibly replacing $\alpha_{p}$ by one of the $\alpha_{i}$ we may assume that for all $1\leq i\leq p-1$ we either have $\overline{\alpha_{i}}=0$ or $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{i})\subseteq\mathfrak{n}_{H}$. We now construct a root $\alpha^{\prime\prime}=n_{2}\alpha_{2}+\cdots+n_{p}\alpha_{p}$, $n_{i}\geq 1$, such that $\overline{\alpha^{\prime\prime}}\neq 0$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime\prime})\not\subseteq\mathfrak{n}_{H}$, then $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{g}(\mathfrak{a}_{G};\alpha^{\prime\prime})]\neq\{0\}$ by Lemma 1.15 since $(\alpha,\alpha^{\prime\prime})=n_{2}(\alpha_{1},\alpha_{2})<0$. By Lemma 1.14 this implies $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{n}^{-\sigma}]\neq\{0\}$.

We inductively construct a root $\beta_{k}=n_{k}\alpha_{k}+\cdots+n_{p}\alpha_{p}$ ($2\leq k\leq p$) with $\overline{\beta_{k}}\neq 0$ and $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k})\not\subseteq\mathfrak{n}_{H}$. Note that for $\beta_{k}$ of the above form we always have $\overline{\beta_{k}}=n_{k}\overline{\alpha_{k}}+\cdots+n_{p}\overline{\alpha_{p}}\neq 0$ since $\overline{\alpha_{i}}$ is either $=0$ or a positive root, and by assumption $\overline{\alpha_{p}}\neq 0$. For $k=p$ we can choose the simple root $\beta_{p}=\alpha_{p}$. Now assume $\beta_{k+1}=n_{k+1}\alpha_{k+1}+\cdots+n_{p}\alpha_{p}$ has been constructed with $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\not\subseteq\mathfrak{n}_{H}$. Then by Lemma 1.14 there are four possibilities for $\beta_{k+1}$ and $\alpha_{k}$:

1. (1)

$\sigma\beta_{k+1}=\beta_{k+1}$ and $\overline{\alpha_{k}}\neq 0$. Then $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\cap\mathfrak{n}^{-\sigma}\neq\{0\}$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k})\subseteq\mathfrak{n}_{H}$, hence by Lemma 1.15

[TABLE]

so that $\beta_{k}=\alpha_{k}+\beta_{k+1}$ satisfies $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k})\not\subseteq\mathfrak{n}_{H}$. 2. (2)

$\sigma\beta_{k+1}=\beta_{k+1}$ and $\overline{\alpha_{k}}=0$. Then $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\cap\mathfrak{n}^{-\sigma}\neq\{0\}$ and $\sigma\alpha_{k}=-\alpha_{k}$. Hence, we have for any $X\in\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\cap\mathfrak{n}^{-\sigma}$ and $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k})$, $X,Y\neq 0$, that $0\neq[X,Y]\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k}+\beta_{k+1})$ by Lemma 1.15 and $\sigma[X,Y]=-[X,\sigma Y]\in\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{k}+\beta_{k+1})$. Therefore, $\sigma[X,Y]\neq[X,Y]$ and hence $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k}+\beta_{k+1})\not\subseteq\mathfrak{n}_{H}$, so that we can choose $\beta_{k}=\alpha_{k}+\beta_{k+1}$. 3. (3)

$\sigma\beta_{k+1}\neq\beta_{k+1}$ and $\overline{\alpha_{k}}\neq 0$. Then $\{X-\sigma X:X\in\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\}\subseteq\mathfrak{n}^{-\sigma}$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k})\subseteq\mathfrak{n}_{H}$. Hence, we have for any $X\in\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})$ and $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k})$, $X,Y\neq 0$, that $0\neq[X,Y]\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k}+\beta_{k+1})$ by Lemma 1.15 and $\sigma[X,Y]=[\sigma X,Y]\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k}+\sigma\beta_{k+1})$. Therefore, $\sigma[X,Y]\neq[X,Y]$ and hence $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k}+\beta_{k+1})\not\subseteq\mathfrak{n}_{H}$, so that we can choose $\beta_{k}=\alpha_{k}+\beta_{k+1}$. 4. (4)

$\sigma\beta_{k+1}\neq\beta_{k+1}$ and $\overline{\alpha_{k}}=0$. Then $\{X-\sigma X:X\in\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})\}\subseteq\mathfrak{n}^{-\sigma}$ and $\sigma\alpha_{k}=-\alpha_{k}$. Let $X\in\mathfrak{g}(\mathfrak{a}_{G};\beta_{k+1})$ and $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha_{k})$, $X,Y\neq 0$; then by Lemma 1.15 we have $[X,Y]\neq 0$.

1. (a)

If $\sigma[X,Y]\neq[X,Y]$ then we can choose $\beta_{k}=\alpha_{k}+\beta_{k+1}$ as in (1), (2) and (3). 2. (b)

If $\sigma[X,Y]=[X,Y]$ then $\sigma(\alpha_{k}+\beta_{k+1})=\alpha_{k}+\beta_{k+1}$ so that $\sigma\beta_{k+1}=2\alpha_{k}+\beta_{k+1}$. In this case we choose $\beta_{k}=\sigma\beta_{k+1}=2\alpha_{k}+n_{k+1}\alpha_{k+1}+\cdots+n_{p}\alpha_{p}$ which is clearly a positive root. Further, $\mathfrak{g}(\mathfrak{a}_{G};\beta_{k})\not\subseteq\mathfrak{n}_{H}$ since $\sigma\beta_{k}=\beta_{k+1}\neq\beta_{k}$.

Inductively, for $k=2$ this produces the desired root $\alpha^{\prime\prime}=\beta_{2}$. ∎

We can finally finish the proof of Theorem 1.3 by choosing $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ according to the three cases in Lemma 1.14. Write $\beta=\overline{\alpha}$.

1. (1)

If $\sigma\alpha\neq\alpha$ we can write $X_{\beta}=Y-\sigma Y$ for some $Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$. Using this $Y$ in the above construction, we have by Lemma 1.13

[TABLE]

Both sides are contained in $\mathfrak{n}=\mathfrak{n}^{\sigma}\oplus\mathfrak{n}^{-\sigma}$, and taking $\mathfrak{n}^{-\sigma}$-components gives $2X_{\beta}=\operatorname{ad}(Z_{M}+Z_{A})X_{\beta}=\operatorname{ad}(Z_{M})X_{\beta}+\beta(Z_{A})X_{\beta}$. Note that we assume $X_{\beta}\neq 0$ by the reduction in Section 1.6.1. Then $\operatorname{ad}(Z_{M})X_{\beta}=(2-\beta(Z_{A}))X_{\beta}$ so that $Z_{M}$ acts by a scalar on $X_{\beta}$. Since $Z_{M}\in\mathfrak{m}_{G}\cap\mathfrak{m}_{H}$ and $M_{G}\cap M_{H}$ is compact, this scalar has to be imaginary, so that $\beta(Z_{A})=2$. In particular, $Z_{A}\neq 0$. 2. (2)

If $\sigma\alpha=\alpha$ and $\mathfrak{g}^{-\sigma}(\mathfrak{a}_{G};\alpha)=\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)$ we take $Y=X_{\beta}$. Then Lemma 1.13 implies

[TABLE]

The same argument as in (1) shows $\beta(Z_{A})=1$ and in particular $Z_{A}\neq 0$. 3. (3)

If (1) and (2) do not hold for any such $\alpha$, we have $\sigma\alpha=\alpha$ and $\mathfrak{g}(\mathfrak{a}_{G};\alpha)\subseteq\mathfrak{n}_{H}$ for all $\alpha>0$ with $w^{-1}\alpha>0$, $\overline{\alpha}\neq 0$. Then Lemma 1.17 implies that there exists an $\alpha$ with $[\mathfrak{g}(\mathfrak{a}_{G};\alpha),\mathfrak{n}^{-\sigma}]\neq\{0\}$. Let $0\neq Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ and $0\neq B_{i}\in\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta_{i})$ ($i=1,2$) with $\beta_{1},\beta_{2}\in\Delta(\mathfrak{n}^{-\sigma})$, such that $[Y,B_{1}]=B_{2}$. By Lemma 1.11 there exists $ma\in(M_{G}\cap M_{H})A_{H}$ such that $\operatorname{Ad}(ma)B_{i}=\pm X_{\beta_{i}}$ ($i=1,2$). Hence, $[\operatorname{Ad}(ma)Y,X_{\beta_{1}}]=\pm X_{\beta_{2}}$. Replacing $Y$ by $\operatorname{Ad}(ma)Y\in\mathfrak{g}(\mathfrak{a}_{G};\alpha)$ we may therefore assume that $[Y,X_{\beta_{1}}]=\pm X_{\beta_{2}}$.

Note that

[TABLE]

and $\operatorname{Ad}(e^{-tY})X=e^{-t\operatorname{ad}(Y)}X\in\mathfrak{n}^{-\sigma}$ so that $n_{t}=e^{tY}$ and $X_{t}=\operatorname{Ad}(m_{t}a_{t})X=e^{-t\operatorname{ad}(Y)}X$. Hence, $Z_{N}=Y$ and $[Z_{M}+Z_{A},X]=-[Y,X]$. Taking the $\beta_{2}$-component gives

[TABLE]

By the same argument as in (1) and (2) we find $\beta_{2}(Z_{A})=\mp 1$ and hence $Z_{A}\neq 0$.

This finishes the proof of Theorem 1.3.∎

2. Spherical matrix coefficients

We study matrix coefficients of finite-dimensional representations of $G$ which are equivariant under the action of $P_{H}\times P_{G}$ by left and right multiplication. Such matrix coefficients correspond to finite-dimensional spherical representations of $G$ whose restriction to $H$ contains a spherical representation, and we show that there exist enough such representations (see Theorem 2.6). In Section 3 these matrix coefficients are used to explicitly construct symmetry breaking operators.

2.1. Reduction to complex connected groups

Since both $G$ and $H$ might be disconnected, their finite-dimensional representations are not easily described in terms of highest weights. To overcome this difficulty we first reduce the construction of matrix coefficients to the case of complex connected groups. Since $G$ is of Harish-Chandra class, there exist

• •

a complex connected linear reductive group $\mathbb{G}_{\mathbb{C}}$ with Lie algebra $\mathfrak{g}_{\mathbb{C}}$,

• •

an antiholomorphic involution $\tau:\mathbb{G}_{\mathbb{C}}\to\mathbb{G}_{\mathbb{C}}$ such that the derived involution $\tau:\mathfrak{g}_{\mathbb{C}}\to\mathfrak{g}_{\mathbb{C}}$ is the conjugation with respect to the real form $\mathfrak{g}$,

• •

a homomorphism $\mu:G\to\mathbb{G}$ from $G$ to the real form $\mathbb{G}=\mathbb{G}_{\mathbb{C}}^{\tau}$ of $\mathbb{G}_{\mathbb{C}}$ with finite kernel and cokernel.

Note that the Lie algebra of $\mathbb{G}$ is equal to $\mathfrak{g}$. We denote by $\mathbb{H}_{\mathbb{C}}$ the complex connected subgroup of $\mathbb{G}_{\mathbb{C}}$ with Lie algebra $\mathfrak{h}_{\mathbb{C}}$ and by $\mathbb{H}=\mu(H)_{0}=\mu(H_{0})\subseteq\mathbb{H}_{\mathbb{C}}$ the connected subgroup of $\mathbb{H}_{\mathbb{C}}$ with Lie algebra $\mathfrak{h}$. Then the finite-dimensional holomorphic representations of $\mathbb{G}_{\mathbb{C}}$ and $\mathbb{H}_{\mathbb{C}}$ are classified in terms of their highest weights, and via the homomorphism $\mu$ they give rise to finite-dimensional representations of $G$ and $H_{0}$.

The image $\mathbb{K}=\mu(K)$ of the maximal compact subgroup of $G$ under $\mu$ is a maximal compact subgroup of $\mathbb{G}$, and the intersection $\mathbb{H}\cap\mathbb{K}$ is maximal compact in $\mathbb{H}$. Further, let

[TABLE]

then $\mathbb{P}_{G}=\mu(P_{G})=\mathbb{M}_{G}\mathbb{A}_{G}\mathbb{N}_{G}$ is a minimal parabolic subgroup of $\mathbb{G}$.

2.2. The Cartan–Helgason Theorem

We now recall the classification of irreducible finite-dimensional spherical representations of $\mathbb{G}$ in terms of their highest weights, the so-called Cartan–Helgason Theorem. Recall that a representation of $\mathbb{G}$ is called spherical if it contains a non-zero $\mathbb{K}$-invariant vector.

We choose a maximal abelian subalgebra $\mathfrak{t}_{G}$ in $\mathfrak{m}_{G}$; then $\mathfrak{j}_{G}=\mathfrak{t}_{G}\oplus\mathfrak{a}_{G}$ is a Cartan subalgebra of $\mathfrak{g}$ and $\mathfrak{j}_{G,\mathbb{C}}$ is a Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}$. Roots in $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ are real on $\mathfrak{a}_{G}$ and imaginary on $\mathfrak{t}_{G}$. Fix a positive system $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ such that the non-zero restrictions to $\mathfrak{a}_{G}$ are contained in $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$. With respect to this data, the irreducible finite-dimensional representations of $\mathbb{G}$ are classified by their highest weights in $\mathfrak{j}_{G,\mathbb{C}}^{\vee}$.

Recall the following theorem (see e.g. [13, Theorem 8.49]):

Theorem 2.1** (Cartan–Helgason).**

For an irreducible finite-dimensional representation $\varphi$ of $\mathbb{G}$ the following statements are equivalent:

1. (1)

$\varphi$* has a non-zero $\mathbb{K}$-fixed vector.* 2. (2)

$\mathbb{M}_{G}$* acts by the $1$-dimensional trivial representation in the highest restricted weight space of $\varphi$.* 3. (3)

The highest weight of $\varphi$ vanishes on $\mathfrak{t}_{G}$, and its restriction to $\mathfrak{a}_{G}$ is contained in the set

[TABLE]

Let $\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})\subseteq\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})$ denote the subset of all $\lambda$ for which there exists an irreducible finite-dimensional $\mathbb{G}$-representation $(\varphi_{\lambda},V_{\lambda})$ of highest weight $\lambda$. Then $\operatorname{span}_{\mathbb{R}}\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})=\operatorname{span}_{\mathbb{R}}\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})=\mathfrak{a}_{G}^{\vee}$, and if $\mathbb{G}$ is simply connected even $\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})=\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})$. Theorem 2.1 immediately gives the action of $\mathbb{P}_{G}$ on the highest weight space of $V_{\lambda}$:

Corollary 2.2**.**

For every $\lambda\in\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})$ the minimal parabolic subgroup $\mathbb{P}_{G}=\mathbb{M}_{G}\mathbb{A}_{G}\mathbb{N}_{G}$ acts on the highest weight space of $V_{\lambda}$ by the character ${\rm\bf 1}\otimes e^{\lambda}\otimes{\rm\bf 1}$.

Here ${\rm\bf 1}$ denotes the trivial representation of $\mathbb{M}_{G}$ and $\mathbb{N}_{G}$, respectively, and $e^{\lambda}$ is the character of $\mathbb{A}_{G}=\exp(\mathfrak{a}_{G})$ given by $e^{\lambda}(e^{X})=e^{\lambda(X)}$, $X\in\mathfrak{a}_{G}$.

Now, for an irreducible finite-dimensional representation $(\varphi,V)$ of $\mathbb{G}$ we denote by $(\varphi^{\vee},V^{\vee})$ the contragredient representation on the dual space $V^{\vee}=\operatorname{Hom}_{\mathbb{C}}(V,\mathbb{C})$ given by

[TABLE]

The following statement is standard:

Lemma 2.3**.**

1. (1)

$(\varphi,V)$* has a non-zero $\mathbb{K}$-fixed vector if and only if $(\varphi^{\vee},V^{\vee})$ has a non-zero $\mathbb{K}$-fixed vector.* 2. (2)

For $\lambda\in\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})$ let $\lambda^{\vee}\in\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})$ be defined by $(\varphi_{\lambda}^{\vee},V_{\lambda}^{\vee})\simeq(\varphi_{\lambda^{\vee}},V_{\lambda^{\vee}})$, then the map

[TABLE]

is the restriction to $\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})$ of a linear map $\mathfrak{a}_{G}^{\vee}\to\mathfrak{a}_{G}^{\vee}$.

2.3. Matrix coefficients

As for $\mathbb{G}$, we denote by $\Lambda^{+}_{\mathbb{H}}(\mathfrak{h},\mathfrak{a}_{H})\subseteq\mathfrak{a}_{H}^{\vee}$ the set of all highest weights $\nu$ of irreducible finite-dimensional representations $(\psi_{\nu},W_{\nu})$ of $\mathbb{H}$ which have a one-dimensional $\mathbb{P}_{H}$-invariant subspace isomorphic to ${\rm\bf 1}\otimes e^{\nu}\otimes{\rm\bf 1}$, where $\mathbb{P}_{H}=\mathbb{M}_{H}\mathbb{A}_{H}\mathbb{N}_{H}$ is the corresponding minimal parabolic subgroup of $\mathbb{H}$.

Lemma 2.4**.**

Let $\lambda\in\Lambda_{\mathbb{G}}^{+}(\mathfrak{g},\mathfrak{a}_{G})$ and $\nu\in\Lambda_{\mathbb{H}}^{+}(\mathfrak{h},\mathfrak{a}_{H})$ and pick non-zero highest weight vectors $v_{0}\in V_{\lambda}$ and $\phi_{0}\in W_{\nu}^{\vee}$. Then for every $0\neq\eta\in\operatorname{Hom}_{\mathbb{H}}(\varphi_{\lambda}|_{\mathbb{H}},\psi_{\nu})$ the function

[TABLE]

is non-zero, real-analytic and satisfies

[TABLE]

for $g\in\mathbb{G}$, $man\in\mathbb{P}_{G}$ and $m^{\prime}a^{\prime}n^{\prime}\in\mathbb{P}_{H}$.

Proof.

It is clear that $f$ is real analytic as a matrix coefficient of a finite-dimensional representation. Further, $f$ is non-zero since $(\varphi_{\lambda},V_{\lambda})$ is irreducible. By Corollary 2.2 and Lemma 2.3 we have $\varphi_{\lambda}(man)v_{0}=a^{\lambda}v_{0}$ and $\psi_{\nu}^{\vee}(m^{\prime}a^{\prime}n^{\prime})\phi_{0}=(a^{\prime})^{\nu^{\vee}}\phi_{0}$ and the claim follows. ∎

Note that since $(\varphi_{\lambda},V_{\lambda})$ and $(\psi_{\nu},W_{\nu})$ extend to holomorphic representations of the complex connected groups $\mathbb{G}_{\mathbb{C}}$ and $\mathbb{H}_{\mathbb{C}}$, we have $\operatorname{Hom}_{\mathbb{H}}(\varphi_{\lambda}|_{\mathbb{H}},\psi_{\nu})=\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})$. Abusing notation, we also write $(\varphi_{\lambda},V_{\lambda})$ and $(\psi_{\nu},W_{\nu})$ for the Lie algebra representations of $\mathfrak{g}$ and $\mathfrak{h}$ for arbitrary $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{H})$ and $\nu\in\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$.

To obtain matrix coefficients with the same properties as in Lemma 2.4, but for the pair $(G,H)$ instead of $(\mathbb{G},\mathbb{H})$, we use the homomorphism $\mu:G\to\mathbb{G}$.

Proposition 2.5**.**

Assume that $(G,H)$ is a strongly spherical reductive pair. Then for each pair $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$ with $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})\neq\{0\}$ there exists $k\geq 1$ and a non-zero real-analytic function $F:G\to\mathbb{R}$, $F\geq 0$, satisfying

[TABLE]

for $g\in G$, $man\in P_{G}$ and $m^{\prime}a^{\prime}n^{\prime}\in P_{H}$.

Proof.

First note that there exists $k\geq 1$ such that $k\lambda\in\Lambda^{+}_{\mathbb{G}}(\mathfrak{g},\mathfrak{a}_{G})$ and $k\nu\in\Lambda^{+}_{\mathbb{H}}(\mathfrak{h},\mathfrak{a}_{H})$. Then also $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{k\lambda}|_{\mathfrak{h}},\psi_{k\nu})\neq\{0\}$ and by Lemma 2.4 there exists a non-zero real-analytic function $f:\mathbb{G}\to\mathbb{C}$ with

[TABLE]

for $g\in\mathbb{G}$, $man\in\mathbb{P}_{G}$ and $m^{\prime}a^{\prime}n^{\prime}\in\mathbb{P}_{H}$. Replacing $f$ by $|f|^{2}$ we may further assume that $f:\mathbb{G}\to\mathbb{R}$ and $f\geq 0$. We consider the non-zero real-analytic function $f\circ\mu:G\to\mathbb{R}$ which satisfies (2.1) at least for $m^{\prime}\in M_{H,0}\subseteq\mu^{-1}(\mathbb{M}_{H})\subseteq M_{H}$. Since the component group $M_{H}/M_{H,0}$ of $M_{H}$ is finite, we can form the finite sum

[TABLE]

and this clearly defines a real-analytic function $F:G\to\mathbb{R}$, $F\geq 0$, with the equivariance property (2.1). Finally, $F$ is non-zero since $f\circ\mu\geq 0$ and $f\circ\mu\neq 0$. ∎

The main result of this section asserts that for all strongly spherical reductive pairs $(G,H)$ there exist enough pairs $(\lambda,\nu)\in\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ with $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})\neq\{0\}$:

Theorem 2.6**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then the set of pairs $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$ such that $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})\neq\{0\}$ spans $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$.

Remark 2.7**.**

One can use the local structure theorem for real spherical varieties by Knop–Krötz–Schlichtkrull [14] to give a classification-free proof of Theorem 2.6. In fact, let $Z=(G\times H)/\operatorname{diag}(H)$; then using [14, Theorem 2.8] it is easy to see that for $X\in\mathfrak{n}^{-\sigma}$ contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit, the minimal parabolic subgroup $P=e^{-X}\overline{P}_{G}e^{X}\times P_{H}$ is $Z$-adapted in the sense of [14, Definition 2.7]. Further, the Levi subgroup $L=e^{-X}M_{G}A_{G}e^{X}\times M_{H}A_{H}$ of $P$ satisfies $L\cap\operatorname{diag}(H)=\operatorname{diag}(M)$, where $M=(M_{G}\cap M_{H})^{X}$ is the stabilizer of $X$. This implies $\mathfrak{a}_{Z}=\operatorname{Ad}(e^{-X})\mathfrak{a}_{G}\times\mathfrak{a}_{H}$ in the notation of [14, Section 2.3] and therefore $\operatorname{rank}(Z)=\dim\mathfrak{a}_{Z}=\dim\mathfrak{a}_{G}+\dim\mathfrak{a}_{H}$. By [14, Remark 3.5] the statement of Theorem 2.6 follows.

However, since the explicit form of the integral kernels of symmetry breaking operators plays an important role in the classification of symmetry breaking operators (see e.g. [2, 3, 22]), we prove Theorem 2.6 using the classification of strongly spherical reductive pairs. From this one can explicitly determine the matrix coefficients which serve as building blocks for the integral kernels of symmetry breaking operators.

Before we come to the proof of this result, let us see how it can be used to show the implication (1)$\Rightarrow$(2) in Theorem 1.3:

Corollary 2.8**.**

If the double coset $P_{H}gP_{G}$ is open, then the projection of $\mathfrak{p}_{H}\cap\operatorname{Ad}(g)\mathfrak{p}_{G}$ to $\mathfrak{a}_{H}$ is trivial.

Proof.

Let $P_{H}gP_{G}$ be an open double coset and $Z=Z_{M}+Z_{A}+Z_{N}\in\mathfrak{p}_{H}\cap\operatorname{Ad}(g)\mathfrak{p}_{G}$. Put $X=\operatorname{Ad}(g)^{-1}Z=X_{M}+X_{A}+X_{N}\in\mathfrak{p}_{G}$. For any $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$ with $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})\neq\{0\}$ let $F$ be as in Proposition 2.5. Since $F$ is non-zero and real-analytic, it is non-zero on the open set $P_{H}gP_{G}$. In particular, $F(g)\neq 0$ and for all $t\in\mathbb{R}$ we have

[TABLE]

so that $\lambda(X_{A})+\nu^{\vee}(Z_{A})=0$. Since the pairs $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$ satisfying $\operatorname{Hom}_{\mathfrak{h}}(\varphi_{\lambda}|_{\mathfrak{h}},\psi_{\nu})\neq\{0\}$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, this implies $X_{A}=0$ and $Z_{A}=0$ and the proof is complete. ∎

Note that the statement in Theorem 2.6 only depends on the pair of Lie algebras $(\mathfrak{g},\mathfrak{h})$. If we define

[TABLE]

then we have to show that $\Lambda(\mathfrak{g},\mathfrak{h})$ spans $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$. Note that $\Lambda(\mathfrak{g},\mathfrak{h})$ is a subsemigroup of $\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$. We first reduce Theorem 2.6 to the case of $\mathfrak{g}$ semisimple and $(\mathfrak{g},\mathfrak{h})$ symmetric and then use the classification in Theorem 1.6 to show the statement case-by-case.

Lemma 2.9**.**

Assume that Theorem 2.6 holds for $\mathfrak{g}$ semisimple, then it holds for $\mathfrak{g}$ reductive.

Proof.

As in Section 1.2 we write $\mathfrak{g}=\mathfrak{g}_{\rm n}\oplus\mathfrak{g}_{\rm el}$ and $p:\mathfrak{g}\to\mathfrak{g}_{\rm n}$ for the canonical projection. Then $\mathfrak{a}_{G}=\mathfrak{a}_{G,\rm n}\oplus\mathfrak{a}_{G,\rm el}$ with $\mathfrak{a}_{G,\rm n}=\mathfrak{a}_{G}\cap\mathfrak{g}_{\rm n}$ and $\mathfrak{a}_{G,\rm el}=\mathfrak{a}_{G}\cap\mathfrak{g}_{\rm el}$, and by Lemma 1.4 (1) the restriction $p|_{\mathfrak{a}_{H}}:\mathfrak{a}_{H}\to\mathfrak{a}_{G,\rm n}$ is injective. Further, by Lemma 1.4 (2) the pair $(\mathfrak{g}_{\rm n},p(\mathfrak{h}))$ is also strongly spherical, so by assumption $\Lambda(\mathfrak{g}_{\rm n},p(\mathfrak{h}))$ spans $\mathfrak{a}_{G,\rm n}^{\vee}\times p(\mathfrak{a}_{H})^{\vee}$. Now for every pair $(\lambda_{0},\nu_{0})\in\Lambda(\mathfrak{g}_{\rm n},p(\mathfrak{h}))$ the pair $(\lambda,\nu)\in\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ with $\lambda|_{\mathfrak{a}_{G,\rm n}}=\lambda_{0}$, $\lambda|_{\mathfrak{a}_{G,\rm el}}=0$ and $\nu=\nu_{0}\circ p|_{\mathfrak{a}_{H}}$ is contained in $\Lambda(\mathfrak{g},\mathfrak{h})$. Hence, the span of $\Lambda(\mathfrak{g},\mathfrak{h})$ contains at least $\mathfrak{a}_{G,\rm n}^{\vee}\times\mathfrak{a}_{H}^{\vee}$. Further, for every $\lambda_{1}\in\mathfrak{a}_{G,\rm el}^{\vee}$ the representation $\varphi_{\lambda}$ is one-dimensional and hence its restriction is an $\mathfrak{h}\cap\mathfrak{k}$-spherical representation of $\mathfrak{h}$ with highest weight $\nu_{1}\in\mathfrak{a}_{H}^{\vee}$, so that $(\lambda_{1},\nu_{1})\in\Lambda(\mathfrak{g},\mathfrak{h})$. This shows that $\Lambda(\mathfrak{g},\mathfrak{h})$ indeed spans $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$. ∎

Lemma 2.10**.**

Assume that Theorem 2.6 holds for $(\mathfrak{g},\mathfrak{h})$ symmetric, then it holds for $(\mathfrak{g},\mathfrak{h})$ reductive.

Proof.

By the previous lemma we may assume that $\mathfrak{g}$ is semisimple. Then, thanks to Corollary 1.7, there exists a non-trivial involution $\sigma$ of $\mathfrak{g}$ such that $\mathfrak{h}\subseteq\mathfrak{g}^{\sigma}$, and $\mathfrak{h}$ and $\mathfrak{g}^{\sigma}$ differ only in compact factors. Hence, $\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})=\Lambda^{+}(\mathfrak{g}^{\sigma},\mathfrak{a}_{H})$ and $\Lambda(\mathfrak{g},\mathfrak{g}^{\sigma})\subseteq\Lambda(\mathfrak{g},\mathfrak{h})$. ∎

2.4. Finite-dimensional branching

We now prove Theorem 2.6 case-by-case for all symmetric pairs in the classification in Section 1.2. For this we first fix some notation.

Let $\mathfrak{j}_{H}\subseteq\mathfrak{h}$ be a Cartan subalgebra of $\mathfrak{h}$ and extend it to a Cartan subalgebra $\mathfrak{j}_{H}\subseteq\mathfrak{j}_{G}\subseteq\mathfrak{g}$ of $\mathfrak{g}$. Note that we no longer assume that $\mathfrak{a}_{G}\subseteq\mathfrak{j}_{G}$ and $\mathfrak{a}_{H}\subseteq\mathfrak{j}_{H}$. Choose a system of positive roots $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ for $\mathfrak{g}$ such that

[TABLE]

is a system of positive roots for $\mathfrak{h}$. Denote by $\varpi_{1},\ldots,\varpi_{s}$ the fundamental weights for $\mathfrak{g}$ with respect to $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ and by $\zeta_{1},\ldots,\zeta_{t}$ the fundamental weights for $\mathfrak{h}$ with respect to $\Sigma^{+}(\mathfrak{h}_{\mathbb{C}},\mathfrak{j}_{H,\mathbb{C}})$. Then any dominant integral weight of $\mathfrak{g}$ with respect to $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ is of the form $\varpi=k_{1}\varpi_{1}+\cdots+k_{s}\varpi_{s}$, $k_{1},\ldots,k_{s}\in\mathbb{N}$, and we write $F^{\mathfrak{g}}(\varpi)$ for the corresponding finite-dimensional representation of $\mathfrak{g}$. Analogous notation is used for $\mathfrak{h}$ and ideals of $\mathfrak{h}$. To simplify some statements we further put $\zeta_{0}:=0$ so that $F^{\mathfrak{h}}(\zeta_{0})$ is the trivial representation of $\mathfrak{h}$.

We make use of the Satake diagrams for $\mathfrak{g}$ and $\mathfrak{h}$ (see e.g. [10, Chapter X, Appendix F] for details). From the Satake diagram the highest weights belonging to spherical representations can be read off. In fact, for every simple root $\alpha_{i}$ whose vertex in the Satake diagram is white and not linked to any other vertex by an arrow, the representations $F^{\mathfrak{g}}(2k\varpi_{i})$ ($k\in\mathbb{N}$) are spherical. If the vertices of two simple roots $\alpha_{i}$ and $\alpha_{j}$ are white and linked by an arrow, then $2k(\varpi_{i}+\varpi_{j})$ ($k\in\mathbb{N}$) are highest weights of spherical representations. Moreover, if $F^{\mathfrak{g}}(\varpi)$ and $F^{\mathfrak{g}}(\varpi^{\prime})$ are spherical, then $F^{\mathfrak{g}}(\varpi+\varpi^{\prime})$ is spherical. In many cases we compute the explicit branching for $F^{\mathfrak{g}}(\varpi_{i})$ resp. $F^{\mathfrak{g}}(\varpi_{i}+\varpi_{j})$ and then use the semigroup property of $\Lambda(\mathfrak{g},\mathfrak{h})$ to conclude that the spherical representation $F^{\mathfrak{g}}(2\varpi_{i})$ resp. $F^{\mathfrak{g}}(2(\varpi_{i}+\varpi_{j}))$ contains a certain spherical $\mathfrak{h}$-representation.

The following reduction from complex Lie algebras to split real forms allows to minimize the number of different cases:

Lemma 2.11**.**

Let $(\mathfrak{g},\mathfrak{h})$ be a reductive pair with $\mathfrak{g}$ and $\mathfrak{h}$ split. If the statement in Theorem 2.6 holds for $(\mathfrak{g},\mathfrak{h})$, then it also holds for $(\mathfrak{g}_{\mathbb{C}},\mathfrak{h}_{\mathbb{C}})$ viewed as real Lie algebras.

Proof.

Since $\mathfrak{g}$ and $\mathfrak{h}$ are split, we can choose $\mathfrak{j}_{G}=\mathfrak{a}_{G}$ and $\mathfrak{j}_{H}=\mathfrak{a}_{H}$. Let $\mathfrak{u}=\mathfrak{k}+i\mathfrak{p}\subseteq\mathfrak{g}_{\mathbb{C}}$; then $\mathfrak{u}$ is maximally compact in $\mathfrak{g}_{\mathbb{C}}$ with complement $\mathfrak{u}^{\perp}=i\mathfrak{k}+\mathfrak{p}$. Further, $\mathfrak{a}_{G,\mathbb{C}}$ is a (real) Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}$ which splits into $\mathfrak{a}_{G,\mathbb{C}}=i\mathfrak{a}_{G}+\mathfrak{a}_{G}$ with $i\mathfrak{a}_{G}\subseteq\mathfrak{u}$ and $\mathfrak{a}_{G}\subseteq\mathfrak{u}^{\perp}$. By the Cartan–Helgason Theorem, the highest weights of $\mathfrak{u}$-spherical representations of $\mathfrak{g}_{\mathbb{C}}$ vanish on $i\mathfrak{a}_{G}$ and their restrictions to $\mathfrak{a}_{G}$ are contained in $\Lambda^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{a}_{G})=\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})$. If $(\varphi_{\lambda},V_{\lambda})$ denotes a $\mathfrak{k}$-spherical representation of $\mathfrak{g}$ with highest weight $\lambda\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})$, then a $\mathfrak{u}$-spherical representation of $\mathfrak{g}_{\mathbb{C}}$ with highest weight $2\lambda$ is given by $V_{\lambda}\otimes V_{\lambda}$ where $\mathfrak{g}_{\mathbb{C}}$ acts by

[TABLE]

Now let $(\lambda,\nu)\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$. Then for any $A\in\operatorname{Hom}_{\mathfrak{h}}(V_{\lambda},W_{\nu})$ we clearly have $A\otimes A\in\operatorname{Hom}_{\mathfrak{h}_{\mathbb{C}}}(V_{\lambda}\otimes V_{\lambda},W_{\nu}\otimes W_{\nu})$. Hence, $(\lambda,\nu)\in\Lambda(\mathfrak{g},\mathfrak{h})$ implies $(2\lambda,2\nu)\in\Lambda(\mathfrak{g}_{\mathbb{C}},\mathfrak{h}_{\mathbb{C}})$ and the claim follows. ∎

Finally, we prove Theorem 2.6 case-by-case for all strongly spherical symmetric pairs in the classification of Theorem 1.6:

A) Trivial case

This case $\mathfrak{g}=\mathfrak{h}$ is by assumption excluded.

C) Compact case

Let $\mathfrak{g}$ be the Lie algebra of a compact simple Lie group; then also $\mathfrak{h}$ is the Lie algebra of a compact group and $\mathfrak{a}_{G}=\mathfrak{a}_{H}=\{0\}$ so that $\Lambda(\mathfrak{g},\mathfrak{h})=\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}=\{0\}\times\{0\}$ holds trivially.

D) Compact subgroup case

Let $\mathfrak{h}=\mathfrak{k}$ be the Lie algebra of a maximal compact subgroup $K$ of a non-compact simple Lie group $G$ with Lie algebra $\mathfrak{g}$. Then $\mathfrak{a}_{H}=\{0\}$ and the only spherical representation of $\mathfrak{h}$ is the trivial representation $W_{0}=\mathbb{C}$. By definition, for every $\lambda\in\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})$ the representation $V_{\lambda}$ contains a $\mathfrak{k}$-fixed vector, hence also $V_{\lambda}^{\vee}$ contains a $\mathfrak{k}$-fixed vector by Lemma 2.3. But $\mathfrak{k}$-fixed vectors in $V_{\lambda}^{\vee}=\operatorname{Hom}_{\mathbb{C}}(V_{\lambda},\mathbb{C})$ are simply $\mathfrak{k}$-equivariant homomorphisms from $V_{\lambda}$ to the trivial representation $W_{0}$ of $\mathfrak{k}$. Thus, $\Lambda(\mathfrak{g},\mathfrak{h})=\Lambda^{+}(\mathfrak{g},\mathfrak{a}_{G})\times\{0\}$ spans $\mathfrak{a}_{G}^{\vee}\times\{0\}=\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$.

E1) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,p+q),\operatorname{\mathfrak{so}}(1,p)+\operatorname{\mathfrak{so}}(q))$

The Satake diagrams of $\mathfrak{g}$ and $\operatorname{\mathfrak{so}}(1,p)\subseteq\mathfrak{h}$ are

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq s\}$ and additionally $\{\pm e_{i}:1\leq i\leq s\}$ if $p+q$ is even. Choose the simple roots $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq s-1$) and $\alpha_{s}=e_{s}$ for $p+q$ even and $\alpha_{s}=e_{s-1}+e_{s}$ for $p+q$ odd. We distinguish two cases:

• •

Assume first that $p+q$ is even or $p$ is odd; then $\mathfrak{j}_{G}=\mathfrak{j}_{H}$. If we choose the simple roots for $\operatorname{\mathfrak{so}}(1,p)\subseteq\mathfrak{h}$ as $\beta_{i}=\alpha_{i}$ ($1\leq i\leq t-1$) and $\beta_{t}=e_{t}$ for $p$ even and $\beta_{t}=e_{t-1}+e_{t}$ for $p$ odd, then $\beta_{i}=\alpha_{i}$ ($t+1\leq i\leq s-1$) and $\beta_{s}=e_{s-1}+e_{s}$ for $q$ even and $\beta_{s}=e_{s}$ for $q$ odd are the simple roots for $\operatorname{\mathfrak{so}}(q)\subseteq\mathfrak{h}$.

• •

Assume now that $p+q$ is odd and $p$ is even; then we can choose $\mathfrak{j}_{H}\subseteq\mathfrak{j}_{G}$ such that $e_{s}|_{\mathfrak{j}_{H,\mathbb{C}}}=0$ and the simple roots for $\operatorname{\mathfrak{so}}(1,p)\subseteq\mathfrak{h}$ are $\beta_{i}=\alpha_{i}|_{\mathfrak{j}_{H,\mathbb{C}}}$ ($1\leq i\leq t-1$) plus $\beta_{t}=e_{t}|_{\mathfrak{j}_{H,\mathbb{C}}}$, and the simple roots for $\operatorname{\mathfrak{so}}(q)\subseteq\mathfrak{h}$ are $\beta_{i}=\alpha_{i}|_{\mathfrak{j}_{H,\mathbb{C}}}$ ($t+1\leq i\leq s-1$).

Consider the fundamental weight $\varpi_{1}=e_{1}$. Clearly $e_{1}|_{\mathfrak{j}_{H,\mathbb{C}}}=\zeta_{1}$ is also a highest weight for $\operatorname{\mathfrak{so}}(1,p)\subseteq\mathfrak{h}$ and hence $(2\varpi_{1},2\zeta_{1})\in\Lambda(\mathfrak{g},\mathfrak{h})$. Further, from the Satake diagram for the real form $\operatorname{\mathfrak{so}}(p+1,q)$ of $\mathfrak{g}_{\mathbb{C}}\simeq\operatorname{\mathfrak{so}}(p+q+1,\mathbb{C})$, it follows that $F^{\mathfrak{g}}(2\varpi_{1})$ is $\mathfrak{h}$-spherical and hence also $(2\varpi_{1},0)\in\Lambda(\mathfrak{g},\mathfrak{h})$. Clearly $(2\varpi_{1},2\zeta_{1})$ and $(2\varpi_{1},0)$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $2$-dimensional.

E2) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{su}}(1,p+q),\mathfrak{s}(\mathfrak{u}(1,p)+\mathfrak{u}(q)))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm(e_{i}-e_{j}):1\leq i<j\leq p+q+1\}$ in the vector space $\{x\in\mathbb{R}^{p+q+1}:x_{1}+\cdots+x_{p+q+1}=0\}$ and choose the simple roots $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq p+q$). If we choose the simple roots $\beta_{i}=\alpha_{i}$ ($1\leq i\leq p$) for $\operatorname{\mathfrak{su}}(1,p)\subseteq\mathfrak{h}$, then $\beta_{i}=\alpha_{i}$ ($p+2\leq i\leq p+q$) are simple roots for $\operatorname{\mathfrak{su}}(q)\subseteq\mathfrak{h}$ and the fundamental weight $\varpi_{p+1}$ describes a character of $\mathfrak{u}(1)\subseteq\mathfrak{h}$.

Consider the dominant integral weight $\varpi_{1}+\varpi_{p+q}=e_{1}-e_{p+q+1}$. In the Weyl group orbit of $e_{1}-e_{p+q+1}$ the weight $e_{1}-e_{p+1}=\zeta_{1}+\zeta_{p}$ is a highest weight for $\mathfrak{h}$ and hence $(2(\varpi_{1}+\varpi_{p+q}),2(\zeta_{1}+\zeta_{p}))\in\Lambda(\mathfrak{g},\mathfrak{h})$. Further, from the Satake diagram for the real form $\operatorname{\mathfrak{su}}(p+1,q)$ of $\mathfrak{g}_{\mathbb{C}}\simeq\operatorname{\mathfrak{sl}}(p+q+1,\mathbb{C})$ it follows that $F^{\mathfrak{g}}(2(\varpi_{1}+\varpi_{p+q}))$ is $\mathfrak{h}$-spherical and hence also $(2(\varpi_{1}+\varpi_{p+q}),0)\in\Lambda(\mathfrak{g},\mathfrak{h})$. Clearly $(2(\varpi_{1}+\varpi_{p+q}),2(\zeta_{1}+\zeta_{p}))$ and $(2(\varpi_{1}+\varpi_{p+q}),0)$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $2$-dimensional.

E3) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{sp}}(1,p+q),\operatorname{\mathfrak{sp}}(1,p)+\operatorname{\mathfrak{sp}}(q)))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq p+q+1\}\cup\{\pm 2e_{i}:1\leq i\leq p+q+1\}$ and choose the simple roots $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq p+q$) and $\alpha_{p+q+1}=2e_{p+q+1}$. If we choose the simple roots $\beta_{i}=\alpha_{i}$ ($1\leq i\leq p$) and $\beta_{p+1}=2e_{p+1}$ for $\operatorname{\mathfrak{sp}}(1,p)\subseteq\mathfrak{h}$, then $\beta_{i}=\alpha_{i}$ ($p+2\leq i\leq p+q+1$) are simple roots for $\operatorname{\mathfrak{sp}}(q)\subseteq\mathfrak{h}$.

Consider the fundamental weight $\varpi_{2}=e_{1}+e_{2}$. Clearly $e_{1}+e_{2}=\zeta_{2}$ is also a highest weight for $\operatorname{\mathfrak{sp}}(1,p)\subseteq\mathfrak{h}$ and hence $(2\varpi_{2},2\zeta_{2})\in\Lambda(\mathfrak{g},\mathfrak{h})$. Further, from the Satake diagram for the real form $\operatorname{\mathfrak{sp}}(p+1,q)$ of $\mathfrak{g}_{\mathbb{C}}\simeq\operatorname{\mathfrak{sp}}(p+q+1,\mathbb{C})$, it follows that $F^{\mathfrak{g}}(2\varpi_{2})$ is $\mathfrak{h}$-spherical and hence also $(2\varpi_{2},0)\in\Lambda(\mathfrak{g},\mathfrak{h})$. Clearly $(2\varpi_{2},2\zeta_{2})$ and $(2\varpi_{2},0)$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $2$-dimensional.

E4) $(\mathfrak{g},\mathfrak{h})=(\mathfrak{f}_{4(-20)},\operatorname{\mathfrak{so}}(8,1))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

By [26] we have

[TABLE]

and hence $(2\varpi_{4},2\zeta_{1}),(2\varpi_{4},0)\in\Lambda(\mathfrak{g},\mathfrak{h})$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $2$-dimensional.

F1) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{sl}}(n+1,\mathbb{C}),\operatorname{\mathfrak{gl}}(n,\mathbb{C}))$

This follows from F3) with Lemma 2.11.

F2) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(n+1,\mathbb{C}),\operatorname{\mathfrak{so}}(n,\mathbb{C}))$

This follows from F5) with Lemma 2.11.

F3) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{sl}}(n+1,\mathbb{R}),\operatorname{\mathfrak{gl}}(n,\mathbb{R}))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We choose $\beta_{i}=\alpha_{i}$ ($1\leq i\leq n-1$) as simple roots for $\operatorname{\mathfrak{sl}}(n,\mathbb{R})\subseteq\mathfrak{h}$, then $\zeta_{n}:=\varpi_{n}$ defines a non-trivial character of $\mathbb{R}\subseteq\mathfrak{h}$. By Appendix A.1 the pairs $(2\varpi_{i},2\zeta_{i}+2\tfrac{i}{n}\zeta_{n})$ ($1\leq i\leq n-1$), $(2\varpi_{i},2\zeta_{i-1}-2\tfrac{n-i+1}{n}\zeta_{n})$ ($2\leq i\leq n$) and $(2\varpi_{1},-2\zeta_{n}),(2\varpi_{n},2\zeta_{n})$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and they clearly span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $2n$-dimensional.

F4) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{su}}(p,q+1),\mathfrak{u}(p,q))$

Set $s={\textup{min}}(p,q+1)$ and $t={\textup{min}}(p,q)$, then the Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We choose the simple roots $\beta_{i}=\alpha_{i}$ ($1\leq i\leq p+q-1$) for $\operatorname{\mathfrak{su}}(p,q)\subseteq\mathfrak{h}$, then $\zeta_{p+q}:=\varpi_{p+q}$ is trivial on $\operatorname{\mathfrak{su}}(p,q)$ and describes a character of $\mathfrak{u}(1)\subseteq\mathfrak{h}$. We use Appendix A.1 in what follows:

• •

($p\leq q$) Then $s=t=p$ and the pairs $(2(\varpi_{i}+\varpi_{p+q-i+1}),2(\zeta_{i}+\zeta_{p+q-i})),(2(\varpi_{i}+\varpi_{p+q-i+1}),2(\zeta_{i-1}+\zeta_{p+q-i+1}))$ ($1\leq i\leq p$) are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2p$.

• •

($p\geq q+1$) Then $s=q+1$ and $t=q$, so that the pairs $(2(\varpi_{i}+\varpi_{p+q-i+1}),2(\zeta_{i}+\zeta_{p+q-i})),(2(\varpi_{i}+\varpi_{p+q-i+1}),2(\zeta_{i-1}+\zeta_{p+q-i+1}))$ ($1\leq i\leq q$) and $(2(\varpi_{q+1}+\varpi_{p}),2(\zeta_{q}+\zeta_{p}))$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2q+1$.

F5) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(p,q+1),\operatorname{\mathfrak{so}}(p,q))$

For $p+q=2m$ even and $s={\textup{min}}(p,q+1)$, $t={\textup{min}}(p,q)$ the Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We use Appendix A.2 in what follows:

• •

($p+2<q$) Then $s=t=p\leq m-2$ and the pairs $(2\varpi_{i},2\zeta_{i}),(2\varpi_{i},2\zeta_{i-1})$ ($1\leq i\leq p$) are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2p$.

• •

($p+2=q$) Again $s=t=p$, but now $t=m-1$. The pairs $(2\varpi_{i},2\zeta_{i}),(2\varpi_{i},2\zeta_{i-1})$ ($1\leq i\leq p-1$) and $(2\varpi_{p},2(\zeta_{p}+\zeta_{p+1})),(2\varpi_{p},2\zeta_{p-1})$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2p$.

• •

($p=q$) Then $s=t=m=p$ and the pairs $(2\varpi_{i},2\zeta_{i}),(2\varpi_{i},2\zeta_{i-1})$ ($1\leq i\leq p-2$) and $(2\varpi_{p-1},2\zeta_{p-2}),(2\varpi_{p-1},2(\zeta_{p-1}+\zeta_{p})),(2\varpi_{p},2\zeta_{p}),(2\varpi_{p},2\zeta_{p-1})$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2p$.

• •

($p=q+2$) Here, $s=q+1$, $t=q$ and $t<s=m$. The pairs $(2\varpi_{i},2\zeta_{i}),(2\varpi_{i},2\zeta_{i-1})$ ($1\leq i\leq q-1$) and $(2\varpi_{q},2\zeta_{q-1}),(2\varpi_{q},2(\zeta_{q}+\zeta_{q+1})),(4\varpi_{q+1},2(\zeta_{q}+\zeta_{q+1}))$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2q+1$.

• •

($p>q+2$) Again $s=q+1$, $t=q$, but now $t<s<m$. The pairs $(2\varpi_{i},2\zeta_{i}),(2\varpi_{i},2\zeta_{i-1})$ ($1\leq i\leq q$) and $(2\varpi_{q+1},2\zeta_{q})$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+t=2q+1$.

The case $p+q=2m-1$ odd is treated similarly.

G1) $(\mathfrak{g},\mathfrak{h})=(\mathfrak{g}^{\prime}+\mathfrak{g}^{\prime},\operatorname{diag}\mathfrak{g}^{\prime})$ with $\mathfrak{g}^{\prime}$ simple compact

This is the same situation as in C).

G2) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(1,n)+\operatorname{\mathfrak{so}}(1,n),\operatorname{diag}\operatorname{\mathfrak{so}}(1,n))$

The Satake diagram of $\operatorname{\mathfrak{so}}(1,n)$ is

[TABLE]

The spherical representations of $\operatorname{\mathfrak{so}}(1,n)$ are of the form $F^{\operatorname{\mathfrak{so}}(1,n)}(2k\varpi_{1})$, $k\in\mathbb{N}$, and hence the spherical representations of $\mathfrak{g}$ are of the form $F^{\operatorname{\mathfrak{so}}(1,n)}(2k_{1}\varpi_{1})\boxtimes F^{\operatorname{\mathfrak{so}}(n,1)}(2k_{2}\varpi_{1})$, $k_{1},k_{2}\in\mathbb{N}$. The representation $F^{\operatorname{\mathfrak{so}}(1,n)}(\varpi_{1})$ is self-dual, and hence the trivial representation $F^{\operatorname{\mathfrak{so}}(1,n)}(0)$ is contained in the tensor product $F^{\operatorname{\mathfrak{so}}(1,n)}(\varpi_{1})\otimes F^{\operatorname{\mathfrak{so}}(1,n)}(\varpi_{1})$. Further, clearly

[TABLE]

Hence, $((2\varpi_{1},0),2\varpi_{1}),((0,2\varpi_{1}),2\varpi_{1}),((2\varpi_{1},2\varpi_{1}),0)\in\Lambda(\mathfrak{g},\mathfrak{h})$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is $3$-dimensional.

H1) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}(2,2n),\mathfrak{u}(1,n))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq n+1\}$ with simple roots $\alpha_{i}=e_{i}-e_{i+1}$, $i=1,\ldots,n$ and $\alpha_{n+1}=e_{n}+e_{n+1}$. Choose the simple roots $\beta_{i}=\alpha_{i}$ ($i=1,\ldots,n$) for $\operatorname{\mathfrak{su}}(1,n)\subseteq\mathfrak{h}$; then $\zeta_{n+1}:=\varpi_{n+1}=\frac{1}{2}(e_{1}+\cdots+e_{n+1})$ defines a non-trivial character of $\mathfrak{u}(1)\subseteq\mathfrak{h}$.

First consider the fundamental weight $\varpi_{2}=e_{1}+e_{2}$; then the Weyl group orbit of $\varpi_{2}$ is equal to the set of roots $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq n+1\}$. Hence, the weights $e_{1}+e_{2}$, $-e_{n}-e_{n+1}$ and $e_{1}-e_{n+1}$ of $F^{\mathfrak{g}}(\varpi_{2})$ are highest weights for $\mathfrak{h}$ so that the representations $F^{\mathfrak{h}}(e_{1}+e_{2})$, $F^{\mathfrak{h}}(-e_{n}-e_{n+1})$ and $F^{\mathfrak{h}}(e_{1}-e_{n+1})$ are contained in $F^{\mathfrak{g}}(\varpi_{2})|_{\mathfrak{h}}$. Note that $e_{1}-e_{n+1}=\zeta_{1}+\zeta_{n}$. Using the Weyl dimension formula we find

[TABLE]

Using the Kostant Branching Formula it is further easy to see that the remaining one-dimensional representation in $F^{\mathfrak{g}}(\varpi_{2})|_{\mathfrak{h}}$ is the trivial representation so that $(2\varpi_{2},2(\zeta_{1}+\zeta_{n})),(2\varpi_{2},0)\in\Lambda(\mathfrak{g},\mathfrak{h})$.

Next consider the highest weight $2\varpi_{1}=2e_{1}$ of $\mathfrak{g}$. Similar considerations to above show that $F^{\mathfrak{g}}(2\varpi_{1})|_{\mathfrak{h}}$ contains $F^{\mathfrak{h}}(\zeta_{1}+\zeta_{n})$ so that $(4\varpi_{1},2(\zeta_{1}+\zeta_{n}))\in\Lambda(\mathfrak{g},\mathfrak{h})$.

Together the three pairs $(4\varpi_{1},2(\zeta_{1}+\zeta_{n})),(2\varpi_{2},2(\zeta_{1}+\zeta_{n})),(2\varpi_{2},0)\in\Lambda(\mathfrak{g},\mathfrak{h})$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ which is $3$-dimensional.

H2) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{su}}^{*}(2n+2),\operatorname{\mathfrak{su}}^{*}(2n)+\mathbb{R}+\operatorname{\mathfrak{su}}(2))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm(e_{i}-e_{j}):1\leq i<j\leq 2n+2\}$ in the vector space $V=\{x\in\mathbb{R}^{2n+2}:x_{1}+\cdots+x_{2n+2}=0\}$. To simplify notation, denote by $\pi(x)$ the orthogonal projection of $x\in\mathbb{R}^{2n+2}$ to $V$. We choose the simple roots $\alpha_{i}=e_{i}-e_{i+1}$ for $\mathfrak{g}$ and the simple roots $\beta_{i}=\alpha_{i}$ ($i=1,\ldots,2n-1$) for $\operatorname{\mathfrak{su}}^{*}(2n)$. Then $\zeta_{2n}:=\varpi_{2n}$ describes a character of $\mathbb{R}\subseteq\mathfrak{h}$ and $\alpha_{2n+1}$ is the non-trivial root for $\operatorname{\mathfrak{su}}(2)\subseteq\mathfrak{h}$. Let $\zeta_{2n+1}:=\tfrac{1}{2}\alpha_{2n+1}$ denote the fundamental weight for $\operatorname{\mathfrak{su}}(2)$. Then a general irreducible representation of $\mathfrak{h}$ takes the form $F^{\operatorname{\mathfrak{su}}^{*}(2n)}(\ell_{1}\zeta_{1}+\cdots+\ell_{2n-1}\zeta_{2n-1})\boxtimes F^{\mathbb{R}}(\ell_{2n}\zeta_{2n})\boxtimes F^{\operatorname{\mathfrak{su}}(2)}(\ell_{2n+1}\zeta_{2n+1})$ with $\ell_{1},\ldots,\ell_{2n-1},\ell_{2n+1}\in\mathbb{N}$ and $\ell_{2n}\in\mathbb{C}$.

Consider the fundamental weight $\varpi_{i}=\pi(e_{1}+\cdots+e_{i})=\tfrac{2n-i+2}{2n+2}(e_{1}+\cdots+e_{i})-\tfrac{i}{2n+2}(e_{i+1}+\cdots+e_{2n+2})$ of $\mathfrak{g}$. In the Weyl group orbit of $\varpi_{i}$ the three weights $\pi(e_{1}+\cdots+e_{i})$, $\pi(e_{1}+\cdots+e_{i-1}+e_{2n+1})$ and $\pi(e_{1}+\cdots+e_{i-2}+e_{2n+1}+e_{2n+2})$ are highest weights for $\mathfrak{h}$. Moreover,

[TABLE]

and

[TABLE]

so that the representations

[TABLE]

are contained in $F^{\mathfrak{g}}(\varpi_{i})|_{\mathfrak{h}}$. Using the Weyl Dimension Formula we find

[TABLE]

This implies that

[TABLE]

are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$, and since $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ is $2n$-dimensional, they form a basis.

H3) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{so}}^{*}(2n+2),\operatorname{\mathfrak{so}}^{*}(2n)+\operatorname{\mathfrak{so}}(2))$

The Satake diagrams of $\mathfrak{g}$ and $\mathfrak{h}$ are

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq n+1\}$ with simple roots $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq n$) and $\alpha_{n+1}=e_{n}+e_{n+1}$. Then we may choose the simple roots for $\mathfrak{h}$ as $\beta_{i}=\alpha_{i+1}$ ($1\leq i\leq n$). The fundamental weight $\zeta_{n+1}=\varpi_{1}=e_{1}$ is trivial on $\operatorname{\mathfrak{so}}^{*}(2n)\subseteq\mathfrak{h}$ and hence defines a non-trivial character of $\operatorname{\mathfrak{so}}(2)\subseteq\mathfrak{h}$.

Consider the fundamental weight $\varpi_{i}=e_{1}+\cdots+e_{i}$, $i=1,\ldots,n-1$. By branching in stages, first from $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{so}}(2n+2,\mathbb{C})$ to $\operatorname{\mathfrak{so}}(2n+1,\mathbb{C})$, then from $\operatorname{\mathfrak{so}}(2n+1,\mathbb{C})$ to $\operatorname{\mathfrak{so}}(2n,\mathbb{C})=\operatorname{\mathfrak{so}}^{*}(2n)_{\mathbb{C}}$, it follows that

[TABLE]

To find the action of the $\operatorname{\mathfrak{so}}(2)$-factor we consider the weights of $F^{\mathfrak{g}}(\varpi_{i})$ which are given by

[TABLE]

Among these, clearly $e_{2}+\cdots+e_{i+1}=\zeta_{i}$, $\pm e_{1}+e_{2}+\cdots+e_{i}=\pm\zeta_{n+1}+\zeta_{i-1}$ and $e_{2}+\cdots+e_{i-1}=\zeta_{i-2}$ are highest weights for $\mathfrak{h}$ and hence

[TABLE]

Similarly, for the highest weight $\varpi_{n}+\varpi_{n+1}=e_{1}+\cdots+e_{n}$ one obtains

[TABLE]

and for the highest weight $2\varpi_{n+1}=e_{1}+\cdots+e_{n+1}$, accordingly

[TABLE]

Now, if $n=2s-1$ is odd, then

[TABLE]

are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+(s-1)=2s-1$. If $n=2s$ is even, then

[TABLE]

are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $s+s=2s$.

H4) $(\mathfrak{g},\mathfrak{h})=(\operatorname{\mathfrak{sp}}(p,q+1),\operatorname{\mathfrak{sp}}(p,q)+\operatorname{\mathfrak{sp}}(1))$

Let $s={\textup{min}}(p,q+1)$ and $t={\textup{min}}(p,q)$; then the Satake diagrams of $\mathfrak{g}$ and $\operatorname{\mathfrak{sp}}(p,q)$ are

[TABLE]

[TABLE]

We realize the root system of $\mathfrak{g}$ as $\{\pm e_{i}\pm e_{j}:1\leq i<j\leq p+q+1\}\cup\{\pm 2e_{i}:1\leq i\leq p+q+1\}$ and choose $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq p+q$) and $\alpha_{p+q+1}=2e_{p+q+1}$ as simple roots. We further choose $\beta_{i}=\alpha_{i}$ ($1\leq i\leq p+q-1$) and $\beta_{p+q}=2e_{p+q}$ as simple roots for $\operatorname{\mathfrak{sp}}(p,q)\subseteq\mathfrak{h}$; then $\beta_{p+q+1}=\alpha_{p+q+1}$ is a simple root for $\operatorname{\mathfrak{sp}}(1)$ and $\zeta_{p+q+1}=\varpi_{p+q+1}=e_{p+q+1}$ the corresponding fundamental weight. A general irreducible representation of $\mathfrak{h}$ takes the form $F^{\operatorname{\mathfrak{sp}}(p,q)}(\ell_{1}\zeta_{1}+\cdots+\ell_{p+q}\zeta_{p+q})\boxtimes F^{\operatorname{\mathfrak{sp}}(1)}(\ell_{p+q+1}\zeta_{p+q+1})$ with $\ell_{1},\ldots,\ell_{p+q+1}\in\mathbb{N}$.

We consider the fundamental weight $\varpi_{i}=e_{1}+\cdots+e_{i}$. Then by [45, Theorem 3.3] we have

[TABLE]

for $i=1,\ldots,p+q$, and

[TABLE]

Hence, the following pairs in $\Lambda(\mathfrak{g},\mathfrak{h})$ span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$:

• •

($p\leq q$) Then $s=t=p$, so that $(2\varpi_{2i},2\zeta_{2i}),(2\varpi_{2i},2\zeta_{2i-2})$ ($1\leq i\leq p$) are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$.

• •

($p\geq q+1$) Then $s=q+1$ and $t=q$, so that $(2\varpi_{2i},2\zeta_{2i}),(2\varpi_{2i},2\zeta_{2i-2})$ ($1\leq i\leq q$) and $(2\varpi_{2q+2},2\zeta_{2q})$ are contained in $\Lambda(\mathfrak{g},\mathfrak{h})$ and span $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$.

H5) $(\mathfrak{g},\mathfrak{h})=(\mathfrak{e}_{6(-26)},\operatorname{\mathfrak{so}}(9,1)+\mathbb{R})$

The Satake diagrams of $\mathfrak{g}$ and $\operatorname{\mathfrak{so}}(9,1)\subseteq\mathfrak{h}$ are

[TABLE]

[TABLE]

We choose the simple roots for $\mathfrak{g}$ such that $\alpha_{1}=\beta_{1}$, $\alpha_{2}=\beta_{4}$, $\alpha_{3}=\beta_{2}$, $\alpha_{4}=\beta_{3}$, $\alpha_{5}=\beta_{5}$, then $\zeta_{6}:=\varpi_{6}$ defines a non-trivial character of $\mathbb{R}\subseteq\mathfrak{h}$. A general irreducible representation of $\mathfrak{h}$ is of the form $F^{\operatorname{\mathfrak{so}}(9,1)}(\ell_{1}\zeta_{1}+\cdots+\ell_{5}\zeta_{5})\boxtimes F^{\mathbb{R}}(\ell_{6}\zeta_{6})$ for $\ell_{1},\ldots,\ell_{5}\in\mathbb{N}$ and $\ell_{6}\in\mathbb{C}$.

By writing $\varpi_{i}$ as a linear combination of the simple roots $\alpha_{j}$ (see e.g. [13, Appendix C.2], and writing $\zeta_{i}$ as a linear combination of the simple roots $\beta_{j}$, we find that

[TABLE]

The spherical representations of $\mathfrak{g}$ have highest weights $2k_{1}\varpi_{1}+2k_{2}\varpi_{6}$, $k_{1},k_{2}\in\mathbb{N}$, and the spherical representations of $\mathfrak{h}$ have highest weights $2\ell_{1}\zeta_{1}+\ell_{2}\zeta_{6}$, $\ell_{1}\in\mathbb{N}$ and $\ell_{2}\in\mathbb{C}$.

By [26] we have

[TABLE]

To determine the action of the center of $\mathfrak{h}$ on each summand we employ the weight space decomposition (using e.g. LiE) and find that the weights of $F^{\mathfrak{g}}(\varpi_{1})$ which are highest weights for $\operatorname{\mathfrak{so}}(9,1)$ are

[TABLE]

so that

[TABLE]

Taking contragredient representations further gives

[TABLE]

Hence,

[TABLE]

and they clearly generate $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, which is of dimension $4$.

3. Lower multiplicity bounds – Construction of symmetry breaking operators

We use the results of Sections 1 and 2 to construct for every strongly spherical reductive pair $(G,H)$ intertwining operators between spherical principal series representations of $G$ and $H$ in terms of their distribution kernels (see Theorem 3.3).

3.1. Principal series representations and symmetry breaking operators

For $(\xi,V_{\xi})\in\widehat{M}_{G}$ and $\lambda\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}$ we define the principal series representation

[TABLE]

realized as the left-regular representation on the space $C^{\infty}(G/P,\mathcal{V}_{\xi,\lambda})$ of smooth sections of the vector bundle $\mathcal{V}_{\xi,\lambda}=G\times_{P_{G}}V_{\xi,\lambda}\to G/P_{G}$ associated to the representation $V_{\xi,\lambda}=\xi\otimes e^{\lambda+\rho_{\mathfrak{n}_{G}}}\otimes{\rm\bf 1}$. Here, $\rho_{\mathfrak{n}_{G}}=\frac{1}{2}\operatorname{tr}\operatorname{ad}_{\mathfrak{n}_{G}}\in\mathfrak{a}_{G}^{\vee}$, the half sum of all positive roots. Denote by $\mathcal{V}_{\xi,\lambda}^{*}=\mathcal{V}_{\xi^{\vee},-\lambda}$ the dual bundle of $\mathcal{V}_{\xi,\lambda}$.

Similarly, we define for $(\eta,W_{\eta})\in\widehat{M}_{H}$ and $\nu\in\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ principal series representations of $H$ by

[TABLE]

and write $W_{\eta,\nu}=\eta\otimes e^{\nu+\rho_{\mathfrak{n}_{H}}}\otimes{\rm\bf 1}$.

Consider the space

[TABLE]

of intertwining operators between principal series of $G$ and $H$, also referred to as symmetry breaking operators by Kobayashi [18]. By the Schwartz Kernel Theorem, symmetry breaking operators can be studied in terms of their distribution kernels.

Proposition 3.1** ([22, Proposition 3.2]).**

The natural map

[TABLE]

which is characterized by $A\varphi(h)=\langle K^{A},\varphi(h\operatorname{\hskip 0.50003pt\cdot\hskip 0.50003pt})\rangle$, is an isomorphism.

As in [22] we use generalized functions rather than distributions, so that $\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})$ can be identified with the dual space of $C_{c}^{\infty}(G/P_{G},\mathcal{V}_{\xi,\lambda})$ and $L^{1}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})\hookrightarrow\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})$.

To be able to apply the theory of Bernstein–Sato identities, we first translate line bundle valued distributions on $G/P_{G}$ to scalar-valued distributions on $G$. Consider the surjective linear map

[TABLE]

where $dp$ denotes a left-invariant measure on $P_{G}$. Then its transpose $\flat^{\top}:\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})\to\mathcal{D}^{\prime}(G)\otimes V_{\xi}^{\vee}$ is injective and embeds $(\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})\otimes W_{\eta,\nu})^{P_{H}}$ into

[TABLE]

We show that this map is actually an isomorphism:

Lemma 3.2**.**

The natural map

[TABLE]

is an isomorphism.

Proof.

It suffices to show that the map is surjective, so let $u\in\mathcal{D}^{\prime}(G)_{(\xi,\lambda),(\eta,\nu)}$. Write $g=kan\in KA_{G}N_{G}=G$ for some maximal compact subgroup $K\subseteq G$ by the Iwasawa decomposition; then the distribution $a^{-(\lambda-\rho_{G})}u(g)$ is right $A_{G}N_{G}$-invariant and therefore of the form $v\otimes 1\otimes 1$ according to the decomposition $G\simeq K\times A_{G}\times N_{G}$ with $v\in\mathcal{D}^{\prime}(K)\otimes\operatorname{Hom}_{\mathbb{C}}(V_{\xi},W_{\eta})=(\mathcal{D}^{\prime}(K)\otimes V_{\xi}^{\vee})\otimes W_{\eta}$. Define $w\in\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}^{*}_{\xi,\lambda})\otimes W_{\eta,\nu}$ by

[TABLE]

then it is easy to show that $w$ is $P_{H}$-invariant and maps to $u$ via the map $\flat^{\top}$. ∎

Now let us first consider the case where both $\xi\in\widehat{M}_{G}$ and $\eta\in\widehat{M}_{H}$ are the trivial representation. We use the notation

[TABLE]

For the statement of the main result of this section we denote by $\mathfrak{a}_{G,+}^{\vee}$ and $\mathfrak{a}_{H,+}^{\vee}$ the positive Weyl chambers corresponding to $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ and $\Sigma^{+}(\mathfrak{h},\mathfrak{a}_{H})$.

Theorem 3.3**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for every open double coset $\Omega=P_{H}gP_{G}\in P_{H}\backslash G/P_{G}$ there exists a family of distributions $K_{\lambda,\nu}\in\mathcal{D}^{\prime}(G)$, unique up to scalar multiples and depending holomorphically on $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$, such that

1. (1)

$K_{\lambda,\nu}\in\mathcal{D}^{\prime}(G)_{\lambda,\nu}$* and $\operatorname{supp}K_{\lambda,\nu}\subseteq\overline{\Omega}$ for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$,* 2. (2)

$K_{\lambda,\nu}\in L^{1}_{\rm loc}(G)$* and $\operatorname{supp}K_{\lambda,\nu}=\overline{\Omega}$ for $\operatorname{Re}(\lambda-\rho_{\mathfrak{n}_{G}})\in\mathfrak{a}_{G,+}^{\vee}$ and $\operatorname{Re}(\nu+\rho_{\mathfrak{n}_{H}})\in-\mathfrak{a}_{H,+}^{\vee}$.*

By holomorphic dependence on the parameters $(\lambda,\nu)$ we mean that for every $\varphi\in C^{\infty}(G)$ the map $(\lambda,\nu)\mapsto\langle K_{\lambda,\nu},\varphi\rangle$ is a holomorphic function on $\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$.

The proof of Theorem 3.3 is carried out in Section 3.3.

3.2. Lower bounds for multiplicities

Before we come to the proof, let us observe that Theorem 3.3 implies lower bounds for multiplicities of intertwining operators. For this denote by $(P_{H}\backslash G/P_{G})_{\rm open}\subseteq P_{H}\backslash G/P_{G}$ the collection of open double cosets and by $\#(P_{H}\backslash G/P_{G})_{\rm open}$ its cardinality (which is finite by Proposition 1.2).

Corollary 3.4**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

Proof.

We choose representatives $x_{1},\ldots,x_{n}\in G$ for the open double cosets in $P_{H}\backslash G/P_{G}$. By Theorem 3.3 there exist holomorphic families of distributions $K_{\lambda,\nu}^{i}$, $1\leq i\leq n$, such that $\operatorname{supp}K_{\lambda,\nu}^{i}=\overline{P_{H}x_{i}P_{G}}$ for $\operatorname{Re}(\lambda-\rho_{\mathfrak{n}_{G}})\in\mathfrak{a}_{G,+}^{\vee}$ and $\operatorname{Re}(\nu+\rho_{\mathfrak{n}_{H}})\in-\mathfrak{a}_{H,+}^{\vee}$. Let $\lambda_{+}\in\mathfrak{a}_{G,+}^{\vee}$ and $\nu_{+}\in\mathfrak{a}_{H,+}^{\vee}$; then for fixed $(\lambda_{0},\nu_{0})\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have $\operatorname{Re}(\lambda_{0}+z\lambda_{+})\in\mathfrak{a}_{G,+}^{\vee}$ and $\operatorname{Re}(\nu_{0}-z\nu_{+})\in-\mathfrak{a}_{H,+}^{\vee}$ for $z\in\mathbb{C}$, $\operatorname{Re}(z)\gg 0$. This implies that the holomorphic functions

[TABLE]

satisfy $\operatorname{supp}f_{i}(z)=\overline{P_{H}x_{i}P_{G}}$ whenever $\operatorname{Re}(z)\gg 0$. Hence, $f_{1}(z),\ldots,f_{n}(z)$ are generically linearly independent and the claim follows from the next lemma combined with Proposition 3.1 and Lemma 3.2. ∎

Lemma 3.5**.**

Let $f_{1},\ldots,f_{n}:\mathbb{C}\to V$ be holomorphic functions with values in a complete locally convex topological vector space $V$. If $f_{1}(z),\ldots,f_{n}(z)$ are generically linearly independent, then there exists $A=(a_{ij})\in\operatorname{GL}(n,\mathbb{C})$ and $m_{1},\ldots,m_{n}\geq 0$ such that the functions

[TABLE]

are holomorphic in $z=0$ and $g_{1}(0),\ldots,g_{n}(0)$ are linearly independent.

Proof.

Rearrange $f_{1},\ldots,f_{n}$ such that $f_{1}(0),\ldots,f_{p}(0)$ are linearly independent and the remaining $f_{p+1}(0),\ldots,f_{n}(0)$ are linear combinations of $f_{1}(0),\ldots,f_{p}(0)$. We now show that $f_{p+1}(z)$ can be replaced by a renormalized linear combination of $f_{1}(z),\ldots,f_{p+1}(z)$ such that $f_{1}(0),\ldots,f_{p+1}(0)$ are linearly independent. Applying this argument recursively to $f_{p+1},\ldots,f_{n}$ shows the statement.

So assume that

[TABLE]

We form the new function $f_{p+1}^{1}(z)=f_{p+1}(z)-\sum_{i=1}^{p}\lambda_{i}^{0}f_{i}(z)$; then $f_{p+1}^{1}(0)=0$ and renormalizing

[TABLE]

gives a holomorphic function $\tilde{f}_{p+1}^{1}(z)$ such that $f_{1}(z),\ldots,f_{p}(z),\tilde{f}_{p+1}^{1}(z)$ are still generically linearly independent. If now $f_{1}(0),\ldots,f_{p}(0),\tilde{f}_{p+1}^{1}(0)$ are linearly independent, we are done, otherwise we have

[TABLE]

As before, we form $f_{p+1}^{2}(z)=\tilde{f}_{p+1}^{1}(z)-\sum_{i=1}^{p}\lambda_{i}^{1}f_{i}(z)$; then $f_{p+1}^{2}(0)=0$ and renormalizing

[TABLE]

gives a holomorphic function $\tilde{f}_{p+1}^{2}(z)$ such that $f_{1}(z),\ldots,f_{p}(z),\tilde{f}_{p+1}^{2}(z)$ are still generically linearly independent. We repeat this procedure as long as possible. If the procedure does not terminate, we obtain $\lambda_{i}^{0},\lambda_{i}^{1},\lambda_{i}^{2},\ldots\in\mathbb{C}$, $1\leq i\leq p$, and holomorphic functions $\tilde{f}_{p+1}^{k}(z)$ with

[TABLE]

where we put $\tilde{f}_{p+1}^{0}=f_{p+1}$. Form the holomorphic functions (the convergence of the sums follows by applying continuous linear functionals to the above identities)

[TABLE]

then

[TABLE]

for all $z\in\mathbb{C}$. But this implies that $f_{1}(z),\ldots,f_{p}(z),f_{p+1}(z)$ are linearly dependent for all $z\in\mathbb{C}$, contradicting the assumption. Hence, the procedure has to terminate at some stage, which implies that $f_{1}(0),\ldots,f_{p}(0),\tilde{f}_{p+1}^{k}(0)$ are linearly independent for some $k$, proving our claim. ∎

3.3. Bernstein–Sato identities and meromorphic extension

Write

[TABLE]

By Proposition 2.5 and Theorem 2.6 there exists a basis $\{(\lambda_{j},\nu_{j})\}_{j=1,\ldots,r}\subseteq\Lambda^{+}(\mathfrak{g},\mathfrak{a})\times\Lambda^{+}(\mathfrak{h},\mathfrak{a}_{H})$ of $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ and non-zero real-analytic functions $F_{j}:G\to\mathbb{R}$, $F_{j}\geq 0$, such that

[TABLE]

for $g\in G$, $man\in P_{G}$ and $m^{\prime}a^{\prime}n^{\prime}\in P_{H}$.

Now, for $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$ with $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})\geq 0$ we define

[TABLE]

Let $\Omega\in P_{H}\backslash G/P_{G}$ be an open double coset and write $\chi_{\Omega}:G\to\{0,1\}$ for its characteristic function. Then it remains to show that $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}$ defines a family of distributions on $G$ which extends meromorphically in $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$. For this we use Bernstein–Sato identities.

By [38, 39] there exists a differential operator $D_{s_{1},\ldots,s_{r}}$ on $G$ depending polynomially on $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$ such that

[TABLE]

where

[TABLE]

with $(a_{i,1},\ldots,a_{i,r})\in\mathbb{C}^{r}\setminus\{0\}$, $b_{i}\in\mathbb{C}$ and $c\neq 0$. This identity immediately shows the meromorphic extension of the family $K_{s_{1},\ldots,s_{r}}\in\mathcal{D}^{\prime}(G)$. However, if we want to meromorphically extend $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}$, we have to control $K_{s_{1},\ldots,s_{r}}$ at the boundary $\partial\Omega$ of $\Omega$.

Lemma 3.6**.**

For every double coset $\omega\subseteq\partial\Omega$ there exists $1\leq j\leq r$ such that

[TABLE]

Proof.

Let $g\in\omega$; then by the equivariance property (3.1) it suffices to show that $F_{j}(g)=0$ for some $1\leq j\leq r$. Since $\omega\subseteq\partial\Omega$, the double coset $P_{H}gP_{G}$ is not open. Using Theorem 1.3 we find an element $Z=Z_{M}+Z_{A}+Z_{N}\in\mathfrak{p}_{H}$ with $Z_{A}\neq 0$ such that $X=\operatorname{Ad}(g^{-1})Z=X_{M}+X_{A}+X_{N}\in\mathfrak{p}_{G}$. This implies

[TABLE]

for all $t\in\mathbb{R}$. Now assume that $F_{j}(g)\neq 0$ for all $1\leq j\leq r$; then $K_{s_{1},\ldots,s_{r}}(g)\neq 0$ for all $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$ with $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})\geq 0$. Hence

[TABLE]

or equivalently

[TABLE]

for $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$ with $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})\geq 0$. This implies $\lambda_{j}(X_{A})+\nu_{j}^{\vee}(Z_{A})=0$ for all $1\leq j\leq r$. But since the pairs $(\lambda_{j},\nu_{j})$ form a basis of $\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$, the pairs $(\lambda_{j},\nu_{j}^{\vee})$ also form a basis and hence $Z_{A}=0$ which gives a contradiction. ∎

By the previous lemma we have $K_{s_{1},\ldots,s_{r}}(g)=0$ whenever $g\in\partial\Omega$ and $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})>0$. This implies a regularity statement for the functions $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}$:

Lemma 3.7**.**

Let $M$ be a smooth manifold, $\Omega\subseteq M$ open and $\varphi\in C^{\infty}(M)$ with $\varphi\geq 0$, $\varphi|_{\partial\Omega}=0$. Then $\chi_{\Omega}\varphi^{s}\in C^{k}(\mathbb{R})$ whenever $\operatorname{Re}s>k$.

The proof is an easy calculus exercise. We can now proceed to prove Theorem 3.3.

Proof of Theorem 3.3.

Let $\Omega\in P_{H}\backslash G/P_{G}$ be an open double coset, then $K_{s_{1},\ldots,s_{r}}|_{\partial\Omega}=0$ for $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})>0$ by Lemma 3.6. By Lemma 3.7 this implies that $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}\in C^{k}(G)$ for $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})>k$. Since the Bernstein–Sato operator $D_{s_{1},\ldots,s_{r}}$ is of finite order, say $k\geq 0$, this implies that $D_{s_{1},\ldots,s_{r}}[\chi_{\Omega}K_{s_{1},\ldots,s_{r}}]$ is a continuous function for $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})>k$. Replacing $K_{s_{1},\ldots,s_{r}}$ in the Bernstein–Sato identity (3.2) by $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}$ yields

[TABLE]

which holds on $\Omega$ by (3.2), and trivially also on $K\setminus\overline{\Omega}$ since both sides vanish. As observed above, for $\operatorname{Re}(s_{1}),\ldots,\operatorname{Re}(s_{r})>k$ both sides are continuous functions, and hence (3.3) holds everywhere. This shows the meromorphic continuation of $\chi_{\Omega}K_{s_{1},\ldots,s_{r}}\in\mathcal{D}^{\prime}(G)$ to $(s_{1},\ldots,s_{r})\in\mathbb{C}^{r}$, or more precisely the holomorphic continuation of

[TABLE]

Finally, the change of variables

[TABLE]

constructs the desired family $K_{\lambda,\nu}$. The equivariance property extends by analytic continuation. ∎

4. Upper multiplicity bounds – Invariant distributions

In this section we establish upper bounds for the multiplicities $\dim\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})$ using Bruhat’s theory of invariant distributions. In particular, we obtain that for $\xi$ and $\eta$ the trivial representations, the intertwining operators constructed in Section 3 form a basis of $\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$ for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$.

4.1. Upper bounds for multiplicities

Let $M\subseteq M_{G}\cap M_{H}$ denote the stabilizer of a generic element $X\in\mathfrak{n}^{-\sigma}$. More precisely,

[TABLE]

where $X\in\mathfrak{n}^{-\sigma}$ with $X=\sum_{\beta\in\Delta(\mathfrak{n}^{-\sigma})}X_{\beta}$, $X_{\beta}\in\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)\setminus\{0\}$. Since for every $\beta\in\Delta(\mathfrak{n}^{-\sigma})$, the group $M_{G}\cap M_{H}$ acts either transitively on the unit sphere $S_{\beta}\subseteq\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)$ or with possibly two orbits if $\dim\mathfrak{g}^{-\sigma}(\mathfrak{a}_{H};\beta)=1$, it follows that for generic $X,Y\in\mathfrak{n}^{-\sigma}$ the stabilizers $(M_{G}\cap M_{H})^{X}$ and $(M_{G}\cap M_{H})^{Y}$ are conjugate in $M_{G}\cap M_{H}$.

The main statement of this section is the following theorem:

Theorem 4.1**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ satisfying the generic condition

[TABLE]

we have

[TABLE]

In particular, for $\xi$ and $\eta$ the trivial representations, we have generically

[TABLE]

Here $\xi^{g}$ denotes the representation of $gM_{G}g^{-1}$ given by $\xi^{g}(m)=\xi(g^{-1}mg)$.

Remark 4.2**.**

If $\mathfrak{m}_{G}\cap\mathfrak{m}_{H}=\{0\}$ then condition (4.1) in Theorem 4.1 can be replaced by the weaker condition

[TABLE]

This follows from the proof in Section 4.2.2.

Remark 4.3**.**

From [21, Corollary 2.7 and Lemma 5.2] one can deduce the following estimate:

[TABLE]

for $\lambda\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}$ and $-\nu\in\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ satisfying a certain regularity and a positivity condition. Note that Theorem 4.1 only requires a regularity condition and no positivity condition. Here $W(\mathfrak{a}_{G}\oplus\mathfrak{a}_{H})\simeq W(\mathfrak{a}_{G})\times W(\mathfrak{a}_{H})$ denotes the Weyl group of the pair $(\mathfrak{g}\oplus\mathfrak{h},\mathfrak{a}_{G}\oplus\mathfrak{a}_{H})$ and by [21, Corollary E] we have $\#W(\mathfrak{a}_{G}\oplus\mathfrak{a}_{H})\geq\#(P_{H}\backslash G/P_{G})_{\rm open}$. Note that this inequality is not sharp. For instance, for the multiplicity one pairs in Fact II we have $\#(P_{H}\backslash G/P_{G})_{\rm open}=1$ (see Lemma 6.3) while the order of the Weyl group $W(\mathfrak{a}_{G}\oplus\mathfrak{a}_{H})$ goes to infinity as the rank increases. Therefore, the estimate in Theorem 4.1 is sharper than the one derived from [21] and holds for an open dense subset of parameters $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$, whereas [21] requires the parameters to be contained in some Weyl chamber.

Before we prove the theorem, note that combined with Corollary 3.4 it implies the following formula for the generic multiplicities:

Corollary 4.4**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ satisfying the generic condition (4.1) we have

[TABLE]

The rest of this section is devoted to the proof of Theorem 4.1.

4.2. Invariant distributions

Recall from Proposition 3.1 and Lemma 3.2 that

[TABLE]

Upper bounds for spaces of invariant distributions such as $\mathcal{D}^{\prime}(G)_{(\xi,\lambda),(\eta,\nu)}$ are provided by Bruhat’s theory of invariant distributions (see e.g. [46, Chapter 5.2] for a detailed account; see also [29, Section 3.4] for an application in a similar setting). This method applies to our situation since the group $P_{H}\times P_{G}$ that acts on $G$ has only finitely many orbits. In this case, Bruhat’s theory implies the following upper bound:

[TABLE]

where for a Lie group $S$ we denote by $\delta_{S}(x)=|\det\operatorname{Ad}(x^{-1})|$, $x\in S$, its modular function, and $V(g)=\mathfrak{g}/(\operatorname{Ad}(g)\mathfrak{p}_{G}+\mathfrak{p}_{H})$. Clearly $\delta_{P_{G}}(man)=a^{-2\rho_{\mathfrak{n}_{G}}}$ and $\delta_{P_{H}}(man)=a^{-2\rho_{\mathfrak{n}_{H}}}$. We estimate the contributions from open and non-open orbits separately.

4.2.1. Open orbits

If $P_{H}gP_{G}$ is an open orbit, we may choose $g=e^{X}\tilde{w}_{0}$ with $X\in\mathfrak{n}^{-\sigma}$ generic. Then $P_{H}\cap gP_{G}g^{-1}=M$ and $(\xi\otimes e^{\lambda-\rho_{\mathfrak{n}_{G}}}\otimes{\rm\bf 1})^{g}|_{M}=\xi^{\tilde{w}_{0}}|_{M}$. Further, since $M$ is compact, $\delta_{P_{H}}=\delta_{gP_{G}g^{-1}}=\delta_{P_{H}\cap gP_{G}g^{-1}}=1$ on $M$. Finally, $S^{m}(V(g))=\{0\}$ for $m>0$, because $V(g)=\{0\}$ in this case, and $S^{0}(V(g))=\mathbb{C}$ is the trivial representation. This implies that

[TABLE]

4.2.2. Non-open orbits

In this case it is more convenient to work with

[TABLE]

where $W(g)=\mathfrak{g}/(\mathfrak{p}_{G}+\operatorname{Ad}(g^{-1})\mathfrak{p}_{H})$. If $P_{H}gP_{G}$ is a non-open orbit, then by Theorem 1.3 the stabilizer $P_{H}\cap gP_{G}g^{-1}$ of $gP_{G}$ in $P_{H}$ contains a one-parameter subgroup $\exp(\mathbb{R}Z)$, where $Z=Z_{M}+Z_{A}+Z_{N}\in(\mathfrak{m}_{G}\cap\mathfrak{m}_{H})+\mathfrak{a}_{H}+\mathfrak{n}_{H}$ with $Z_{A}\neq 0$. Note that

[TABLE]

with $X_{M}=\operatorname{Ad}(\tilde{w}^{-1})Z_{M}\in\mathfrak{m}_{G}$ and $X_{A}=\operatorname{Ad}(\tilde{w}^{-1})Z_{A}\in\mathfrak{a}_{G}$, $X_{A}\neq 0$. Then

[TABLE]

Further, $\delta_{P_{G}}(e^{tX})=e^{-2t\rho_{\mathfrak{n}_{G}}(X_{A})}$ and $\delta_{g^{-1}P_{H}g}(e^{tX})=e^{-2t\rho_{\mathfrak{n}_{H}}(Z_{A})}$.

Lemma 4.5**.**

$\delta_{P_{G}\cap g^{-1}P_{H}g}(e^{tX})=\exp(-t\sum_{\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})}n_{\alpha}\alpha(X_{A}))$* for some integers $0\leq n_{\alpha}\leq\dim\mathfrak{g}(\mathfrak{a}_{G};\alpha)$.*

Proof.

We compute $\delta_{P_{G}\cap g^{-1}P_{H}g}(e^{X})$ for $X=X_{M}+X_{A}+X_{N}\in\mathfrak{m}_{G}+\mathfrak{a}_{G}+\mathfrak{n}_{G}$ using the formula

[TABLE]

Choose a basis $(Y_{\alpha})_{\alpha}\cup(Y_{\beta})_{\beta}\cup(Y_{\gamma})_{\gamma}$ of $\mathfrak{p}_{G}\cap\operatorname{Ad}(g^{-1})\mathfrak{p}_{H}$ with the following properties:

1. (1)

$Y_{\alpha}=Y_{\alpha,M}+Y_{\alpha,A}+Y_{\alpha,N}\in\mathfrak{m}_{G}+\mathfrak{a}_{G}+\mathfrak{n}_{G}$ such that $(Y_{\alpha,M})_{\alpha}$ is an orthogonal basis of the projection of $\mathfrak{p}_{G}\cap\operatorname{Ad}(g^{-1})\mathfrak{p}_{H}$ to $\mathfrak{m}_{G}$. 2. (2)

$Y_{\beta}=Y_{\beta,A}+Y_{\beta,N}\in\mathfrak{a}_{G}+\mathfrak{n}_{G}$ such that $(Y_{\beta,A})_{\beta}$ is linearly independent. 3. (3)

$Y_{\gamma}=Y_{\gamma,N}\in\mathfrak{n}_{G}$ such that $(Y_{\gamma,N})_{\gamma}$ is a basis of $\mathfrak{n}_{G}\cap\operatorname{Ad}(g^{-1})\mathfrak{p}_{H}$.

Here orthogonal bases are taken with respect to any $\mathfrak{m}_{G}$-invariant inner product on $\mathfrak{g}$. We now study the action of $\operatorname{ad}(X)$ on $Y_{\alpha}$, $Y_{\beta}$ and $Y_{\gamma}$. First,

[TABLE]

Note that $[X_{M},Y_{\alpha,M}]\perp Y_{\alpha,M}$ since the inner product is $\mathfrak{m}_{G}$-invariant. Hence, the coefficient of $Y_{\alpha}$ in the expression of $\operatorname{ad}(X)Y_{\alpha}$ as a linear combination of the basis elements is zero. This implies that the contribution of the basis elements $Y_{\alpha}$ to the trace is trivial. Next we have

[TABLE]

so that $\operatorname{ad}(X)Y_{\beta}$ is a linear combination of the basis elements $Y_{\gamma}$. Therefore, also the basis elements $Y_{\beta}$ do not contribute to the trace and we have

[TABLE]

We now specify a basis $(Y_{\gamma})_{\gamma}$ of $\mathfrak{n}_{G}\cap\operatorname{Ad}(g^{-1})\mathfrak{p}_{H}$. Order the positive roots $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})=\{\alpha_{1},\ldots,\alpha_{n}\}$ such that $[\mathfrak{n}_{G},\mathfrak{g}(\mathfrak{a}_{G};\alpha_{j})]\subseteq\bigoplus_{i=j+1}^{n}\mathfrak{g}(\mathfrak{a}_{G};\alpha_{i})$. Then we can choose the basis $\{Y_{\gamma}=Y_{j,k}:1\leq j\leq n,1\leq k\leq n_{j}\}$ such that

• •

$Y_{j,k}=\sum_{i=j}^{n}Y_{j,k}^{i}\in\bigoplus_{i=j}^{n}\mathfrak{g}(\mathfrak{a}_{G};\alpha_{i})$ for all $1\leq j\leq n$, $1\leq k\leq n_{j}$,

• •

$(Y_{j,k}^{j})_{k=1,\ldots,n_{j}}$ are mutually orthogonal with respect to a $\mathfrak{m}_{G}$-invariant inner product on $\mathfrak{g}(\mathfrak{a}_{G};\alpha_{j})$.

We have

[TABLE]

Since $[X_{M},Y_{j,k}^{j}]\perp Y_{j,k}^{j}$ we obtain

[TABLE]

so that

[TABLE]

Lemma 4.6**.**

Let $P\in S^{m}(W(g))$, $P\neq 0$, with $\operatorname{ad}(X)P=\lambda P$ for some $\lambda\in\mathbb{C}$; then we have $\operatorname{Re}\lambda=-\sum_{\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})}m_{\alpha}\alpha(X_{A})$ for some integers $m_{\alpha}\geq 0$. If additionally $X_{M}=0$ then $\lambda=-\sum_{\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})}m_{\alpha}\alpha(X_{A})$.

Proof.

First note that $W(g)=\mathfrak{g}/(\mathfrak{p}_{G}+\operatorname{Ad}(g)^{-1}\mathfrak{p}_{H})\simeq\overline{\mathfrak{n}}_{G}/(\overline{\mathfrak{n}}_{G}\cap(\mathfrak{p}_{G}+\operatorname{Ad}(g)^{-1}\mathfrak{p}_{H}))$. We order the positive roots $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})=\{\alpha_{1},\ldots,\alpha_{n}\}$ such that $[\mathfrak{n}_{G},\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{j})]\subseteq\bigoplus_{i=j+1}^{n}\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{i})$. Then we can choose a basis $\{Y_{j,k}:1\leq j\leq n,1\leq k\leq n_{j}\}$ of $\overline{\mathfrak{n}}_{G}\cap(\mathfrak{p}_{G}+\operatorname{Ad}(g)^{-1}\mathfrak{p}_{H})$ such that

• •

$Y_{j,k}=\sum_{i=j}^{n}Y_{j,k}^{i}\in\bigoplus_{i=j}^{n}\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{i})$ for all $1\leq j\leq n$, $1\leq k\leq n_{j}$,

• •

$(Y_{j,k}^{j})_{k=1,\ldots,n_{j}}\subseteq\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{j})$ are linearly independent for every $1\leq j\leq n$.

For every $1\leq j\leq n$ we extend $(Y_{j,k}^{j})_{k=1,\ldots,n_{j}}$ to a $\mathbb{C}$-basis $(Y_{j,k}^{j})_{k=1,\ldots,n_{j}}\cup(Z_{j,k})_{k=1,\ldots,m_{j}}$ of $\mathfrak{g}(\mathfrak{a}_{G};-\alpha_{j})_{\mathbb{C}}$; then the equivalence classes $\overline{Z}_{j,k}=Z_{j,k}+(\overline{\mathfrak{n}}_{G}\cap(\mathfrak{p}_{G}+\operatorname{Ad}(g)^{-1}\mathfrak{p}_{H}))$, $1\leq j\leq n$, $1\leq k\leq m_{j}$, form a basis of $\overline{\mathfrak{n}}_{G}/(\overline{\mathfrak{n}}_{G}\cap(\mathfrak{p}_{G}+\operatorname{Ad}(g)^{-1}\mathfrak{p}_{H}))$. Since $X_{M}\in\mathfrak{m}_{G}$ is contained in a maximal torus in $\mathfrak{m}_{G}$, we may assume that every $Z_{j,k}$ is an eigenvector of $\operatorname{ad}(X_{M})$ with imaginary eigenvalue: $\operatorname{ad}(X_{M})Z_{j,k}=\sqrt{-1}\lambda_{j,k}Z_{j,k}$. Further, by definition we have $\operatorname{ad}(X_{A})Z_{j,k}=\alpha_{j}(X_{A})Z_{j,k}$, so that

[TABLE]

Now assume $P\in S^{m}(W(g))$, $P\neq 0$, with $\operatorname{ad}(X)P=\lambda P$. Write $P$ as a linear combination of the basis elements $\overline{Z}_{j_{1},k_{1}}\cdots\overline{Z}_{j_{m},k_{m}}$ of $S^{m}(W(g))$; then there exist $1\leq j_{1}\leq\ldots\leq j_{m}\leq n$ and $1\leq k_{i}\leq n_{j_{i}}$ such that the coefficient of $\overline{Z}_{j_{1},k_{1}}\cdots\overline{Z}_{j_{m},k_{m}}$ in $P$ is non-zero. Choose such $(j_{1},k_{1}),\ldots,(j_{m},k_{m})$ with the property that $j_{1},\ldots,j_{m}$ are minimal. Considering only the coefficient of $\overline{Z}_{j_{1},k_{1}}\cdots\overline{Z}_{j_{m},k_{m}}$ in $\operatorname{ad}(X)P=\lambda P$ we obtain

[TABLE]

which implies the claim. ∎

Now assume that $\phi\in V_{\xi}^{\vee}\otimes W_{\eta}\otimes S^{m}(W(g))$, $\phi\neq 0$, is contained in the space of invariants (4.3). Since $X_{M}$ resp. $Z_{M}$ is contained in a maximal torus in $\mathfrak{m}_{G}$ resp. $\mathfrak{m}_{H}$, there exist bases $(v_{i})_{i}$ of $V_{\xi}^{\vee}$ and $(w_{j})_{j}$ of $W_{\eta}$ such that $\xi^{\vee}(X_{M})v_{i}=\sqrt{-1}a_{i}v_{i}$ and $\eta(Z_{M})w_{j}=\sqrt{-1}b_{j}w_{j}$ with $a_{i},b_{j}\in\mathbb{R}$. Write $\phi$ in terms of this basis as

[TABLE]

with $P_{i,j}\in S^{m}(W(g))$; then $X\cdot\phi=0$ implies, using Lemma 4.5

[TABLE]

At least one $P_{i,j}$ is non-trivial and hence it follows from Lemma 4.6 that

[TABLE]

Hence, the space (4.3) of invariants must be trivial if (4.1) is satisfied. This completes the proof of Theorem 4.1.

5. Application to Shintani functions

In [17] Kobayashi established a connection between symmetry breaking operators and Shintani functions of a pair $(G,H)$ of real reductive groups. Combining his results with Corollary 3.4 we prove lower bounds for the dimension of the space of Shintani functions (see Theorem 5.3).

5.1. Shintani functions for real reductive groups

Let $\mathfrak{t}_{G}\subseteq\mathfrak{m}_{G}$ and $\mathfrak{t}_{H}\subseteq\mathfrak{m}_{H}$ be Cartan subalgebras of $\mathfrak{m}_{G}$ and $\mathfrak{m}_{H}$; then $\mathfrak{j}_{G}=\mathfrak{t}_{G}+\mathfrak{a}_{G}\subseteq\mathfrak{g}$ and $\mathfrak{j}_{H}=\mathfrak{t}_{H}+\mathfrak{a}_{H}\subseteq\mathfrak{h}$ are maximally split Cartan subalgebras of $\mathfrak{g}$ and $\mathfrak{h}$. We identify $\mathfrak{j}_{G,\mathbb{C}}^{\vee}\simeq\mathfrak{t}_{G,\mathbb{C}}^{\vee}\oplus\mathfrak{a}_{G,\mathbb{C}}^{\vee}$ and $\mathfrak{j}_{H,\mathbb{C}}^{\vee}\simeq\mathfrak{t}_{H,\mathbb{C}}^{\vee}\oplus\mathfrak{a}_{H,\mathbb{C}}^{\vee}$. Let us choose positive systems $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})\subseteq\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ and $\Sigma^{+}(\mathfrak{h}_{\mathbb{C}},\mathfrak{j}_{H,\mathbb{C}})\subseteq\Sigma(\mathfrak{h}_{\mathbb{C}},\mathfrak{j}_{H,\mathbb{C}})$ such that the restriction of a positive root to $\mathfrak{a}_{G}$ resp. $\mathfrak{a}_{H}$ is either zero or contained in $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ resp. $\Sigma^{+}(\mathfrak{h},\mathfrak{a}_{H})$. Write $\rho_{\mathfrak{g}}$ resp. $\rho_{\mathfrak{h}}$ for the half sum of all roots in $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ resp. $\Sigma^{+}(\mathfrak{h}_{\mathbb{C}},\mathfrak{j}_{H,\mathbb{C}})$. Then $\rho_{\mathfrak{g}}=\rho_{\mathfrak{m}_{G}}+\rho_{\mathfrak{n}_{G}}$ with $\rho_{\mathfrak{m}_{G}}=\rho_{\mathfrak{g}}|_{\mathfrak{t}_{G}}$, and similarly $\rho_{\mathfrak{h}}=\rho_{\mathfrak{m}_{H}}+\rho_{\mathfrak{n}_{H}}$. Further, let $W(\mathfrak{j}_{G,\mathbb{C}})$ and $W(\mathfrak{j}_{H,\mathbb{C}})$ denote the Weyl groups of $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ and $\Sigma(\mathfrak{h}_{\mathbb{C}},\mathfrak{j}_{H,\mathbb{C}})$.

The Harish-Chandra isomorphism provides a natural identification

[TABLE]

where $Z(\mathfrak{g}_{\mathbb{C}})$ denotes the center of the universal enveloping algebra $U(\mathfrak{g}_{\mathbb{C}})$ of $\mathfrak{g}_{\mathbb{C}}$. We use the same notation $\chi_{\Xi}\in\operatorname{Hom}_{\mathbb{C}-\rm alg}(Z(\mathfrak{h}_{\mathbb{C}}),\mathbb{C})$ for $\Xi\in\mathfrak{j}_{H,\mathbb{C}}^{\vee}$. The left resp. right action of $U(\mathfrak{g}_{\mathbb{C}})$ on $C^{\infty}(G)$ will be denoted by $L_{u}$ resp. $R_{u}$, $u\in U(\mathfrak{g}_{\mathbb{C}})$.

Definition 5.1** (see [17, Definition 1.1], [34]).**

A function $f\in C^{\infty}(G)$ is called a Shintani function of type $(\Lambda,\Xi)\in\mathfrak{j}_{G,\mathbb{C}}^{\vee}\times\mathfrak{j}_{H,\mathbb{C}}^{\vee}$ if $f$ satisfies the following three properties:

1. (1)

$f(k^{\prime}gk)=f(g)$ for any $k^{\prime}\in H\cap K$, $k\in K$. 2. (2)

$R_{u}f=\chi_{\Lambda}(u)f$ for any $u\in Z(\mathfrak{g}_{\mathbb{C}})$. 3. (3)

$L_{v}f=\chi_{\Xi}(v)f$ for any $v\in Z(\mathfrak{h}_{\mathbb{C}})$.

The space of Shintani functions of type $(\Lambda,\Xi)$ is denoted by $\operatorname{Sh}(\Lambda,\Xi)$. We write $\operatorname{Sh}_{\rm mod}(\Lambda,\Xi)$ for the subspace of Shintani functions of moderate growth.

For the definition of moderate growth see e.g. [17, Definition 3.3]. The following result is due to Kobayashi:

Fact 5.2** (see [17, Theorem 1.2 and 8.1]).**

Let $(G,H)$ be a pair of real reductive algebraic groups.

1. (1)

$\dim\operatorname{Sh}(\Lambda,\Xi)<\infty$* for all $(\Lambda,\Xi)\in\mathfrak{j}_{G,\mathbb{C}}^{\vee}\times\mathfrak{j}_{H,\mathbb{C}}^{\vee}$ if and only if $(G,H)$ is strongly spherical.* 2. (2)

$\dim\operatorname{Sh}(\Lambda,\Xi)\neq 0$* implies $\Lambda\in W(\mathfrak{j}_{G,\mathbb{C}})(\rho_{\mathfrak{m}_{G}}+\mathfrak{a}_{G,\mathbb{C}}^{\vee})$ and $\Xi\in W(\mathfrak{j}_{H,\mathbb{C}})(\rho_{\mathfrak{m}_{H}}+\mathfrak{a}_{H,\mathbb{C}}^{\vee})$.*

In view of this statement we abuse notation and write

[TABLE]

5.2. Lower bounds for $\dim\operatorname{Sh}(\lambda,\nu)$

The following statement gives more detailed information about $\dim\operatorname{Sh}(\lambda,\nu)$:

Theorem 5.3**.**

Assume that $(G,H)$ is a strongly spherical reductive pair such that $(\mathfrak{g},\mathfrak{h})$ is non-trivial and indecomposable. Then for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

and for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have

[TABLE]

Proof.

Let $\lambda_{+}\in W(\mathfrak{a}_{G})\lambda$ and $\nu_{-}\in W(\mathfrak{a}_{H})(-\nu)$ such that $\operatorname{Re}\langle\lambda_{+},\alpha\rangle\geq 0$ for all $\alpha\in\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$ and $\operatorname{Re}\langle\nu_{-},\beta\rangle\leq 0$ for all $\beta\in\Sigma^{+}(\mathfrak{h},\mathfrak{a}_{H})$. Then by [17, Theorem 8.2 and Remark 8.3] there exists a natural embedding

[TABLE]

which is an isomorphism for generic $(\lambda,\nu)$. Now the statement follows from Corollaries 3.4 and 4.4. ∎

Remark 5.4**.**

According to Kobayashi [17, Remark 10.2 (4)] it is plausible that $\operatorname{Sh}_{\rm mod}(\lambda,\nu)=\operatorname{Sh}(\lambda,\nu)$ for all $(\lambda,\nu)\in\mathfrak{j}_{G,\mathbb{C}}^{\vee}\times\mathfrak{j}_{H,\mathbb{C}}^{\vee}$ if the pair $(G,H)$ is strongly spherical. In this case, Theorem 5.3 would imply

[TABLE]

for generic $(\lambda,\nu)$.

Remark 5.5**.**

For some strongly spherical pairs $(G,H)$ of low rank, the space of Shintani functions was studied in more detail, and in some cases its dimension was computed for all parameters $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$. The pairs $(\operatorname{GL}(2,\mathbb{F}),\operatorname{GL}(1,\mathbb{F})\times\operatorname{GL}(1,\mathbb{F}))$, $\mathbb{F}=\mathbb{R},\mathbb{C}$, were studied by Hirano [11, 12], the pairs $(\operatorname{O}(1,n+1),\operatorname{O}(1,n))$ by Kobayashi [17], the pairs $(\operatorname{U}(1,n+1),\operatorname{U}(1,n)\times\operatorname{U}(1))$ by Tsuzuki [42, 43, 44], the pairs $(\operatorname{Sp}(2,\mathbb{R}),\operatorname{Sp}(1,\mathbb{R})\times\operatorname{Sp}(1,\mathbb{R}))$ and $(\operatorname{Sp}(2,\mathbb{R}),\operatorname{SL}(2,\mathbb{C}))$ by Moriyama [32, 33]. and the pair $(\operatorname{SL}(3,\mathbb{R}),\operatorname{GL}(2,\mathbb{R}))$ by Sono [40].

6. Multiplicity one pairs

For the pairs

[TABLE]

it was proven by Sun–Zhu [41] that $\dim\operatorname{Hom}_{H}(\pi|_{H},\tau)\leq 1$ for all irreducible smooth admissible Fréchet representations $\pi$ of $G$ and $\tau$ of $H$ of moderate growth. In this section we determine for which principal series representations $\pi=\pi_{\xi,\lambda}$ and $\tau=\tau_{\eta,\nu}$ the multiplicities are $=1$, at least for generic $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$. The main tool to pass from spherical principal series $\pi_{\lambda}$ and $\tau_{\nu}$ to general vector-valued principal series $\pi_{\xi,\lambda}$ and $\tau_{\eta,\nu}$ is a variant of the Jantzen–Zuckerman translation principle. This method was previously used in the construction of symmetry breaking operators in [28] and we briefly recall the main steps, adapted to our setting.

6.1. The translation principle

Let $(\varphi,E)$ be an irreducible finite-dimensional representation of $G$; then the space

[TABLE]

of $\overline{\mathfrak{n}}_{G}$-fixed vectors is an irreducible representation of $M_{G}A_{G}$, and we obtain a $\overline{P}_{G}$-equivariant embedding $E^{\prime}\hookrightarrow E$. Similarly, we let $(\psi,F)$ be an irreducible finite-dimensional representation of $H$ and consider the irreducible $M_{H}A_{H}$-representation $F^{\prime}=F^{\overline{\mathfrak{n}}_{H}}$. Then the projection $F\twoheadrightarrow F^{\prime}$ onto the highest restricted weight space is $P_{H}$-equivariant, where we let $N_{H}$ act trivially on $F^{\prime}$. We assume that $\operatorname{Hom}_{H}(E|_{H},F)\neq\{0\}$.

Now suppose we are given an $H$-intertwining operator

[TABLE]

for $V$ and $W$ finite-dimensional representations of $P_{G}$ and $P_{H}$. Tensoring with a non-trivial $H$-intertwiner $\eta:E\to F$ gives an intertwiner

[TABLE]

from which we want to construct an intertwiner between certain vector-valued principal series.

For this, we first consider the representation $\operatorname{Ind}_{P_{G}}^{G}(V)\otimes E$. Since $\tilde{w}_{0}P_{G}\tilde{w}_{0}^{-1}=\overline{P}_{G}$, the map $f\mapsto f(\operatorname{\hskip 0.50003pt\cdot\hskip 0.50003pt}\tilde{w}_{0}^{-1})$ defines an isomorphism

[TABLE]

Next, we make use of the isomorphism

[TABLE]

where we view both sides as $(V\otimes E)$-valued smooth functions on $G$. Now, the $\overline{P}_{G}$-equivariant embedding $E^{\prime}\hookrightarrow E$ induces an embedding

[TABLE]

which we compose with the isomorphism

[TABLE]

to obtain an embedding

[TABLE]

Similarly, without having to invoke $\tilde{w}_{0}$, we have a surjection

[TABLE]

where the second map is induced by the $P_{H}$-equivariant projection $F\to F^{\prime}$.

Finally, composing (6.2) with the embedding (6.3) and the surjection (6.4) defines an $H$-intertwining operator

[TABLE]

Now let $V={\rm\bf 1}\otimes e^{\lambda}\otimes{\rm\bf 1}$ and $W={\rm\bf 1}\otimes e^{\nu}\otimes{\rm\bf 1}$, and write $(E^{\prime})^{\tilde{w}_{0}^{-1}}\simeq\xi\otimes e^{\lambda_{0}}\otimes{\rm\bf 1}$ and $F^{\prime}=\eta\otimes e^{\nu_{0}}\otimes{\rm\bf 1}$ for $\xi\in\widehat{M}_{G}$, $\eta\in\widehat{M}_{H}$ and $(\lambda_{0},\nu_{0})\in\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$. Then $A\mapsto\Phi(A)$ defines a map

[TABLE]

Recall that by Proposition 3.1 every intertwining operator $A\in\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})$ is given by a distribution kernel $K_{A}\in(\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda}^{*})\otimes W_{\eta,\nu})^{P_{H}}$. To describe how the distribution kernel of $\Phi(A)$ arises from the distribution kernel of $A$ we denote by $i:E^{\prime}\hookrightarrow E$ the $\overline{P}_{G}$-equivariant embedding and by $p:F\twoheadrightarrow F^{\prime}$ the $P_{H}$-equivariant quotient. Further let $\mathcal{E}^{\prime}=G\times_{P_{G}}(E^{\prime})^{\tilde{w}_{0}^{-1}}$ and denote by $(\mathcal{E}^{\prime})^{\vee}=G\times_{P_{G}}((E^{\prime})^{\tilde{w}_{0}^{-1}})^{\vee}$ the contragredient bundle.

Theorem 6.1** ([28, Section 2]).**

The map

[TABLE]

has the property that the distribution kernel $K_{\Phi(A)}\in(\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\xi,\lambda+\lambda_{0}}^{*})\otimes W_{\eta,\nu+\nu_{0}})^{P_{H}}$ is given in terms of the distribution kernel $K_{A}\in(\mathcal{D}^{\prime}(G/P_{G},\mathcal{V}_{\lambda}^{*})\otimes W_{\nu})^{P_{H}}$ as

[TABLE]

where $\sigma\in(C^{\infty}(G/P_{G},(\mathcal{E}^{\prime})^{\vee})\otimes F^{\prime})^{P_{H}}$ is given by

[TABLE]

Note that in the statement we identify $\mathcal{V}_{\lambda}^{*}\otimes(\mathcal{E}^{\prime})^{\vee}\simeq\mathcal{V}_{\xi,\lambda+\lambda_{0}}^{*}$ and $W_{\nu}\otimes F^{\prime}\simeq W_{\eta,\nu+\nu_{0}}$.

Corollary 6.2**.**

Let $\xi\in\widehat{M}_{G}$ and $\eta\in\widehat{M}_{H}$ and assume that there exist irreducible finite-dimensional representations $E$ of $G$ and $F$ of $H$ such that $\xi^{\tilde{w}_{0}}\simeq E^{\mathfrak{n}_{G}}|_{M_{G}}$, $\eta\simeq F^{\mathfrak{n}_{H}}|_{M_{H}}$ and $\operatorname{Hom}_{H}(E|_{H},F)\neq\{0\}$. Then there exist $(\lambda_{0},\nu_{0})\in\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ and a linear map

[TABLE]

which is on the level of distribution kernels given by tensoring with a fixed non-trivial real-analytic section. In particular, $\operatorname{supp}(K_{\Phi(A)})=G/P_{G}$ whenever $\operatorname{supp}K_{A}=G/P_{G}$.

Proof.

Replacing $E$ and $F$ by their twists with the Cartan involution $\theta$ of $G$ we may assume that $\xi^{\tilde{w}_{0}}\simeq E^{\overline{\mathfrak{n}}_{G}}|_{M_{G}}$, $\eta\simeq F^{\overline{\mathfrak{n}}_{H}}|_{M_{H}}$ and $\operatorname{Hom}_{H}(E|_{H},F)\neq\{0\}$. Then $E^{\prime}=E^{\overline{\mathfrak{n}}_{G}}$ and $F^{\prime}=F^{\overline{\mathfrak{n}}_{H}}$ satisfy $(E^{\prime})^{\tilde{w}_{0}^{-1}}|_{M_{G}}\simeq\xi$ and $F^{\prime}|_{M_{H}}\simeq\eta$, so that the statement follows from Theorem 6.1. It remains to show that $\sigma$ is a non-trivial real-analytic section. That $\sigma$ is non-trivial is a consequence of the irreducibility of $E$. Further, as a matrix coefficient of a finite-dimensional representation it is clearly real analytic. ∎

6.2. Iwasawa decompositions

To construct finite-dimensional representations $E$ of $G$ and $F$ of $H$ for the translation principle, we first give the explicit Iwasawa decompositions for all pairs $(G,H)$ in (6.1) (following the notation of Section 1.3), compute the generic stabilizer $M\subseteq M_{G}\cap M_{H}$ of an element of $\mathfrak{n}^{-\sigma}$ defined in Section 4.1, and prove in particular the following statement:

Lemma 6.3**.**

Let $(G,H)$ be one of the pairs in (6.1); then $\#(P_{H}\backslash G/P_{G})_{\rm open}=1$.

Note that Lemma 6.3 also follows from Corollary 4.4 combined with Fact II. However, since we need the relevant structure theory of the pairs $(G,H)$ in Section 6.3, we include an independent proof here.

6.2.1. $(G,H)=(\operatorname{GL}(n+1,\mathbb{C}),\operatorname{GL}(n,\mathbb{C}))$

Let $G=\operatorname{GL}(n+1,\mathbb{C})$, $n\geq 2$, and define a Cartan involution on $G$ by $\theta(g)=(g^{*})^{-1}$; then $K=G^{\theta}=\operatorname{U}(n+1)$. We embed $H=\operatorname{GL}(n,\mathbb{C})$ in the upper-left corner of $G$; then $\theta$ leaves $H$ invariant and $H\cap K=H^{\theta}=\operatorname{U}(n)$. We choose the maximal abelian subalgebra

[TABLE]

of $\mathfrak{h}^{-\theta}$ and extend it to the maximal abelian subalgebra

[TABLE]

of $\mathfrak{g}^{-\theta}$. Then

[TABLE]

Further,

[TABLE]

and such an element of $\mathfrak{n}^{-\sigma}$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit if and only if $z_{1},\ldots,z_{n}\neq 0$. Hence $M=\{{\rm\bf 1}\}$ is trivial.

6.2.2. $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$

Let $G=\operatorname{GL}(n+1,\mathbb{R})$, $n\geq 2$, and define a Cartan involution on $G$ by $\theta(g)=(g^{\top})^{-1}$; then $K=G^{\theta}=\operatorname{O}(n+1)$. We embed $H=\operatorname{GL}(n,\mathbb{R})$ in the upper left corner of $H$; then $\theta$ leaves $H$ invariant and $H\cap K=H^{\theta}=\operatorname{O}(n)$. We choose the maximal abelian subalgebra

[TABLE]

of $\mathfrak{h}^{-\theta}$ and extend it to the maximal abelian subalgebra

[TABLE]

of $\mathfrak{g}^{-\theta}$. Then

[TABLE]

Further,

[TABLE]

and such an element of $\mathfrak{n}^{-\sigma}$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit if and only if $x_{1},\ldots,x_{n}\neq 0$. Hence $M=\{{\rm\bf 1}\}$ is trivial.

6.2.3. $(G,H)=(\operatorname{U}(p,q+1),\operatorname{U}(p,q))$

Let $G=\operatorname{U}(p,q+1)$, $p,q\geq 1$, and define a Cartan involution on $G$ by $\theta(g)=(g^{*})^{-1}$, then $K=G^{\theta}=\operatorname{U}(p)\times\operatorname{U}(q+1)$. We embed $H=\operatorname{U}(p,q)$ in the upper-left corner of $G$; then $\theta$ leaves $H$ invariant and $H\cap K=H^{\theta}=\operatorname{U}(p)\times\operatorname{U}(q)$. Assume that $p\geq q+1$; the case $p\leq q$ is handled similarly (see Section 6.2.5 for a related computation). We choose the maximal abelian subalgebra

[TABLE]

of $\mathfrak{g}^{-\theta}$; then $\mathfrak{a}_{H}=\mathfrak{a}_{G}\cap\mathfrak{h}$ is maximal abelian in $\mathfrak{h}^{-\theta}$. Write $e_{i}\in\mathfrak{a}_{G}^{\vee}$ for the functional mapping a matrix of the above form to $t_{i}$. Then

[TABLE]

with root spaces given by

[TABLE]

We choose the positive system

[TABLE]

then

[TABLE]

and

[TABLE]

The action of $M_{G}\cap M_{H}\simeq\operatorname{U}(1)^{q}\times\operatorname{U}(p-q-1)$ is given by the natural action of $\operatorname{U}(1)^{q}$ on $z\in\mathbb{C}^{q}$, and the second factor $\operatorname{U}(p-q-1)$ acts trivially. Hence, the element $X(t)$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit if and only if $t_{1},\ldots,t_{q}\neq 0$ and then $M=\operatorname{U}(p-q-1)$. Further, $\operatorname{U}(1)^{q}\subseteq M_{G}\cap M_{H}$ acts transitively on the product $(S^{1})^{q}\subseteq\mathbb{C}^{q}$ of the unit spheres $S^{1}\subseteq\mathbb{C}$ and therefore there is a unique open orbit.

6.2.4. $(G,H)=(\operatorname{SO}(n+1,\mathbb{C}),\operatorname{SO}(n,\mathbb{C}))$

Let $G=\operatorname{SO}(n+1,\mathbb{C})$, $n\geq 2$, and define a Cartan involution on $G$ by $\theta(g)=(g^{*})^{-1}=\overline{g}$; then $K=G^{\theta}=\operatorname{SO}(n+1)$. We embed $H=\operatorname{SO}(n,\mathbb{C})$ in the upper-left corner of $G$; then $\theta$ leaves $H$ invariant and $H\cap K=H^{\theta}=\operatorname{SO}(n)$. Write $n=2m$ if $n$ is even and $n=2m-1$ if it is odd. We choose the maximal abelian subalgebra

[TABLE]

of $\mathfrak{g}^{-\theta}$, where

[TABLE]

Then $\mathfrak{a}_{H}=\mathfrak{a}_{G}\cap\mathfrak{h}$ is maximal abelian in $\mathfrak{h}^{-\theta}$. Write $e_{i}\in\mathfrak{a}_{G}^{\vee}$ for the functional mapping a matrix of the above form to $t_{i}$. Then

[TABLE]

with root spaces given by

[TABLE]

We choose the positive system

[TABLE]

then

[TABLE]

and

[TABLE]

The action of

[TABLE]

is given by the natural action of $\operatorname{SO}(2)\simeq\operatorname{U}(1)$ on $t_{i}\in\mathbb{C}$ by rotation. Hence, the element $X(t)$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit if and only if $t_{1},\ldots,t_{m-1}\neq 0$ resp. $t_{1},\ldots,t_{m}\neq 0$ and then $M=\{{\rm\bf 1}\}$. Further, $\operatorname{SO}(2)$ acts transitively on the unit sphere in $\mathbb{C}$ and hence $M_{G}\cap M_{H}$ acts transitively on the product of the unit spheres in $\mathfrak{n}^{-\sigma}$, so there is a unique open orbit.

6.2.5. $(G,H)=(\operatorname{SO}(p,q+1),\operatorname{SO}(p,q))$

Let $G=\operatorname{SO}(p,q+1)$, $p,q\geq 1$, and define a Cartan involution on $G$ by $\theta(g)=(g^{\top})^{-1}$; then $K=G^{\theta}=\operatorname{S}(\operatorname{O}(p)\times\operatorname{O}(q+1))$. We embed $H=\operatorname{SO}(p,q)$ in the upper-left corner of $G$; then $\theta$ leaves $H$ invariant and $H\cap K=H^{\theta}=\operatorname{S}(\operatorname{O}(p)\times\operatorname{O}(q))$. Assume that $p\leq q$; the case $p\geq q+1$ is handled similarly (see Section 6.2.3 for a related computation). We choose the maximal abelian subalgebra

[TABLE]

of $\mathfrak{g}^{-\theta}$ and $\mathfrak{h}^{-\theta}$. Write $e_{i}\in\mathfrak{a}_{G}^{\vee}$ for the functional mapping a matrix of the above form to $t_{i}$. Then

[TABLE]

with root spaces given by

[TABLE]

We choose the positive system

[TABLE]

then

[TABLE]

and

[TABLE]

The action of $M_{G}\cap M_{H}\simeq\operatorname{O}(1)^{p}\times\operatorname{SO}(q-p)$ is given by the natural action of $\operatorname{O}(1)^{p}$ on $t\in\mathbb{R}^{p}$ by sign changes of the coordinates, and the second factor $\operatorname{SO}(q-p)$ acts trivially. Hence, the element $X(t)$ is contained in an open $(M_{G}\cap M_{H})A_{H}$-orbit if and only if $t_{1},\ldots,t_{p}\neq 0$ and then $M=\operatorname{SO}(q-p)$. Further, $\operatorname{O}(1)^{p}\subseteq M_{G}\cap M_{H}$ acts transitively on the product $\{\pm 1\}^{p}\subseteq\mathbb{R}^{p}$ of the unit spheres $\{\pm 1\}\subseteq\mathbb{R}$ and therefore there is a unique open orbit.

6.3. Finite-dimensional branching

Using case-by-case arguments, we prove in this subsection the following statement:

Proposition 6.4**.**

Let $(G,H)$ be one of the pairs in (6.1) and $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$. Then

1. (1)

$\dim\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\dim\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\leq 1$, 2. (2)

whenever $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$, there exist finite-dimensional representations $E$ of $G$ and $F$ of $H$ such that $\xi\simeq E^{\mathfrak{n}_{G}}|_{M_{G}}$, $\eta\simeq F^{\mathfrak{n}_{H}}|_{M_{H}}$ and $\operatorname{Hom}_{H}(E|_{H},F)\neq\{0\}$.

In all five cases, $G$ and $H$ have connected complexifications $G_{\mathbb{C}}$ and $H_{\mathbb{C}}$, and we can parametrize irreducible finite-dimensional representations of $G_{\mathbb{C}}$ and $H_{\mathbb{C}}$ by their highest weights $\lambda\in\mathfrak{j}_{G,\mathbb{C}}^{\vee}$ and $\nu\in\mathfrak{j}_{H,\mathbb{C}}^{\vee}$ for $\mathfrak{j}_{G}\subseteq\mathfrak{g}$ and $\mathfrak{j}_{H}\subseteq\mathfrak{h}$ Cartan subalgebras. Denote the restrictions of the corresponding representations to $G$ and $H$ by $F^{G}(\lambda)$ and $F^{H}(\nu)$.

In each case, we use the notation introduced in Section 6.2.

6.3.1. $(G,H)=(\operatorname{GL}(n+1,\mathbb{C}),\operatorname{GL}(n,\mathbb{C}))$

We extend $\mathfrak{a}_{G}$ to the Cartan subalgebra $\mathfrak{j}_{G}=\mathfrak{t}_{G}\oplus\mathfrak{a}_{G}$ with

[TABLE]

Let $f_{i}\in\mathfrak{t}_{G,\mathbb{C}}^{\vee}$ ($1\leq i\leq n+1$) be the linear functionals mapping a diagonal matrix as above to $\sqrt{-1}t_{i}$. Then the root system $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ is of the form

[TABLE]

and putting

[TABLE]

we have $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\pm(\varepsilon_{i}^{\prime}-\varepsilon_{j}^{\prime}),\pm(\varepsilon_{i}^{\prime\prime}-\varepsilon_{j}^{\prime\prime}):1\leq i<j\leq n+1\}$. Further, the positive system $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\varepsilon_{i}^{\prime}-\varepsilon_{j}^{\prime},\varepsilon_{i}^{\prime\prime}-\varepsilon_{j}^{\prime\prime}:1\leq i<j\leq n+1\}$ is compatible with the positive system $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

We choose the complexification $G_{\mathbb{C}}=\operatorname{GL}(n+1,\mathbb{C})\times\operatorname{GL}(n+1,\mathbb{C})$ with embedding $G\hookrightarrow G_{\mathbb{C}}$ given by $g\mapsto(g,\overline{g})$; then $\mathfrak{j}_{G,\mathbb{C}}$ is a Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{gl}}(n+1,\mathbb{C})+\operatorname{\mathfrak{gl}}(n+1,\mathbb{C})$ and the irreducible finite-dimensional representations of $G_{\mathbb{C}}$ are parametrized by their highest weights $\lambda=\lambda_{1}^{\prime}\varepsilon_{1}^{\prime}+\cdots+\lambda_{n+1}^{\prime}\varepsilon_{n+1}^{\prime}+\lambda_{1}^{\prime\prime}\varepsilon_{1}^{\prime\prime}+\cdots+\lambda_{n+1}^{\prime\prime}\varepsilon_{n+1}^{\prime\prime}$, $\lambda_{1}^{\prime}\geq\ldots\geq\lambda_{n+1}^{\prime}$, $\lambda_{1}^{\prime\prime}\geq\ldots\geq\lambda_{n+1}^{\prime\prime}$, $\lambda_{i}^{\prime},\lambda_{i}^{\prime\prime}\in\mathbb{Z}$. Clearly, an element $g=\exp(\sqrt{-1}\operatorname{diag}(t_{1},\ldots,t_{n+1}))\in M_{G}$ acts on the highest weight space of $F^{G}(\lambda)$ by $e^{\sqrt{-1}(\lambda_{1}^{\prime}-\lambda_{1}^{\prime\prime})t_{1}}\cdots e^{\sqrt{-1}(\lambda_{m}^{\prime}-\lambda_{m}^{\prime\prime})t_{m}}$.

In the same way we parametrize irreducible finite-dimensional representations of $H_{\mathbb{C}}=\operatorname{GL}(n,\mathbb{C})\times\operatorname{GL}(n,\mathbb{C})$ by highest weight $\nu=\nu_{1}^{\prime}\varepsilon_{1}^{\prime}+\cdots+\nu_{n}^{\prime}\varepsilon_{n}^{\prime}+\nu_{1}^{\prime\prime}\varepsilon_{1}^{\prime\prime}+\cdots+\nu_{n}^{\prime\prime}\varepsilon_{n}^{\prime\prime}$, $\nu_{1}^{\prime}\geq\ldots\geq\nu_{n}^{\prime}$, $\nu_{1}^{\prime\prime}\geq\ldots\geq\nu_{n}^{\prime\prime}$, $\nu_{i}^{\prime},\nu_{i}^{\prime\prime}\in\mathbb{Z}$. On its highest weight space an element of the form $h=\exp(\sqrt{-1}\operatorname{diag}(t_{1},\ldots,t_{n},0))\in M_{H}$ acts by $e^{\sqrt{-1}(\nu_{1}^{\prime}-\nu_{1}^{\prime\prime})t_{1}}\cdots e^{\sqrt{-1}(\nu_{n}^{\prime}-\nu_{n}^{\prime\prime})t_{n}}$.

Now, $M_{G}=\operatorname{U}(1)^{n+1}$, $M_{H}=\operatorname{U}(1)^{n}$ and $M=\{{\rm\bf 1}\}$. Hence, irreducible representations of $M_{G}$ and $M_{H}$ are one-dimensional and $\dim\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\dim\operatorname{Hom}_{M}(\xi_{M},\eta|_{M})=1$ for all $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$. Every $\xi\in\widehat{M}_{G}$ has the form $\xi(\exp(\sqrt{-1}\operatorname{diag}(t_{1},\ldots,t_{n+1})))=e^{\sqrt{-1}\xi_{1}t_{1}}\cdots e^{\sqrt{-1}\xi_{n+1}t_{n+1}}$ with $\xi_{1},\ldots,\xi_{n+1}\in\mathbb{Z}$. Similarly, every $\eta\in\widehat{M}_{H}$ has the form $\eta(\exp(\sqrt{-1}\operatorname{diag}(t_{1},\ldots,t_{n},0)))=e^{\sqrt{-1}\eta_{1}t_{1}}\cdots e^{\sqrt{-1}\eta_{n}t_{n}}$ with $\eta_{1},\ldots,\eta_{n}\in\mathbb{Z}$. By the above observations $\xi\simeq F^{G}(\lambda)^{\mathfrak{n}_{G}}|_{M_{G}}$ if and only if $\lambda_{i}^{\prime}-\lambda_{i}^{\prime\prime}=\xi_{i}$ ($1\leq i\leq n+1$). Further, $\eta\simeq F^{H}(\nu)^{\mathfrak{n}_{H}}|_{M_{H}}$ if and only if $\nu_{i}^{\prime}-\nu_{i}^{\prime\prime}=\eta_{i}$ ($1\leq i\leq n$). Moreover, we have $\operatorname{Hom}_{H}(F^{G}(\lambda),F^{H}(\nu))\neq\{0\}$ if and only if

[TABLE]

We first choose $(\lambda_{n+1}^{\prime},\lambda_{n+1}^{\prime\prime})\in\mathbb{Z}\times\mathbb{Z}$ with $\lambda_{n+1}^{\prime}-\lambda_{n+1}^{\prime\prime}=\xi_{n+1}$. Next we choose $(\nu_{n}^{\prime},\nu_{n}^{\prime\prime})\in\mathbb{N}\times\mathbb{N}$ with $\nu_{n}^{\prime}\geq\lambda_{n+1}^{\prime}$, $\nu_{n}^{\prime\prime}\geq\lambda_{n+1}^{\prime\prime}$ and $\nu_{n}^{\prime}-\nu_{n}^{\prime\prime}=\eta_{n}$. Iterating this procedure constructs (not necessarily unique) highest weights $\lambda$ and $\nu$ with the desired properties.

6.3.2. $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$

We choose the natural complexification $G_{\mathbb{C}}=\operatorname{GL}(n+1,\mathbb{C})$; then $\mathfrak{a}_{G,\mathbb{C}}$ is a maximal torus in $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{gl}}(n+1,\mathbb{C})$ and the irreducible finite-dimensional representations of $G_{\mathbb{C}}$ are parametrized by their highest weights $\lambda=\lambda_{1}e_{1}+\cdots+\lambda_{n+1}e_{n+1}$, $\lambda_{1}\geq\ldots\geq\lambda_{n+1}$, $\lambda_{i}\in\mathbb{Z}$. For $H$ we use similar notation: $\nu=\nu_{1}e_{1}+\cdots+\nu_{n}e_{n}$, $\nu_{1}\geq\ldots,\nu_{n}$, $\nu_{i}\in\mathbb{Z}$.

Note that an element $g=\operatorname{diag}((-1)^{k_{1}},\ldots,(-1)^{k_{n+1}})\in M_{G}$ acts on the highest weight space $F^{G}(\lambda)^{\mathfrak{n}_{G}}$ by $(-1)^{k_{1}\lambda_{1}+\cdots+k_{n+1}\lambda_{n+1}}$. Similarly, $h=\operatorname{diag}((-1)^{\ell_{1}},\ldots,(-1)^{\ell_{n}},1)\in M_{H}$ acts on $F^{H}(\nu)^{\mathfrak{n}_{H}}$ by $(-1)^{\ell_{1}\nu_{1}+\cdots+\ell_{n}\nu_{n}}$. By the classical branching laws, the restriction of $F^{G}(\lambda)$ to $H$ contains all representations $F^{H}(\nu)$ with

[TABLE]

Now, $M_{G}=\operatorname{O}(1)^{n+1}$, $M_{H}=\operatorname{O}(1)^{n}$ and $M=\{{\rm\bf 1}\}$. Hence, irreducible representations of $M_{G}$ and $M_{H}$ are one-dimensional and $\dim\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\dim\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})=1$ for all $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$. Every $\xi\in\widehat{M}_{G}$ has the form $\xi(\operatorname{diag}((-1)^{k_{1}},\ldots,(-1)^{k_{n+1}}))=(-1)^{k_{1}\xi_{1}+\cdots+k_{n+1}\xi_{n+1}}$ with $\xi_{i}\in\mathbb{Z}/2\mathbb{Z}$. Similarly, every representation $\eta\in\widehat{M}_{H}$ has the form $\eta(\operatorname{diag}((-1)^{\ell_{1}},\ldots,(-1)^{\ell_{n}},1))=(-1)^{\ell_{1}\eta_{1}+\cdots+\ell_{n}\eta_{n}}$ with $\eta_{i}\in\mathbb{Z}/2\mathbb{Z}$. By the above observations, $\xi\simeq F^{G}(\lambda)^{\mathfrak{n}_{G}}|_{M_{G}}$ if and only if $\xi_{i}=\lambda_{i}+2\mathbb{Z}$. Further, $\eta\simeq F^{H}(\nu)^{\mathfrak{n}_{H}}|_{M_{H}}$ if and only if $\eta_{i}=\nu_{i}+2\mathbb{Z}$. It is clear that for fixed $\xi_{1},\ldots,\xi_{n+1},\eta_{1},\ldots,\eta_{n}\in\mathbb{Z}/2\mathbb{Z}$ there always exist integers $\lambda_{1},\ldots,\lambda_{n+1}$ and $\nu_{1},\ldots,\nu_{n}$ satisfying the interlacing condition (6.5) and $\xi_{i}=\lambda_{i}+2\mathbb{Z}$, $\eta_{i}=\nu_{i}+2\mathbb{Z}$.

6.3.3. $(G,H)=(\operatorname{U}(p,q+1),\operatorname{U}(p,q))$

Assume that $p\geq q+1$; the case $p\leq q$ is handled similarly. We extend $\mathfrak{a}_{G}$ to the Cartan subalgebra $\mathfrak{j}_{G}=\mathfrak{t}_{G}\oplus\mathfrak{a}_{G}$ with

[TABLE]

Let $f_{i}\in\mathfrak{t}_{G,\mathbb{C}}^{\vee}$ ($1\leq i\leq q+1$) be the linear functionals mapping a diagonal matrix as above to $\sqrt{-1}t_{i}$, and $g_{i}\in\mathfrak{t}_{G,\mathbb{C}}^{\vee}$ ($1\leq i\leq p-q-1$) the ones mapping to $\sqrt{-1}s_{i}$. Then the root system $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ is of the form

[TABLE]

and putting

[TABLE]

we have $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\pm(\varepsilon_{i}-\varepsilon_{j}):1\leq i<j\leq p+q+1\}$. Further, the positive system $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\varepsilon_{i}-\varepsilon_{j}:1\leq i<j\leq p+q+1\}$ is compatible with the positive system $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

We choose the complexification $G_{\mathbb{C}}=\operatorname{GL}(p+q+1,\mathbb{C})$, then $\mathfrak{j}_{G,\mathbb{C}}$ is a Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{gl}}(p+q+1,\mathbb{C})$ and the irreducible finite-dimensional representations of $G_{\mathbb{C}}$ are parametrized by their highest weights $\lambda=\lambda_{1}\varepsilon_{1}+\cdots+\lambda_{p+q+1}\varepsilon_{p+q+1}$, $\lambda_{1}\geq\ldots\geq\lambda_{p+q+1}$, $\lambda_{i}\in\mathbb{Z}$.

An element $g=\operatorname{diag}(z_{1},\ldots,z_{q+1},{\rm\bf 1}_{p-q-1},z_{1},\ldots,z_{q+1})\in M_{G}$ acts on the highest restricted weight space of $F^{G}(\lambda)$ by $z_{1}^{\lambda_{1}+\lambda_{p+q+1}}\cdots z_{q+1}^{\lambda_{q+1}+\lambda_{p+1}}$. Further, $\operatorname{U}(p-q-1)\subseteq M_{G}$ has roots $\pm(\varepsilon_{i}-\varepsilon_{j})$ ($q+2\leq i<j\leq p$) and therefore its action on the highest restricted weight space is given by $F^{\operatorname{U}(p-q-1)}(\lambda_{q+2}\varepsilon_{q+2}+\cdots+\lambda_{p}\varepsilon_{p})$.

In the same way we parametrize irreducible finite-dimensional representations of $H_{\mathbb{C}}=\operatorname{GL}(p+q,\mathbb{C})$ by their highest weights $\nu=\nu_{1}\varepsilon_{1}+\cdots+\nu_{p+q}\varepsilon_{p+q}$. An element of the form $h=\operatorname{diag}(z_{1},\ldots,z_{q},{\rm\bf 1}_{p-q},z_{1},\ldots,z_{q})\in M_{H}$ acts on the highest restricted weight space by $z_{1}^{\nu_{1}+\nu_{p+q}}\cdots z_{q}^{\nu_{q}+\nu_{p+1}}$ and $\operatorname{U}(p-q)\subseteq M_{H}$ acts by $F^{\operatorname{U}(p-q)}(\nu_{q+1}\varepsilon_{q+1}+\cdots+\nu_{p}\varepsilon_{p})$.

Now, $M_{G}=\operatorname{U}(1)^{q+1}\times\operatorname{U}(p-q-1)$, $M_{H}=\operatorname{U}(1)^{q}\times\operatorname{U}(p-q)$ and $M=\operatorname{U}(p-q-1)$. An irreducible representation $\xi\in\widehat{M}_{G}$ is of the form $\xi=\xi^{\prime}\boxtimes\xi^{\prime\prime}$ with

[TABLE]

$\xi_{1}^{\prime},\ldots,\xi_{q+1}^{\prime}\in\mathbb{Z}$, and $\xi^{\prime\prime}=F^{\operatorname{U}(p-q-1)}(\xi_{1}^{\prime\prime}\varepsilon_{q+2}+\cdots+\xi_{p-q-1}^{\prime\prime}\varepsilon_{p})$, $\xi_{1}^{\prime\prime}\geq\ldots\geq\xi_{p-q-1}^{\prime\prime}$. Similarly, every $\eta\in\widehat{M}_{H}$ has the form $\eta=\eta^{\prime}\boxtimes\eta^{\prime\prime}$ with

[TABLE]

$\eta_{1}^{\prime},\ldots,\eta_{q}^{\prime}\in\mathbb{Z}$, and $\eta^{\prime\prime}=F^{\operatorname{U}(p-q)}(\eta_{1}^{\prime\prime}\varepsilon_{q+1}+\cdots+\eta_{p-q}^{\prime\prime}\varepsilon_{p})$, $\eta_{1}^{\prime\prime}\geq\ldots\geq\eta_{p-q}^{\prime\prime}$.

This implies that we have to put

[TABLE]

Since $\tilde{w}_{0}=\operatorname{diag}({\rm\bf 1}_{p},-{\rm\bf 1}_{q+1})$ commutes with $M_{G}$ we have $\xi^{\tilde{w}_{0}}=\xi$, so that $\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})$, and the condition $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$ is equivalent to the condition $\operatorname{Hom}_{\operatorname{U}(p-q-1)}(\xi^{\prime\prime},\eta^{\prime\prime}|_{\operatorname{U}(p-q-1)})\neq\{0\}$ which is in turn equivalent to

[TABLE]

Therefore, the already chosen $\lambda_{i}$’s and $\nu_{j}$’s satisfy the necessary interlacing condition for $\operatorname{Hom}_{H}(F^{G}(\lambda),F^{H}(\nu))\neq\{0\}$. It remains to show that one can choose the remaining $\lambda_{i}$’s and $\nu_{j}$’s such that the interlacing condition still holds and additionally $\lambda_{i}+\lambda_{p+q-i+2}=\xi_{i}^{\prime}$ and $\nu_{j}+\nu_{p+q-j+1}=\eta_{j}^{\prime}$, which is an easy exercise.

6.3.4. $(G,H)=(\operatorname{SO}(n+1,\mathbb{C}),\operatorname{SO}(n,\mathbb{C}))$

Assume that $n=2m$ is even; the case of odd $n$ is treated similarly. We extend $\mathfrak{a}_{G}$ to the Cartan subalgebra $\mathfrak{j}_{G}=\mathfrak{t}_{G}\oplus\mathfrak{a}_{G}$ with

[TABLE]

Let $f_{i}\in\mathfrak{t}_{G,\mathbb{C}}^{\vee}$ ($1\leq i\leq m$) be the linear functionals mapping a diagonal matrix as above to $\sqrt{-1}t_{i}$. Then the root system $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ is of the form

[TABLE]

and putting

[TABLE]

we have $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\pm\varepsilon_{i}^{\prime}\pm\varepsilon_{j}^{\prime},\pm\varepsilon_{i}^{\prime\prime}\pm\varepsilon_{j}^{\prime\prime}:1\leq i<j\leq m\}\cup\{\pm\varepsilon_{i}^{\prime},\pm\varepsilon_{i}^{\prime\prime}:1\leq i\leq m\}$. Further, the positive system $\Sigma^{+}(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})=\{\varepsilon_{i}^{\prime}\pm\varepsilon_{j}^{\prime},\varepsilon_{i}^{\prime\prime}\pm\varepsilon_{j}^{\prime\prime}:1\leq i<j\leq m\}\cup\{\varepsilon_{i}^{\prime},\varepsilon_{i}^{\prime\prime}:1\leq i\leq m\}$ is compatible with the positive system $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

We choose the complexification $G_{\mathbb{C}}=\operatorname{SO}(n+1,\mathbb{C})\times\operatorname{SO}(n+1,\mathbb{C})$ with embedding $G\hookrightarrow G_{\mathbb{C}}$ given by $g\mapsto(g,\overline{g})$; then $\mathfrak{j}_{G,\mathbb{C}}$ is a Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{so}}(n+1,\mathbb{C})+\operatorname{\mathfrak{so}}(n+1,\mathbb{C})$ and the irreducible finite-dimensional representations of $G_{\mathbb{C}}$ are parametrized by their highest weights $\lambda=\lambda_{1}^{\prime}\varepsilon_{1}^{\prime}+\cdots+\lambda_{m}^{\prime}\varepsilon_{m}^{\prime}+\lambda_{1}^{\prime\prime}\varepsilon_{1}^{\prime\prime}+\cdots+\lambda_{m}^{\prime\prime}\varepsilon_{m}^{\prime\prime}$, $\lambda_{1}^{\prime}\geq\ldots\geq\lambda_{m-1}^{\prime}\geq|\lambda_{m}^{\prime}|$, $\lambda_{1}^{\prime\prime}\geq\ldots\geq\lambda_{m-1}^{\prime\prime}\geq|\lambda_{m}^{\prime\prime}|$, $\lambda_{i}^{\prime},\lambda_{i}^{\prime\prime}\in\mathbb{Z}$. An element $g=\exp(\sqrt{-1}\operatorname{diag}(D(t_{1}),\ldots,D(t_{m})))\in M_{G}$ acts on the highest weight space of $F^{G}(\lambda)$ by $e^{\sqrt{-1}(\lambda_{1}^{\prime}-\lambda_{1}^{\prime\prime})t_{1}}\cdots e^{\sqrt{-1}(\lambda_{m}^{\prime}-\lambda_{m}^{\prime\prime})t_{m}}$.

In the same way, we parametrize irreducible finite-dimensional representations of $H_{\mathbb{C}}=\operatorname{SO}(n,\mathbb{C})\times\operatorname{SO}(n,\mathbb{C})$ by their highest weights $\nu=\nu_{1}^{\prime}\varepsilon_{1}^{\prime}+\cdots+\nu_{m-1}^{\prime}\varepsilon_{m-1}^{\prime}+\nu_{1}^{\prime\prime}\varepsilon_{1}^{\prime\prime}+\cdots+\nu_{m-1}^{\prime\prime}\varepsilon_{m-1}^{\prime\prime}$, $\nu_{1}^{\prime}\geq\ldots\geq\nu_{m-1}^{\prime}$, $\nu_{1}^{\prime\prime}\geq\ldots\geq\nu_{m-1}^{\prime\prime}$, $\nu_{i}^{\prime},\nu_{i}^{\prime\prime}\in\mathbb{Z}$. On its highest weight space, an element $h=\exp(\sqrt{-1}\operatorname{diag}(D(t_{1}),\ldots,D(t_{m-1}),0))\in M_{H}$ acts by $e^{\sqrt{-1}(\nu_{1}^{\prime}-\nu_{1}^{\prime\prime})t_{1}}\cdots e^{\sqrt{-1}(\nu_{m-1}^{\prime}-\nu_{m-1}^{\prime\prime})t_{m-1}}$.

Now, $M_{G}=\operatorname{SO}(2)^{m}$, $M_{H}=\operatorname{SO}(2)^{m-1}$ and $M=\{{\rm\bf 1}\}$. Hence, irreducible representations of $M_{G}$ and $M_{H}$ are one-dimensional and $\dim\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\dim\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})=1$ for all $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$. Every $\xi\in\widehat{M}_{G}$ has the form

[TABLE]

with $\xi_{1},\ldots,\xi_{m}\in\mathbb{Z}$. Similarly, every $\eta\in\widehat{M}_{H}$ has the form

[TABLE]

with $\eta_{1},\ldots,\eta_{m-1}\in\mathbb{Z}$. By the above observations, $\xi\simeq F^{G}(\lambda)^{\mathfrak{n}_{G}}|_{M_{G}}$ if and only if $\lambda_{i}^{\prime}-\lambda_{i}^{\prime\prime}=\xi_{i}$ ($1\leq i\leq m$). Further, $\eta\simeq F^{H}(\nu)^{\mathfrak{n}_{H}}|_{M_{H}}$ if and only if $\nu_{i}^{\prime}-\nu_{i}^{\prime\prime}=\eta_{i}$ ($1\leq i\leq m-1$). Moreover, we have $\operatorname{Hom}_{H}(F^{G}(\lambda),F^{H}(\nu))\neq\{0\}$ if and only if

[TABLE]

We first choose $(\lambda_{m}^{\prime},\lambda_{m}^{\prime\prime})\in\mathbb{Z}\times\mathbb{Z}$ with $\lambda_{m}^{\prime}-\lambda_{m}^{\prime\prime}=\xi_{m}$. Next we choose $(\nu_{m-1}^{\prime},\nu_{m-1}^{\prime\prime})\in\mathbb{N}\times\mathbb{N}$ with $\nu_{m-1}^{\prime}\geq|\lambda_{m}^{\prime}|$, $\nu_{m-1}^{\prime\prime}\geq|\lambda_{m}^{\prime\prime}|$ and $\nu_{m-1}^{\prime}-\nu_{m-1}^{\prime\prime}=\eta_{m-1}$. Iterating this procedure shows the claim.

6.3.5. $(G,H)=(\operatorname{SO}(p,q+1),\operatorname{SO}(p,q))$

Assume that $p\leq q$; the case $p\geq q+1$ is handled similarly. We further assume that $q-p=2m$ is even, leaving the odd case to the reader. We extend $\mathfrak{a}_{G}$ to the Cartan subalgebra $\mathfrak{j}_{G}=\mathfrak{t}_{G}\oplus\mathfrak{a}_{G}$ with

[TABLE]

Let $e_{p+i}\in\mathfrak{t}_{G,\mathbb{C}}^{\vee}$ ($1\leq i\leq m$) be the linear functional mapping a matrix as above to $\sqrt{-1}t_{i}$. Then the root system $\Sigma(\mathfrak{g}_{\mathbb{C}},\mathfrak{j}_{G,\mathbb{C}})$ is of the form

[TABLE]

and the positive system

[TABLE]

is compatible with the positive system $\Sigma^{+}(\mathfrak{g},\mathfrak{a}_{G})$.

We choose the complexification $G_{\mathbb{C}}=\operatorname{SO}(p+q+1,\mathbb{C})$, then $\mathfrak{j}_{G,\mathbb{C}}$ is a Cartan subalgebra of $\mathfrak{g}_{\mathbb{C}}=\operatorname{\mathfrak{so}}(p+q+1,\mathbb{C})$ and the irreducible finite-dimensional representations of $G_{\mathbb{C}}$ are parametrized by their highest weights $\lambda=\lambda_{1}e_{1}+\cdots+\lambda_{p+m}e_{p+m}$, where $\lambda_{1}\geq\ldots\geq\lambda_{p+m}\geq 0$, $\lambda_{i}\in\mathbb{Z}$. An element $g=\operatorname{diag}((-1)^{k_{1}},\ldots,(-1)^{k_{p}},(-1)^{k_{1}},\ldots,(-1)^{k_{p}},{\rm\bf 1}_{q-p-1})\in M_{G}$ acts on the highest restricted weight space of $F^{G}(\lambda)$ by $(-1)^{k_{1}\lambda_{1}+\cdots+k_{p}\lambda_{p}}$. Further, $\operatorname{SO}(q-p+1)\subseteq M_{G}$ acts on the highest restricted weight space by $F^{\operatorname{SO}(q-p+1)}(\lambda_{p+1}e_{p+1}+\cdots+\lambda_{p+m}e_{p+m})$.

In the same way we parametrize irreducible finite-dimensional representations of $H_{\mathbb{C}}=\operatorname{SO}(p+q,\mathbb{C})$ by their highest weights $\nu=\nu_{1}e_{1}+\cdots+\nu_{p+m}e_{p+m}$. An element $h=\operatorname{diag}(x_{1},\ldots,x_{p},x_{1},\ldots,x_{p},{\rm\bf 1}_{q-p+1})\in M_{H}$ acts by $x_{1}^{\nu_{1}}\cdots x_{p}^{\nu_{p}}$ and $\operatorname{SO}(q-p)\subseteq M_{H}$ acts by $F^{\operatorname{SO}(q-p)}(\nu_{p+1}e_{p+1}+\cdots+\nu_{p+m}e_{p+m})$.

Now, $M_{G}=\operatorname{O}(1)^{p}\times\operatorname{SO}(q-p+1)$, $M_{H}=\operatorname{O}(1)^{p}\times\operatorname{SO}(q-p)$ and $M=\operatorname{SO}(q-p)$. An irreducible representation $\xi\in\widehat{M}_{G}$ is of the form $\xi=\xi^{\prime}\boxtimes\xi^{\prime\prime}$ with

[TABLE]

$\xi_{1}^{\prime},\ldots,\xi_{p}^{\prime}\in\mathbb{Z}/2\mathbb{Z}$, and $\xi^{\prime\prime}=F^{\operatorname{SO}(q-p+1)}(\xi_{1}^{\prime\prime}e_{p+1}+\cdots+\xi_{m}^{\prime\prime}e_{p+m})$, $\xi_{1}^{\prime\prime}\geq\ldots\geq\xi_{m}^{\prime\prime}\geq 0$. Similarly, every $\eta\in\widehat{M}_{H}$ has the form $\eta=\eta^{\prime}\boxtimes\eta^{\prime\prime}$ with

[TABLE]

$\eta_{1}^{\prime},\ldots,\eta_{p}^{\prime}\in\mathbb{Z}/2\mathbb{Z}$, and $\eta^{\prime\prime}=F^{\operatorname{SO}(q-p)}(\eta_{1}^{\prime\prime}e_{p+1}+\cdots+\eta_{m}^{\prime\prime}e_{p+m})$, $\eta_{1}^{\prime\prime}\geq\ldots\geq\eta_{m-1}^{\prime\prime}\geq|\eta_{m}^{\prime\prime}|$.

This implies that we have to put

[TABLE]

Since $\tilde{w}_{0}=\operatorname{diag}(-{\rm\bf 1}_{p},{\rm\bf 1}_{q+1})$ commutes with $M_{G}$ we have $\xi^{\tilde{w}_{0}}=\xi$, so that $\operatorname{Hom}_{M}(\xi^{\tilde{w}_{0}}|_{M},\eta|_{M})=\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})$, and the condition $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$ is equivalent to the condition $\operatorname{Hom}_{\operatorname{SO}(q-p)}(\xi^{\prime\prime}|_{\operatorname{SO}(q-p)},\eta^{\prime\prime})\neq\{0\}$ which is in turn equivalent to

[TABLE]

Therefore, the already chosen $\lambda_{i}$’s and $\nu_{j}$’s satisfy the necessary interlacing condition for $\operatorname{Hom}_{H}(F^{G}(\lambda),F^{H}(\nu))\neq\{0\}$. It remains to show that one can choose the remaining $\lambda_{i}$’s and $\nu_{j}$’s such that the interlacing condition holds and additionally $\xi_{i}^{\prime}=\lambda_{i}+2\mathbb{Z}$ and $\eta_{j}^{\prime}=\nu_{j}+2\mathbb{Z}$, which is an easy exercise.

6.4. Generic multiplicities

Using the intertwining operators between spherical principal series constructed in Section 3 and the upper multiplicity bounds obtained in Section 4, we prove the following generic multiplicity formula for general principal series representations of multiplicity one pairs:

Theorem 6.5**.**

Assume that $(G,H)$ is one of the multiplicity one pairs in (6.1). Then for all $(\xi,\eta)\in\widehat{M}_{G}\times\widehat{M}_{H}$ and $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we have the lower multiplicity bound

[TABLE]

and for $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ satisfying the generic condition (4.1) we have

[TABLE]

Proof.

By Lemma 6.3 we have $\#(P_{H}\backslash G/P_{G})_{\rm open}=1$. Hence, it follows from Theorem 4.1 and Proposition 6.4 (1) that

[TABLE]

for $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ satisfying (4.1). It therefore suffices to show that for $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$ we have $\dim\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})\geq 1$ for all $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$.

By Proposition 6.4 (2) and Corollary 6.2 there exist $(\lambda_{0},\nu_{0})\in\mathfrak{a}_{G}^{\vee}\times\mathfrak{a}_{H}^{\vee}$ and a linear map

[TABLE]

which is on the level of distribution kernels given by tensoring with a fixed non-trivial real-analytic section. We apply this map to the holomorphic family of intertwining operators obtained in Theorem 3.3. More precisely, by Theorem 3.3 there exists a family $K_{\lambda,\nu}$ of distribution kernels of intertwining operators $A_{\lambda,\nu}\in\operatorname{Hom}_{H}(\pi_{\lambda}|_{H},\tau_{\nu})$, depending holomorphically on $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$, such that generically $\operatorname{supp}K_{\lambda,\nu}=G/P_{G}$. Then the distribution kernels of $\Phi(A_{\lambda,\nu})$ depend holomorphically on $(\lambda,\nu)\in\mathfrak{a}_{G,\mathbb{C}}^{\vee}\times\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ since they are given by tensoring the holomorphic family $K_{\lambda,\nu}$ with a fixed smooth section. Further, by Corollary 6.2 they are generically supported on $G/P_{G}$ and hence the holomorphic family $\Phi(A_{\lambda,\nu})\in\operatorname{Hom}_{H}(\pi_{\xi,\lambda+\lambda_{0}}|_{H},\tau_{\eta,\nu+\nu_{0}})$ is non-trivial. Now the desired lower multiplicity bound follows from Lemma 3.5. ∎

Combining Theorem 6.5 with the multiplicity one statement in Fact II we immediately obtain the following corollary:

Corollary 6.6**.**

Let $(G,H)$ be one of the pairs in (6.1) and assume that $\pi_{\xi,\lambda}$ and $\tau_{\eta,\nu}$ are irreducible. Then, if $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$, we have

[TABLE]

6.5. The Gross–Prasad conjecture for complex orthogonal groups

In 1992 B. Gross and D. Prasad [8] formulated a conjecture about the multiplicities $\dim_{H}(\pi|_{H},\tau)$ for the reductive pair $(G,H)=(\operatorname{SO}(n+1),\operatorname{SO}(n))$ over local and global fields. For the field $k=\mathbb{C}$ the local conjecture takes the following form:

Conjecture 6.7** ([8, Conjecture 11.5]).**

Let $(G,H)=(\operatorname{SO}(n+1,\mathbb{C}),\operatorname{SO}(n,\mathbb{C}))$ and assume that $\pi_{\xi,\lambda}$ and $\tau_{\eta,\nu}$ are irreducible; then $\dim\operatorname{Hom}_{H}(\pi_{\xi,\lambda}|_{H},\tau_{\eta,\nu})=1$.

Using our results from the previous section we can prove this conjecture. It follows from the following more general statement:

Corollary 6.8**.**

Let $(G,H)$ be one of the pairs in (6.1) and assume that $p=q$ or $p=q+1$ in the case of indefinite unitary or orthogonal groups. Then, if the representations $\pi_{\xi,\lambda}$ and $\tau_{\eta,\nu}$ are irreducible, we have

[TABLE]

Proof.

In all cases we have $M=\{{\rm\bf 1}\}$ so that $\operatorname{Hom}_{M}(\xi|_{M},\eta|_{M})\neq\{0\}$ for all $\xi\in\widehat{M}_{G}$ and $\eta\in\widehat{M}_{H}$. Then the statement follows from Corollary 6.6. ∎

7. An example: $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$

For the multiplicity one pair $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$ we describe the meromorphic families of intertwining operators between principal series as constructed in Section 6 explicitly. Over $p$-adic fields such operators were previously constructed by Murase–Sugano [34] (see also the recent work by Neretin [35] in the context of finite-dimensional representations) and the formulas for the distribution kernels turn out to be formally the same in the real case.

7.1. Distribution kernels

For $1\leq p\leq n+1$ and $1\leq q\leq n$, we define the following polynomial functions on $M((n+1)\times(n+1),\mathbb{R})$:

[TABLE]

With the representative

[TABLE]

for the longest Weyl group element $w_{0}\in W(\mathfrak{a}_{G})$, we then consider the functions $g\mapsto\Phi_{k}(\tilde{w}_{0}g),\Psi_{k}(\tilde{w}_{0}g)$ on $G$. For $d=\operatorname{diag}(d_{1},\ldots,d_{n+1})\in M_{G}A_{G}$ and $n\in N_{G}$ we have

[TABLE]

We identify $\widehat{M}_{G}\simeq(\mathbb{Z}/2\mathbb{Z})^{n+1}$ by mapping $\xi=(\xi_{1},\ldots,\xi_{n+1})\in(\mathbb{Z}/2\mathbb{Z})^{n+1}$ to the character

[TABLE]

Similarly $\widehat{M}_{H}\simeq(\mathbb{Z}/2\mathbb{Z})^{n}$. Further, we identify $\mathfrak{a}_{G,\mathbb{C}}^{\vee}\simeq\mathbb{C}^{n+1}$ by $\lambda\mapsto(\lambda(E_{1,1}),\ldots,\lambda(E_{n+1,n+1}))$ and similarly $\mathfrak{a}_{H,\mathbb{C}}^{\vee}\simeq\mathbb{C}^{n}$. Then for $\xi=(\xi_{1},\ldots,\xi_{n+1})\in(\mathbb{Z}/2\mathbb{Z})^{n+1}\simeq\widehat{M}_{G}$, $\eta=(\eta_{1},\ldots,\eta_{n})\in(\mathbb{Z}/2\mathbb{Z})^{n}\simeq\widehat{M}_{H}$ and $\lambda\in\mathbb{C}^{n+1}\simeq\mathfrak{a}_{G,\mathbb{C}}^{\vee}$, $\nu\in\mathbb{C}^{n}\simeq\mathfrak{a}_{H,\mathbb{C}}^{\vee}$ we put

[TABLE]

where

[TABLE]

and

[TABLE]

Here we have used the notation

[TABLE]

Then $K_{(\xi,\lambda),(\eta,\nu)}$ satisfies

[TABLE]

for $g\in G$, $man\in P_{G}$, $m^{\prime}a^{\prime}n^{\prime}\in P_{H}$. Hence, the functions $K_{(\xi,\lambda),(\eta,\nu)}$ define a meromorphic family of intertwining operators $A_{(\xi,\lambda),(\eta,\nu)}:\pi_{\xi,\lambda}|_{H}\to\tau_{\eta,\nu}$ by

[TABLE]

Remark 7.1**.**

It is easy to see that the functions $g\mapsto\Phi_{k}(g),\Psi_{k}(g)$ are matrix coefficients for the irreducible finite-dimensional representation of $\operatorname{GL}(n+1,\mathbb{R})$ on $\bigwedge^{k}\mathbb{R}^{n+1}$.

Appendix A Finite-dimensional branching rules for strong Gelfand pairs

We list the explicit branching rules for some small representations of the strong Gelfand pairs $(\operatorname{\mathfrak{sl}}(n+1,\mathbb{C}),\operatorname{\mathfrak{gl}}(n,\mathbb{C}))$ and $(\operatorname{\mathfrak{so}}(n+1,\mathbb{C}),\operatorname{\mathfrak{so}}(n,\mathbb{C}))$. The classical branching rules for these pairs can be found e.g. in [6, Chapter 8].

A.1. $(\operatorname{\mathfrak{sl}}(n+1,\mathbb{C}),\operatorname{\mathfrak{gl}}(n,\mathbb{C}))$

We label the Dynkin diagrams of $\operatorname{\mathfrak{sl}}(n+1,\mathbb{C})$ and $\operatorname{\mathfrak{sl}}(n,\mathbb{C})$ as usual:

[TABLE]

[TABLE]

Realize the root system of $\operatorname{\mathfrak{sl}}(n+1,\mathbb{C})$ as $\{\pm(e_{i}-e_{j}):1\leq i<j\leq n+1\}$ in the vector space $V=\{x\in\mathbb{R}^{n+1}:x_{1}+\cdots+x_{n+1}=0\}$. To simplify notation, denote by $\pi(x)$ the orthogonal projection of $x\in\mathbb{R}^{n+1}$ to $V$. We choose the simple roots $\alpha_{i}=e_{i}-e_{i+1}$ ($1\leq i\leq n$) for $\operatorname{\mathfrak{sl}}(n+1,\mathbb{C})$ and the simple roots $\beta_{i}=\alpha_{i}$ ($i=1,\ldots,n-1$) for $\operatorname{\mathfrak{sl}}(n,\mathbb{C})$. Denote by $\varpi_{1},\ldots,\varpi_{n}$ the corresponding fundamental weights for $\operatorname{\mathfrak{sl}}(n+1,\mathbb{C})$ and by $\zeta_{1},\ldots,\zeta_{n-1}$ the fundamental weights for $\operatorname{\mathfrak{sl}}(n,\mathbb{C})$. Further put $\zeta_{n}:=\varpi_{n}$, then $\zeta_{n}$ describes a character of $\mathfrak{z}(\operatorname{\mathfrak{gl}}(n,\mathbb{C}))\simeq\mathbb{C}$.

Consider the fundamental weight $\varpi_{i}=\pi(e_{1}+\cdots+e_{i})$. From the classical branching laws we know that $F^{\mathfrak{g}}(\varpi_{i})|_{\mathfrak{h}}$ decomposes into the direct sum of the two $\mathfrak{h}$-representations with highest weights $\pi(e_{1}+\cdots+e_{i})$ and $\pi(e_{1}+\cdots+e_{i-1}+e_{n+1})$. Using

[TABLE]

it follows that

[TABLE]

Now consider the fundamental weight $\varpi_{i}+\varpi_{n-i+1}=(e_{1}+\cdots+e_{i})-(e_{n-i+2}+\cdots+e_{n+1})$ ($1\leq i\leq\frac{n}{2}$). From the classical branching laws we know that $F^{\mathfrak{g}}(\varpi_{i}+\varpi_{n-i+1})|_{\mathfrak{h}}$ decomposes into the direct sum of the four $\mathfrak{h}$-representations with highest weights

[TABLE]

so that

[TABLE]

Note that for $i=\frac{n}{2}$ with $n=2m$ even, the formula still holds and we have $\zeta_{i}+\zeta_{n-i}=2\zeta_{m}$. Similarly one obtains for $i=\frac{n+1}{2}$ with $n=2m-1$ odd,

[TABLE]

A.2. $(\operatorname{\mathfrak{so}}(n+1,\mathbb{C}),\operatorname{\mathfrak{so}}(n,\mathbb{C}))$

We label the Dynkin diagrams of $\operatorname{\mathfrak{so}}(n+1,\mathbb{C})$ and $\operatorname{\mathfrak{so}}(n,\mathbb{C})$ as usual:

[TABLE]

From the classical branching rules it follows that, for $n=2m$ even,

[TABLE]

and for $n=2m-1$ odd,

[TABLE]