Branching laws of unitary representations associated to minimal elliptic orbits for indefinite orthogonal group O(p,q)

TL;DR

This paper provides a comprehensive analysis of how certain unitary representations of indefinite orthogonal groups decompose when restricted to specific subgroups, including explicit constructions and formulas.

Contribution

It offers a complete description of the discrete spectra in the branching laws for representations associated with minimal elliptic orbits, including explicit holographic operators and a Parseval-type formula.

Findings

Complete description of discrete spectra in branching laws

Explicit construction of holographic operators

Proof of a closed Parseval-type formula

Abstract

We give a complete description of the discrete spectra in the branching law $\Pi|_{G'}$ with respect to the pair $(G,G')=(O(p,q), O(p',q') \times O(p'',q''))$ for irreducible unitary representations $\Pi$ of $G$ that are "geometric quantization" of minimal elliptic coadjoint orbits. We also construct explicitly all holographic operators and prove a closed Parseval-type formula.

Tables

Table 1. Table 7.1 : Triples ( 𝔤 ℂ , 𝔥 ℂ , 𝔤 ℂ ′ ) subscript 𝔤 ℂ subscript 𝔥 ℂ superscript subscript 𝔤 ℂ ′ ({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime}) with 𝔤 ℂ subscript 𝔤 ℂ {\mathfrak{g}}_{\mathbb{C}} simple in Theorem 7.6

${𝔤}_{ℂ}$	${𝔥}_{ℂ}$	${𝔤}_{ℂ}^{\prime }$
$𝔰{𝔩}_n$	$𝔤{𝔩}_{n-1}$	$𝔰{𝔩}_p\oplus 𝔰{𝔩}_q\oplus ℂ$
$𝔰{𝔩}_{2m}$	$𝔤{𝔩}_{2m-1}$	$𝔰{𝔭}_m$
$𝔰{𝔩}_6$	$𝔰{𝔭}_3$	$𝔰{𝔩}_4\oplus 𝔰{𝔩}_2\oplus ℂ$
$𝔰{𝔬}_n$	$𝔰{𝔬}_{n-1}$	$𝔰{𝔬}_p\oplus 𝔰{𝔬}_q$
$𝔰{𝔬}_{2m}$	$𝔰{𝔬}_{2m-1}$	$𝔤{𝔩}_m$
$𝔰{𝔬}_{2m}$	$𝔰{𝔬}_{2m-2}\oplus ℂ$	$𝔤{𝔩}_m$
$𝔰{𝔭}_n$	$𝔰{𝔭}_{n-1}\oplus 𝔰{𝔭}_1$	$𝔰{𝔭}_p\oplus 𝔰{𝔭}_q$
$𝔰{𝔭}_n$	$𝔰{𝔭}_{n-2}\oplus 𝔰{𝔭}_2$	$𝔰{𝔭}_{n-1}\oplus 𝔰{𝔭}_1$
${𝔢}_6$	${𝔣}_4$	$𝔰{𝔬}_{10}\oplus ℂ$
${𝔣}_4$	$𝔰{𝔬}_9$	$𝔰{𝔬}_9$

Equations

$${\dim }_{\mathbb{C}}{\mathrm{Hom}}_{G^{\prime }}(\pi ,\Pi {|}_{G^{\prime }})\le 1\text{for all π∈G′},$$

$$\mathrm{Disc}(\Pi {|}_{G^{\prime }}):=\{\pi \in \widehat{G^{\prime }}:{\mathrm{Hom}}_{G^{\prime }}(\pi ,\Pi {|}_{G^{\prime }})\ne \{0\}\},$$

$$\underset{(\delta ,\epsilon )\in \{-+,++,+-\}}{⨁}\underset{({\lambda }^{\prime },{\lambda }^{\prime \prime })\in {\Lambda }_{\delta \epsilon }(\lambda )}{{\sum }^{\oplus }}{\pi }_{\delta ,{\lambda }^{\prime }}^{p^{\prime },q^{\prime }}⊠{\pi }_{\epsilon ,{\lambda }^{\prime \prime }}^{p^{\prime \prime },q^{\prime \prime }}\text{(Hilbert direct sum).}$$

$${\Lambda }_{-+}(\lambda )$$

$${\Lambda }_{++}(\lambda )$$

$${\Lambda }_{+-}(\lambda )$$

$$\Pi {|}_{G^{\prime }}=\underset{n\in \mathbb{N}}{{\sum }^{\oplus }}{\pi }_{+,\lambda +n+\frac{1}{2}}^{p,q-1}⊠(\mathrm{sgn}{)}^n,$$

$$\pi ={\pi }_{+,\lambda -n-\frac{1}{2}}^{p-1,q}⊠(\mathrm{sgn}{)}^n\text{for some 0≤n<λ−21}.$$

$${\mathrm{Hom}}_{G^{\prime }}({\Pi }^{\infty }{|}_{G^{\prime }},{\pi }^{\infty })\ne \{0\}\text{if (λ′,λ′′)∈Λδ,ε(λ)}.$$

$$T_{\delta \epsilon ,\lambda }^{{\lambda }^{\prime },{\lambda }^{\prime \prime }}:{\pi }_{\delta ,{\lambda }^{\prime }}^{p^{\prime },q^{\prime }}⊠{\pi }_{\epsilon ,{\lambda }^{\prime \prime }}^{p^{\prime \prime },q^{\prime \prime }}\to {\pi }_{+,\lambda }^{p,q},$$

$$A_{+}(p,q):=$$

$$A_{-}(p,q):=$$

$$\rho :=\frac{1}{2}(p+q-2).$$

$$b\equiv$$

$$\delta \equiv$$

$$(\lambda ,\frac{p+q}{2}-2,\frac{p+q}{2}-3,\cdots ,\frac{p+q}{2}-[\frac{p+q}{2}])$$

$$\{\begin{array}{ll}{\mathcal{H}}^b({\mathbb{R}}^p)⊠1& \text{if b≥0},\\ 1⊠1& \text{if b≤0}.\end{array}$$

$$A_{+}(p,q)=\{\frac{1}{2},-\frac{1}{2}\}$$

$$\{{\pi }_{+,\lambda }^{p,q}:\lambda \in A_{+}(p,q)\}\cup \{{\pi }_{-,\lambda }^{p,q}:\lambda \in A_{-}(p,q).\}$$

$$\{\pm f_i\pm f_j:1\le i<j\le [\frac{p+q}{2}]\}(\cup \{\pm f_i:1\le i\le [\frac{p+q}{2}]\}(\text{p+q: odd})).$$

$$\Delta (\mathfrak{u},{\mathfrak{j}}_{\mathbb{C}})=\{f_1\pm f_j:2\le j\le [\frac{p+q}{2}]\}(\cup \{f_1\}(\text{p+q: odd})).$$

$$L\simeq SO(2)\times O(p-2,q).$$

$${\mathcal{R}}_{\mathfrak{q}}^{p-2}({\mathbb{C}}_{\lambda f_1})\simeq A_{\mathfrak{q}}(\lambda -\rho ).$$

$${\mathcal{O}}_{\nu }\equiv {\mathcal{O}}_{+,\nu }:={\mathrm{Ad}}^{\ast }(G)(\nu f_1)(\subset \sqrt{-1}{\mathfrak{g}}^{\ast }),$$

$${\mathcal{L}}_{\lambda +\rho }\to {\mathcal{O}}_{\lambda }\simeq G/L,$$

$$H_{\overline{\partial }}^j(G/L,{\mathcal{L}}_{\lambda +\rho })$$

$$\mathrm{DIM}{\pi }_{\epsilon ,\lambda }^{p,q}=\frac{1}{2}\dim {\mathcal{O}}_{\epsilon ,\lambda }=p+q-2\text{for ε=±}.$$

$$MA\simeq GL(1,\mathbb{R})\times O(p-1,q-1).$$

$$S^{p-1}\times S^{q-1}\simeq G/P_{+}\to G/P$$

$$\Xi (K,b):=\underset{\begin{array}{c}m,n\in \mathbb{N}\\ m-n\in 2\mathbb{N}+b\end{array}}{⨁}{\mathcal{H}}^m({\mathbb{R}}^p)⊠{\mathcal{H}}^n({\mathbb{R}}^q)\text{(algebraic direct sum).}$$

$$g_{{\mathbb{R}}^{p,q}}:=d{x}_1^2+\cdots +d{x}_p^2-d{y}_1^2-\cdots -d{y}_q^2.$$

$$X(p,q)\equiv X(p,q{)}_{+}:=\{(x,y)\in {\mathbb{R}}^{p+q}:|x{|}^2-|y{|}^2=1\},$$

Full text

MSC 2010: Primary 22E46; Secondary 22E45, 53D50, 58J42, 53C50.

1 Introduction and main results

In this article, we determine the discrete spectra of the restriction $\Pi|_{G^{\prime}}$ of an irreducible unitary representation of $G$ to a subgroup $G^{\prime}$, where

1. $\bullet$

$\Pi$ is “attached to” a minimal elliptic coadjoint orbit (Section 2), 2. $\bullet$

$(G,G^{\prime})=(O(p,q),O(p^{\prime},q^{\prime})\times O(p^{\prime\prime},q^{\prime\prime}))$ with $p=p^{\prime}+p^{\prime\prime}$ and $q=q^{\prime}+q^{\prime\prime}$.

We denote by $\widehat{G^{\prime}}$ the set of equivalence classes of irreducible unitary representations of $G^{\prime}$ (unitary dual). In Theorem 1.1 we prove a multiplicity-free theorem asserting

[TABLE]

and give a complete description of the discrete spectra for the branching:

[TABLE]

where $\operatorname{Hom}_{G^{\prime}}(\,,\,)$ denotes the space of continuous $G^{\prime}$-homomorphisms.

The irreducible unitary representations $\Pi$ in consideration are of various aspects such as

1. $\bullet$

they are “geometric quantization” of indefinite Kähler manifolds (Section 2.3); 2. $\bullet$

they are “discrete series representations” for pseudo-Riemannian space forms (Section 2.5), [F79, S83]; 3. $\bullet$

they are “unitarization” of the Zuckerman derived functor modules that are cohomological induction from a maximal $\theta$-stable parabolic subalgebra ${\mathfrak{q}}$ (Section 2.2), [V87, VZ84].

The representations $\Pi$ of $G=O(p,q)$ are parametrized by $\varepsilon\in\{\pm\}$ and $\lambda\in A_{\varepsilon}(p,q)$, see Definition-Theorem 2.1, and will be denoted by $\pi_{\varepsilon,\lambda}^{p,q}.$

Our first main result gives a description of the discrete part (cf. Section 6.1) of the restriction $\Pi|_{G^{\prime}}$. Without loss of generality, we assume $\varepsilon=+$.

Theorem 1.1**.**

For $\lambda\in A_{+}(p,q)$, we set $\Pi=\pi_{+,\lambda}^{p,q}$, the irreducible unitary representation of $G=O(p,q)$, as in Definition-Theorem 2.1. Then the discrete part of the restriction $\Pi|_{G^{\prime}}$ is a multiplicity-free direct sum of irreducible unitary representations of the subgroup $G^{\prime}=O(p^{\prime},q^{\prime})\times O(p^{\prime\prime},q^{\prime\prime})$ as follows:

[TABLE]

Here the parameter set $\Lambda_{\delta,\varepsilon}(\lambda)$ is defined for $\lambda\in A_{+}(p,q)$ by

[TABLE]

We note that $\Lambda_{++}(\lambda)$ is a finite set, whereas $\Lambda_{{+-}}(\lambda)$ (also $\Lambda_{{-+}}(\lambda)$) is an infinite set unless it is empty.

Our proof is geometric and constructive. It is outlined as follows. First, we divide the pseudo-Riemannian space form $G/H=O(p,q)/O(p-1,q)$ into three regions (up to conull set) according to orbit types labeled by ${-+}$, ${++}$, ${+-}$ of the subgroup $G^{\prime}$. Second, we introduce $G^{\prime}$-intertwining operators (holographic operators) from each irreducible summand of (1.1) to the original representation $\pi_{+,\lambda}^{p,q}$ by realizing these representations in the space of eigenfunctions of the Laplacian on pseudo-Riemannian space forms (Theorem 4.3). The final step is to prove the exhaustion of (1.1), which is carried out by a careful estimate of the boundary behaviours of solutions that “holographic operators” must satisfy (Section 5).

Here is an example of Theorem 1.1 when $(p^{\prime\prime},q^{\prime\prime})=(1,0)$ and $(0,1)$.

Example 1.2**.**

Suppose $p\geq 2$ and $q\geq 1$. Let $\Pi:=\pi_{+,\lambda}^{p,q}\in\widehat{G}$ for $\lambda\in A_{+}(p,q)$.

1. (1)

([K93])* If $(p^{\prime\prime},q^{\prime\prime})=(0,1)$, then $\Lambda_{{-+}}(\lambda)=\Lambda_{{++}}(\lambda)=\emptyset$ and*

[TABLE]

where $\operatorname{sgn}$ stands for the nontrivial character of $O(1)\simeq O(1,0)$. 2. (2)

If $(p^{\prime\prime},q^{\prime\prime})=(1,0)$, then $\Lambda_{{-+}}(\lambda)=\Lambda_{{+-}}(\lambda)=\emptyset$. Moreover, $\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})\neq\{0\}$ if and only if $\pi\in\widehat{G^{\prime}}$ is of the form

[TABLE]

In the general case where $p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime}\geq 2$ and $\lambda>2$, all the three parameter sets $\Lambda_{{-+}}(\lambda)$, $\Lambda_{{++}}(\lambda)$, and $\Lambda_{{+-}}(\lambda)$ are nonempty (Section 6).

As a corollary of Theorem 1.1 and its proof, we find a necessary and sufficient condition on the quadruple $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ for the restriction $\Pi|_{G^{\prime}}$ to have the following properties:

1. $\bullet$

$\Pi|_{G^{\prime}}$ is discretely decomposable (Theorem 6.4), 2. $\bullet$

the discrete part (1.1) is at most a finite sum (Theorem 6.3), 3. $\bullet$

$\Pi|_{G^{\prime}}$ contains only continuous spectrum (Theorem 6.2).

Our results can be also applied to the existence problem of symmetry breaking operators between smooth representations of $G$ and its subgroup $G^{\prime}$. Let $\Pi^{\infty}$ be the Fréchet space of smooth vectors of the unitary representation $\Pi$ of $G$, and $\pi^{\infty}$ that of a unitary representation $\pi$ of the subgroup $G^{\prime}$.

Corollary 1.3**.**

Let $\Pi=\pi_{+,\lambda}^{p,q}\in\widehat{G}$ for $\lambda\in A_{+}(p,q)$ and $\pi=\pi_{\delta,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{\varepsilon,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}}\in\widehat{G^{\prime}}$ for some $(\delta,\varepsilon)=(-,+)$, $(+,+)$, or $(+,-)$. Then we have:

[TABLE]

The second main theorem in this article is a quantitative result: for every $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta,\varepsilon}(\lambda)$, we construct explicitly in a geometric model of representations a holographic operator (an injective $G^{\prime}$-intertwining operator)

[TABLE]

and find a closed formula of its operator norm (Theorem 4.3).

Branching laws in the same setting with specific choices of $p^{\prime}$, $p^{\prime\prime}$, $q^{\prime}$, $q^{\prime\prime}$ have been studied over 25 years:

1. $\bullet$

When $(p^{\prime\prime},q^{\prime})=(0,0)$, Theorem 1.1 is nothing but the $K$-type formula, and can be computed by a generalized Blattner formula of the Zuckerman derived functor modules [V87, K92], see also Faraut [F79], Howe–Tan [HT93]. 2. $\bullet$

When $p^{\prime\prime}=0$, the restriction $\Pi|_{G^{\prime}}$ is discretely decomposable (Theorem 6.4). In this case, Theorem 1.1 gives the whole branching law of the restriction $\Pi|_{G^{\prime}}$, which was determined in [K93, Thm. 3.3]. The special case $(p,q)=(3,3)$ with $(p^{\prime\prime},q^{\prime\prime})=(0,1)$ was also studied in [ØS08]. 3. $\bullet$

When $(q^{\prime},q^{\prime\prime})=(1,0)$ (hence $q=1$), the branching law of $\Pi|_{G^{\prime}}$ was obtained in [MO15]. In this case, $\Pi|_{G^{\prime}}$ contains also continuous spectrum. 4. $\bullet$

In the case $p^{\prime\prime}=q=1$, an analogous result to (1.2) was studied in [KS18b, Thms. 4.1 and 4.2] when $\Pi^{\infty}$ and $\pi^{\infty}$ are cohomologically induced representations from more general parabolic subalgebras. 5. $\bullet$

If $(p^{\prime\prime},q^{\prime\prime})=(1,0)$ or $(0,1)$, then $\operatorname{Hom}_{G^{\prime}}(\Pi^{\infty}|_{G^{\prime}},\pi^{\infty})$ is at most of one-dimensional by the general result of Sun and Zhu [SZ12]. In this case, the discrete spectra (1.1) are stated in Example 1.2, and some part of them have been obtained recently in Ørsted and Speh [ØS19] by a different approach under the constraints that $b(\lambda)\geq 0$ (see (2.4) for notation).

For general $p^{\prime}$, $q^{\prime}$, $p^{\prime\prime}$, $q^{\prime\prime}$, the complete classification of discrete spectra (Theorem 1.1), and the construction of all holographic operators with a Parseval-type theorem (Theorems 4.3 and 5.1) were presented at the conference “Analyse harmonique sur les groupes de Lie et les espaces symétriques” en l’honneur de Jacques Faraut held in Nancy-Strasbourg in June, 2005, however, the manuscript [K02] has not been published.

Because of growing interest in branching problems for reductive groups in recent years, I come to think that the results and the methods here might be of some help for further perspectives such as a possible generalization of the Gross–Prasad conjecture for nontempered representations (e.g. [GP92, KS18b, ØS19]) as well as analytic representation theory.

$\langle$Acknowledgements$\rangle$ The author was partially supported by Grant-in-Aid for Scientific Research (A) (18H03669), Japan Society for the Promotion of Science.

Notation: ${\mathbb{N}}=\{0,1,2,\dots\}$ and ${\mathbb{N}}_{+}=\{1,2,\dots\}$.

2 Irreducible unitary representations

attached to minimal elliptic orbits

In this section, we discuss a certain family of irreducible unitary representations of $G=O(p,q)$, denoted by $\pi_{\varepsilon,\lambda}^{p,q}$ with parameter $\varepsilon=\pm$ and $\lambda\in A_{\varepsilon}(p,q)$ defined as below:

[TABLE]

The representations $\pi_{\varepsilon,\lambda}^{p,q}$ are a generalization of the finite-dimensional representations of the compact group $O(p)$ on the space ${\mathcal{H}}^{m}({\mathbb{R}}^{p})$ of spherical harmonics (see Remark 2.2 (1)). These unitary representations $\pi_{\varepsilon,\lambda}^{p,q}$ have been treated from various aspects in scattered literatures ([F79, HT93, K92, K93, KØ03, ØS08, ØS19, S83]). For the convenience of the reader, we summarize a number of realizations of the representations $\pi_{\varepsilon,\lambda}^{p,q}$ when $\varepsilon=+$ in Section 2.1.

Throughout this section, we adopt the same notation as in [KØ03].

2.1 Summary: four realizations of $\pi_{\varepsilon,\lambda}^{p,q}$

We use the German lower case letter ${\mathfrak{g}}$, ${\mathfrak{k}}$, $\cdots$, to denote the Lie algebras of $G$, $K$, $\cdots$, and write ${\mathfrak{Z}}({\mathfrak{g}})$ for the center of the enveloping algebra of the complexified Lie algebra ${\mathfrak{g}}_{\mathbb{C}}={\mathfrak{g}}\otimes_{\mathbb{R}}{\mathbb{C}}$. For ${\mathfrak{g}}={\mathfrak{o}}(p,q)$, we set

[TABLE]

For $\lambda\in A_{+}(p,q)$, we put

[TABLE]

Definition-Theorem 2.1**.**

Let $p\geq 2$ and $q\geq 0$. For any $\lambda\in A_{+}(p,q)$, there exists a unique irreducible unitary representation of $G=O(p,q)$, to be denoted by $\pi_{+,\lambda}^{p,q}$, whose underlying $(\mathfrak{g},K)$-module is given by one of (therefore, any of) the following $(\mathfrak{g},K)$-modules that are isomorphic to each other:

1. (i)

The Zuckerman derived functor module $A_{\mathfrak{q}}(\lambda-\rho)$ (see Section 2.2); 2. (ii)

(geometric quantization of coadjoint orbits) the underlying $(\mathfrak{g},K)$-module of the Dolbeault cohomology $H_{\overline{\partial}}^{p-2}({\mathcal{O}}_{\lambda},{\mathcal{L}}_{\lambda+\rho})$ (see Section 2.3); 3. (iii)

the underlying $(\mathfrak{g},K)$-module of the subrepresentation of the parabolic induction $I_{\delta}(\lambda+\rho)$ with $K$-types $\Xi(K;b)$ (see Section 2.4); 4. (iii)′

the underlying $(\mathfrak{g},K)$-module of the quotient of the parabolic induction $I_{\delta}(-\lambda+\rho)$ with $K$-types $\Xi(K;b)$; 5. (iv)

the underlying $(\mathfrak{g},K)$-module of the discrete series representation $L^{2}(X(p,q))_{\lambda}$ (see Section 2.5) for the symmetric space $X(p,q)=O(p,q)/O(p-1,q)$.

The ${\mathfrak{Z}}({\mathfrak{g}})$-infinitesimal character of $\pi_{+,\lambda}^{p,q}$ is given by

[TABLE]

in the Harish-Chandra parametrization for the standard basis, and the minimal $K$-type of $\pi_{+,\lambda}^{p,q}$ is given by

[TABLE]

The proof of the equivalence is given in [K92, Thm. 3] and [KØ03, Sect. 5.4], see also references therein. Since these rich aspects of the representations $\pi_{\varepsilon,\lambda}^{p,q}$ are the heart of our main results in both the proof and perspectives, we give a brief account on each of these aspects in Sections 2.2–2.5 below.

Remark 2.2**.**

1. (1)

When $q=0$, $\pi_{+,\lambda}^{p,0}$ is an irreducible finite-dimensional representation of the compact group $O(p,0)\simeq O(p)$ on the space ${\mathcal{H}}^{m}({\mathbb{R}}^{p})$ of spherical harmonics of degree $m=\lambda-\frac{p}{2}+1$. 2. (2)

The conditions (iii) and (iii)′* in Definition-Theorem 2.1 make sense for $q>0$; the other conditions for $q\geq 0$.*

For $(p,q)=(1,0)$, $O(p,q)\simeq O(1)$. It is convenient to set

[TABLE]

Via the isomorphism of Lie groups $O(p,q)\simeq O(q,p)$, we define an irreducible unitary representation $\pi_{-,\lambda}^{p,q}$ for $\lambda\in A_{-}(p,q)$ to be the one $\pi_{+,\lambda}^{q,p}$ of $O(q,p)$, where we recall from (2.2) that $A_{-}(p,q)=A_{+}(q,p)$.

By the $K$-type formula (see the condition (iii) in Definition-Theorem 2.1 and by the formula (2.6) of the ${\mathfrak{Z}}({\mathfrak{g}})$-infinitesimal character, the following proposition holds.

Proposition 2.3**.**

Irreducible unitary representations of $G=O(p,q)$ in the following set are not isomorphic to each other:

[TABLE]

2.2 Zuckerman derived functor modules $A_{\mathfrak{q}}(\lambda)$

Let $G=O(p,q)$, and $\theta$ the Cartan involution corresponding to a maximal compact subgroup $K=O(p)\times O(q)$. We take a Cartan subalgebra ${\mathfrak{t}}$ of ${\mathfrak{k}}$, and extend it to that of ${\mathfrak{g}}$, to be denoted by ${\mathfrak{j}}$. Take the standard basis $\{f_{i}:1\leq i\leq[\frac{p+q}{2}]\}$ of ${\mathfrak{j}}_{\mathbb{C}}^{\ast}$ such that the root system $\Delta({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{j}}_{\mathbb{C}})$ is given by

[TABLE]

Let ${\mathfrak{q}}={\mathfrak{l}}_{\mathbb{C}}+{\mathfrak{u}}$ be a $\theta$-stable parabolic subalgebra of ${\mathfrak{g}}_{\mathbb{C}}$ with Levi part ${\mathfrak{l}}_{\mathbb{C}}$ containing ${\mathfrak{j}}_{\mathbb{C}}$ and nilpotent radical ${\mathfrak{u}}$ defined by

[TABLE]

Then the normalizer $L$ of ${\mathfrak{u}}$ in $G$ is given by

[TABLE]

For $\nu\in{\mathbb{Z}}$, we write ${\mathbb{C}}_{\nu f_{1}}$ for the one-dimensional representation of the Levi subgroup $L$ by letting the second factor act trivially. The same letter ${\mathbb{C}}_{\nu f_{1}}$ is used to denote a character of the Lie algebra ${\mathfrak{l}}$ for $\nu\in{\mathbb{C}}$.

Zuckerman introduced cohomological parabolic induction ${\mathcal{R}}_{\mathfrak{q}}^{j}$ ($j\in{\mathbb{N}}$) which is a covariant functor from the category of $({\mathfrak{l}},L\cap K)$-modules (or that of metaplectic $({\mathfrak{l}},L\cap K)\tilde{}$-modules) to that of $(\mathfrak{g},K)$-modules.

We note that ${\mathbb{C}}_{\lambda f_{1}}$ lifts to the metaplectic $({\mathfrak{l}},L\cap K)\tilde{}$-module if and only if ${\mathbb{C}}_{(\lambda+\rho)f_{1}}$ lifts to $L$, namely, $\lambda\in{\mathbb{Z}}+\frac{1}{2}(p+q)$. In particular, for $\lambda\in A_{+}(p,q)$ ($\subset{\mathbb{Z}}+\frac{1}{2}(p+q)$), we obtain $(\mathfrak{g},K)$-modules ${\mathcal{R}}_{\mathfrak{q}}^{j}({\mathbb{C}}_{\lambda f_{1}})$ for $j\in{\mathbb{N}}$, which vanish except for $j=p-2$, and the resulting $(\mathfrak{g},K)$-module is

[TABLE]

Here we have adopted the convention and normalization in [V87, Def. 6.20] for ${\mathcal{R}}_{\mathfrak{q}}^{j}$ and in [VZ84] for $A_{\mathfrak{q}}(\cdot)$. This normalization means that $A_{\mathfrak{q}}(\nu)$ has nonzero $(\mathfrak{g},K)$-cohomologies when $\nu=0$, whereas ${\mathcal{R}}_{\mathfrak{q}}^{j}$ preserves the ${\mathfrak{Z}}({\mathfrak{l}})$- and ${\mathfrak{Z}}({\mathfrak{g}})$-infinitesimal characters in the Harish-Chandra parametrization modulo the Weyl groups $W_{L}$ and $W_{G}$.

The general theory of the Zuckerman cohomological parabolic induction (see [V87] for instance) assures that the $(\mathfrak{g},K)$-module ${\mathcal{R}}_{\mathfrak{q}}^{p-2}({\mathbb{C}}_{\lambda f_{1}})$ is nonzero and irreducible if $\lambda$ is in the “good range” (i.e. if $\lambda>\frac{1}{2}(p+q)-2$), whereas the same condition may fail if the parameter $\lambda$ wanders outside the “good range”. Although our parameter set $A_{+}(p,q)$ contains finitely many $\lambda$ that are outside the good range, the $(\mathfrak{g},K)$-module ${\mathcal{R}}_{\mathfrak{q}}^{p-2}({\mathbb{C}}_{\lambda f_{1}})$ is nonzero and irreducible for all $\lambda\in A_{+}(p,q)$, see [K92, Thm. 3] applied to $r=1$ with the notation therein.

2.3 Geometric quantization of elliptic orbits

Any coadjoint orbit of a Lie group carries a natural symplectic structure. We shall see that the irreducible unitary representation $\pi_{+,\lambda}^{p,q}$ of $G$ may be regarded as a “geometric quantization” of the minimal elliptic coadjoint orbit

[TABLE]

where $\lambda=\nu-\rho$ if we adopt the normalization of the parameter for “quantization” as in [K94b], see below.

As a homogeneous space, ${\mathcal{O}}_{\nu}$ ($\nu\neq 0$) is identified with the homogeneous space $G/L$ where $L$ is the subgroup defined in (2.7). Since the same homogeneous space $G/L$ arises an open $G$-orbit of the complex flag variety $G_{\mathbb{C}}/Q$ where $Q$ is the complex parabolic subgroup with Lie algebra ${\mathfrak{q}}$ (Section 2.2) of the complexified Lie group $G_{\mathbb{C}}$, it carries a $G$-invariant complex structure. Moreover, it admits a $G$-invariant indefinite Kähler metric such that its imaginary part yields the Kostant–Kirillov–Souriau symplectic form.

For $\nu\in{\mathbb{Z}}$, we form a homogeneous line bundle ${\mathcal{L}}_{\nu}:=G\times_{L}{\mathbb{C}}_{\nu f_{1}}$ over $G/L$. For instance, the canonical bundle of $G/L$ is expressed as ${\mathcal{L}}_{2\rho}={\mathcal{L}}_{p+q-2}$. For $\lambda\in{\mathbb{Z}}+\rho$ with $\lambda\neq 0$, we take the Dolbeault cohomologies for the $G$-equivariant holomorphic line bundle

[TABLE]

which carry a natural Fréchet topology by the closed range theorem of the $\overline{\partial}$-operator due to Schmid and Wong [Wo95], and the Fréchet $G$-module

[TABLE]

is a maximal globalization of the $(\mathfrak{g},K)$-module ${\mathcal{R}}_{\mathfrak{q}}^{j}({\mathbb{C}}_{\lambda f_{1}})$. This shows the $(\mathfrak{g},K)$-modules in (i) and (ii) in Theorem 1.1 are isomorphic to each other. If $\lambda\in A_{+}(p,q)$, then the Dolbeault cohomology for $j=p-2$ contains a Hilbert space on which $G$ acts as the unitary representation $\pi_{+,\lambda}^{p,q}$.

For $q\geq 2$, we can consider similar family of minimal elliptic coadjoint orbits ${\mathcal{O}}_{-,\lambda}\simeq G/L_{-}$ with $L_{-}:=O(p,q-2)\times SO(2)$ by switching the role of $p$ and $q$, and we obtain an irreducible unitary representations $\pi_{-,\lambda}^{p,q}$ for $\lambda\in A_{-}(p,q)$ ($=A_{+}(q,p)$).

The irreducible unitary representations $\pi_{\varepsilon,\lambda}^{p,q}$ of $G$ may be interpreted as geometric quantization of the coadjoint orbits ${\mathcal{O}}_{\varepsilon,\lambda}$, and the Gelfand–Kirillov dimension is given by

[TABLE]

2.4 Degenerate principal series representations

The indefinite orthogonal group $G=O(p,q)$ has a maximal (real) parabolic subgroup $P=MAN$, unique up to conjugation, with Levi factor

[TABLE]

Any one-dimensional representation of the first factor $GL(1,{\mathbb{R}})$ is parametrized by $(\varepsilon,\nu)\in\{\pm\}\times{\mathbb{C}}$, which extends to a character $\chi_{\varepsilon,\nu}$ of $MA$ by letting the second factor trivial. We denote by $I_{\varepsilon}(\nu)$ the $G$-module obtained as unnormalized parabolic induction $\operatorname{Ind}_{P}^{G}(\chi_{\varepsilon,\nu})$. Our parameter $\nu$ is chosen in a way that the trivial one-dimensional representation ${\bf{1}}$ of $G$ occurs as the subrepresentation of $I_{+}(0)$, and as the quotient of $I_{+}(2\rho)=I_{+}(p+q-2)$.

Geometrically, the real flag variety $G/P$ has a $G$-equivariant double covering

[TABLE]

where $P_{+}=(GL(1,{\mathbb{R}})_{+}\times O(p-1,q-1))N$ is a normal subgroup of $P$ of index two, and the group $G$ acts conformally on $S^{p-1}\times S^{q-1}$ endowed with the pseudo-Riemannian metric $g_{S^{p-1}}\oplus(-g_{S^{q-1}})$.

We recall that ${\mathcal{H}}^{m}({\mathbb{R}}^{p})$ denotes the space of spherical harmonics of degree $m$. For $p=1$, we consider only $m=0$ and $1$. The orthogonal group $O(p)$ acts irreducibly on ${\mathcal{H}}^{m}({\mathbb{R}}^{p})$, and we shall use the same letter to denote the resulting representation.

For $b\in{\mathbb{Z}}$, we define the following infinite-dimensional $K$-module:

[TABLE]

We recall from Howe–Tan [HT93]:

Proposition 2.4**.**

Suppose $\lambda\in A_{+}(p,q)$. Let $b$ and $\varepsilon$ be as in (2.4) and (2.5).

1. (1)

There is a unique irreducible submodule of $I_{\varepsilon}(\lambda+\rho)$ with $K$-types $\Xi(K,b)$. 2. (2)

There is a unique irreducible quotient of $I_{\varepsilon}(-\lambda+\rho)$ with $K$-types $\Xi(K,b)$. 3. (3)

These two modules are isomorphic to each other.

2.5 Discrete series for semisimple symmetric spaces

We equip ${\mathbb{R}}^{p+q}$ with the standard pseudo-Riemannian structure

[TABLE]

Then $g_{{\mathbb{R}}^{p,q}}$ is nondegenerate on the following hypersurface

[TABLE]

yielding a pseudo-Riemannian structure $g_{X(p,q)}$ of signature $(p-1,q)$ with constant sectional curvature $+1$, sometimes referred to as a pseudo-Riemannian space form of positive curvature. We also set

[TABLE]

Then $X(p,q)_{-}$ has a pseudo-Riemannian structure of signature $(p,q-1)$. There is a natural isomorphism (reversing the signature of the pseudo-Riemannian metric):

[TABLE]

Then $X(p,q)$ is a sphere $S^{p-1}$ if $q=0$, a hyperbolic space if $p=1$, de Sitter manifold if $p=2$, and anti-de Sitter manifold if $q=1$. We note $X(0,q)=\emptyset$.

The group $G=O(p,q)$ acts isometrically and transitively on $X(p,q)_{\pm}$, and we have $G$-diffeomorphims:

[TABLE]

The pseudo-Riemannian metric $g_{X(p,q)}$ induces the Radon measure, and the Laplace–Beltrami operator $\Delta\equiv\Delta_{X(p,q)}$ on $X(p,q)$.

For $\lambda\in{\mathbb{C}}$, we consider a differential equation on $X(p,q)$:

[TABLE]

where $\rho=\frac{1}{2}(p+q-2)$, and set

[TABLE]

Proposition 2.5** (Faraut [F79], Strichartz [S83]).**

$L^{2}(X(p,q))_{\lambda}\neq\{0\}$* if and only if $\lambda\in A_{+}(p,q)$.*

The group $G=O(p,q)$ acts on $L^{2}(X(p,q))_{\lambda}$ as an irreducible unitary representation. Moreover, if $f\in L^{2}(X(p,q))_{\lambda}$ is $K$-finite, then there is an analytic function $a\in C^{\infty}(S^{p-1}\times S^{q-1})$ such that

[TABLE]

3 General scheme

Our approach to the branching laws (Theorem 1.1) is to use analysis on $G^{\prime}$-orbits in the reductive symmetric space $G/H$, as developed in [K94a, K98b] among others. In our setting, $G/H\simeq X(p,q)$ admits principal orbits of the subgroup $G^{\prime}$ (see [K98b, Sect. 8.2]), hence all the discrete spectrum in the branching law $\Pi|_{G^{\prime}}$ can be captured though the analysis on principal $G^{\prime}$-orbits, as formulated in Proposition 3.1 below.

3.1 Principal $G^{\prime}$-orbits in $X(p,q)$

We introduce a $G^{\prime}$-invariant function in the ambient space $\mathbb{R}^{p+q}=\mathbb{R}^{p^{\prime}+p^{\prime\prime}+q^{\prime}+q^{\prime\prime}}$ by

[TABLE]

If $(u^{\prime},u^{\prime\prime},v^{\prime},v^{\prime\prime})\in X(p,q)$, then

[TABLE]

We define three $G^{\prime}$-invariant open sets $X(p,q)_{\delta\varepsilon}$ of $X(p,q)$ by

[TABLE]

Then the disjoint union

[TABLE]

is conull in $X(p,q)$. Accordingly, we have a direct sum decomposition of the Hilbert space:

[TABLE]

which is stable by the action of $G^{\prime}$. We shall see in (4.6)–(4.8) that the isomorphism classes of the isotropy subgroups of the subgroup $G^{\prime}$ at points in $X(p,q)_{\delta\varepsilon}$ are determined uniquely by $(\delta,\varepsilon)$.

3.2 A priori estimate of $\operatorname{Disc}(\Pi|_{G^{\prime}})$

By using the general theory [K98b], we explain the three families of irreducible representations of $G^{\prime}$ occurring in the branching law $\Pi|_{G^{\prime}}$ (Theorem 1.1) arise from the decomposition (3.2).

Proposition 3.1**.**

For $\lambda\in A_{+}(p,q)$, we set $\Pi:=\pi^{{p},{q}}_{+,{\lambda}}\in\widehat{G}$ as in Definition-Theorem 2.1. If $\pi\in\widehat{G^{\prime}}$ satisfies $\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})\neq\{0\}$, then there exist uniquely $(\delta^{\prime},\delta^{\prime\prime})\in\{{-+},{++},{+-}\}$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in A_{\delta^{\prime}}(p^{\prime},q^{\prime})\times A_{\delta^{\prime\prime}}(p^{\prime\prime},q^{\prime\prime})$ such that

[TABLE]

Moreover the following parity condition holds:

[TABLE]

Proof.

The existence follows from the general results proved in [K98b, Thm.8.6]. The uniqueness is clear because these irreducible $G^{\prime}$-modules are mutually inequivalent.

To show the parity condition (3.5), we observe that the central element $-I_{p,q}$ of $G$ acts on $\pi_{\varepsilon,\lambda}^{p,q}$ as a scalar $(-1)^{\lambda-\frac{p-q}{2}\varepsilon+1},$ as one sees from the equivalent condition (iii) in Definition-Theorem 2.1. Since $(-I_{p^{\prime},q^{\prime}})\times(-I_{p^{\prime\prime},q^{\prime\prime}})\in G^{\prime}$ is identified with $-I_{p,q}\in G$, it follows from the assumption $\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})\neq\{0\}$ that

[TABLE]

Then one obtains (3.5) in view of $\lambda^{\prime}\in{\mathbb{Z}}+\frac{p^{\prime}+q^{\prime}}{2},\,\,\lambda^{\prime\prime}\in{\mathbb{Z}}+\frac{p^{\prime\prime}+q^{\prime\prime}}{2}\,\,\text{ and }\,\,\lambda\in{\mathbb{Z}}+\frac{p+q}{2}.$ ∎

The above proof gives useful geometric information on functions that belong to irreducible components of the branching law:

Proposition 3.2**.**

In the setting of Proposition 3.1, suppose $\pi\in\widehat{G^{\prime}}$ satisfies $\operatorname{Hom}_{G^{\prime}}(\pi,L^{2}(X(p,q))_{\lambda})\neq\{0\}$. We set $\kappa:=(\delta^{\prime},\delta^{\prime\prime})\in\{{-+},{++},{+-}\}$ according to (3.4) in Proposition 3.1. Then we have

[TABLE]

for any function $F$ in the image of $\operatorname{Hom}_{G^{\prime}}(\pi,L^{2}(X(p,q))_{\lambda})$.

4 Construction of holographic operators

In this section we construct explicit intertwining operators (holographic operators) from irreducible $G^{\prime}$-modules to irreducible $G$-modules:

[TABLE]

by using a geometric realization of these representations in the $L^{2}$-spaces of pseudo-Riemannian space forms $X(p^{\prime},q^{\prime})_{\delta}$, $X(p^{\prime\prime},q^{\prime\prime})_{\varepsilon}$ and $X(p,q)$, as described in Section 2.5. Moreover, we find a closed formula for the operator norm of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$. The main results of this section are stated in Theorem 4.3.

4.1 Preliminaries

To state the quantitative results (Theorem 4.3), we set

[TABLE]

Lemma 4.1**.**

1. (1)

$V_{{\delta\varepsilon},{\lambda}}^{({\lambda^{\prime}},{\lambda^{\prime\prime}})}>0$* if $\lambda>0$, $\lambda^{\prime},\lambda^{\prime\prime}\geq-\frac{1}{2}$, and $\delta\varepsilon\lambda-\varepsilon\lambda^{\prime}-\delta\lambda^{\prime\prime}>0$. Here $\delta\varepsilon\lambda:=\lambda$ when $\delta=\varepsilon$ and $-\lambda$ when $\delta\neq\varepsilon$.* 2. (2)

$V_{{\delta\varepsilon},{\lambda}}^{({\lambda^{\prime}},{\lambda^{\prime\prime}})}>0$* if $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta\varepsilon}(\lambda)$.*

Proof.

(1) Clear from the definition.

(2)  The second statement is a special case of the first one. See also Lemma 4.2 for an alternative proof. ∎

4.2 Jacobi functions and Jacobi polynomials

Let us consider the differential operator

[TABLE]

We recall that for $\lambda$, $\lambda^{\prime}$, $\lambda^{\prime\prime}\in{\mathbb{C}}$ with $\lambda^{\prime\prime}\neq-1,-2,\cdots$, the Jacobi function $\varphi_{i\lambda}^{(\lambda^{\prime\prime},\lambda^{\prime})}(t)$ is the unique even solution to the following differential equation

[TABLE]

such that $\varphi(0)=1$, see Koornwinder [Kw84], for instance. We note that $\varphi_{i\lambda}^{(\lambda^{\prime\prime},\lambda^{\prime})}(t)=\varphi_{-i\lambda}^{(\lambda^{\prime\prime},\lambda^{\prime})}(t).$ By the change of variables $z=-\sinh^{2}t$, $g(z):=\varphi(t)$ satisfies the hypergeometric differential equation

[TABLE]

with

[TABLE]

The hypergeometric differential equation (4.3) has a regular singularity $z=0$, and its exponents are [math], $-\lambda^{\prime\prime}$. For $\lambda^{\prime\prime}\neq 0$, we denote by $g_{1(0)}(z)$ and $g_{2(0)}(z)$ the unique solutions to (4.3) such that

[TABLE]

We set

[TABLE]

If $\lambda^{\prime\prime}\neq 0,-1,-2,\cdots$, then $u_{1(0)}(t)$ is the Jacobi function $\varphi_{i\lambda}^{\lambda^{\prime\prime},\lambda^{\prime}}(t)$ (see (4.5)), and thus we have

[TABLE]

where ${}_{2}F_{1}$ is the Gauss hypergeometric function. We need the following formulæ for the $L^{2}$-norms of the Jacobi functions.

Lemma 4.2** ([KØ03, Lem. 8.2]).**

Suppose $\lambda>0$.

[TABLE]

4.3 Construction of holographic operators

We define the following diffeomorphisms $\Phi_{\delta\varepsilon}$ onto the open subsets $X(p,q)_{\delta\varepsilon}$ by

[TABLE]

By using the following coordinates:

[TABLE]

we introduce linear operators

[TABLE]

as follows:

[TABLE]

Theorem 4.3**.**

Suppose $(\delta,\varepsilon)=(-,+)$, $(+,+)$ or $(+,-)$. Let $\lambda\in A_{+}(p,q)$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta\varepsilon}(\lambda)$. Then $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ induces an injective $G^{\prime}$-intertwining operator:

[TABLE]

Moreover, $(V_{{\delta\varepsilon},{\lambda^{\prime\prime}}}^{({\lambda},{\lambda^{\prime}})})^{-\frac{1}{2}}T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ is an isometry.

The proof of Theorem 4.3 is divided into two parts:

1. $\bullet$

to compute the operator norm of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$, see Proposition 4.4; 2. $\bullet$

to show that $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ is a weak solution to (2.10), see Proposition 4.7.

4.4 Operator norms of the holographic operators

We prove that the linear operator $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ is a scalar multiple of an isometric operator, and find its $L^{2}$-norm. We do not need that $h$ satisfies a differential equation in the proposition below.

Proposition 4.4**.**

Suppose $(\delta,\varepsilon)=(-,+)$, $(+,+)$, or $(+,-)$. If $\lambda>0$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta\varepsilon}(\lambda)$, then $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ is an isometry upto scaling:

[TABLE]

for all $h\in L^{2}(X(p^{\prime},q^{\prime})_{\delta}\times X(p^{\prime\prime},q^{\prime\prime})_{\varepsilon})$.

Proof.

With respect to the diffeomorphisms (4.6)–(4.8), the invariant measure $d\mu$ on $X(p,q)$ is expressed as

[TABLE]

where

[TABLE]

Hence the proof of Proposition 4.4 is reduced to Lemma 4.2. ∎

4.5 Construction of smooth solutions on open sets

Since the Laplacian $\Delta_{X(p,q)}$ is not an elliptic differential operator unless the signature of $g_{X(p,q)}$ is definite (*i.e., * $p=1$ or $q=0$), eigenfunctions (in the distribution sense) of the Laplacian are not necessarily real analytic on $X(p,q)$. In fact, when $p\geq 2$ and $q\geq 1$, one sees from the proof of Corollary 6.5 that $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ is never real analytic on the whole space $X(p,q)$ if $h\not\equiv 0$ and $p^{\prime}p^{\prime\prime}\neq 0$.

We begin by considering the restriction of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ to the open set $X(p,q)_{\delta\varepsilon}$ (Section 3.1) for each $(\delta,\varepsilon)=(-,+)$, $(+,+)$, or $(+,-)$.

Proposition 4.5**.**

Suppose $\lambda,\lambda^{\prime},\lambda^{\prime\prime}\in{\mathbb{C}}$ such that $\lambda^{\prime},\lambda^{\prime\prime}\neq-1,-2,\cdots$. Then for any $h\in C^{\infty}(X(p^{\prime},q^{\prime})_{\delta})_{\lambda^{\prime}}\otimes C^{\infty}(X(p^{\prime\prime},q^{\prime\prime})_{\varepsilon})_{\lambda^{\prime\prime}}$, $F(x):=T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h(x)$ satisfies the differential equation (2.10) on the open set $X(p,q)_{\delta\varepsilon}$.

Proof.

Suppose $(\delta,\varepsilon)=(+,-)$. We set

[TABLE]

where we set $\rho^{\prime}=\frac{p^{\prime}+q^{\prime}-2}{2},\ \rho^{\prime\prime}=\frac{p^{\prime\prime}+q^{\prime\prime}-2}{2}.$ We note that $\rho=\rho^{\prime}+\rho^{\prime\prime}+1.$

A short computation shows that

[TABLE]

under the transform $S_{\lambda^{\prime},\lambda^{\prime\prime}}$ defined by

[TABLE]

Via the diffeomorphism $\Phi_{+-}$ (4.8), the Laplacian $\Delta_{X(p,q)}$ takes the form:

[TABLE]

in $X(p,q)_{{+-}}$. Therefore, for nonzero $h^{\prime}\in C^{\infty}(X(p^{\prime},q^{\prime}))_{\lambda^{\prime}}$ and $h^{\prime\prime}\in C^{\infty}(X(q^{\prime\prime},p^{\prime\prime}))_{\lambda^{\prime\prime}}$, $F_{{+-}}(z^{\prime},z^{\prime\prime},t):=h^{\prime}(z^{\prime})h^{\prime\prime}(z^{\prime\prime})(S_{\lambda^{\prime},\lambda^{\prime\prime}}\varphi)(t)$ satisfies

[TABLE]

if and only if $\varphi$ satisfies the Jacobi differential equation (4.2). Thus Proposition 4.5 is shown for $(\delta,\varepsilon)=(+,-)$.

The proof for $(\delta,\varepsilon)=(-,+)$ is essentially the same, and that for $(\delta,\varepsilon)=(+,+)$ goes similarly. In this case, the Laplacian takes the form:

[TABLE]

on $X(p,q)_{{++}}$ in the coordinates via $\Phi_{{++}}$, where we set

[TABLE]

By the change of variables $z=\sin^{2}\theta$, the function

[TABLE]

satisfies the same hypergeometric equation (4.3), with regular singularities: the exponents at $z=0$ are [math], $-\lambda^{\prime\prime}$; and those at $z=1$ are [math], $-\lambda^{\prime}$. ∎

4.6 Boundary $\partial X(p,q)_{\delta\varepsilon}$

By definition (4.9), $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ is the extension of a solution to the differential equation (2.10) in the open domain $X(p,q)_{\delta\varepsilon}$ (see Proposition 4.5) to the whole manifold $X(p,q)$ by zero outside the domain. In order to prove a precise condition for such an extension to give a weak solution to (2.10) in $L^{2}(X(p,q))$, we need an estimate of the solution near the boundary.

In this section we study the boundary $\partial X(p,q)_{\delta\varepsilon}$. We observe that

[TABLE]

Since $\partial X(p,q)_{-+}$ is similar to $\partial X(p,q)_{+-}$, we take a closer look at

[TABLE]

We note that the singular part $\partial X(p,q)_{+-}^{\operatorname{sing}}$ is diffeomorphic to $X(p^{\prime},q^{\prime})$ and that the map $\Phi_{{+-}}$ extended to $t=0$ in (4.8) surjects $\partial X(p,q)_{+-}^{\operatorname{sing}}$:

[TABLE]

On the other hand, the regular part $\partial X(p,q)_{+-}^{\operatorname{reg}}$ is a hypersurface in $X(p,q)$. In a neighbourhood $U$ of a point at $\partial X(p,q)_{{+-}}^{\operatorname{reg}}$, we set

[TABLE]

and take coordinates on $U$ $(\subset X(p,q))$ by

[TABLE]

where $z^{\prime}=(x^{\prime},y^{\prime})\in X(p^{\prime},q^{\prime})$, $\omega^{\prime\prime}\in S^{p^{\prime\prime}-1}$, and $\eta^{\prime}\in S^{q^{\prime\prime}-1}$. Then $U\cap X(p,q)_{{+-}}$ is given by $\xi_{1}>0$, whereas $U\cap X(p,q)_{{++}}$ is given by $\xi_{1}<0$.

Lemma 4.6**.**

In the coordinates (4.15), the Laplacian $\Delta_{X(p,q)}$ takes the form

[TABLE]

where $P$ and $Q$ are differential operators of variables $\xi_{2}$, $x^{\prime}$, $y^{\prime}$, $\omega^{\prime\prime}$ and $\eta^{\prime\prime}$ with smooth coefficients.

Proof.

The coordinates (4.15) are obtained from $\Phi_{{+-}}(z^{\prime},z^{\prime\prime},t)$, see (4.8), successively by the following two steps:

[TABLE]

By change of coordinates in the first step, the Laplacian $\Delta_{X(p,q)}$ takes the form (4.14) with the second term replaced by

[TABLE]

where we set

[TABLE]

Then the change of variables $(t,s)\mapsto(\xi_{1},\xi_{2})$ in the second step yields

[TABLE]

whence the lemma by short computations. ∎

4.7 Extension as a weak solution in $L^{2}(X(p,q))$

The proof of Theorem 4.3 will be completed if the image of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ gives weak solutions to the differential equation (2.10).

Proposition 4.7**.**

Suppose $(\delta,\varepsilon)=(-,+)$, $(+,+)$, or $(+,-)$. Assume $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta\varepsilon}(\lambda)$. Then for any $h\in L^{2}(X(p^{\prime},q^{\prime})_{\delta})_{\lambda^{\prime}}\widehat{\otimes}L^{2}(X(p^{\prime\prime},q^{\prime\prime})_{\varepsilon})_{\lambda^{\prime\prime}}$, $F:=T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ is a weak solution to the differential equation (2.10) on $X(p,q)$.

Proof.

Since the Laplacian $\Delta$ is a closed operator on $L^{2}(X(p,q))$, and since $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ is a bounded operator by Proposition 4.4, it suffices to prove the assertion for a dense subspace of the Hilbert space. Thus we may and do assume that $h$ is a $K^{\prime}$-finite function. Then $F$ is real analytic on $X(p,q)_{\delta\varepsilon}$ and satisfies (2.10) in $X(p,q)_{\delta\varepsilon}$ in the usual sense by Proposition 4.5.

In order to prove that $F$ is a weak solution to (2.10) in the whole manifold $X(p,q)$, we consider the boundary $\partial X(p,q)_{\delta\varepsilon}$, and explain the case $(\delta,\varepsilon)=(+,-)$. We may and do assume that $p^{\prime\prime}>0$. In fact, if $p^{\prime\prime}=0$, then $X(p,q)_{{++}}=X(p,q)_{{+-}}=\emptyset$ and $T_{{+-},\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h|_{X(p,q)_{{+-}}}$ extends to a smooth function on $X(p,q)$.

Suppose $p^{\prime\prime}>0$. Then $\lambda^{\prime\prime}\in A_{-}(p^{\prime\prime},q^{\prime\prime})$ satisfies $\lambda^{\prime\prime}>0$. In order to prove that $F$ is a weak solution to (2.10), it suffices to verify it near the boundary $\partial X(p,q)_{{+-}}=\partial X(p,q)_{{+-}}^{\operatorname{reg}}\cup\partial X(p,q)_{{+-}}^{\operatorname{sing}}$.

Case I. First, we deal with a neighbourhood $U$ of a point at $\partial X(p,q)_{{+-}}^{\operatorname{reg}}$. We take coordinates of $U$ as in (4.15). We recall that the boundary $U\cap\partial X(p,q)_{{+-}}$ is given by $\xi_{1}=0$ where $\xi_{2}>0$. Then $\Phi_{{+-}}(z^{\prime},z^{\prime\prime},t)$ with $z^{\prime\prime}=(\omega^{\prime\prime}\sinh s,\eta^{\prime\prime}\cosh s)$, see (4.17), approaches to boundary points in $\partial X(p,q)_{{+-}}^{\operatorname{reg}}$, as $t\to 0$ and $s\to\infty$ with constraints

[TABLE]

because

[TABLE]

Then it follows from (2.11) that the $K^{\prime}$-finite function $h$ has an asymptotic behavior

[TABLE]

as $s\to\infty$ for some analytic function $a(z^{\prime},\omega^{\prime\prime},\eta^{\prime\prime})$, and therefore $F=T_{{+-},\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}h$ in $U\cap X(p,q)_{{+-}}$ behaves as

[TABLE]

near the boundary $\xi_{1}\downarrow 0$, whereas $F\equiv 0$ for $\xi_{1}<0$. Since $\lambda^{\prime\prime}>0$ and since $\Delta_{X(p,q)}$ takes the form (4.16), the distribution $\Delta_{X(p,q)}F$ is actually a locally integrable function on $U$. Since $F$ solves (2.10) in $U\setminus\partial X(p,q)_{{+-}}$ in the usual sense, so does $F$ in $U$ in the distribution sense.

Case II. Next, we deal with a neighbourhood $U$ of a point at $\partial X(p,q)_{{+-}}^{\operatorname{sing}}$. In this case, we use $(z^{\prime},z^{\prime\prime},t)\in X(p^{\prime},q^{\prime})\times X(q^{\prime\prime},p^{\prime\prime})\times[0,\infty)$ as coordinates of $U\cap\overline{X(p,q)_{{+-}}}$ via $\Phi_{{+-}}$.

Since $F$ behaves as $O(t^{\lambda^{\prime\prime}-\rho^{\prime\prime}})$ when $t$ tends to zero, so does $Y_{1}F$ as $O(t^{\lambda^{\prime\prime}-\rho^{\prime\prime}-1})$ and $Y_{1}Y_{2}F$ as $O(t^{\lambda^{\prime\prime}-\rho^{\prime\prime}-2})$ for any vector fields $Y_{1}$, $Y_{2}$ on $X(p,q)$. In view of the formula (4.12) of the measure $d\mu_{+-}(t)$, these functions belong to $L_{\operatorname{loc}}^{1}({\mathbb{R}},d\mu_{+-}(t))$ if

[TABLE]

which is automatically satisfied because $\lambda^{\prime\prime}>0$. Thus $F$ is a weak solution to (2.10) near the boundary $\partial X(p,q)_{\delta\varepsilon}$ when $(\delta,\varepsilon)=(+,-)$.

The other cases $(\delta,\varepsilon)=(+,+)$ and $(-,+)$ are similar. Thus Proposition 4.7 is proved. ∎

5 Exhaustion of holographic operators

Let $\Pi\in\widehat{G}$ be any discrete series representation for the pseudo-Riemannian space form $G/H\simeq X(p,q)$. In this section we prove that discrete spectra of the restriction $\Pi|_{G^{\prime}}$ are exhausted by (1.1) counted with multiplicities, hence complete the proof of Theorem 1.1.

To be precise, we recall from Proposition 2.5 that any $\Pi\in{\operatorname{Disc}({G/H})}$ is of the form $\Pi=\pi_{+,\lambda}^{p,q}$ for some $\lambda\in A_{+}(p,q)$, and from Proposition 3.1 that $\pi\in\widehat{G^{\prime}}$ satisfying $\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})\neq\{0\}$ must be of the form $\pi=\pi_{\delta,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{\varepsilon,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}}$ for some $(\lambda^{\prime},\lambda^{\prime\prime})\in A_{\delta}(p^{\prime},q^{\prime})\times A_{\varepsilon}(p^{\prime\prime},q^{\prime\prime})$ with $(\delta,\varepsilon)\in\{(-,+),(+,+),(+,-)\}$. We show that $(\lambda^{\prime},\lambda^{\prime\prime})$ is actually an element of $\Lambda_{\delta\varepsilon}(\lambda)$. More strongly, we prove:

Theorem 5.1**.**

Suppose that $\lambda\in A_{+}(p,q)$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in A_{\delta}(p^{\prime},q^{\prime})\times A_{\varepsilon}(p^{\prime\prime},q^{\prime\prime})$. Then, we have

[TABLE]

We already know in [K93] that the direct sum (1.1) equals the whole restriction $\Pi|_{G^{\prime}}$ if $p^{\prime}=0$ or $p^{\prime\prime}=0$. In this case, $\Pi=\pi_{+,\lambda}^{p,q}$ is $K^{\prime}$-admissible (cf. Section 6.5), and the multiplicity of each $K^{\prime}$-type occurring in $\Pi$ coincides with that in (1.1). Hence the restriction $\Pi|_{G^{\prime}}$ is discretely decomposable and is isomorphic to the direct sum (1.1). Thus, we shall assume $p^{\prime}p^{\prime\prime}>0$ from now on.

The rest of this section is devoted to the proof of Theorem 5.1 in the case $p^{\prime}p^{\prime\prime}>0$ and $(\delta,\varepsilon)=(+,-)$. The other cases where $(\delta,\varepsilon)=(-,+)$ or $(+,+)$ are similar.

5.1 Kummer’s relation

The hypergeometric differential equation (4.3) has a regular singularity also at $z=\infty$, and its exponents are $\frac{1}{2}(\lambda^{\prime}+\lambda^{\prime\prime}+1-\lambda)$ and $\frac{1}{2}(\lambda^{\prime}+\lambda^{\prime\prime}+1+\lambda)$. Suppose $\lambda\neq 0$. We write $g_{(\infty)}^{+}(z)$ and $g_{(\infty)}^{-}(z)$ for the unique solutions to (4.3) such that

[TABLE]

and set

[TABLE]

Lemma 5.2** (Kummer’s relation).**

Suppose $\lambda\neq 0,-1,-2,\dots$ and $\lambda^{\prime\prime}\neq 0$.

1. (1)

There exist uniquely $a(\lambda^{\prime},\lambda^{\prime\prime},\lambda)$, $b(\lambda^{\prime},\lambda^{\prime\prime},\lambda)\in{\mathbb{C}}$ such that

[TABLE] 2. (2)

If $\lambda^{\prime\prime}\neq 0,-1,-2,\dots$, then

[TABLE]

Moreover, if $\lambda^{\prime\prime}\not\in{\mathbb{Z}}$, then $a(\lambda^{\prime},\lambda^{\prime\prime},\lambda)=b(\lambda^{\prime},-\lambda^{\prime\prime},\lambda)$.

Proof.

The first statement is clear because $g_{1(0)}(z)$ and $g_{2(0)}(z)$ are linearly independent solutions to (4.3).

To see the second statement, we begin with the generic case where $\lambda\not\in\{0,-1,-2,\cdots\}$ and $\lambda^{\prime\prime}\not\in{\mathbb{Z}}$. Then we have

[TABLE]

and Kummer’s relation [Er53, 2.9 (39)] shows $a(\lambda^{\prime},\lambda^{\prime\prime},\lambda)=b(\lambda^{\prime},-\lambda^{\prime\prime},\lambda)$ with the formula (5.4) for $b(\lambda^{\prime},\lambda^{\prime\prime},\lambda)$.

When $\lambda^{\prime\prime}=m\in{\mathbb{N}}_{+}$, $g_{1(0)}(z)$ remains to be the same (5.5) but $g_{2(0)}(z)$ does not take the form (5.6). In fact, $g_{2(0)}(z)$ contains a logarithmic term, and is given by the analytic continuation:

[TABLE]

where $P_{m}\equiv P_{m}(\lambda^{\prime},\lambda)\in{\mathbb{C}}$ is determined by

[TABLE]

Then the change of basis may alter the coefficient $a(\lambda^{\prime},\lambda^{\prime\prime},\lambda)$ in (5.3) but leaves $b(\lambda^{\prime},\lambda^{\prime\prime},\lambda)$ invariant. Thus the lemma is proved. ∎

For $\lambda^{\prime},\lambda^{\prime\prime}\in{\mathbb{R}}$, we set a measure $d\mu^{\lambda^{\prime},\lambda^{\prime\prime}}$ on ${\mathbb{R}}$ by

[TABLE]

We note that $d\mu_{{+-}}(t)=d\mu^{\rho^{\prime},\rho^{\prime\prime}}(t)$, see (4.12), and

[TABLE]

by the definition of the transform (4.13) of $S_{\lambda^{\prime},\lambda^{\prime\prime}}$.

We need the following:

Lemma 5.3**.**

Suppose $\lambda>0$, $\lambda^{\prime}>-1$, $\lambda^{\prime\prime}>-1$. Then $u_{(\infty)}^{-}(t)\in L^{2}((0,\infty),d\mu^{\lambda^{\prime},\lambda^{\prime\prime}}(t))$ if and only if $-1<\lambda^{\prime\prime}<1$ or $\lambda^{\prime}-\lambda^{\prime\prime}-\lambda-1\in 2{\mathbb{N}}$.

Proof.

By the asymptotic behavior (5.1) of $g_{(\infty)}^{-}(z)$ as $z\to\infty$, we have

[TABLE]

because $\lambda>0$. Likewise, by the asymptotic behavior (4.4) of $g_{1(0)}(z)$ and $g_{2(0)}(z)$ as $z\to 0$,

[TABLE]

In view of the Kummer’s relation (5.3),

[TABLE]

belongs to $L^{2}((0,\infty),d\mu^{\lambda^{\prime},\lambda^{\prime\prime}}(t))$ if and only if $-1<\lambda^{\prime\prime}<1$ or $b(\lambda^{\prime},\lambda^{\prime\prime},\lambda)=0$. The latter condition amounts to $\lambda^{\prime}-\lambda^{\prime\prime}-\lambda-1\in 2{\mathbb{N}}$ by Lemma 5.2 (2). Thus the lemma is proved. ∎

5.2 Possible form of holographic operators

In this section we examine a possible form for a holographic operator $\pi\to\Pi|_{G^{\prime}}$, and find a necessary condition on the parameter for $\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})$ to be nonzero. We begin with the following:

Lemma 5.4**.**

Let $\lambda\in A_{+}(p,q)$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in A_{+}(p^{\prime},q^{\prime})\times A_{-}(p^{\prime\prime},q^{\prime\prime})$. Suppose $T\in\operatorname{Hom}_{G^{\prime}}(\pi_{+,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{-,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}},\pi_{+,\lambda}^{p,q}|_{G^{\prime}})$. Then in the geometric realizations of these representations on pseudo-Riemannian space forms (Section 2.5), $T$ must be of the following form: there exists $c\in{\mathbb{C}}$ such that

[TABLE]

for all $h\in L^{2}(X(p^{\prime},q^{\prime}))_{\lambda^{\prime}}\widehat{\otimes}L^{2}(X(q^{\prime\prime},p^{\prime\prime}))_{\lambda^{\prime\prime}}$.

Remark 5.5**.**

We have used the Jacobi function $u_{1(0)}(t)=\varphi_{i\lambda}^{(\lambda^{\prime\prime},\lambda^{\prime})}(t)$ (4.5) for the definition of the holographic operator $T_{{+-},\lambda}^{\lambda^{\prime\prime},\lambda^{\prime}}$ in (4.9) instead of $u_{(\infty)}^{-}(t)$ as in Lemma 5.4. It is a part of Theorem 5.1 to show that $u_{1(0)}(t)$ is proportional to $u_{(\infty)}^{-}(t)$ if $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{+}(\lambda)$.

Proof of Lemma 5.4.

For any $h$ in $\pi^{{p^{\prime}},{q^{\prime}}}_{+,{\lambda^{\prime}}}\boxtimes\pi^{{p^{\prime\prime}},{q^{\prime\prime}}}_{-,{\lambda^{\prime\prime}}}$, we have $\operatorname{Supp}Th\subset\overline{X(p,q)_{+-}}$ by Proposition 3.2.

Suppose that $h$ is $K^{\prime}$-finite. We set

[TABLE]

where $S_{\lambda^{\prime},\lambda^{\prime\prime}}^{-1}$ (see (4.13)) is applied to the last variable $t$. Then the following differential equations are satisfied:

[TABLE]

where $\Delta_{X(p^{\prime},q^{\prime})}$ acts on $z^{\prime}$-variables, and $\Delta_{X(q^{\prime},p^{\prime})}$ on $z^{\prime\prime}$-variables.

As in the proof of Proposition 4.5, the differential equation (2.10) yields the following differential equation (in the sense of distribution):

[TABLE]

where $L_{{+-}}$ is defined in (4.1). Since $\lambda\neq 0$, the solution $\psi_{+-}(z^{\prime},z^{\prime\prime},t)$ is a linear combination of the basis $u_{(\infty)}^{+}(t)$ and $u_{(\infty)}^{-}(t)$. Hence $\psi_{{+-}}$ is of the form

[TABLE]

for some real analytic functions $h_{+}(z^{\prime},z^{\prime\prime})$ and $h_{-}(z^{\prime},z^{\prime\prime})$ on $X(p^{\prime},q^{\prime})\times X(q^{\prime\prime},p^{\prime\prime})$. We observe that under the assumption $\lambda>0$ we have

[TABLE]

Since $\operatorname{Supp}Th\subset\overline{X(p,q)_{{+-}}}$, the formula (4.10) of the invariant measure on $X(p,q)$ and the definition (4.13) of $S_{\lambda^{\prime},\lambda^{\prime\prime}}$ imply

[TABLE]

Thus we conclude from $Th\in L^{2}(X(p,q))$ that $h_{+}(z^{\prime},z^{\prime\prime})=0$. In turn, we have

[TABLE]

Since $T$ is a continuous map between the Hilbert spaces, we have

[TABLE]

if $T\neq 0$. Moreover, $h\mapsto h_{-}$ is a $({\mathfrak{g}}^{\prime},K^{\prime})$-endomorphism of the irreducible $({\mathfrak{g}}^{\prime},K^{\prime})$-module $(\pi_{+,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{-,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}})_{K^{\prime}}$, whence there exists $c\in{\mathbb{C}}$ such that $h_{-}=ch$ for all $K^{\prime}$-finite vectors $h$ by Schur’s lemma. Since $T$ is a continuous map, we obtain Lemma 5.4. ∎

Next, we show that the condition $Th\in L^{2}(X(p,q))$ leads us to the following:

Proposition 5.6**.**

Retain $(\delta,\varepsilon)=(+,-)$. Suppose $\lambda\in A_{+}(p,q)$ and $(\lambda^{\prime},\lambda^{\prime\prime})\in A_{\delta}(p^{\prime},q^{\prime})\times A_{\varepsilon}(p^{\prime\prime},q^{\prime\prime})$. If $\operatorname{Hom}_{G^{\prime}}(\pi_{\delta,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{\varepsilon,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}},\pi_{+,\lambda}^{p,q}|_{G^{\prime}})\neq\{0\}$, then $\lambda^{\prime\prime}=\frac{1}{2}$ or $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{\delta\varepsilon}(\lambda)$.

In Section 5.3, we treat the case $\lambda^{\prime\prime}=\frac{1}{2}$.

Proof.

As we have seen (5.11) in the proof of Lemma 5.4, $u_{(\infty)}^{-}(t)\in L^{2}((0,\infty),d\mu^{\lambda^{\prime},\lambda^{\prime\prime}}(t))$. Hence $-1<\lambda^{\prime\prime}<1$ or $\lambda^{\prime}-\lambda^{\prime\prime}-\lambda-1\in 2{\mathbb{N}}$ by Lemma 5.3. Since $\lambda^{\prime\prime}\in A_{-}(p^{\prime\prime},q^{\prime\prime})$ with $p^{\prime\prime}>0$ (see (2.2)), the only possible $\lambda^{\prime\prime}$ with $\lambda^{\prime\prime}<1$ is $\lambda^{\prime\prime}=\frac{1}{2}$. (We note that $\lambda^{\prime\prime}=-\frac{1}{2}$ occurs only when $(p^{\prime\prime},q^{\prime\prime})=(0,1)$.) Thus Proposition 5.6 is proved. ∎

5.3 The case $\lambda^{\prime\prime}=\frac{1}{2}$

The case $\lambda^{\prime\prime}=\frac{1}{2}$ is delicate because there exists a continuous $G^{\prime}$-homomorphism

[TABLE]

such that the image of $T$ consists of weak solutions to (2.10) in $L^{2}(X(p,q)_{{+-}})$ without the assumption $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{{+-}}(\lambda)$. However, we shall see that $Th$ cannot be a weak solution to (2.10) in $L^{2}(X(p,q))$ unless $(\lambda^{\prime},\lambda^{\prime\prime})\in\Lambda_{{+-}}(\lambda)$. For this, it suffices to show the following:

Lemma 5.7**.**

In the setting of Lemma 5.4, suppose $\lambda^{\prime\prime}=\frac{1}{2}$ and $(\lambda^{\prime},\lambda^{\prime\prime})\not\in\Lambda_{{+-}}(\lambda)$. Then the distribution $\Delta_{X(p,q)}(Th)$ is not a locally integrable function on $X(p,q)$ for any nonzero $K^{\prime}$-finite function $h$.

Proof.

We consider a neighbourhood $U$ at a point of $\partial X(p,q)_{{+-}}^{\operatorname{reg}}$, and use the coordinates (4.15) as in Section 4.6. Then $Th=0$ if $\xi_{1}<0$. Let us examine the behavior of $Th$ in $U\cap\overline{X(p,q)_{{+-}}}$ near the boundary as $\xi_{1}\downarrow 0$.

Let $\psi_{{+-}}$ be as in (5.8). Since $(\lambda^{\prime},\lambda^{\prime\prime})\not\in\Lambda_{{+-}}(\lambda)$, the coefficient $b(\lambda^{\prime},\lambda^{\prime\prime},\lambda)$ in (5.3) does not vanish. Hence there exist $A\in{\mathbb{C}}$ and $B\neq 0$ such that

[TABLE]

We recall from (4.19) that $h(z^{\prime},z^{\prime\prime})$ has an asymptotic behavior

[TABLE]

for some real analytic function of $(z^{\prime},\omega^{\prime\prime},\eta^{\prime\prime})\in X(p^{\prime},q^{\prime})\times S^{p^{\prime\prime}-1}\times S^{q^{\prime\prime}-1}$ as $s\to\infty$ in the coordinates $z^{\prime\prime}=(\omega^{\prime\prime}\sinh s,\eta^{\prime\prime}\cosh s)$.

Combining these two asymptotic behaviours as $s\to\infty$ and $t\to 0$ with $\xi_{2}=e^{s}\sinh t$ away from 0 and infinity, we obtain the asymptotic behavior of $Th$ near the boundary $\partial X(p,q)_{{+-}}^{\operatorname{reg}}$:

[TABLE]

where the first term is given by

[TABLE]

In view of $\lambda^{\prime\prime}=\frac{1}{2}$, the proof of the lemma is reduced to the following. ∎

Lemma 5.8**.**

Let $U$ be an open subset of ${\mathbb{R}}^{n}$, and $P$ a differential operator on $U$ of the form

[TABLE]

such that $P^{\prime}$ and $P^{\prime\prime}$ are differential operators of variables $\xi^{\prime}=(\xi_{2},\cdots,\xi_{n})$ with smooth coefficients in $\xi=(\xi_{1},\xi^{\prime})$. Suppose that $f(\xi)$ is a locally integrable function on $U$ of the form

[TABLE]

for some smooth function $F$. Then the distribution $P(\xi_{1}f)$ is a continuous function in $U$. Furthermore, $f$ is a weak solution to $Pf=0$ only when $P(\xi_{1}f)|_{\xi_{1}=0}\equiv 0$.

Proof.

The first assertion is clear. Moreover we have $P(\xi_{1}f)|_{\xi_{1}=0}=P_{1}F(0,\xi^{\prime}).$

For the second assertion, we observe that $f$ is a smooth function on $U^{\operatorname{reg}}:=U\setminus\{\xi_{1}\neq 0\}$. Hence, in order to show $Pf\neq 0$ in the distribution sense, it suffices to show that $Pf$ does not belong to $L_{\operatorname{loc}}^{1}(U)$ when $P_{1}F(0,\xi^{\prime})\not\equiv 0$. We introduce a locally integrable function $\widetilde{f}$ on $U$ by

[TABLE]

Clearly, the distribution

[TABLE]

is not locally integrable unless $P_{1}F(0,\xi^{\prime})\not\equiv 0$. Since $(P-\frac{\partial}{\partial\xi_{1}}P_{1})f\in L_{\operatorname{loc}}^{1}(U)$ and $\frac{\partial}{\partial\xi_{1}}P_{1}(f-\widetilde{f})\in L_{\operatorname{loc}}^{1}(U)$, we conclude that $Pf\not\in L_{\operatorname{loc}}^{1}(U)$. Thus the lemma is proved. ∎

6 Further analysis of the branching laws

In this section we discuss further analytic aspects of the branching laws of the restriction $\Pi|_{G^{\prime}}$ of a discrete series representation $\Pi\in\operatorname{Disc}(G/H)$ ($\subset\widehat{G}$), see Section 6.1 for notation.

6.1 Generalities: discrete part of unitary representations

Any unitary representation $\pi$ of a reductive Lie group $L$ has a unique irreducible decomposition:

[TABLE]

where $d\mu$ is a Borel measure on the unitary dual $\widehat{L}$, and $n_{\pi}\colon\widehat{L}\to{\mathbb{N}}\cup\{\infty\}$ is a measurable function (multiplicity).

In what follows, we use the same letter to denote a representation space with the representation. Then the Hilbert direct sum

[TABLE]

is identified with the maximal closed $G$-submodule of $\pi$ which is discretely decomposable. We say that the unitary representation $\pi_{\operatorname{disc}}$ is the discrete part of the unitary representation $\pi$, and its orthogonal complement $\pi_{\operatorname{cont}}$ in $\pi$ is the continuous part of $\pi$.

The unitary representation $\pi$ is discretely decomposable if $\pi=\pi_{\operatorname{disc}}$, whereas $\pi=\pi_{\operatorname{cont}}$ (i.e., $\pi_{\operatorname{disc}}=\{0\}$) means that the irreducible decomposition (6.1) does not contain any discrete spectrum.

The irreducible decomposition (6.1) is called the Plancherel formula when $\pi$ is the regular representation on $L^{2}(X)$ where $X$ is an $L$-space with invariant measure; it is called the branching law when $\pi$ is the restriction $\Pi|_{L}$ of a unitary representation $\Pi$ of a group $G$ containing $L$ as a subgroup. The support $\{\sigma\in\widehat{L}:\operatorname{Hom}_{L}(\sigma,\pi)\neq\{0\}\}$ will be denoted by

[TABLE]

We consider the restriction $\Pi\in\operatorname{Disc}(G/H)$ $(\subset\widehat{G})$ to the subgroup $G^{\prime}$. The unitary representation $\Pi|_{G^{\prime}}$ of the subgroup $G^{\prime}$ splits into the discrete and continuous parts:

[TABLE]

We ask

Question 6.1**.**

Let $H$, $G^{\prime}$ be reductive subgroups of $G$ and $\Pi\in{\operatorname{Disc}({G/H})}$.

1. (1)

When $(\Pi|_{G^{\prime}})_{\operatorname{disc}}=\{0\}$? 2. (2)

When $\#(\operatorname{Disc}(\Pi|_{G^{\prime}}))<\infty$? 3. (3)

When $(\Pi|_{G^{\prime}})_{\operatorname{cont}}=\{0\}$?

We note that if $G^{\prime}=H$ and if $\Pi\in\operatorname{Disc}(G/H)$ then the underlying $({\mathfrak{g}},K)$-module $\Pi_{K}$ is never discretely decomposable as a $({\mathfrak{g}}^{\prime},K^{\prime})$-module, see [K98a, Thm. 6.2].

6.2 Criteria

for $(\Pi|_{G^{\prime}})_{\operatorname{disc}}=\{0\}$ and $(\Pi|_{G^{\prime}})_{\operatorname{cont}}=\{0\}$

We retain the previous setting where

[TABLE]

From now, we assume

[TABLE]

Then Proposition 2.5 and Theorem 1.1 may be restated as:

[TABLE]

In particular $\operatorname{Disc}(G/H)\neq\emptyset$.

Here are answers to Question 6.1 (1)–(3):

Theorem 6.2** (purely continuous spectrum).**

The following two conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ are equivalent:

1. (i)

$\operatorname{Disc}(\Pi|_{G^{\prime}})=\emptyset$* for any $\Pi\in\operatorname{Disc}(G/H)$;* 2. (ii)

$(p^{\prime},p^{\prime\prime})=(1,1)$, $(p^{\prime},q^{\prime})=(1,1)$ or $(p^{\prime\prime},q^{\prime\prime})=(1,1)$.

As a weaker property than Theorem 6.2, we have:

Theorem 6.3** (at most finitely many discrete summands).**

The following three conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ are equivalent:

1. (i)

$\#\operatorname{Disc}(\Pi|_{G^{\prime}})<\infty$* for any $\Pi\in\operatorname{Disc}(G/H)$;* 2. (ii)

$\#\operatorname{Disc}(\Pi|_{G^{\prime}})<\infty$* for some $\Pi\in\operatorname{Disc}(G/H)$;* 3. (iii)

$p^{\prime}p^{\prime\prime}>0$, $\operatorname{min}(p^{\prime\prime},q^{\prime})\leq 1$ and $\operatorname{min}(p^{\prime},q^{\prime\prime})\leq 1$.

As an opposite extremal case to Theorem 6.2, we have:

Theorem 6.4** (discretely decomposable restriction).**

The following three conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ are equivalent:

1. (i)

The restriction $\Pi|_{G^{\prime}}$ is discretely decomposable for any $\Pi\in\operatorname{Disc}(G/H)$; 2. (ii)

The restriction $\Pi|_{G^{\prime}}$ is discretely decomposable for some $\Pi\in\operatorname{Disc}(G/H)$; 3. (iii)

$p^{\prime}=0$* or $p^{\prime\prime}=0$.*

For a unitary representation $\Pi$ of $G$, the space $\Pi^{\infty}$ of smooth vectors (as a representation of $G$) is smaller in general than the space $(\Pi|_{G^{\prime}})^{\infty}$ of smooth vectors as a representation of the subgroup $G^{\prime}$. This difference detects discrete decomposability of the restriction $\Pi|_{G^{\prime}}$ as follows.

Corollary 6.5**.**

Let $\Pi\in{\operatorname{Disc}({G/H})}$. Then the following two conditions are equivalent:

1. (i)

The restriction $\Pi|_{G^{\prime}}$ contains continuous spectrum in the branching law; 2. (ii)

There does not exist a closed $G^{\prime}$-irreducible submodule $W$ in $\Pi$ such that $W\cap\Pi^{\infty}\neq\{0\}$.

6.3 Proof of Theorem 6.3: finitely many summands

We begin with the proof of Theorem 6.3.

Lemma 6.6**.**

In the setting (6.2), the following three conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ and $\lambda\in A_{+}(p,q)$ are equivalent:

1. (i)

$\Lambda_{{+-}}(\lambda)\neq\emptyset$; 2. (ii)

$\#\Lambda_{{+-}}(\lambda)=\infty$; 3. (iii)

$p^{\prime\prime}=0$* or “$p^{\prime}\geq 2$ and $q^{\prime\prime}\geq 2$”.*

Proof.

Direct from the definition of $\Lambda_{{+-}}(\lambda)$ in Section 1. ∎

We note that the conditions (i) and (ii) in Lemma 6.6 do not depend on the choice of $\lambda\in A_{+}(p,q)$. An analogous result holds for $\Lambda_{{-+}}(\lambda)$ by switching the role of $(p^{\prime},q^{\prime})$ and $(p^{\prime\prime},q^{\prime\prime})$. Hence we have:

Lemma 6.7**.**

The following three conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ and $\lambda\in A_{+}(p,q)$ are equivalent:

1. (i)

$\Lambda_{{-+}}(\lambda)\cup\Lambda_{{+-}}(\lambda)\neq\emptyset$; 2. (ii)

$\#(\Lambda_{{-+}}(\lambda)\cup\Lambda_{{+-}}(\lambda))=\infty$; 3. (iii)

$p^{\prime}p^{\prime\prime}=0$, $\operatorname{min}(p^{\prime\prime},q^{\prime})\geq 2$, or $\operatorname{min}(p^{\prime},q^{\prime\prime})\geq 2$.

Since $\#\Lambda_{{++}}(\lambda)<\infty$ for any $\lambda$, Theorem 6.3 follows immediately from Lemma 6.7.

6.4 Nonexistence condition of discrete spectrum: proof of Theorem 6.2

In this section, we discuss about when the restriction $\Pi|_{G^{\prime}}$ decomposes into continuous spectrum, and give a proof of Theorem 6.2.

We begin with the following observation on elementary combinatorics:

Lemma 6.8**.**

The condition (ii) in Theorem 6.2 is equivalent to the condition:

[TABLE]

Proof.

Clear from the definitions (2.1) and (2.2) of $A_{\pm}(p,q)$. ∎

Thus the implication (ii) $\Rightarrow$ (i) in Theorem 6.2 follows readily from Theorem 1.1 and Lemma 6.8.

In order to prove the opposite implication, we need another elementary combinatorics as below. The proof is direct from the definition of $\Lambda_{{++}}(\lambda)$.

Lemma 6.9**.**

In the setting (6.2), assume further that $p^{\prime},p^{\prime\prime}\geq 2$. Then for $\lambda\in A_{+}(p,q)$, we have the following:

1. (1)

$\Lambda_{{++}}(\lambda)=\emptyset$*  if $\lambda<2$ or if “$\lambda=2$ and $p^{\prime}\equiv q^{\prime}\mod 2$”;* 2. (2)

$\Lambda_{{++}}(\lambda)\neq\emptyset$*  if $\lambda>2$ or if “$\lambda=2$ and $p^{\prime}\not\equiv q^{\prime}\mod 2$”.*

We are ready to complete the proof of Theorem 6.2.

Proof of the implication (i) $\Rightarrow$ (ii) in Theorem 6.2.

Suppose that $\operatorname{Disc}(\Pi|_{G^{\prime}})=\emptyset$ for any $\Pi\in\operatorname{Disc}(G/H)$. Then Theorem 6.3 tells

[TABLE]

On the other hand, it follows from Lemma 6.9 (2) that $\Lambda_{{++}}(\lambda)\neq\emptyset$ for $\lambda>2$ if $\operatorname{min}(p^{\prime},p^{\prime\prime})\geq 2$. Hence we get $\min(p^{\prime},p^{\prime\prime})\leq 1$. Without loss of generality, we may and do assume $p^{\prime}=1$. In turn, the condition (6.3) imply

[TABLE]

As we saw in Example 1.2, $\operatorname{Disc}(\pi_{+,\lambda}^{p,q}|_{G^{\prime}})\neq\emptyset$ for any $\lambda\in A_{+}(p,q)$ with $\lambda\geq 1$ if $(p^{\prime},q^{\prime})=(1,0)$. Hence $(p^{\prime},q^{\prime})\neq(1,0)$. Thus the implication (i) $\Rightarrow$ (ii) in Theorem 6.2 is proved. ∎

6.5 Proof of Theorem 6.4

and Corollary 6.5

In the category of $(\mathfrak{g},K)$-modules, analogous results to Theorem 6.4 and Corollary 6.5 are known in a general setting, which we now recall:

Proposition 6.10**.**

Let $(G,G^{\prime})$ be a reductive symmetric pair. For $\Pi\in\widehat{G}$ of which the underlying $(\mathfrak{g},K)$-module $\Pi_{K}$ is a Zuckerman derived functor module $A_{\mathfrak{q}}(\lambda)$. Then the following four conditions are equivalent:

1. (i)

$\Pi_{K}$* is discretely decomposable as a $({\mathfrak{g}}^{\prime},K^{\prime})$-module ([K98a, Def. 1.1]).* 2. (ii)

$\Pi_{K}$* is $K^{\prime}$-admissible, namely, $\dim_{\mathbb{C}}\operatorname{Hom}_{K^{\prime}}(\tau,\Pi_{K})<\infty$ for any $\tau\in\widehat{K^{\prime}}$.* 3. (iii)

There exists a $G^{\prime}$-irreducible closed subspace $\pi$ of $\Pi$ such that $\pi\cap\Pi_{K}\neq\{0\}$. 4. (iv)

There exists a $G^{\prime}$-irreducible closed subspace $\pi$ of $\Pi$ such that $\pi\cap\Pi_{K}$ is dense in the Hilbert space $\pi$.

Proof.

The equivalence (i) $\Leftrightarrow$ (ii) is proved in [K98a, Thm. 4.2]. The equivalence (i) $\Leftrightarrow$ (iii) $\Leftrightarrow$ (iv) follows from [K98a, Lem. 1.5]. ∎

The equivalence holds without the assumption $\Pi_{K}\simeq A_{\mathfrak{q}}(\lambda)$. See also [KO${}_{\text{Y}}$15, K19].

Back to our setting, we know from the classification theory [KO${}_{\text{Y}}$12]:

Lemma 6.11**.**

The following three conditions on $(p^{\prime},p^{\prime\prime},q^{\prime},q^{\prime\prime})$ are equivalent:

1. (i)

$\Pi_{K}$* is discretely decomposable as a $({\mathfrak{g}}^{\prime},K^{\prime})$-module for any $\Pi\in{\operatorname{Disc}({G/H})}$;* 2. (ii)

$\Pi_{K}$* is discretely decomposable as a $({\mathfrak{g}}^{\prime},K^{\prime})$-module for some $\Pi\in{\operatorname{Disc}({G/H})}$;* 3. (iii)

$p^{\prime}=0$* or $p^{\prime\prime}=0$.*

Since the discrete decomposability in the category of $(\mathfrak{g},K)$-module implies the discrete decomposability of the unitary representation, the implication (iii) $\Rightarrow$ (i) ($\Rightarrow$ (ii)) in Theorem 6.4 follows from Lemma 6.11.

To prove the converse implication (ii) $\Rightarrow$ (iii) in Theorem 6.4, the following lemma is crucial.

Lemma 6.12**.**

Let $G/H=O(p,q)/O(p-1,q)$ $(=X(p,q))$. Then the direct sum $\bigoplus_{\Pi\in{\operatorname{Disc}({G/H})}}\Pi$ is $K$-admissible.

Proof.

This follows from the classification of ${\operatorname{Disc}({G/H})}$ in Proposition 2.5 and from the $K$-type formula of $\Pi$ as seen in the condition (iii) of Definition-Theorem 2.1. ∎

Combining Lemma 6.12 with Theorem 1.1, we have

Proposition 6.13**.**

For any $\Pi\in{\operatorname{Disc}({G/H})}$, $(\Pi|_{G^{\prime}})_{\operatorname{disc}}$ is $K^{\prime}$-admissible.

We are ready to complete the proof of Theorem 6.4.

Proof of the implication (ii) $\Rightarrow$ (iii) in Theorem 6.4.

Suppose that the restriction $\Pi|_{G^{\prime}}$ is discretely decomposable as a unitary representation of the subgroup $G^{\prime}$, i.e., $\Pi|_{G^{\prime}}=(\Pi|_{G^{\prime}})_{\operatorname{disc}}$. Then $\Pi$ is $K^{\prime}$-admissible by Proposition 6.13, and so is the underlying $(\mathfrak{g},K)$-module $\Pi_{K}$. Hence $p^{\prime}=0$ or $p^{\prime\prime}=0$ by Lemma 6.11. Thus Theorem 6.4 is proved. ∎

Proof of Corollary 6.5.

By Theorem 6.4, the condition (i) in Corollary 6.5 is equivalent to the following:

(i) $p^{\prime}p^{\prime\prime}\neq 0$,

whereas the condition (ii) is clearly equivalent to

(ii)′ For any $\pi\in\widehat{G^{\prime}}$ and any $\iota\in\operatorname{Hom}_{G^{\prime}}(\pi,\Pi|_{G^{\prime}})$, $\iota(\pi)\cap\Pi^{\infty}=\{0\}$.

Let us prove the equivalence (i)′ $\Leftrightarrow$ (ii)′.

(ii)′ $\Rightarrow$ (i)′: Suppose $p^{\prime}p^{\prime\prime}=0$. Then $\iota(\pi)\cap\Pi_{K}\neq\{0\}$ by Proposition 6.10, whence $\iota(\pi)\cap\Pi^{\infty}\neq\{0\}$ because $\Pi_{K}\subset\Pi^{\infty}$.

(i)′ $\Rightarrow$ (ii)′: Conversely, suppose $\iota\colon\pi\to\Pi|_{G^{\prime}}$ is a nonzero continuous $G^{\prime}$-homomorphism for some $\pi\in\widehat{G^{\prime}}$. Then $\pi$ must be of the form $\pi_{\delta,\lambda^{\prime}}^{p^{\prime},q^{\prime}}\boxtimes\pi_{\varepsilon,\lambda^{\prime\prime}}^{p^{\prime\prime},q^{\prime\prime}}$ for some $(\delta,\varepsilon)$ and $(\lambda^{\prime},\lambda^{\prime\prime})$, and $\iota$ must be a scalar multiple of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ by Theorems 1.1 and 4.3. If $p^{\prime}p^{\prime\prime}\neq 0$, then it follows from the definition of $X(p,q)_{\delta\varepsilon}$ in Section 3.1 that at least two of the open sets $X(p,q)_{{-+}}$, $X(p,q)_{{++}}$, $X(p,q)_{{+-}}$ are nonempty, and thus $\operatorname{Image}T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}\cap C^{\infty}(X(p,q))=\{0\}$ by the definition of $T_{\delta\varepsilon,\lambda}^{\lambda^{\prime},\lambda^{\prime\prime}}$ in Section 4.3. Since $\Pi^{\infty}\subset C^{\infty}(X(p,q))$, this shows that $\iota(\pi)\cap\Pi^{\infty}=\{0\}$. Therefore, we have shown the implication (i)′ $\Rightarrow$ (ii)′. ∎

7 Appendix —multiplicity in branching laws

As viewed in [K15], we divide branching problems into the following three stages:

Stage A: Abstract features of the restriction;

Stage B: Branching laws (irreducible decomposition of restrictions);

Stage C: Construction of symmetry breaking/holographic operators.

The role of Stage A is to develop an abstract theory on the restriction of representations as generally as possible. In turn, we could expect a detailed study of the restriction in Stages B and C in the specific settings that are a priori guaranteed to be “nice” in Stage A. Conversely, new results and methods in Stage C may indicate a further fruitful direction of branching problems including Stage A.

The present article has focused on analytic problems in Stages B and C in the setting where the triple $H\subset G\supset G^{\prime}$ is given by

[TABLE]

Then one might wonder what are the abstract features (Stage A) which have arisen from this article, and also might be curious about a possible generalization beyond the setting (7.1). The spectral property of the branching laws is such an aspect, which we discussed in Section 6. Another aspect of Theorem 1.1 is the multiplicity-free property:

[TABLE]

Here, for $\Pi\in\widehat{G}$, the multiplicity $m_{\Pi}(\pi)$ of $\pi\in\widehat{G^{\prime}}$ as the discrete spectrum of the (unitary) restriction $\Pi|_{G^{\prime}}$ is defined by

[TABLE]

In this Appendix, we give a flavor of some multiplicity estimates (Stage A) in a broader setting than (7.1), for instance, when

[TABLE]

In what follows, we treat not only discrete series representations $\Pi\in\operatorname{Disc}(G/H)$ but also non-unitary representations that have a non-trivial $H$-period (or is $H$-distinguished) as well. We recall that there is a canonical equivalence of categories between the category ${\mathcal{HC}}$ of $({\mathfrak{g}},K)$-modules of finite length and the category ${\mathcal{M}}$ of smooth admissible representations of moderate growth by the Casselman–Wallach globalization theory [Wa92, Chap. 11]. Denote by $\operatorname{Irr}(G)$ the set of irreducible objects in ${\mathcal{M}}$. The unitary dual $\widehat{G}$ may be thought of as a subset of $\operatorname{Irr}(G)$ by taking smooth vectors:

[TABLE]

For $\Pi^{\infty}\in\operatorname{Irr}(G)$ and $\pi^{\infty}\in\operatorname{Irr}(G^{\prime})$, we set

[TABLE]

In general, for any $\Pi\in\widehat{G}$, one has $m_{\Pi}(\pi)\leq m_{\Pi^{\infty}}(\pi^{\infty})$ for all $\pi\in\widehat{G^{\prime}}$, and $m_{\Pi}(\pi)\leq n_{\Pi}(\pi)\leq m_{\Pi^{\infty}}(\pi^{\infty})$ a.e. $\pi\in\widehat{G^{\prime}}$ with respect to the measure for the disintegration (6.1) of the (unitary) restriction $\Pi|_{G^{\prime}}$, where we recall $n_{\Pi}\colon\widehat{G^{\prime}}\to{\mathbb{N}}\cup\{\infty\}$ is the measurable function which gives the multiplicity in (6.1).

For a closed subgroup $H$ of $G$, we define

[TABLE]

where $\Pi^{-\infty}$ denotes the representation on the space of distribution vectors.

Then $\operatorname{Disc}(G/H)$ may be thought of as a subset of $\operatorname{Irr}(G)_{H}$ via (7.4).

Now we address the following:

Problem 7.1**.**

Find a criterion for a triple $H\subset G\supset G^{\prime}$ with bounded multiplicity property for the restriction: there exists $C>0$ such that

[TABLE]

Note that the condition (7.5) immediately implies

[TABLE]

We also note that (7.2) is nothing but (7.6) with $C=1$.

We recall some general results in the setting where $H=\{e\}$ from [KO${}_{\text{T}}$15, Thms. C and D] and [K98a, Thm. 4.2] (see also Proposition 6.10):

Bounded multiplicity: $(G_{\mathbb{C}}\times G_{\mathbb{C}}^{\prime})/\operatorname{diag}G_{\mathbb{C}}^{\prime}$ is spherical iff

[TABLE]

Finite multiplicity: $(G\times G^{\prime})/\operatorname{diag}G^{\prime}$ is real spherical iff

[TABLE]

Admissible restriction: If $\Pi_{K}$ is discretely decomposable as a $({\mathfrak{g}}^{\prime},K^{\prime})$-module and if $(G,G^{\prime})$ is a symmetric pair, then

[TABLE]

(This generalizes Harish-Chandra’s admissibility theorem for compact $G^{\prime}$.)

In these cases, explicit criteria lead us to the classification theory. The criterion [KO${}_{\text{T}}$15] for (7.7) depends only on the complexification $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})$, hence the classification for (7.7) for simple ${\mathfrak{g}}_{\mathbb{C}}$ simple reduces to a classical result [Kr76]:

[TABLE]

In this case, one can take $C=1$ for most of the real forms [SZ12]. On the other hand, irreducible symmetric pairs $({\mathfrak{g}},{\mathfrak{g}}^{\prime})$ satisfying (7.8) were classified in [KM14]. The triples $(A_{\mathfrak{q}}(\lambda),{\mathfrak{g}},{\mathfrak{g}}^{\prime})$ having discretely decomposable restrictions $A_{\mathfrak{q}}(\lambda)|_{{\mathfrak{g}}^{\prime}}$ were classified in [KO${}_{\text{Y}}$12].

We now consider the setting (7.3). In this generality, (7.6) may fail. The following example is a reformulation of [K00, Ex. 5.5] (cf. [K08, Sect. 6.3]).

Example 7.2**.**

$(G,H,G^{\prime})=(SO(5,{\mathbb{C}}),SO(3,2),SO(3,2))$. Then for any $\Pi\in\operatorname{Disc}(G/H)$ there exists $\pi\in\widehat{G^{\prime}}$ such that $m_{\Pi}(\pi)=\infty$. (In this case, the disintegration $\Pi|_{G^{\prime}}$ contains continuous spectrum, see (7.9).)

As we shall see in Observation 7.10 (1) below, the bounded multiplicity property (7.5) often holds if $\operatorname{rank}G/H=1$, but not always:

Example 7.3**.**

Let $(G,H,G^{\prime})=(SU(3),U(2),SO(3))$. Then (7.6) fails because $m_{\Pi_{n}}(\pi_{n})=[\frac{n}{2}]+1$ where $\Pi_{n}\in\operatorname{Disc}(G/H)$ and $\pi_{n}\in\widehat{G^{\prime}}$ are of dimensions $(n+1)^{3}$ and $2n+1$, respectively.

Example 7.4**.**

Let $(G,H,G^{\prime})=(SL(3,{\mathbb{R}}),GL(2,{\mathbb{R}}),SO(3))$. Then (7.6) fails because $\sup_{\pi\in\widehat{G^{\prime}}}m_{\Pi}(\pi)=\infty$ for any $\Pi\in\operatorname{Disc}(G/H)$.

The last example may be compared with the following:

Example 7.5**.**

Let $(G,H,G^{\prime})=(SL(4,{\mathbb{R}}),Sp(2,{\mathbb{R}}),SO(4))$. Then (7.6) holds because $\sup_{\pi\in\widehat{G^{\prime}}}m_{\Pi}(\pi)=1$ for any $\Pi\in\operatorname{Disc}(G/H)$.

To describe an answer to Problem 7.1 (Stage A) which covers not only discrete series representations $\Pi\in\operatorname{Disc}(G/H)$ but also any irreducible representations $\Pi^{\infty}$ realized in $C^{\infty}(G/H)$, we fix some notation. Denote by $\sigma$ the involution of $G$ that defines a symmetric pair $(G,H)$. We use the same letter $\sigma$ to denote the complex linear extension of its differential. We write $G_{\mathbb{C}}$ for a complexification of $G$, and $G_{U}$ for a compact real form of $G_{\mathbb{C}}$. Let ${\mathfrak{j}}_{\mathbb{C}}$ be a maximal semisimple abelian subspace in ${\mathfrak{g}}_{\mathbb{C}}^{-\sigma}=\{X\in{\mathfrak{g}}_{\mathbb{C}}:\sigma X=-X\}$, and $Q_{\mathbb{C}}$ a parabolic subgroup of $G_{\mathbb{C}}$ with Levi part $Z_{G_{\mathbb{C}}}({\mathfrak{j}}_{\mathbb{C}})$.

Theorem 7.6**.**

Suppose that $(G,H)$ is a reductive symmetric pair, and $G^{\prime}$ an (algebraic) reductive subgroup of $G$. Then the following three conditions on the triple $(G,H,G^{\prime})$ are equivalent:

1. (i)

${}^{\exists}C>0$, $m_{\Pi^{\infty}}(\pi^{\infty})\leq C$  ${}^{\forall}\Pi^{\infty}\in\operatorname{Irr}(G)_{H}$ and ${}^{\forall}\pi^{\infty}\in\operatorname{Irr}(G^{\prime})$. 2. (ii)

$G_{\mathbb{C}}/Q_{\mathbb{C}}$* is $G_{\mathbb{C}}^{\prime}$-spherical.* 3. (iii)

$G_{\mathbb{C}}/Q_{\mathbb{C}}$* is $G_{U}^{\prime}$-strongly visible.*

See [T21] (see also [K05, Cor. 15]) for the equivalence (ii) $\Leftrightarrow$ (iii).

Remark 7.7**.**

The multiplicity-freeness (7.2) holds for compact forms.

It should be mentioned that the bounded multiplicity property (i) depends a priori on the real form $(G,H,G^{\prime})$, however, Theorem 7.6 tells that its criterion (ii) (or equivalently (iii)) can be stated only by the complexification of the Lie algebras $({\mathfrak{g}},{\mathfrak{h}},{\mathfrak{g}}^{\prime})$. Here is a complete classification of such triples $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})$ when ${\mathfrak{g}}_{\mathbb{C}}$ is simple:

Corollary 7.8** (classification).**

Assume ${\mathfrak{g}}_{\mathbb{C}}$ is simple in the setting (7.3). Then the bounded multiplicity property (7.5) holds for the triple $(G,H,G^{\prime})$ iff the complexified Lie algebras $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})$ are in Table 7.1 up to automorphisms. In the table, $p$, $q$ are arbitrary subject to $n=p+q$.

Here by “automorphisms” we mean inner automorphisms for $({\mathfrak{g}},{\mathfrak{h}})$ and $({\mathfrak{g}},{\mathfrak{g}}^{\prime})$ separately and outer autormorphisms for $({\mathfrak{g}},{\mathfrak{h}},{\mathfrak{g}}^{\prime})$ simultaneously. Thus in Table 7.1, we have omitted some cases such as $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})=({\mathfrak{so}}_{8},{\mathfrak{spin}}_{7})$, $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})=({\mathfrak{so}}_{8},{\mathfrak{gl}}_{4},{\mathfrak{so}}_{6}\oplus{\mathfrak{so}}_{2})$ or $({\mathfrak{sl}}_{4},{\mathfrak{sp}}_{2},{\mathfrak{sl}}_{2}\oplus{\mathfrak{sl}}_{2}\oplus{\mathbb{C}})$.

The right-hand side of Table 7.1 collects the case (7.10), where a stronger bounded multiplicity theorem (7.7) holds. The left-hand side includes:

Example 7.9**.**

The setting (7.1) for Theorem 1.1 is a real form of $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})=({\mathfrak{so}}_{n},{\mathfrak{so}}_{n-1},{\mathfrak{so}}_{p}\oplus{\mathfrak{so}}_{q})$ in the fourth row of the left-hand side in Table 7.1.

From Corollary 7.8, one sees the following:

Observation 7.10**.**

(1)* The bounded multiplicity (7.5) holds for any triple $(G,H,G^{\prime})$ with $\operatorname{rank}G/H=1$ except for the following two cases: $({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{h}}_{\mathbb{C}},{\mathfrak{g}}_{\mathbb{C}}^{\prime})=({\mathfrak{sl}}_{n},{\mathfrak{gl}}_{n-1},{\mathfrak{so}}_{n})\text{ or }({\mathfrak{f}}_{4},{\mathfrak{so}}_{9},{\mathfrak{sp}}_{3}\oplus{\mathfrak{sl}_{2}).}$

(2) The bounded multiplicity (7.5) may hold even when $\operatorname{rank}G/H>1$ and $\operatorname{rank}G/G^{\prime}>1$.*

Theorem 7.6 also gives a criterion for two reductive symmetric pairs $(G,H_{1})$ and $(G,H_{2})$ with the following bounded multiplicity property of tensor product representations.

Theorem 7.11** (tensor product).**

Suppose that $(G,H_{j})$ $(j=1,2)$ are reductive symmetric pairs, and that ${Q_{j}}_{\mathbb{C}}$ are parabolic subgroups of $G_{\mathbb{C}}$ as in Theorem 7.6. Then the following three conditions on the triple $(G,H_{1},H_{2})$ are equivalent:

1. (i)

There exists $C>0$ such that

[TABLE] 2. (ii)

$(G_{\mathbb{C}}\times G_{\mathbb{C}})/({Q_{1}}_{\mathbb{C}}\times{Q_{2}}_{\mathbb{C}})$* is $G_{\mathbb{C}}$-spherical via the diagonal action.* 3. (iii)

$(G_{\mathbb{C}}\times G_{\mathbb{C}})/({Q_{1}}_{\mathbb{C}}\times{Q_{2}}_{\mathbb{C}})$* is $G_{U}$-strongly visible via the diagonal action.*

By the classification of strongly visible actions [K07], one concludes from Theorem 7.6 that such examples for groups of type A are rare:

Example 7.12** (tensor product).**

Suppose ${\mathfrak{g}}_{\mathbb{C}}={\mathfrak{sl}}(n,{\mathbb{C}})$. Then (7.11) holds iff $({\mathfrak{g}}_{\mathbb{C}},{{\mathfrak{h}}_{1}}_{\mathbb{C}},{{\mathfrak{h}}_{2}}_{\mathbb{C}})$ is isomorphic to $({\mathfrak{sl}}_{2},{\mathfrak{so}}_{2},{\mathfrak{so}}_{2})$ or $({\mathfrak{sl}}_{4},{\mathfrak{sp}}_{2},{\mathfrak{sp}}_{2}).$

For groups of type BD, one has:

Example 7.13** (tensor product).**

Let $G=O(p,q)$, and $H_{1}$, $H_{2}$ be $O(p-1,q)$ or $O(p,q-1)$. Then (7.11) holds. In particular, the tensor product $\Pi_{\delta,\lambda}^{p,q}\otimes\Pi_{\varepsilon,\nu}^{p,q}$ decomposes into irreducible unitary representations with uniformly bounded multiplicities for any $\delta,\varepsilon\in\{+,-\}$, $\lambda\in A_{\delta}(p,q)$, $\nu\in A_{\varepsilon}(p,q)$.

Example 7.14** (tensor product).**

Let $G=O(2p,2q)$ with $p+q=4$, $H=O(2p-1,2q)$, and $G^{\prime}=U(p,q)$. Then (7.11) holds.

Proofs of the assertions in Appendix will be given in another paper.