Branching laws for discrete series of some affine symmetric spaces

TL;DR

This paper investigates the restriction of discrete series representations of affine symmetric spaces, using period integrals and symmetry-breaking operators, and explores conjectures related to Arthur packets and the Gross-Prasad conjectures.

Contribution

It introduces non-vanishing symmetry-breaking operators for discrete spectrum representations and discusses conjectures on restrictions within Arthur packets.

Findings

Constructed explicit symmetry-breaking operators for certain representations.

Established non-vanishing conditions for these operators.

Proposed conjectures relating to Arthur and Vogan packets.

Abstract

In this paper we study branching laws for certain unitary representations. This is done on the smooth vectors via a version of the {\it period integrals}, studied in number theory, and also closely connected to the {\it symmetry-breaking operators}, introduced by T.~Kobayashi. We exhibit non-vanishing symmetry breaking operators for the restriction of a representation $\Pi$ in the discrete spectrum for real hyperboloids to representations of smaller orthogonal groups. In the last part we discuss some conjectures for the restriction of representations in Arthur packets containing the representation $\Pi$ and the corresponding Arthur-Vogan packets to smaller orthogonal groups; these are inspired by the Gross-Prasad conjectures.

Equations

$$\text{mbox}{Hom}_{G^{\prime }}({\Pi }_{|G^{\prime }}^{\infty },{\pi }^{\infty }).$$

$$\text{mbox}{Hom}_{G^{\prime }}({\Pi }_{|G^{\prime }}^{\infty }\otimes ({\pi }^{\vee }{)}^{\infty },\mathbb{C}).$$

$$I=I(v,w)={\int }_{X^{\prime }}v({\dot{g}}^{\prime }{e}_p)\cdot w({\dot{g}}^{\prime }{e}_p)d\dot{g}$$

$$G=S{O}_o(p,q)\text{mbox}withp,q\ge 4\text{mbox}andH=S{O}_o(p,q-1).$$

$$a_1,a_2,\dots ,a_m$$

$$b_1,b_2,\dots ,b_n$$

$$a_1>b_1>a_2>b_2>\dots$$

$$b_1\ge a_1>a_2>b_2>\dots$$

$$(a,0,0,\dots ,0)+\rho =(a+\frac{p+q}{2},\frac{p+q-2}{2},\dots )$$

$$(a+\frac{p+q}{2},\frac{p+q-2}{2},\dots )$$

$$(b+\frac{p+q-1}{2},\frac{p+q-3}{2},\dots )$$

$$\text{mbox}{dimHom}_{G^{\prime }}({\Pi }_{|G^{\prime }}^{\infty },{\pi }^{\infty })\ne 0.$$

$$(a+\frac{p+q}{2},\frac{p+q-2}{2},\dots )$$

$$(b+\frac{p+q-1}{2},\frac{p+q-3}{2},\dots )$$

$$\text{mbox}{dimHom}_{G^{\prime }}({\Pi }_{|G^{\prime }}^{\infty },{\pi }^{\infty })\ne 0$$

$$(s,0,\dots ,o)+\rho$$

$$\text{mbox}{Hom}_{G^1}({\Pi }_s^{\infty },{\pi }^{\infty })\ne 0$$

$$\text{mbox}{Hom}_{G^1}({\Pi }_{as}^{\infty },{\pi }^{\infty })=0$$

$$\text{mbox}{Hom}_{G^1}({\Pi }_{as}^{\infty },{\pi }^{\infty })\ne 0$$

$$\text{mbox}{Hom}_{G^1}({\Pi }_s^{\infty },{\pi }^{\infty })=0$$

$$Q(X,X)=x_1^2+...+x_p^2-x_{p+1}^2-....-x_{p+q}^2$$

$$\{\xi \in {\mathbb{R}}^{p+q}|Q(\xi ,\xi )=-1\}$$

$$G=KJH$$

$$X(p,q)=(K/M_H)J_{+}$$

$$G^1=K^{1}J(H\cap G^1)\text{mbox}and{G}^2=K^{2}J(H\cap G^2)$$

$$\{\xi \in {\mathbb{R}}^{p+q}|Q(\xi ,\xi )=1,{\xi }_p=0\}=X(p-1,q)\subset X(p,q)$$

$$\{\xi \in {\mathbb{R}}^{p+q}|Q(\xi ,\xi )=1,{\xi }_{p+1}=0\}=X(p,q-1)\subset X(p,q)$$

$$X(p-1,q)=(K_{G^1}/M_{G^1})J_{+}$$

$$X(p,q-1)=(K_{G^2}/M_{G^2})J_{+}$$

$$\Phi (y,y^{\prime },t)=(y_1\sinh (t),\dots y_p\sinh (t),y_1^{\prime }\cosh (t),y_2^{\prime }\cosh (t),\dots ,y_q^{\prime }\cosh (t))$$

Full text

Research by B.Speh partially supported by NSF grant DMS-1500644

*Dedicated to B. Kostant *

Mathematic Subject Classification (2010): Primary 22E46; Secondary 53C30, 22E30 22E45, 22E50

I. Introduction

The restriction of a finite-dimensional irreducible representation $\Pi_{G}$ of a connected compact Lie group $G$ to a connected Lie subgroup $H$ is a classical problem. For example, the restriction of irreducible representations of $SU(n+1)$ to $SU(n)$ or of $SO(n+1)$ to the subgroup $SO(n)$ can be expressed as a combinatorial pattern satisfied by the highest weights of the irreducible representation $\Pi_{G}$ of the large group and of the irreducible representations $\pi_{H}$ appearing in the restriction [34]. More generally determining the restriction of an irreducible representation of a compact connected Lie group to a closed connected subgroup is transformed to a combinatorial problem involving highest weights by a famous branching theorem by B. Kostant [22].

Considerable efforts have been devoted recently to understanding the restriction or branching laws for infinite dimensional irreducible representations $\Pi$ of a semisimple Lie group $G$ to a symmetric subgroup $G^{\prime}$ and more recently to other subgroups. There are many different techniques for dealing with this question. The orbit method and geometric quantization considered by B. Kostant are useful tools to determine the restriction of the representation $\Pi$ to a non-compact reductive subgroup $G^{\prime}$ (see [12]) if the restriction of $\Pi$ is $G^{\prime}$-admissible, i.e. if it is a direct sum of irreducible representations with finite multiplicities of the subgroup $G^{\prime}$. For example this is the case if we restrict a holomorphic discrete series representation $\Pi$ to a large subgroup. See [28].

To determine explicit branching laws, one usually uses specific models of the representations involved. In the present paper we work with representations in the way they arise explicitly as discrete series of $L^{2}(G/H)$ of a symmetric space $G/H$, and we refer to those as Flensted-Jensen representations. We investigate the restriction of discrete series representations of symmetric spaces $G/H$, i.e of irreducible subrepresentations $\Pi$ of $G$ in $L^{2}(G/H)$, to a symmetric subgroup $G^{\prime}$. The restriction of $\Pi$ to $G^{\prime}$ is not always a direct sum of irreducible representations of $G^{\prime}$ and thus as in [18] and in [19] for irreducible representations $\pi$ of $G^{\prime}$ we consider

[TABLE]

Here, as the notation suggests, we work in the smooth category and consider $G^{\prime}$-equivariant continuous homomorphisms in the topology on the $C^{\infty}$-vectors introduced by Casselman-Wallach [33] and Bernstein-Kroetz [5]. Equivalently we can consider periods

[TABLE]

where $(\pi^{\vee})^{\infty}$ is the smooth part of the contragredient representation of $\pi$.

In our case we shall consider a linear functional on $\Pi^{\infty}\otimes(\pi^{\vee})^{\infty}$ defined by a period integral over $G^{\prime}$-orbits on $X=G/H$; namely

[TABLE]

where $X^{\prime}=G^{\prime}/H^{\prime},\,H^{\prime}=G^{\prime}\cap H$ and $H$ is the stabilizer of the base point $e_{p}$. This linear functional is $G^{\prime}$-invariant provided it converges for all $C^{\infty}$-vectors. We obtain a branching law by proving that this linear functional is not zero. For discrete series of a semisimple Lie group a similar linear functional was used by J. Vargas [32] to obtain in some cases a branching law for unitary restriction of discrete series representations of $G$ to a subgroup $G^{\prime}$. We emphasize that here we work in the category of smooth representations.

In this article we illustrate the main ideas in one specific example (making formulas quite explicit). We assume that

[TABLE]

and consider representations $\Pi$ in the discrete spectrum of $L^{2}(G/H)$ of the de Sitter space $X=G/H$ and representations $\pi^{\vee}$ in the discrete spectrum of $L^{2}(G^{\prime}/H^{\prime})$ where

1. (1)

$G^{\prime}=SO_{o}(p-1,q)$ and $H^{\prime}=SO_{o}(p-1,q-1),$ or 2. (2)

$G^{\prime}=SO_{o}(p,q-1)$ and $H^{\prime}=SO_{o}(p,q-2)$

We discuss two basic questions for our period integrals, namely their convergence and their non-vanishing on $\Pi^{\infty}\otimes(\pi^{\vee})^{\infty}$. For this we use the work of M. Flensted-Jensen [9] about discrete series for affine symmetric spaces $G/H$, criteria for existence, and explicit formulas for test functions in the minimal $K$-types. We show that for certain pairs of representations our period integrals do not vanish for functions in the minimal K-types and that they define in this case a $G^{\prime}$-invariant linear functional on the underlying $({\mathfrak{g}},K)$-modules. Then we show that the period integral $I$ in fact defines a continuous $G^{\prime}$-invariant linear functional on the $C^{\infty}$-vectors. In the last step, i.e. to show that this linear functional is continuous, we use that the smooth vectors in $\Pi\times\pi$ form an irreducible module for the algebra of Schwartz functions on $G,G^{\prime}$ [5], [33]. Thus we obtain the ”automatic continuity” of the period integral $I$.

Following the notation in [29] we shall (see more details below) parametrize the representations in the discrete series of $G/H$, $G^{\prime}/H^{\prime}$ and $G^{\prime\prime}/H^{\prime\prime}$ (cases (1) and (2) above) by constants $a,\ b,\ c\ \in{{\mathbb{Z}}_{+}}{/2}$ respectively. Note that if $p+q$ is even then $a$ is integral and $b$, $c$ are half integral. If $p+q$ is odd then $a$ is half integral and $b$, $c$ are integral.

Theorem I.1**.**

Under the above assumptions

1. (1)

Suppose that $G^{\prime}=SO_{o}(p-1,q)$ and $H^{\prime}=SO_{o}(p-1,q-1).$ The period integral is nontrivial on the minimal K-type of $\Pi_{a}\otimes\pi_{b}$ if $b<a$. 2. (2)

Suppose that $G^{\prime}=SO_{o}(p,q-1)$ and $H^{\prime}=SO_{o}(p,q-2)$. The period integral is nontrivial on the minimal K-type of $\Pi_{a}\otimes\pi_{c}$ iff $c=a+1/2$.

We can now formulate our branching results.

We say that two finite sequences of half integers

[TABLE]

[TABLE]

have an interlacing property of finite type if

[TABLE]

We say that the sequences have a interlacing property of infinite type 1 if

[TABLE]

Note that the infinitesimal character of the representation $\Pi_{a}$ is

[TABLE]

Theorem I.2**.**

Under the above assumptions

1. (1)

Suppose that $G^{\prime}=SO_{o}(p-1,q)$ and $H^{\prime}=SO_{o}(p-1,q-1),$ Suppose that $\Pi$ and $\pi$ are Flensted Jensen representations with infinitesimal characters

[TABLE]

[TABLE]

If the infinitesimal characters of $\Pi$ and $\pi$ have an interlacing property of finite type then

[TABLE] 2. (2)

Suppose that $G^{\prime}=SO_{o}(p,q-1)$ and $H^{\prime}=SO_{o}(p,q-2)$. Suppose that $\Pi$ and $\pi$ are Flensted Jensen representations with the infinitesimal characters

[TABLE]

[TABLE]

Then

[TABLE]

implies that $b\geq a+1/2$ and thus the infinitesimal characters have an interlacing property of infinite type 1.

In the last section we look at our results from a different perspective using ideas coming from automorphic forms, the trace formula and the Gross-Prasad conjectures [10], [20], [24]. They suggest to consider not individual groups and representations but rather packets of representations of $G=SO_{o}(p,q)$, i.e. an Arthur packet $\mathcal{C}_{p,q}$ which contains discrete series representation $\Pi_{s}$ of $L^{2}(G/H)$. Instead of considering one group $SO_{o}(p,q)$ it might be helpful to consider a family of groups namely the pure inner forms containing $SO_{o}(p,q)$. Instead of considering a Arthur packet ${\mathcal{C}_{p,q}}$ of representations it might best to formulate the results in the language of Arthur-Vogan packets $\mathcal{A}V$. For motivation and details see [10], [2] and section V.

We fix a regular infinitesimal character.

[TABLE]

of a finite dimensional representation of $SO(p+q)$. In [3] J. Adams and J. Johnson introduced stable packets of cohomologically induced representations which are usually referred to as Arthur packets. The Arthur packet containing a Flensted-Jensen representation $\Pi_{s}$ in the discrete spectrum of $SO_{o}(p,q)/SO_{o}(p,q-1)$ contains also another representation $\Pi_{as}$ . The representations are cohomologically induced from parabolic subgroups with Levi $SO(2,0)\times SO_{o}(p-1,q)$, respectively. $SO_{o}(p,q-1)\times SO(0,2)$ and the representation $\Pi_{as}$ is contained in the discrete spectrum of $SO_{o}(p,q)/SO_{o}(p-1,q)$.

We restrict to 2 subgroups $G^{1}=SO_{o}(p-1,q)$ and $G^{2}=SO_{o}(p,q-1)$ which are not in the same inner class of orthogonal groups. For each of these subgroups we have corresponding Arthur packets of representations. $L$-functions and trace formula considerations strongly suggest that the multiplicities depend on the inner class of the groups $G^{i}$ and the Arthur packet of representations.

Our computations and the results of [17] support this; they show that the restriction of exactly one of the representations in our Arthur packet is direct sum, the other one is not admissible.

Based on our computations we expect furthermore a multiplicity one result similar to the famous Gross-Prasad conjecture, namely as follows:

Suppose that $\pi$ is a representations of $G^{1}$ in an Arthur packet which contains a Flensted-Jensen representation in the discrete spectrum of $G^{1}/(H^{1}\cap G^{1})$. If

[TABLE]

then

[TABLE]

and if

[TABLE]

then

[TABLE]

We expect that an analogous statement holds for the group $G^{2}$.

Acknowledgement. That this paper is devoted to B. Kostant is only a small token of the gratitude and debt that we feel and owe for much inspiration and mathematical energy. B. Speh would like to thank the department of mathematics at Aarhus University for its hospitality and support during this research. We also thank T. Kobayashi for some private communications, and in particular remarks on the topic of branching laws for the representations considered here.

II. Generalities

In this section we recall the results about symmetric spaces by M. Flensted-Jensen [9], Flensted Jenson functions and Flensted Jensen representations [30], [29]. We also discuss the restriction of representations and introduce symmetry breaking operators as well as periods. The notation here is inspired by [18].

II.1. Notation and geometric background

Consider the quadratic form

[TABLE]

and let $G=G_{p,q}=SO_{o,}(p,q)$ be the identity component of the automorphism group $O(p,q)$ of the form $Q(X,X)$.

Recall from the introduction that we assume in this article $p,q\geq 4$. We denote the stabilizer of $e_{p+q}$ by $H=H_{p,q-1}$, the stabilizer of $e_{1}$ by $G^{1}=G^{1}_{p-1,q}$. and the stabilizer of $e_{p+1}$ by $G^{2}=G^{\prime 2}_{p,q-1}=O(p,q-1)$. The maximal compact subgroups of $G,G^{1},G^{2}$ and $H$ are denoted by $K$, $K^{1}$, $K^{2}$ and $K_{H}$, and we denote the Lie algebras by the coresponding small gothic letters. The connected subgroup with Lie algebra ${\mathbb{R}}(E_{p+q,p}+E_{p,p+q})$, is denoted by $J$.

The indefinite hyperbolic space $X(p,q)$ is defined by

[TABLE]

It is a rank one symmetric space and isomorphic to $G/H$. As a product

[TABLE]

and

[TABLE]

where $M_{H}$ is the intersection of the centralizer of $J$ in $K$ with the compact group $K_{H}$ (see [29]) and $J_{+}\subset J$ is isomorphic to ${\mathbb{R}}^{+}$. (Strictly speaking, we are here parametrizing an open dense subset of $X(p,q)$ but this we ignore here and below.)

We observe

Lemma II.1**.**

Under these assumptions

1. (1)

[TABLE] 2. (2)

The set

[TABLE]

is an orbit of $G^{1}$ and isomorphic to $G^{1}/G^{1}\cap H$ . The set

[TABLE]

is an orbit of $G^{2}$ and isomorphic to $G^{2}/G^{2}\cap H$. 3. (3)

We have

[TABLE]

respectively

[TABLE]

where $M_{G^{i}}$ i = 1,2 is the intersection of the centralizer of $J$ in $K^{i}$ with the compact group $K_{H}=K\cap H$ (see **[29]** 8.4.2). 4. (4)

$M_{G}=SO(p-1)\times SO(q-1)$* and so *

$X(p,q)=(S^{p-1}\times S^{q-1}){\mathbb{R}}^{+}$*, *

$X(p-1,q)=(S^{p-2}\times S^{q-1}){\mathbb{R}}^{+}$* and *

$X(p,q-1)=(S^{p-1}\times S^{q-2}){\mathbb{R}}^{+}.$**

On $X(p,q)$ we have the coordinates

[TABLE]

where $\sum(y_{i})^{2}=1$ , $\sum(y_{i}^{\prime})^{2}=1$ and

[TABLE]

II.2. Analytic background

The universal enveloping algebra $U({\mathfrak{g}})$ acts on the $C^{\infty}$-functions on the hyperbolic space $X(p,q)$. The algebra ${\mathcal{S}}(G)$ of Schwartz functions acts by convolutions as well. (See [5] and below for the definition of ${\mathcal{S}}(G)$, for more details see III.3.)

For background on the parameters relevant for the discrete series of an affine symmetric space, see Schlichtkrull [30].

Suppose that $\lambda$ satisfies the condition 8.8 and 8.9 in [29] and let $F(\lambda,x)\in\mathcal{C}^{\infty}(G/H)$ be the function constructed by M. Flensted-Jensen (see 7.3 in [29]). In our example

[TABLE]

where $\lambda>0$ and $a=\lambda-1+\frac{p-q}{2}$, and $f_{a}$ is a spherical harmonic of degree $a$. Flensted Jensen proved that the Flensted-Jensen function $F(\lambda,x)$ has the following properties:

1. (1)

$F(\lambda,x)$ is square integrable. 2. (2)

$F(\lambda,x)$ decays rapidly on $J_{+}$ in the coordinates $(K/M_{H})\ J_{+}$, i.e. it is a Schwartz function as a function on $J_{+}$ . 3. (3)

$F(\lambda,x)$ generates an irreducible submodule $({\mathfrak{g}},K)$-module $W(\lambda)\subset\mathcal{C}^{\infty}(G/H)$. 4. (4)

$F(\lambda,x)$ as $K$-module generates the minimal $K$-type. 5. (5)

The unitary completion of the $({\mathfrak{g}},K)$ module $W(\lambda)$ is a unitary representation, namely it is a unitary irreducible subrepresentation $L^{2}$–spectrum of $X(p,q)$.

Furthermore $F(\lambda,x)$ generates an irreducible ${\mathcal{S}}(G)$ -module ${W}(\lambda)^{\infty}$ which is the Casselman-Wallach realization of the smooth representation of $G$ defined by the $({\mathfrak{g}},K)$-module $W(\lambda)$ [5].

We refer to the $({\mathfrak{g}},K)$-module $W(\lambda)$ and its completion $W(\lambda)^{\infty}$ as Flensted-Jensen representations. We denote the Flensted-Jensen representations of $G^{1}/G^{1}\cap H$ by $V(\nu)$ and of $G^{2}/G^{2}\cap H$ by $U(\mu)$.

Remark II.2*.*

By lemma 4.5 in [15] the functions in the $({\mathfrak{g}},K)$-module of a Flensted-Jensen representation decay at least as rapidly in the $J_{+}$ direction as the Flensted-Jensen function.

II.3. Restriction of representations

We consider in this article the restriction of a representation $\Pi^{\infty}$ of a reductive Lie group $G$ to a noncompact reductive subgroup ${G^{\prime}}$. Since the restriction is usually not a direct sum of irreducible representations we consider instead for an irreducible representation symmetry breaking operators in (the smooth category)

[TABLE]

or eguivalently

[TABLE]

where $\pi^{\vee}$ is the contragredient representation of an irreducible representation $\pi$ of $G^{\prime}$.(See [19].)

Suppose that $G^{\prime}$ and $H$ are the fixed points of 2 different involutions of a reductive group $G$, both of which commute with the Cartan involution $\theta$. Let $\Pi$ be a representation on the discrete spectrum of $G/H$. Assume that

[TABLE]

If $G^{\prime}/H\cap G^{\prime}\subset G/H$ is not a principal orbit, then the restriction of $\Pi$ may not be a direct sum of irreducible representations, and $\pi$ may not be a Flensted-Jensen representation in the discrete spectrum of $L^{2}(G^{\prime}/G^{\prime}\cap H)$. Consider the following example: Note that for $G=SO_{o}(p,q)$, $H=SO_{o}(p,q-1)$ and a Flensted-Jensen representation $\Pi$ for G/H we have

[TABLE]

For the subgroup $G^{2}$ which is conjugate to $H$ in G we also have

[TABLE]

but the trivial representation is not a Flensted-Jensen representation in $L^{2}(G^{2}/G^{2}\cap H)$.

II.4. Flensted-Jensen representations of orthogonal groups and their parameters

We assume in this subsection that $G=SO_{o}(p,q)$ and $H=SO_{0}(p,q-1)$ is the stabilizer of $e_{p+q}$. We recall the Langlands parameters, the $\theta$-stable parameter and the infinitesimal character of the Flensted-Jensen representations in $L^{2}(G/H)$ following the exposition in [29].

Let $\sigma$ be the involution of ${\mathfrak{g}}$ with fixed points ${\mathfrak{h}}$ and let ${\mathfrak{q}}$ be its -1 eigenspace. We have $H\cap K=SO(p)\times SO(q-1)$ and so $K/H\cap K$ is the sphere $SO(q)/SO(q-1)$. Then ${\mathbb{R}}(E_{p+q,,p+1}-E_{p+1,p+q})$ is a maximal abelian subspace ${\mathfrak{t}}$ of semisimple elements in ${\mathfrak{q}}\cap{\mathfrak{k}}$.

For the rest of the subsection we assume that $\lambda\in{\mathfrak{t}}^{*}$ satisfies the assumptions in 4.1 of [29]. Let

[TABLE]

where $\rho$ and $\rho_{c}$ are the half sum of positive roots of ${\mathfrak{t}}$ on ${\mathfrak{g}}$ respectively ${\mathfrak{k}}$. Thus

[TABLE]

is the highest weight vector of a finite dimensional representation $\delta_{\lambda}$ of K with a $(K\cap H)$-fixed vector, i.e. of a finite dimensional representation of $SO(q)$ with an $SO(q-1)$ fixed vector and highest weight $(\mu_{\lambda},0,\dots,0)$

Let $W(\lambda)$ be the Flensted-Jensen representation with parameter $\lambda$. The lowest $K$-type of $W(\lambda)$ has a minimal $K$-type with highest weight $\mu_{\lambda}$ [29] theorem 5.4. The group

[TABLE]

is the Levi subgroup of a $\theta$-stable parabolic subgroup and $\mu_{\lambda}$ is a differential of a one dimensional representation of $L$. $W(\lambda)$ is obtained by cohomological induction from a $\theta$-stable parabolic with Levi $L$ and the character $\lambda$. For more details see IV.1. Following the terminology of [19] we refer to $(L,\lambda)$ as the $\theta$–stable parameter of the Flensted-Jensen representation $W(\lambda)$ and its completion $W(\lambda)^{\infty}$. Thus we conclude

Proposition II.3**.**

Let ${\mathfrak{c}}$ be a Cartan subalgebra of ${\mathfrak{g}}\otimes_{{\mathbb{R}}}{\mathbb{C}}$ . We choose the usual set $\Delta$ of roots and positive roots $\Delta^{+}$. Then the infinitesimal character of $W(\lambda)$ is

[TABLE]

Let ${\mathfrak{a}}$ be a maximal abelian subspace in ${\mathfrak{l}}\cap{\mathbf{p}}$ and $A$ a connected subgroup with Lie algebra ${\mathfrak{a}}$. So $A$ has dimension $min(p,q-2)$. $P_{L}=M_{L}AN_{L}$ is the cuspidal parabolic subgroup of $G$ with $A$ as its split component. If $\lambda$ is positive and large enough, then $W(\lambda)^{\infty}$ is the Langlands subquotient of a principal series representation $I(P_{L},\delta,s)$ induced from $P_{L}$ with minimal K-type $\delta_{\lambda}$, see [30] Theorem 6.1, i.e. it has the Langlands parameter $P_{L},\delta,s$. We can now prove

Lemma II.4**.**

The representation $W(\lambda)$ is self-dual, i.e. it is isomorphic to its contragradient $(W(\lambda)^{\vee}$.

Proof. The map $\Pi\rightarrow\Pi^{\vee}$ is an involution on the set of irreducible representations of a semisimple group $G$ which was investigated in [1] and [4]. The infinitesimal character of $\Pi^{\vee}$ is a Weyl group conjugate of the negative of the infinitesimal character of $\Pi$. Thus irreducible representations and their contragradients have the same infinitesimal character if the Weyl group contains $-1$ [4]. This is satisfied if $p+q$ is odd. If $p+q$ is even then

[TABLE]

is conjugate to

[TABLE]

and thus $W(\lambda)$ and $W(\lambda)^{\vee}$ have the same infinitesimal character. Since the minimal $K$-type is self-dual both representations also have the same minimal $K$-type. By Theorem 5.4 in [30] the minimal $K$-type and the infinitesimal character determine uniquely the Langlands parameter of an irreducible representation. ∎.

III. Period integrals.

In this section we assume that $G=SO_{o}(p,q)$. We consider the restriction of a Flensted-Jensen representation ${W}(\lambda)^{\infty}$ to $G^{i}$, $i=1,2$ and find linear nonzero functionals in

[TABLE]

where $\pi^{\infty}=U(\mu)^{\infty}$ or $\pi^{\infty}={V}(\nu)^{\infty}$ is a Flensted-Jensen representation generated by a Flensted-Jensen function $F(\nu,x)\in C^{\infty}(G^{1}/G^{1}\cap H)$, respectively $F(\mu,x)\in C^{\infty}(G^{2}/G^{2}\cap H)$. For this we consider period integrals of test functions in the lowest $K$-type of ${W}(\lambda)^{\infty}\otimes(\pi^{\vee})^{\infty}$ and show in III.2 that the period integrals defined in the introduction and in II.3 converge for some pairs of representations and are nonzero on the underlying $({\mathfrak{g}},K)$-module. In III.3 we show that the period integrals also converge on the $C^{\infty}$–vectors. In III.4 we show that it defines a continuous linear functional on the Fréchet space using the results of Bernstein-Kroetz.

III.1. Branching and period integrals for compact orthogonal groups

Consider a compact group $K$, symmetric subgroups $\ F,\ K^{\prime},\ F^{\prime}$ so that

[TABLE]

and the manifolds $Y=K/F$ and $Y^{\prime}=K^{\prime}/F^{\prime}$.

Let $\phi$ be a $K$-finite function in $C^{\infty}(Y)$ and $\phi^{\prime}$ a $K^{\prime}$-finite function in $C^{\infty}(Y^{\prime})$. We consider the period integral

[TABLE]

and we want to understand for which pairs of representations it is nonzero in our case.

We assume now

1. (1)

$K=SO(q),F=SO(q-1)$ the stabilizer of the last coordinate. 2. (2)

$K^{\prime}=SO(q-1),F^{\prime}=K^{\prime}\cap F=SO(q-1)$

The manifolds are $Y=K/F,$ $Y^{\prime}=K^{\prime}/F^{\prime}$ are a sphere and the equator. Recall that $L^{2}(X)$ is a direct sum of irreducible representations $\pi_{a}$ of $SO(q)$ with highest weight (a,0,…, 0). We obtain an embedding of $\pi_{a}$ into $L^{2}(X)$ by considering the coefficients $f_{a}(k)=(v,\pi_{a}(k)v_{o})$ where $v_{o}$ is the $F$-fixed vector. The classical branching rules for the restriction of $\pi_{a}$ to $SO(q-1)$ show that

[TABLE]

$\pi_{a}$ is also an irreducible $U({\mathfrak{k}})$-module. This implies

Proposition III.1**.**

Let $v_{a}$ be the highest weight vector of $\pi_{a}$and $v_{b}$ a highest weight vector of $\pi_{b}$ with $0\leq b\leq a$. There exists a $D_{a,b}\in U({\mathfrak{k}})$ so that

[TABLE]

Proof: We may assume that $D_{a,b}f_{a}$ is a multiple of $f_{b}$. ∎

III.2. Periods for the $({\mathfrak{g}},K)$–module $W(\lambda)\otimes U(\mu)$

Let $\lambda,\mu$ be the parameter of the Flensted-Jensen representations of $G$ and $G^{2}$ and let $F(\lambda,x)$, $F(\mu,x^{\prime})$ the corresponding Flensted-Jensen functions. We consider the period integrals

[TABLE]

or more generally

[TABLE]

for $D_{k}\in U({\mathfrak{k}})$. Given $\lambda,\mu$ we consider the following two problems:

1. (1)

For which $\mu$ exists $D_{k}\in U({\mathfrak{k}})$ so that the integral ${\mathcal{P}I}_{p,q}(\mu,\lambda)$ is convergent and non-zero. 2. (2)

Does this period integral define a $G^{i}$-invariant linear functional of $W(\lambda)\otimes V(\mu)$?

Given the assymptotics of the measure and of $F(\lambda)$ and $F(\mu)$ we get convergence if

[TABLE]

and so $\lambda+\mu>-\frac{1}{2}.$ To obtain nonvanishing of the period integral on the minimal $K$-type we need by III.1

[TABLE]

Thus we have

[TABLE]

Since the radial part of the integral over $X(p,q-1)$ only involves a power of $cosh(t)$, this proves (1) in the

Proposition III.2**.**

1. (1)

Suppose that $G^{\prime}=G^{2}=SO_{o}(p.q-1)$ and that $W(\lambda)$ and $U(\mu)$ are the $({\mathfrak{g}},K)$-modules of Flensted-Jensen representations of G/H and $G^{2}/H\cap G^{2}$.

The period integral converges if $\lambda+\mu>-\frac{1}{2}$. It doesn’t vanish on the minimal $(K\otimes K^{\prime})$-type of the $({\mathfrak{g}}\otimes{\mathfrak{g}}^{\prime},K\otimes K^{\prime})$ modules $W(\lambda)\times U(\mu)$ if

[TABLE] 2. (2)

Suppose that $G^{\prime}=G^{1}=SO_{o}(p-1,q)$ and that $W(\lambda)$ and $V(\nu)$ are the $({\mathfrak{g}},K)$-modules of Flensted-Jensen representations of G/H and of $G^{1}/(H\cap G^{1})$. The period integral converges if $\lambda+\nu>-\frac{1}{2}$ and it doesn’t vanish on the minimal K-type if

[TABLE]

Proof: In (2) we consider the period integral

[TABLE]

and proceed as in (1).

∎

Corollary III.3**.**

1. (1)

The $({\mathfrak{g}},K)$–module $W(\lambda)\otimes V(\nu)$ has a nontrivial $G^{1}=SO_{o}(p-1,q)$–invariant linear functional if

[TABLE] 2. (2)

The $({\mathfrak{g}},K)$–module $W(\lambda)\otimes U(\mu)$ has a nontrivial $G^{2}=SO_{o}(p,q-1)$–invariant linear functional if

[TABLE]

Proof: By II.3 all functions in $W(\lambda)$ respectively in $V(\nu)$ and $\ U(\mu)$ decay at least as fast as the Flensted Jensen functions. Hence the integral converges for all functions in $W(\lambda)\otimes V(\nu)$, respectively $W(\lambda)\otimes U(\mu)$. ∎

III.3. Periods on ${W}(\lambda)^{\infty}\otimes{V}(\mu)^{\infty}$

III.3.1. A linear functional on the vector space ${W}(\lambda)^{\infty}\otimes{V}(\mu)^{\infty}$

We assume again that G=$SO_{o}(p,q)$ and $H=SO_{o}(p,q-1)$ and we first ignore the topology on ${W}(\lambda)^{\infty}\otimes{V}(\mu)^{\infty}$.

We recall from [5] and [33] the definition of the algebra ${\mathcal{S}}(G)$ of Schwartz functions: Let $\sigma$ be a finite dimensional faithful representation with a $K$-invariant inner product. Define for $g\in G$ a scale

[TABLE]

and we denote by $L$ and $R$ the left and right regular representation. We call the space of smooth functions (decaying in all derivatives)

[TABLE]

for all $X,Y\in U({\mathfrak{g}})$, $r\in{\mathbb{N}}$ the Schwartz algebra ${\mathcal{S}}(G)$, and give it the topology via the corresponding seminorms. By definition $s(g)=s(g^{-1})$, $s(g_{1}g_{2})\leq s(g_{1})s(g_{2})$ and similarly $s(g_{1}g_{2}^{-1})^{-r}\leq s(g_{1})^{-r}s(g_{2})^{r}$. We deduce that if $f\in{\mathcal{S}}(G)$ then the function $f(g^{-1})$ is also in ${\mathcal{S}}(G)$. Furthermore for $f\in{\mathcal{S}}(G)$ and $w(g)\in W(\lambda)$ the integral

[TABLE]

converges and is in $C^{\infty}(X(p,q))$. The map

[TABLE]

defines a homomorphism of the Fréchet convolution algebra of Schwartz functions ${\mathcal{S}}(G)$ onto ${W}(\lambda)^{\infty}$ [5]. Here we use deep results from the theory of globalizations of Harish-Chandra modules. So we have the identification, also as topological spaces via the open mapping Theorem,

[TABLE]

Furthermore ${W}(\lambda)^{\infty}$ is an irreducible ${\mathcal{S}}(G)$-module. Hence to estimate the growth of a function in ${W}(\lambda)^{\infty}$ we have to estimate the growth of functions of the form $\int_{G}f(g)F(\lambda,g^{-1}x)dg$ or equivalently $\int_{G}f(g)F(\lambda,gx)dg$ with $f\in{\mathcal{S}}(G)$. For this we make the following explicit estimates which they seem to depend on having classical groups and it is not clear whether similar estimates can be made in say exceptional groups. For $g\in G$ we write in block form

[TABLE]

where

[TABLE]

and 2 more relations. In the coordinates

[TABLE]

where $|y^{\prime}|=|y^{\prime\prime}|=1$ (Euclidian norms) on $X(p,q)$ (see II.1) we have

[TABLE]

Now $|A_{3}x^{\prime}+A_{4}x^{\prime\prime}|$ never vanishes and there exists a constant $C$ so that

[TABLE]

(see II.2) so that we need to understand the asymptotics of

[TABLE]

Thus we estimate from below

[TABLE]

with $|y^{\prime}|=1=|y^{\prime\prime}|.$ We note that the sets $\{A_{4}y^{\prime\prime}|\ |y^{\prime\prime}|^{2}=1\}$ and $\{A_{3}y^{\prime}|\ |y^{\prime}|=1\}$ are ellipsoids that never intersect. We can find their axes by looking at the quadratic forms

[TABLE]

with eigenvalues

[TABLE]

respectively

[TABLE]

Note that $A_{4}^{t}A_{4}$ and $A_{4}A_{4}^{t}$ have the same non-zero eigenvalues and similarly $A^{t}_{3}A_{3}$ and $A_{3}A_{3}^{t}$. Hence $\mu_{1}=\lambda_{1}-1,\mu_{2}=\lambda_{2}-1$ etc. Note that for j sufficiently large j some eigenvalues $\mu_{j}$ may be zero.

Lemma III.4**.**

The 2 ellipsoids have parallel axes.

Proof.

Note that for $\lambda_{1}$ and its eigenvector $A_{4}^{t}A_{4}e_{1}=\lambda_{1}e_{1}$ implies that $|A_{4}e_{1}|^{2}=\lambda_{1}$ and thus $A_{4}e_{1}$ is the largest axis of the large ellipsoid. We also have

[TABLE]

so that

[TABLE]

comparing to the similar

[TABLE]

we deduce that that the axis $A_{3}v_{1}$, respectively $A_{4}e_{1}$, of the smaller and the larger ellipsoid are proportional if the multiplicity of the eigenvalue is one. If the multiplicity is higher than one, then can identify the eigenspaces and the axes of the ellipsoids this way. For the next eigenvalue $\lambda_{2}=\mu_{2}+1$ the axis $A_{4}e_{2}$ is perpendicular to $A_{4}e_{1}$ and similarly $A_{3}v_{2}$ is perpendicular to $A_{3}v_{1}$, and again $A_{4}e_{2}$ and $A_{3}v_{2}$ are linearly dependent. ∎

So can conclude

Lemma III.5**.**

Under the above assumptions

[TABLE]

Proof.

Note

[TABLE]

So in particular $Q\geq\mbox{min}_{i}\frac{1}{2\sqrt{\lambda_{i}}}$ and thus

[TABLE]

∎

This gives, using the scales defining ${\mathcal{S}}(G)$ the estimate for a typical function

[TABLE]

from above

Proposition III.6**.**

Every $F\in V(\lambda)^{\infty}={\mathcal{S}(G)}F(\lambda)$ satisfies

[TABLE]

The constant C depends on $f\in{\mathcal{S}}(G)$ and $C\geq\int_{G}|f(g)|s(g)^{a}da$.

Proof.

For the defining representation the Hilbert-Schmidt norm of $g$ dominates the Hilbert-Schmidt norm of $A_{4}$. Hence the scale $s(g)$ dominates the Hilbert-Schmidt norm $\mbox{tr }A^{t}_{4}A_{4}$ and asymptotically for $t\rightarrow\infty$

[TABLE]

approaches $|A_{3}y^{\prime}+A_{4}y^{\prime\prime}|$. ∎

In a similar way we can estimate the growth of functions in $V(\nu)^{\infty}$ and $U(\mu)^{\infty}$. Hence we can conclude

Corollary III.7**.**

The period integral

[TABLE]

of functions $f,\ f^{\prime}$ in $W(\lambda)^{\infty}$, $V(\mu)^{\infty}$ respectively $U(\nu)^{\infty}$ converges if the integral $I$ over the corresponding Flensted-Jensen functions converges and it defines a linear functional $I$ on $W(\lambda)^{\infty}\otimes V(\mu)^{\infty}$, and $W(\lambda)^{\infty}\otimes U(\nu)^{\infty}$ respectively.

III.3.2. Continuity of the period $I$

To complete the proof it remains to show that the linear functional

[TABLE]

is continuous.

Note the map

[TABLE]

[TABLE]

is continuous. So if $f_{n}\rightarrow f$ in ${\mathcal{S}}(G)$ then the estimates imply that $I_{f_{n}}\rightarrow I_{f}$ pointwise and dominated by $C\,e^{-at}$ for some $C$.

Hence for fixed $F^{\prime}\in V(\mu)^{\infty}$

[TABLE]

for $n\rightarrow\infty$. Hence we have continuity in the first variable. A similar argument shows that we have continuity in the second variable. Since we are working in Fréchet spaces the period integral is continuous in both variables.

Thus we conclude

Theorem III.8**.**

1. (1)

The representation ${W}(\lambda)^{\infty}\otimes{V}(\nu)^{\infty}$ has a nontrivial $G^{1}=SO_{o}(p-1,q)$–invariant linear functional which is continuous in the Casselman-Wallach topology if

[TABLE] 2. (2)

The representation

${W}(\lambda)^{\infty}\otimes{U}(\mu)^{\infty}$* has a nontrivial $G^{2}=SO_{o}(p,q-1)$–invariant linear functional which is continuous in the Casselman-Wallach topology if*

[TABLE]

Remark III.9*.*

This result shows that the linear functional defined by a period integral on the $({\mathfrak{g}},K)$-module $W(\lambda)\boxtimes V(\nu)$ extends ”automatically” to a continuous linear functional on the completion in the spirit of the automatic continuity theorems by Casselman [8] and Delorme, van den Ban [7].

IV. Branching of ${W}(\lambda)^{\infty}$ to $G^{1}$ and $G^{2}$.

In this section we restate the results of the previous section using the terminology of symmetry breaking in terms of Langlands parameters as well as the interlacing property of the infinitesimal characters.

IV.1. Review of cohomological induced representations

We summarize cohomological parabolic induction. A basic reference is the book by A.W. Knapp and D. Vogan [11]. We begin with a connected real reductive Lie group $G$. Let $K$ be a maximal compact subgroup, and $\theta$ the corresponding Cartan involution. Let $H_{K}$ be a Cartan subgroup of K. Given an element $X\in\mathfrak{h}_{K}$, the complexified Lie algebra ${\mathfrak{g}}_{\mathbb{C}}={\operatorname{Lie}}(G)\otimes_{\mathbb{R}}{\mathbb{C}}$ is decomposed into the eigenspaces of $\sqrt{-1}\operatorname{ad}(X)$, and we write

[TABLE]

for the sum of the eigenspaces with negative, zero, and positive eigenvalues. Then ${\mathfrak{q}}:={\mathfrak{l}}_{\mathbb{C}}+{\mathfrak{u}}$ is a $\theta$-stable parabolic subalgebra with Levi subgroup

[TABLE]

The homogeneous space $G/L$ is endowed with a $G$-invariant complex manifold structure with holomorphic cotangent bundle $G\times_{L}{\mathfrak{u}}$. As an algebraic analogue of Dolbeault cohomology groups for $G$-equivariant holomorphic vector bundle over $G/L$, Zuckerman introduced a cohomological parabolic induction functor ${\mathcal{R}}_{\mathfrak{q}}^{j}(\cdot\otimes{\mathbb{C}}{\rho({\mathfrak{u}})})$ ($j\in{\mathbb{N}}$) from the category of $({\mathfrak{l}},L\cap K)$-modules to the category of $({\mathfrak{g}},K)$-modules.

Now assume that $G=SO_{o}(p,q)$ and

[TABLE]

is the Levi subgroup of a $\theta$-stable parabolic subgroup ${\mathfrak{q}}={\mathfrak{l}}+{\mathfrak{n}}$ and $\lambda$ is a differential of a one dimensional representation $\lambda$ of $L$ . We assume that the positive roots have positive inner product with the roots in $\lambda$. Then $W(\lambda)$ is cohomologically induced from ($L,\lambda$). Since we assume that $p$ and $q$ are larger than $4$, $L$ is not compact and thus the representations $W(\lambda)$, $V(\nu)$ and $U(\mu)$ are not tempered.

IV.2. Branching Theorem

Lemma IV.1**.**

1. (1)

The restriction of ${W}(\lambda)^{\infty}$ to $G^{1}$ is admissible. 2. (2)

The restriction of ${W}(\lambda)^{\infty}$ to $G^{2}$ is not admissible.

Proof.

It follows from the tables in [17]. ∎

Recall that in the Casselman-Wallach realization of irreducible representations $\Pi$, $\pi$ of $G$, respectively of $G^{\prime}\subset G$

[TABLE]

and

[TABLE]

are isomorphic, where $\pi^{\vee}$ is the contragredient representation of an irreducible representation $\pi$ of $G^{\prime}$. So using Lemma II.4 we can summarize the results of the last section in the Branching Theorem**

Theorem IV.2**.**

Under the previous assumptions

1. (1)

If $G^{\prime}=G^{2}=SO_{o}(p.q-1)$ and

[TABLE]

then

[TABLE] 2. (2)

If $G^{1}=SO_{o}(p-1,q)$ and

[TABLE]

then

[TABLE]

Remark IV.3*.*

Using different techniques similar results were obtained in [23].

IV.3. Different formulations of the Branching Theorem

IV.3.1. A graphic formulation of the Branching Theorem.

The restriction of the representation $W(\lambda)$ to $G^{2}$ can be represented by the following diagram. The representations of G are in the first row, the representations of $G^{i}$ in the second row. Representations are connected by an arrow if their multiplicity is one.

Restriction to $G^{2}$:* Let that $V(\mu)$ be a representation of $G^{2}$ with $\mu+\frac{1}{2}\leq\lambda$. Then*

[TABLE]

Restriction to $G^{1}$**. Suppose that $\nu=\lambda+1/2$

[TABLE]

Proof.

In the first case the arrow in the restriction to the trivial representation is the $H=SO_{o}(p,q-1)$ linear functional defining by the embedding of ${W}(\lambda)^{\infty}$ into the Schwartz space on $G/H$. ∎

Remark: The restriction of ${W}(\lambda)^{\infty}$ follows the same pattern as the restriction of representations of height $1$ for $O(n,1)$ to $O(n-1,1)$ [21].

IV.3.2. The Branching Theorem using interlacing patterns

Interlacing patterns of highest weights of finite dimensional representations and of $\theta$-stable parameters of irreducible representations of O(n,1) have been used successfully to express the restriction of irreducible representations as well as in the Gross Prasad conjectures. [20], [21].

We say that two finite sequences

[TABLE]

[TABLE]

$n=m-1,$ or $n=m$ have an interlacing property of finite type if

[TABLE]

We say that the sequences have a interlacing property of infinite type 1 if

[TABLE]

Note that the infinitesimal character of the representation ${W}(\lambda)^{\infty}$ is

[TABLE]

Here we choose a root system so that and positive with respect to the positive root with respect to $(\lambda,0,0,\dots,0)$ and the roots of $so(q)\oplus so(p)$.

Theorem IV.4**.**

Under the above assumptions

1. (1)

Let $\Pi={W}(\lambda)$ and $\pi={V}(\nu)$ be Flensted Jensen representations of G/H respecitively $G^{1}/(H\cap G^{1})$ with the infinitesimal characters

[TABLE]

[TABLE]

Then

[TABLE]

implies that $\nu>\lambda$ and thus the infinitesimal characters have an interlacing property of infinite type 1. 2. (2)

Suppose that $\Pi={W}(\lambda)$ and $\pi={U}(\mu)$ are Flensted Jensen representations of $G/H$ respectively $G^{2}/(G^{2}\cap H)$ with the infinitesimal characters

[TABLE]

[TABLE]

If the infinitesimal characters have an interlacing property of finite type, then

[TABLE]

V. Another perspective and some speculations.

In this section we rephrase our previous results and reexamine them in the context of Arthur packets and Arthur Vogan packets.

To simplify the notation and the considerations we assume in this section that the groups $G$, $G^{1}$ and $G^{2}$ have discrete series representations, i.e. that p and q are even and that $p\leq q$. Furthermore we assume as in the earlier sections that $4\leq p,$

V.1. Generalities

An Arthur Vogan parameter is a a homomorrphism of the Weil group in the Langlands dual group of G which commutes with a homorphism of $SL(2,{\mathbb{C}})$. This defines a unipotent orbit and a one dimensional representation of the Levi subgroup of a parabolic subgroup of the complexification $G_{\mathbb{C}}$ of G. We obtain a family of one dimensional representations of $\theta$-stable parabolic subgroups of a family of pure inner forms of $G_{\mathbb{C}}$. By cohomological induction we obtain a family of irreducible representations of pure inner forms compatible G. For details, definitions and results see the article by J. Adams and J. Johnson [3] or the book by Adams-Barbasch-Vogan [2].

The family of pure inner forms of $SO_{o}(p,q)$ consists of the groups $SO_{o}(p-2r,q+2r),$ $q<2r<p$ and $SO_{o}(p+2s,q-2s)$. Similarly we obtain a family of pure inner forms compatible with $G^{1}$ and $G^{2}$ and observe that these inner classes are disjoint.

Examples:

If both $p$ and $q$ are even then the pure inner class of pure inner forms of $SO_{o}(p,q)$ contains both the compact groups $SO(p+q,0)$ and $SO(0,p+q).$

The class of pure inner forms of $SO_{0}(2n,2m+1)$ contains the compact group $SO(0,2n+2m+1)$ and and $SO_{o}(2n+2m,1)$ the orthogonal group of real rank one.

Using lemma 2.10 of J. Adams and J. Johnson [3] we can show, that if $G=SO_{o}(p,q)$ with both $p,q$ even and $4\leq p\leq q$ then the number of inequivalent irreducible representations in an Arthur packet containing ${W}(\lambda)^{\infty}$ is 2 as follows: We denote by $T$ a compact Cartan subgroup of $SO(p,q)$ and ${\mathfrak{t}}_{\mathbb{C}}$ its complexifed Lie algebra. The Lie algebra of $G_{\mathbb{C}}$ is noted by by ${\mathfrak{g}}_{\mathbb{C}}$ and ${\mathfrak{l}}_{\mathbb{C}}={\mathfrak{l}}\otimes{\mathbb{C}}={\mathbb{C}}\oplus so(p-2,q)$. The corresponding Weyl groups are $W(G,T)$, $W({\mathfrak{g}}_{\mathbb{C}},{\mathfrak{t}}_{\mathbb{C}})$ and $W({\mathfrak{l}}_{\mathbb{C}},{\mathfrak{t}}_{\mathbb{C}})$. The representations in an Arthur packet are parametrized by the double cosets

[TABLE]

Lemma V.1**.**

$\mathbb{S}$* has 2 disjoint double cosets.*

Proof.

Note that the coset $W({\mathfrak{g}}_{\mathbb{C}})/W({\mathfrak{l}}_{\mathbb{C}},{\mathfrak{t}}_{\mathbb{C}})$ has $m=2[\frac{p+q}{2}]$ elements and has representants $P_{i}$ acting on a basis $e_{i}$ of ${\mathbb{C}}^{m}$ by permuting $e_{1}$ and $e_{i}$ and by sign changes composed with permutations. W(G,T) is the Weyl group of $SO(p)\times SO(q)$. It has 2 orbits on $W({\mathfrak{g}}_{\mathbb{C}})/W({\mathfrak{l}}_{\mathbb{C}},{\mathfrak{t}}_{\mathbb{C}})$ with representants Identity and $P_{1}$. ∎

Remark V.2*.*

A similar proof shows that $\mathbb{S}$ contains exactly 2 disjoint double cosets if $p$ or $q$ is even and $4\leq p\leq q$.

**Examples:

**For $q=1,p=2m,\ m>I$ the results in [19] show that in this case the Arthur packet containing $W(\lambda)^{\infty}$ contains only one element, a representation of height one.

If $p=0$, then the Arthur packet contains exactly one finite dimensional spherical subrepresentation of $L^{2}(SO(0,q)/SO(0,q-1))$.

Consider now a maximal parabolic subgroup $P_{\mathbb{C}}$ of $SO(p+q,{\mathbb{C}})$ with a Levi $L_{\mathbb{C}}=SO(p+q-2)~{}{\mathbb{C}}^{*}$. If $4\leq p\leq q$ it defines two $\theta$-stable parabolic subgroups $P_{1}$ and $P_{2}$ of $SO_{o}(p,q)$ with Levi $L_{1}=SO(p,q-2)SO(0,2)$ respectively $L_{2}=SO(2,0)SO(p-2,q)$, which are both in the same family of pure inner forms. If two irreducible representations are cohomological induced from characters of $P_{1}$ and $P_{2}$ and have the same infinitesimal character then they are in the same Vogan packet [3].

We denote the representation in the Arthur packet corresponding to the double coset of the identity is denoted by ${W}(\lambda)^{\infty}$ and the representation corresponding to the other coset by ${W}_{as}(\lambda)^{\infty}$. ${W}(\lambda)^{\infty}$ is in the discrete spectrum of $L^{2}(SO_{o}(p+q)/SO_{o}(p,q-1)$ whereas the representation ${W}_{s}(\lambda)^{\infty}$ is also a Flensted Jensen representation in the discrete spectrum of $L^{2}(SO_{o}(p,q)/SO_{o}(p-1,q))$.

An Arthur Vogan packet containg the Arthur packet $\{{W}_{as}(\lambda)^{\infty}$. $\hat{W}(\lambda)^{\infty}\}$ is a disjoint union of the Arthur packets of the Flensted-Jensen representations of the pure inner forms of $SO(p,q)$.

Example The case $G=SO_{0}(3,3)$ was partially discussed in [26]. We consider there a representation cohomological induced from the trivial character of $L=SO(2,0)SO_{o}(1,3)$, i.e. a Flensted-Jensen representation of $SO_{0}(3,3)/SO_{o}(2,3)$, i.e. in our notation $W(0)^{\infty}_{as}$. The other representation is the Arthur packet is cohomological induced from the trivial representation of $L=SO(0,2)SO_{o}(3,1)$.

The pure inner forms of $SO_{0}(3,3)$ are $SO_{o}(5,1)$ and $SO_{o}(1,5)$. We are considering the representations in the discrete spectrum of $SO_{0}(1,5)/SO_{o}(1,4)$ respectively $SO_{o}(5,1)/SO(4,1)$, since a symmetric space has no discrete spectrum. These representations are cohomological induced from $L=SO_{o}(1,3)SO(0,2)$ and $L=SO(2,0)SO_{0}(3,1)$.

Thus the Arthur packet consists of 2 representations and the Arthur Vogan packet contains 4 representations.

V.2. Restriction of representations in Arthur packets

We fix a regular infinitesimal character

[TABLE]

of $SO(p+q,{\mathbb{C}})$ satisfying conditions 8.8 and 8.9 in [30].

$L$-functions and trace formula considerations strongly suggest that this symmetry breaking depends on the inner class of the groups $G^{i}$ and the Arthur packet of representations. Our computations and the results of [17] support this. Recall that the restriction of a representations to a subgroup H is called admissible if it is direct sum of representations of H.

Proposition V.3**.**

1. (1)

The restriction to $G^{1}$ of exactly one of the representations $\{{W}(\lambda)^{\infty},\ {W}_{as}(\lambda)^{\infty}\}$ in the Arthur packet is admissible, the other one is not admissible. 2. (2)

The restriction to $G^{2}$ of exactly one of the representations $\{{W}(\lambda)^{\infty},\ {W}_{as}(\lambda)^{\infty}\}$ in the Arthur packet is admissible, the other one is not admissible. 3. (3)

Each representation in the Arthur packet is admissible for exactly one of the groups $G^{i}$, $i\in\{1,\ 2\}$

Proof.

For ${W}(\lambda)^{\infty}$ this follows from the branching theorem and for ${W}_{as}(\lambda)^{\infty}$ it follows from the table in [17]. More precise information about the restriction of ${W}_{as}(\lambda)^{\infty}$ in the unitary category is contained in [14] supporting the conjecture. ∎

Remark V.4*.*

The complete branching law for the restriction of $W_{as}(\lambda)$ to $G^{2}$ can be found in [14] theorem 3.3 and information about the discrete spectrum of the restriction of the unitary representation with Harish-Chandra-module $W_{as}(\lambda)$ to $G^{1}$ are in [16].

V.3.

Based on our computations we expect a stronger result similar to the famous Gross-Prasad conjecture. Our assumptions imply that for the groups $G^{i}$, i=1,2 we have an Arthur packets with 2 representations $\{{V}(\nu)^{\infty},{V}_{as}(\nu)^{\infty}\}$ , respectively $\{U(\mu)^{\infty},U_{as}(\mu)^{\infty}\}$.

Conjecture for Arthur packets

Suppose that $\pi_{1}$ is a representations of $G^{1}$ in an Arthur packet

[TABLE]

1. (1)

If

[TABLE]

then

[TABLE] 2. (2)

if

[TABLE]

then

[TABLE]

There is an analogous statement for the group $G^{2}$.

Definition If ${G}$ is a pure inner form of $SO_{o}(p,q)$ and ${G}^{i}$ is a pure inner form of $G^{i}$ we call (${G}^{i}$ ${G}$) a relevant pair if ${G}^{i}$ is a subgroup of ${G}$.

Example: The pure inner forms of $SO_{o}(3,3)$ are

[TABLE]

The pure inner forms of $G^{1}$ are

[TABLE]

and relevant pairs

[TABLE]

Conjecture for Arthur-Vogan packets

Let ${\mathcal{A}V}_{G}$ be the Arthur Vogan packet which contains ${W}(\lambda)^{\infty}$ and ${\mathcal{A}V}_{G^{1}}$ the Arthur packet which contains ${V}(\nu)^{\infty}$. Suppose that $\underline{G}$ is a pure inner form of $G=SO_{o}(p,q)$ and $\underline{G}^{1}$ is a pure inner form of $G^{1}=SO_{o}(p-1,q)$ and that $\underline{G}$ and $\underline{G}^{1}$ is a relevant pair.

Assume that $W$ and $W_{a}$ are inequivalent representations of $\underline{G}$ in the Arthur packet $(\mathcal{A}V)_{{G}}$ and $V$ a representation of $\underline{G}^{1}$ in the Arthur-Vogan packet $(\mathcal{A}V)_{G^{1}}$.

1. (1)

If

[TABLE]

then

[TABLE] 2. (2)

If

[TABLE]

then

[TABLE]

We expect that an analogous statement holds for ${\mathcal{A}V}_{G^{2}}$.

Remark V.5*.*

The conjecture for Arthur-Vogan packets is similar but weaker than the Gross Prasad conjectures for Vogan packets of tempered representations [10], [A1].

Remark V.6*.*

If $G=SO_{o}(n+1,1)$ then the Arthur packet has only one representation, $G^{1}=SO(n+1)$ is compact and $G^{2}=SO_{o}(n,1)$ and the results follow from [19].