Real double flag varieties for the symplectic group

TL;DR

This paper investigates the structure of double flag varieties associated with the real symplectic group, classifies orbits, and constructs explicit integral transforms between principal series representations.

Contribution

It provides a detailed classification of orbits and develops explicit integral transforms for the case of maximal parabolic subgroups related to Grassmannians.

Findings

Classification of L-orbits on the double flag variety

Explicit integral transforms between principal series of G and L

Analysis of the geometric structure of the double flag variety

Abstract

In this paper we study a key example of a Hermitian symmetric space and a natural associated double flag variety, namely for the real symplectic group $G$ and the symmetric subgroup $L$, the Levi part of the Siegel parabolic $P_S$. We give a detailed treatment of the case of the maximal parabolic subgroups $Q$ of $L$ corresponding to Grassmannians and the product variety of $G/P_S$ and $L/Q$; in particular we classify the $L$-orbits here, and find natural explicit integral transforms between degenerate principal series of $L$ and $G$.

Equations

$$L=\Bigl{\{}\begin{pmatrix}a&0\\ 0&\,{}^{t}{a}\,^{-1}\end{pmatrix}\Bigm{|}a\in\mathrm{GL}_{n}(\mathbb{R})\Bigr{\}}\simeq\mathrm{GL}_{n}(\mathbb{R}),$$

$$N=\Bigl{\{}\begin{pmatrix}1&z\\ 0&1\end{pmatrix}\Bigm{|}z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\Bigr{\}}\simeq\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$$

$$Q=P_{(d,n-d)}^{\mathrm{GL}}=\Bigl{\{}\begin{pmatrix}\alpha&\xi\\ 0&\beta\end{pmatrix}\Bigm{|}\alpha\in\mathrm{GL}_{d}(\mathbb{R}),\beta\in\mathrm{GL}_{n-d}(\mathbb{R}),\xi\in\mathrm{M}_{d,n-d}(\mathbb{R})\Bigr{\}}.$$

$$G=\mathrm{Sp}_{2n}(\mathbb{R})=\{g\in\mathrm{GL}_{2n}(\mathbb{R})\mid\,{}^{t}{g}\,J_{n}g=J_{n}\}\quad\text{ where }J_{n}=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{0}&{-1_{n}}\\ {1_{n}}&{0}\end{array}\Bigr{)}.$$

$${}^{t}ca-{}^{t}ac$$

$${}^{t}da-{}^{t}bc$$

$$g^{-1}pg=\left(\begin{array}{cc}{}^{t}dxa+{}^{t}dzc-{}^{t}byc& {}^{t}dxb+{}^{t}dzd-{}^{t}byd\\ -{}^{t}cxa-{}^{t}czc+{}^{t}ayc& -{}^{t}cxb-{}^{t}czd+{}^{t}ayd\end{array}\right).$$

$$b=-c,d=a,$$

$${}^{t}ab\in {\mathrm{Sym}}_n(\mathbb{R}),\text{ and }{}^{t}aa+{}^{t}bb=1_n$$

$$K=\left\{\begin{pmatrix}a&b\\ -b&a\end{pmatrix}\Bigm{|}a+ib\in\mathrm{U}(n)\right\}.$$

$$(a+ib{)}^{\ast }(a+ib)$$

$$=({}^{t}aa+{}^{t}bb)+i({}^{t}ab-{}^{t}ba)=1_n.$$

$$G/P_S=\bigcup _{w\in W_G}{P}_{S}w{P}_S/P_S=\underset{w\in W_{P_S}\backslash W_G/W_{P_S}}{⨆}{P}_{S}w{P}_S/P_S,$$

$$\{w_k=(1,\dots ,1,-1,\dots ,-1)=(1^k,(-1{)}^{n-k})\mid 0\le k\le n\}\subset (\mathbb{Z}/2\mathbb{Z}{)}^n.$$

$$w_{k}=\left(\begin{array}[]{cc|cc}&&{-1_{n-k}}&\\ &{1_{k}}&&\\ \hline\cr{1_{n-k}}&&&\\ &&&{1_{k}}\end{array}\right).$$

$$N_{w}=\left\{\begin{pmatrix}1_{n}&0\\ \eta&1_{n}\end{pmatrix}\Bigm{|}\eta=\begin{pmatrix}\zeta&\xi\\ \,{}^{t}{\xi}\,&0_{k}\end{pmatrix},\;\zeta\in\mathop{\mathrm{Sym}}\nolimits_{n-k}(\mathbb{R}),\,\xi\in\mathrm{M}_{n-k,k}(\mathbb{R})\right\}$$

$$\displaystyle w^{-1}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{x}&{z}\\ {0}&{y}\end{array}\Bigr{)}w$$

$$=\left(\begin{array}{cc}{x}_{22}+z_{21}+y_{11}& -x_{21}+z_{22}+y_{12}\\ -x_{12}-z_{11}+y_{21}& x_{11}-z_{12}+y_{22}\end{array}\right)$$

$$\displaystyle=\begin{pmatrix}\begin{array}[]{cc|cc}y_{11}&0&0&y_{12}\\ z_{21}&x_{22}&-x_{21}&z_{22}\\ \hline\cr-z_{11}&-x_{12}&x_{11}&-z_{12}\\ y_{21}&0&0&y_{22}\end{array}\end{pmatrix}$$

$$\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{0}\\ {\eta}&{1}\end{array}\Bigr{)}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\alpha}&{\beta}\\ {0}&{\delta}\end{array}\Bigr{)}=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\alpha}&{\beta}\\ {\eta\alpha}&{\eta\beta+\delta}\end{array}\Bigr{)},$$

$$\eta$$

$$=\left(\begin{array}{cc}-z_{11}& -x_{12}\\ y_{21}& 0\end{array}\right)\left(\begin{array}{cc}{y}_{11}^{-1}& 0\\ -x_{22}^{-1}{z}_{21}{y}_{11}^{-1}& x_{22}^{-1}\end{array}\right)$$

$$=\left(\begin{array}{cc}-z_{11}{y}_{11}^{-1}+x_{12}{x}_{22}^{-1}{z}_{21}{y}_{11}^{-1}& -x_{12}{x}_{22}^{-1}\\ y_{21}{y}_{11}^{-1}& 0\end{array}\right),$$

$$w^{-1}aw=\begin{pmatrix}\begin{array}[]{cc|cc}h^{\prime}_{1}&0&0&h^{\prime}_{2}\\ 0&h_{4}&-h_{3}&0\\ \hline\cr 0&-h_{2}&h_{1}&0\\ h^{\prime}_{3}&0&0&h^{\prime}_{4}\end{array}\end{pmatrix}\\ \text{ where }a=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{h}&{0}\\ {0}&{\,{}^{t}{h}\,^{-1}}\end{array}\Bigr{)},\;h=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{h_{1}}&{h_{2}}\\ {h_{3}}&{h_{4}}\end{array}\Bigr{)},\;\,{}^{t}{h}\,^{-1}=h^{\prime}=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{h^{\prime}_{1}}&{h^{\prime}_{2}}\\ {h^{\prime}_{3}}&{h^{\prime}_{4}}\end{array}\Bigr{)},$$

$$\begin{split}\Bigl{\{}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{z}\\ {0}&{1}\end{array}\Bigr{)}w_{k}P_{S}/P_{S}\Bigm{|}z=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{I_{r,s}}&{0}\\ {0}&{0}\end{array}\Bigr{)}\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R}),\,0\leq r+s\leq n-k\Bigr{\}},\\ \quad\text{ where $I_{r,s}=\mathop{diag}\nolimits(1_{r},-1_{s})$.}\end{split}$$

$$\{\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{z}\\ {0}&{1}\end{array}\Bigr{)}wP_{S}/P_{S}\mid z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\}.$$

$$\biggl{\{}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1_{n}}&{0}\\ {\eta}&{1_{n}}\end{array}\Bigr{)}\biggm{|}\eta=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\zeta}&{0}\\ {0}&{0}\end{array}\Bigr{)},\;\zeta\in\mathop{\mathrm{Sym}}\nolimits_{n-k}(\mathbb{R})\biggr{\}}\subset w^{-1}P_{S}w/(w^{-1}P_{S}w)\cap P_{S}.$$

$$(w^{-1}aw)\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{0}\\ {\eta}&{1}\end{array}\Bigr{)}P_{S}/P_{S}=\begin{pmatrix}\begin{array}[]{cc|cc}h_{1}^{\prime}&0&&\\ 0&1&&\\ \hline\cr h_{1}\zeta&0&h_{1}&0\\ 0&0&0&1\end{array}\end{pmatrix}P_{S}/P_{S}=\begin{pmatrix}\begin{array}[]{cc|cc}1&0&&\\ 0&1&&\\ \hline\cr h_{1}\zeta\,{}^{t}{h_{1}}&0&1&0\\ 0&0&0&1\end{array}\end{pmatrix}P_{S}/P_{S}.$$

$$h_{1}\zeta\,{}^{t}{h_{1}}\,=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{I_{r,s}}&{0}\\ {0}&{0}\end{array}\Bigr{)}$$

$$\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1_{n}}&{0_{n}}\\ {\eta}&{1_{n}}\end{array}\Bigr{)}\in N_{w},\quad\eta=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\zeta}&{\xi}\\ {\,{}^{t}{\xi}}&{0_{k}}\end{array}\Bigr{)}\quad(\zeta\in\mathop{\mathrm{Sym}}\nolimits_{n-k}(\mathbb{R}),\;\xi\in\mathrm{M}_{n-k,k}(\mathbb{R})),$$

$$\eta=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\zeta}&{\xi}\\ {\,{}^{t}{\xi}}&{0}\end{array}\Bigr{)}\mapsto a\mathbin{\overset{w}{\cdot}}\eta=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{A}&{B}\\ {\,{}^{t}{B}}&{0}\end{array}\Bigr{)}\quad\text{ where }\quad\Biggl{\{}\begin{array}[]{l}A=(h_{1}+Bh_{3})\zeta\,{}^{t}{(h_{1}+Bh_{3})}\,,\\[5.16663pt] B=(-h_{2}+h_{1}\xi)(h_{4}-h_{3}\xi)^{-1}.\end{array}$$

$$w^{-1}aw$$

$$\displaystyle=\begin{pmatrix}\begin{array}[]{cc|cc}h^{\prime}_{1}+h^{\prime}_{2}\,{}^{t}{\xi}&0&0&h^{\prime}_{2}\\ -h_{3}\zeta&h_{4}-h_{3}\xi&-h_{3}&0\\ \hline\cr h_{1}\zeta&-h_{2}+h_{1}\xi&h_{1}&0\\ h^{\prime}_{3}+h^{\prime}_{4}\,{}^{t}{\xi}&0&0&h^{\prime}_{4}\end{array}\end{pmatrix}$$

$$\displaystyle=:\Bigl{(}\begin{array}[]{@{\,}c|c@{\,}}{1}&{0}\\ \hline\cr{\eta}&{1}\end{array}\Bigr{)}\Bigl{(}\begin{array}[]{@{\,}c|c@{\,}}{\alpha}&{\beta}\\ \hline\cr{0}&{\delta}\end{array}\Bigr{)}=\Bigl{(}\begin{array}[]{@{\,}c|c@{\,}}{\alpha}&{\beta}\\ \hline\cr{\eta\alpha}&{\eta\beta+\delta}\end{array}\Bigr{)}.$$

$$\eta$$

Full text

K. N.  is supported by JSPS KAKENHI Grant Numbers #16K05070.

Introduction

The geometry of flag varieties over the complex numbers, and in particular double flag varieties, have been much studied in recent years (see, e.g., [FN16], [HT12], [Tra09], [FGT09] etc.). In this paper we focus on a particular case of a real double flag variety with the purpose of understanding in detail (1) the orbit structure under the natural action of the smaller reductive group (2) the construction of natural integral transforms between degenerate principal series representations, equivariant for the same group. Even though aspects of (1) are known from general theory (e.g., [KM14], [KO13] and references therein), the cases we treat here provide new and explicit information; and for (2) we also find new phenomena, using the theory of prehomogeneous vector spaces and relative invariants. In particular the Hermitian case we study has properties complementary to other well-known cases of (2). For this, we refer the readers to [KS15], [MØO16], [KØP11], [CKØP11], [Zha09], [BSKZ14] among others.

Thus in this paper we study a key example of a Hermitian symmetric space and a natural associated double flag variety, namely for the real symplectic group $G$ and the symmetric subgroup $L$, the Levi part of the Siegel parabolic $P_{S}$. We give a detailed treatment of the case of the maximal parabolic subgroup $Q$ of $L$ corresponding to Grassmannians and the product variety of $G/P_{S}$ and $L/Q$; in particular we classify the open $L$-orbits here, and find natural explicit integral transforms between degenerate principal series of $L$ and $G$. We realize these representations in their natural Hilbert spaces and determine when the integral transforms are bounded operators. As an application we also obtain information about the occurrence of finite-dimensional representations of $L$ in both of these generalized principal series representations of $G$ resp. $L$. It follows from general principles, that our integral transforms, depending on two complex parameters in certain half-spaces, may be meromorphically continued to the whole parameter space; and that the residues will provide kernel operators (of Schwartz kernel type, possibly even differential operators), also intertwining (i.e., $L$-equivariant). For general background on integral operators depending meromorphically on parameters, and for equivariant integral operators – introduced by T. Kobayashi as symmetry-breaking operators – as we study here, see [KS15], [MØO16] and [KK79]. However, we shall not pursue this aspect here, and it is our future subject.

It will be clear, that the structure of our example is such that other Hermitian groups, in particular of tube type, will be amenable to a similar analysis; thus we contend ourselves here to give all details for the symplectic group only.

Let us fix notations and explain the content of this paper more explicitly. So let $G=\mathrm{Sp}_{2n}(\mathbb{R})$ be a real symplectic group. We denote a symplectic vector space of dimension $2n$ by $V=\mathbb{R}^{2n}$ with a natural symplectic form defined by $\langle u,v\rangle=\,{}^{t}{u}\,J_{n}v$, where $J_{n}=\begin{pmatrix}0&-1_{n}\\ 1_{n}&0\end{pmatrix}$. Thus, our $G$ is identified with $\mathrm{Sp}(V)$. Let $V^{+}=\mathrm{span}_{\mathbb{R}}\{{e_{1},e_{2},\dots,e_{n}}\}$ spanned by the first $n$ fundamental basis vectors, which is a Lagrangian subspace of $V$. Similarly, we put $V^{-}=\mathrm{span}_{\mathbb{R}}\{{e_{n},e_{n+1},\dots,e_{2n}}\}$, a complementary Lagrangian subspace to $V^{+}$, and we have a complete polarization $V=V^{+}\oplus V^{-}$. The Lagrangians $V^{+}$ and $V^{-}$ are dual to each other by the symplectic form, so that we can and often do identify $V^{-}=(V^{+})^{\ast}$.

Let $P_{S}=\mathop{Stab}\nolimits_{G}(V^{+})=\{g\in G\mid gV^{+}=V^{+}\}$ be the stabilizer of the Lagrangian subspace $V^{+}$. Then $P_{S}$ is a maximal parabolic subgroup of $G$ with Levi decomposition $P_{S}=L\ltimes N$, where $L=\mathop{Stab}\nolimits_{G}(V^{+})\cap\mathop{Stab}\nolimits_{G}(V^{-})$, the stabilizer of the polarization, and $N$ is the unipotent radical of $P_{S}$. We call $P_{S}$ a Siegel parabolic subgroup. Since $G=\mathrm{Sp}(V)$ acts on Lagrangian subspaces transitively, $\Lambda:=G/P_{S}$ is the collection of all Lagrangian subspaces in $V$. We call this space a Lagrangian flag variety and also denote it by $\mathop{LGr}\nolimits(\mathbb{R}^{2n})$.

The Levi subgroup $L$ of $P_{S}$ is explicitly given by

[TABLE]

and we consider it to be $\mathrm{GL}(V^{+})$ which acts on $V^{-}=(V^{+})^{\ast}$ in the contragredient manner. The unipotent radical $N$ of $P_{S}$ is realized in the matrix form as

[TABLE]

via the exponential map. Note that $\begin{pmatrix}a&b\\ 0&\,{}^{t}{a}\,^{-1}\end{pmatrix}\in P_{S}$ if and only if $a\,{}^{t}{b}\,\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$, which in turn equivalent to $a^{-1}b\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$.

Take a maximal parabolic subgroup $Q$ in $L=\mathrm{GL}(V^{+})$ which stabilizes $d$-dimensional isotropic space $U\subset V^{+}$. Then $\Xi_{d}:=L/Q=\mathop{Gr}\nolimits_{d}(V^{+})=\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ is the Grassmannian of $d$-dimensional spaces. Note that, in the standard realization,

[TABLE]

Now, our main concern is a double flag variety $X=\Lambda\times\Xi_{d}=G/P_{S}\times L/Q$ on which $L=\mathrm{GL}_{n}(\mathbb{R})$ acts diagonally. We are strongly interested in the orbit structure of $X$ under the action of $L$ and its applications to representation theory.

Goal and Main Results 0.1**.**

We will consider the following problems.

• $(1)$

To prove there are finitely many $L$-orbits on the double flag variety $X=\Lambda\times\Xi_{d}$. We will give a complete classification of open orbits, and recursive strategy to determine the whole structure of $L$-orbits on $X$. See Theorems 2.7 and 4.3.

• $(2)$

To construct relative invariants on each open orbits. We will use them to define integral transforms between degenerate principal series representations of $L$ and that of $G$. For this, see § 7, especially Theorems 7.1 and 7.2.

Here we will make a short remark on the double flag varieties over the complex number field (or, more correctly, over an algebraically closed fields of characteristic zero).

Let us complexify everything which appears in the setting above, so that $G_{\mathbb{C}}=\mathrm{Sp}_{2n}(\mathbb{C})$ and $L_{\mathbb{C}}\simeq\mathrm{GL}_{n}(\mathbb{C})$. The complexifications of the parabolics are $P_{S,\mathbb{C}}$, the stabilizer of a Lagrangian subspace in the symplectic vector space $\mathbb{C}^{2n}$, and $Q_{\mathbb{C}}$, the stabilizer of a $d$-dimensional vector space in $\mathbb{C}^{n}$. Then it is known that the double flag variety $X_{\mathbb{C}}=G_{\mathbb{C}}/P_{S,\mathbb{C}}\times L_{\mathbb{C}}/Q_{\mathbb{C}}$ has finitely many $L_{\mathbb{C}}$-orbits or $\#Q_{\mathbb{C}}\backslash G_{\mathbb{C}}/P_{S,\mathbb{C}}<\infty$. In this case, one can replace the maximal parabolic $Q_{\mathbb{C}}$ by a Borel subgroup $B_{L,\mathbb{C}}$ of $L_{\mathbb{C}}$, and still there are finitely many $L_{\mathbb{C}}$ orbits in $G_{\mathbb{C}}/P_{S,\mathbb{C}}\times L_{\mathbb{C}}/B_{L,\mathbb{C}}$ (see [NO11] and [HNOO13, Table 2]).

Even if there are only finitely many orbits of a complex algebraic group, say $L_{\mathbb{C}}$, acting on a smooth algebraic variety, there is no guarantee for finiteness of orbits of real forms in general 111 It is known that there is a canonical bijection $L(\mathbb{R})\backslash(L/H)(\mathbb{R})=\ker(H^{1}(C;H)\to H^{1}(C;L))$, where $C=\mathop{Gal}\nolimits(\mathbb{C}/\mathbb{R})$ and $H^{1}(C;H)$ denotes the first Galois cohomology group. See [BJ06, Eq. (II.5.6)].. So our problem over reals seems impossible to be deduced from the results over $\mathbb{C}$.

On the other hand, in the case of the complex full flag varieties, there exists a famous bijection between $K_{\mathbb{C}}$ orbits and $G_{\mathbb{R}}$ orbits called Matsuki correspondence [Mat88]. Both orbits are finite in number. In the case of double flag varieties, there is no such known correspondences. It might be interesting to pursue such correspondences.

Toshiyuki Kobayashi informed us that the finiteness of orbits $\#X/L<\infty$ also follows from general results on visible actions [Kob05]. We thank him for his kind notice.

Acknowledgement. K. N. thanks Arhus University for its warm hospitality during the visits in August 2015 and 2016. Most of this work has been done in those periods.

1. Elementary properties of $G=\mathrm{Sp}_{2n}(\mathbb{R})$

In this section, we will give very well known basic facts on the symplectic group for the sake of fixing notations. We define

[TABLE]

The following lemmas are quite elementary and well known. We just present them because of fixing notations.

Lemma 1.1**.**

If we write g=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}\in\mathrm{GL}_{2n}(\mathbb{R}), then $g$ belongs to $G$ if and only if $\,{}^{t}{a}\,c,\;\,{}^{t}{b}\,d\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ and $\,{}^{t}{a}\,d-\,{}^{t}{c}\,b=1$.

Proof.

We rewrite $\,{}^{t}{g}\,Jg=J$ by coordinates, and get

[TABLE]

which shows the lemma. ∎

Lemma 1.2**.**

If we write g=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}\in G, then g^{-1}=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\,{}^{t}{d}}&{-\,{}^{t}{b}}\\ {-\,{}^{t}{c}}&{\,{}^{t}{a}}\end{array}\Bigr{)}.

Proof.

Since $\,{}^{t}{g}\,Jg=J$, we get g^{-1}=J^{-1}\,{}^{t}{g}\,J=-J\,{}^{t}{g}\,J=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\,{}^{t}{d}}&{-\,{}^{t}{b}}\\ {-\,{}^{t}{c}}&{\,{}^{t}{a}}\end{array}\Bigr{)}. ∎

Lemma 1.3**.**

If we write g=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}\in G and p=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{x}&{z}\\ {0}&{y}\end{array}\Bigr{)}\in P_{S}, then

[TABLE]

Note that, in fact, $y=\,{}^{t}{x}\,^{-1}$.

Proof.

Just a calculation, using Lemma 1.2. ∎

A maximal compact subgroup $K$ of $G$ is given by $K=\mathrm{Sp}_{2n}(\mathbb{R})\cap\mathrm{O}(2n)$.

Lemma 1.4**.**

An element g=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}\in G belongs to $K$ if and only if

[TABLE]

hold. Consequently,

[TABLE]

Proof.

$g\in G$ belongs to $\mathrm{O}(2n)$ if and only if $\,{}^{t}{g}\,=g^{-1}$. From Lemma 1.2, we get $a=d$ and $c=-b$. From 1.1, we get the rest two equalities.

Note that if $a+ib\in\mathrm{U}(n)$

[TABLE]

This last formula is equivalent to the above two equalities. ∎

2. $L$-orbits on the Lagrangian flag variety $\Lambda$

Now, let us begin with the investigation of $L$ orbits on $\Lambda=G/P_{S}$, which should be well-known.

Let us denote the Weyl group of $P_{S}$ by $W_{P_{S}}$, which is isomorphic to $S_{n}$, the symmetric group of $n$-th order. In fact, it coincides with the Weyl group of $L$.

By Bruhat decomposition, we have

[TABLE]

where in the second sum $w$ moves over the representatives of the double cosets. The double coset space $W_{P_{S}}\backslash W_{G}/W_{P_{S}}\simeq S_{n}\backslash(S_{n}\ltimes(\mathbb{Z}/2\mathbb{Z})^{n})/S_{n}$ has a complete system of representatives of the form

[TABLE]

We realize $w_{k}$ in $G$ as

[TABLE]

Lemma 2.1**.**

For $0\leq k\leq n$, we temporarily write $w=w_{k}$. Then $P_{S}wP_{S}/P_{S}=w(w^{-1}P_{S}w)P_{S}/P_{S}\simeq w^{-1}P_{S}w/(w^{-1}P_{S}w\cap P_{S})$ contains $N_{w}$ given below as an open dense subset.

[TABLE]

Proof.

Take \Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{x}&{z}\\ {0}&{y}\end{array}\Bigr{)}\in P_{S} and write w=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}, where a=d=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{0}&{0}\\ {0}&{1_{k}}\end{array}\Bigr{)},\;c=-b=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1_{n-k}}&{0}\\ {0}&{0}\end{array}\Bigr{)}. Then, using the formula in Lemma 1.3, we can calculate as

[TABLE]

Let us rewrite the last formula in the form

[TABLE]

so that we get

[TABLE]

provided that $y_{11}^{-1}$ and $x_{22}^{-1}$ exist (an open condition). Note that we can take $z_{11}$ and $z_{21}$ arbitrary, and also that, if we put $x_{21}=0$ and $y_{12}=0$, we can take $x_{12}$ (which determines $y_{21}$) arbitrary. This shows the last formula (2.9) above exhausts $\eta$ of the form in (2.3). ∎

Remark 2.2*.*

The formula (2.9) actually gives a symmetric matrix. One can check this directly, using $y=\,{}^{t}{x}\,^{-1}$. See also Lemma 2.4 below.

Let us consider $L=\mathrm{GL}_{n}(\mathbb{R})$ action on the $k$-th Bruhat cell $P_{S}w_{k}P_{S}/P_{S}$. It is just the left multiplication. However, if we identify it with $w^{-1}P_{S}w/(w^{-1}P_{S}w)\cap P_{S}$ as in Lemma 2.1, the action of $a\in L$ is given by the left multiplication of $w^{-1}aw$. This conjugation is explicitly given as

[TABLE]

which can be read off from Equation (2.8).

Lemma 2.3**.**

There are exactly $\binom{n-k+2}{2}$ of $L$-orbits on the Bruhat cell $P_{S}w_{k}P_{S}/P_{S}\;(0\leq k\leq n)$. A complete representatives of $L$-orbits is given as

[TABLE]

Proof.

For the brevity, we will write $w=w_{k}$. Firstly, we observe that by the left multiplication of $L$ clearly we can choose orbit representatives from the set

[TABLE]

Then, by the calculations in the proof of Lemma 2.1 and Equation (2.9), it reduces to the subset

[TABLE]

Now let us consider the action of $L$ on this set. Take a=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{h}&{0}\\ {0}&{\,{}^{t}{h}\,^{-1}}\end{array}\Bigr{)}\in L, where $h=\mathop{diag}\nolimits(h_{1},1_{k})\;(h_{1}\in\mathrm{GL}_{n-k}(\mathbb{R}))$. Then, the action of $a$ is the left multiplication of $w^{-1}aw$ as explained above (see Equation (2.10)). As a consequence, it brings to

[TABLE]

Now it is well known that for a suitable choice of $h_{1}\in\mathrm{GL}_{n-k}(\mathbb{R})$, we get

[TABLE]

for a certain signature $(r,s)$ with $r+s\leq n-k$. ∎

Let us explicitly describe the $L$-action on $N_{w}\subset(w^{-1}P_{S}w)/(w^{-1}P_{S}w)\cap P_{S}$.

Lemma 2.4**.**

The action of $w^{-1}aw$ in Equation (2.10) on

[TABLE]

is given by

[TABLE]

So the action on $\xi$-part is linear fractional, while action on $\zeta$-part is a mixture of unimodular and linear fractional action.

Proof.

Take $a\in L$ as in Equation (2.10) and we use the formula of $w^{-1}aw$ there.

[TABLE]

From this, we calculate

[TABLE]

where

[TABLE]

We will rewrite these formulas neatly.

Firstly, we notice it should hold $B=\,{}^{t}{C}\,$. Let us check it. For this, we compare h\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{-\xi}\\ {0}&{1}\end{array}\Bigr{)} and \,{}^{t}{h}\,^{-1}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{0}\\ {\,{}^{t}{\xi}}&{1}\end{array}\Bigr{)}. Using notation $h^{\prime}=\,{}^{t}{h}\,^{-1}$, we calculate both as

[TABLE]

Since (2.18) and (2.30) are mutually inverse, we get

[TABLE]

Now, we calculate

[TABLE]

where in the last equality we use Equation (2.32). This also proves the formula for linear fractional action on $\xi$.

Secondly, we check that $A$ is symmetric.

[TABLE]

This proves that $A$ is symmetric and at the same time the formula of the action on $\zeta$ in the lemma. ∎

Lemma 2.5**.**

For a representative

[TABLE]

of $L$-orbits in the $k$-th Bruhat cell $w^{-1}P_{S}w/(w^{-1}P_{S}w)\cap P_{S}$ (see Lemma 2.3), the stabilizer is given by

[TABLE]

Thus an orbit $\mathcal{O}_{(k;r,s)}$ through $p_{(k;r,s)}$ is isomorphic to $\mathrm{GL}_{n}(\mathbb{R})/H_{(k;r,s)}$, where $H_{(k;r,s)}$ is the collection of $h$ given in Equation (2.36).

Proof.

We put $\xi=0$ in Lemma 2.4, and assume that $a\mathbin{\overset{w}{\cdot}}\eta=\eta$. It gives $B=-h_{2}h_{4}^{-1}=0$ and $A=h_{1}\zeta\,{}^{t}{h_{1}}\,=\zeta$. Here, we assume $h_{4}$ is regular. So, under this hypothesis, we get $h_{2}=0$. Since the stabilizer is a closed subgroup, we must have $h_{2}=0$ in any case (as a matter of fact, actually $h_{4}$ must be regular). ∎

For the later reference, we reinterpret the above lemma by Lagrangian realization. Recall that $G/P_{S}$ is isomorphic to the set of Lagrangian subspaces in $V$ denoted as $\Lambda$. The isomorphism is explicitly given by $G/P_{S}\ni gP_{S}\mapsto g\cdot V^{+}\in\Lambda$, here we identify $V^{+}$ with the space $\mathrm{span}_{\mathbb{R}}\{{\boldsymbol{e}_{1},\dots,\boldsymbol{e}_{n}}\}$ spanned by the first $n$ fundamental vectors in $V=\mathbb{R}^{2n}$. For $v=\sum_{i=1}^{n}c_{i}\boldsymbol{e}_{i}\in V^{+}$, we denote $v^{(n-k)}=\sum_{i=1}^{n-k}c_{i}\boldsymbol{e}_{i}$ and $v_{(k)}=\sum_{j=1}^{k}c_{n-k+j}\boldsymbol{e}_{n-k+j}$ so that $v=v^{(n-k)}+v_{(k)}$.

Lemma 2.6**.**

With the notation introduced above, $L$-orbits on the Lagrangian Grassmannian $\Lambda\simeq G/P_{S}$ has a representatives of the following form.

[TABLE]

Here $0\leq k\leq n$ and $r,s\geq 0$ denote the signature which satisfy $0\leq r+s\leq k$.

Proof.

By Lemma 2.5, we know the representatives of $L$-orbits in the $k$-th Bruhat cell $w^{-1}P_{S}w/(w^{-1}P_{S}w)\cap P_{S}\;\;(w=w_{k})$. They are denoted as $p_{(k;r,s)}$. The corresponding Lagrangian subspace is obtained by $w_{k}\,p_{(k;r,s)}\cdot V^{+}$. If we take $v\in V^{+}$ and write it as $v=v^{(n-k)}+v_{(k)}$ as in just before the lemma, then we obtain

[TABLE]

which proves the lemma. ∎

Theorem 2.7**.**

Let $B_{n}\subset L$ be a Borel subgroup of $L$. A double flag variety $G/P_{S}\times L/B_{n}$ has finitely many $L$-orbits. In other words, $G/P_{S}$ has finitely many $B_{n}$-orbits. In this sense, $G/P_{S}$ is a real $L$-spherical variety.

Proof.

Firstly, we consider the open Bruhat cell, i.e., the case where $k=0$ and $w=w_{0}=J_{n}$. The cell is isomorphic to $w_{0}^{-1}P_{S}w_{0}/(w_{0}^{-1}P_{S}w_{0}\cap P_{S})\simeq N_{w_{0}}$, where

[TABLE]

and the action of $h\in\mathrm{GL}_{n}(\mathbb{R})\simeq L$ is given by the unimodular action: $z\mapsto hz\,{}^{t}{h}\,$. So the complete representatives of $L$-orbits are given by $\{z_{r,s}:=\mathop{diag}\nolimits(1_{r},-1_{s},0)\mid r,s\geq 0,\;r+s\leq n\}$. Let $H_{r,s}\subset L$ be the stabilizer of $z_{r,s}$ (note that $H_{r,s}$ is denote as $H_{(0;r,s)}$ in Lemma 2.5. We omit [math] for brevity). Then an $L$-orbit in the open Bruhat cell is isomorphic to $L/H_{r,s}$. What we must prove is that there are only finitely many $B_{n}$ orbits on $L/H_{r,s}$.

Direct calculations tell that

[TABLE]

where $\mathrm{O}(r,s)$ denotes the indefinite orthogonal group preserving a quadratic form defined by $\mathop{diag}\nolimits(1_{r},-1_{s})$. Note that if $r+s=n$, we simply get $H_{r,s}=\mathrm{O}(r,s)$, which is a symmetric subgroup in $\mathrm{GL}_{n}(\mathbb{R})$. It is well known that a minimal parabolic subgroup $P_{\mathrm{min}}$ has finitely many orbits on $G/H$, where $G$ is a general connected reductive Lie group, and $H$ its symmetric subgroup (i.e., an open subgroup of the fixed point subgroup of a non-trivial involution of $G$). For this, we refer the readers to [Wol74], [Mat79], [Ros79]. Thus, the Borel subgroup $B_{n}$ has an open orbit on $L/\mathrm{O}(r,s)$ when $r+s=n$. This is equivalent to say that $\mathfrak{b}^{\prime}_{n}+\mathfrak{o}(r,s)=\mathfrak{l}$ for some choice $\mathfrak{b}^{\prime}_{n}$ of a Borel subalgebra of \mathfrak{l}=\mathop{\mathrm{missing}}{Lie}\nolimits{}L.

On the other hand, the following is known.

Lemma 2.8**.**

*Let $G$ be a connected reductive Lie group and $P_{\mathrm{min}}$ its minimal parabolic subgroup. For any closed subgroup $H$ of $G$, let us consider an action of $H$ on the flag variety $G/P_{\mathrm{min}}$ by the left translation. Then the followings are equivalent.

• $(1)$

There are finitely many $H$-orbits in $G/P_{\mathrm{min}}$, i.e., we have $\#H\backslash G/P_{\mathrm{min}}<\infty$.

• $(2)$

There exists an open $H$-orbit in $G/P_{\mathrm{min}}$.

• $(3)$

There exists $g\in G$ for which $\mathop{Ad}\nolimits g\cdot\mathfrak{h}+\mathfrak{p}_{\mathrm{min}}=\mathfrak{g}$ holds.

For the proof of this lemma, see [KO13, Remark 2.5 4)]. There is a misprint there, however. So we repeat the remark here. Matsuki [Mat91] observed that the lemma follows if it is valid for real rank one case, while the real rank one case had been already established by Kimelfeld [Kim87]. See also [KS13] for another proof.

Now, in the case where $r+s<n$, since the upper left corner of $H_{r,s}$ is $\mathrm{O}(r,s)$, we can find a Borel subgroup $\mathfrak{b}^{\prime}_{n}$ in $\mathfrak{l}$ for which $\mathfrak{b}^{\prime}_{n}+\mathfrak{h}_{r,s}=\mathfrak{l}$ holds. By the above lemma, $H_{r,s}$ has finitely many orbits in $L/B_{n}$ or $\#B_{n}\backslash L/H_{r,s}<\infty$.

Secondly, let us consider the general Bruhat cell. Then, by Lemma 2.5, we know there are finitely many $L$-orbits and they are isomorphic to $L/H_{(k;r,s)}$. The Lie algebra of $H_{(k;r,s)}$ realized in $\mathrm{GL}_{n}(\mathbb{R})$ is of the following form:

[TABLE]

where \Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{\alpha}&{\beta}\\ {0}&{\gamma}\end{array}\Bigr{)}\in\mathfrak{gl}_{n-k}(\mathbb{R}) and $\delta\in\mathrm{M}_{k,n-k}(\mathbb{R}),\;\eta\in\mathfrak{gl}_{k}(\mathbb{R})$. Let us choose a Borel subalgebra $\mathfrak{b}^{\prime}_{n-k}$ of $\mathfrak{gl}_{n-k}(\mathbb{R})$ such that

[TABLE]

applying the arguments for the open Bruhat cell. Then we can take

[TABLE]

as a Borel subalgebra of $\mathfrak{l}$ which satisfies $\mathfrak{b}_{n}+\mathfrak{h}_{(k;r,s)}=\mathfrak{l}$. Thus Lemma 2.8 tells that $B_{n}$-orbits in $L/H_{(k;r,s)}$ is finite. ∎

Corollary 2.9**.**

For any parabolic subgroup $Q$ of $L$, the double flag variety $X=\Lambda\times\Xi_{d}=G/P_{S}\times L/Q$ has finitely many $L$-orbits, hence it is of finite type.

3. Maslov index

In [KS94], Kashiwara and Shapira described the orbit decomposition of the diagonal action of $G=\mathrm{Sp}_{2n}(\mathbb{R})$ in the triple product $\Lambda^{3}=\Lambda\times\Lambda\times\Lambda$ of Lagrangian Grassmannians. They used an invariant called Maslov index to classify the orbits and concluded that there are only finitely many orbits, i.e., $\#\Lambda^{3}/G<\infty$.

Let us explain the relation of their result and ours.

Fix points $x_{\pm}\in\Lambda$ which are corresponding to the Lagrangian subspaces $V^{\pm}\subset V$. We consider a $G$-stable subspace containing $\{x_{+}\}\times\{x_{-}\}\times\Lambda$, namely Put

[TABLE]

Since all the orbits go through a point $\{x_{+}\}\times\{x_{-}\}\times\{\lambda\}$ for a certain $\lambda\in\Lambda$, $G$-orbit decomposition of $Y$ reduces to orbit decomposition of $\mathop{Stab}\nolimits_{G}(\{x_{+}\}\times\{x_{-}\})$ in $\Lambda=G/P_{S}$.

It is easy to see that the stabilizer $\mathop{Stab}\nolimits_{G}(\{x_{+}\}\times\{x_{-}\})$ is exactly $L$ so that $Y/G\simeq\Lambda/L\simeq L\backslash G/P_{S}$, on the last of which we discussed in § 2. Since $Y\subset\Lambda^{3}$, it has finitely many $G$-orbits due to [KS94], hence $\Lambda=G/P_{S}$ also has finitely many $L$-orbits. A detailed look at [KS94] will also provides the classification of orbits, which we do not carry out here.

However, for proving the finiteness of $B_{n}$-orbits, we need explicit structure of orbits as homogeneous spaces of $L$. This is the main point of our analysis in § 2.

4. Classification of open $L$-orbits in the double flag variety

Let us return back to the original situation of Grassmannians, i.e., our $Q=P_{(d,n-d)}^{\mathrm{GL}}\subset L$ is a maximal parabolic subgroup which stabilizes a $d$-dimensional subspace in $V^{+}$. So the double flag variety $X=G/P_{S}\times L/Q$ is isomorphic to the product of the Lagrangian Grassmannian $\Lambda=\mathop{LGr}\nolimits(\mathbb{R}^{2n})$ and the Grassmannian $\Xi_{d}=\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ of $d$-dimensional subspaces.

In this section, we will describe open $L$-orbits in $X$. To study $L$-orbits in $X=G/P_{S}\times L/Q$, we use the identification

[TABLE]

In this identification, open $L$-orbits corresponds to open $Q$-orbits, since they are of the largest dimension. We already know the description of $L$-orbits on $\Lambda=G/P_{S}$ from § 2. Open $Q$-orbits are necessarily contained in open $L$-orbits, hence we concentrate on the open Bruhat cell $P_{S}w_{0}P_{S}/P_{S}\simeq N_{w_{0}}\simeq\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$. $L$ acts on $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ via unimodular action: $h\cdot z=hz\,{}^{t}{h}\,\;\;(z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R}),\;h\in\mathrm{GL}_{n}(\mathbb{R})\simeq L)$.

The following lemma, Sylvester’s law of inertia, is a special case of Lemma 2.3.

Lemma 4.1**.**

Let $L=\mathrm{GL}_{n}(\mathbb{R})$ act on $N_{w_{0}}=\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ via unimodular action. Then, open orbits are parametrized by the signature $(p,q)$ with $p+q=n$. A complete system of representatives are given by $\{I_{p,q}\mid p,q\geq 0,\;p+q=n\}$, where $I_{p,q}=\mathop{diag}\nolimits(1_{p},-1_{q})$.

Let us denote open $L$-orbits by

[TABLE]

Thus we are looking for open $Q$-orbits in $\Omega(p,q)$. Let us denote $H=\mathop{Stab}\nolimits_{L}(I_{p,q})$, the stabilizer of $I_{p,q}\in\Omega(p,q)$, which is isomorphic to an indefinite orthogonal group $\mathrm{O}(p,q)$. As a consequence $\Omega(p,q)\simeq L/H\simeq\mathrm{GL}_{n}(\mathbb{R})/\mathrm{O}(p,q)$.

Since $\Omega(p,q)\simeq L/H$,

[TABLE]

where $\Xi_{d}=\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ is the Grassmannian of $d$-dimensional subspaces. So our problem of seeking $Q$-orbits in $\Omega(p,q)$ is equivalent to understand $H$-orbits in a partial flag variety $\Xi_{d}$. Since $H$ is a symmetric subgroup fixed by an involutive automorphism of $L$, this problem is ubiquitous in representation theory of real reductive Lie groups.

Let us consider a $d$-dimensional subspace $U=\mathrm{span}_{\mathbb{R}}\{{\boldsymbol{e}_{1},\boldsymbol{e}_{2},\dots,\boldsymbol{e}_{d}}\}\in\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ which is stabilized by $Q$. Take $z\in\Omega(p,q)$, and consider a quadratic form $\mathscr{Q}_{z^{-1}}(v,v)=\,{}^{t}{v}\,z^{-1}v\;(v\in\mathbb{R}^{n})$ associated to $z^{-1}$, which also has the same signature $(p,q)$ as that of $z$. Note that the restriction of $\mathscr{Q}_{z^{-1}}$ to $U$ can be degenerate, and the rank and the signature of \mathscr{Q}_{z^{-1}}\big{|}_{U} is preserved by the action of $Q$. In fact, for $u\in U$ and $m\in Q$, we get

[TABLE]

Since $m^{-1}\in Q$ preserves $U$, the quadratic forms $\mathscr{Q}_{z^{-1}}$ and $\mathscr{Q}_{(m\cdot z)^{-1}}$ have the same rank and the signature when restricted to $U$. So they are clearly invariants of a $Q$-orbit in $\Omega(p,q)$. Put

[TABLE]

where $s+t$ is the rank of \mathscr{Q}_{z^{-1}}\big{|}_{U}. Clearly $0\leq s\leq p,\,0\leq t\leq q$ and $s+t\leq d$ must be satisfied.

Lemma 4.2**.**

$Q$-orbits in $\Omega(p,q)$ are exactly

[TABLE]

given in (4.2). The orbit $\Omega(p,q;s,t)$ is open if and only if $s+t=d=\dim U$, i.e., the quadratic form $\mathscr{Q}_{z^{-1}}$ is non-degenerate when restricted to $U$.

Proof.

The restriction \mathscr{Q}_{z^{-1}}\big{|}_{U} is a quadratic form, and we denote its signature by $(s,t)$. The rank of \mathscr{Q}_{z^{-1}}\big{|}_{U} is $s+t$ and $k=d-(s+t)$ is the dimension of the kernel. Obviously, we must have $0\leq s,t,k\leq d$. Since $\mathscr{Q}_{z^{-1}}$ is non-degenerate with signature $(p,q)$, there exist signature constraints

[TABLE]

These conditions are equivalent to the condition given in the lemma. The signature $(s,t)$ and hence the dimension $k$ of the kernel is invariant under the action of $Q$.

Conversely, if a $d$-dimensional subspace $U_{1}$ of the quadratic space $\mathbb{R}^{n}$ has the same signature $(s,t)$ (and hence $k$), it can be translated into $U$ by the isometry group $\mathrm{O}(p,q)$ by Witt’s theorem. This means the signature concretely classifies $Q$-orbits. ∎

This lemma practically classifies open $L$-orbits on $X=\Lambda\times\Xi_{d}$. However, we rewrite it more intrinsically.

Firstly, we note that, for $z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$, a Lagrangian subspace $\lambda\in\Lambda=G/P_{S}$ in the open Bruhat cell $P_{S}w_{0}P_{S}/P_{S}\simeq N_{w_{0}}$ is given by

[TABLE]

and clearly such $z$ is uniquely determined by $\lambda$. We denote the Lagrangian subspace by $\lambda_{z}$. Also, we denote a $d$-dimensional subspace in $\Xi_{d}=\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ by $\xi$.

Theorem 4.3**.**

Suppose that non-negative integers $p,q$ and $s,t$ satisfies

[TABLE]

Then an open $L$-orbit in $X=\Lambda\times\Xi_{d}$ is given by

[TABLE]

Every open orbit is of this form.

5. Relative invariants

Let us consider the vector space $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\times\mathrm{M}_{n,d}(\mathbb{R})$, on which $\mathrm{GL}_{n}(\mathbb{R})\times\mathrm{GL}_{d}(\mathbb{R})$ acts. The action is given explicitly as

[TABLE]

Let us put $\mathrm{M}^{\circ}_{n,d}(\mathbb{R}):=\{y\in\mathrm{M}_{n,d}(\mathbb{R})\mid\mathop{rank}\nolimits y=d\}$, the subset of full rank matrices in $\mathrm{M}_{n,d}(\mathbb{R})$. Then, a map $\pi:\mathrm{M}^{\circ}_{n,d}(\mathbb{R})\to\Xi_{d}=\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$ defined by $\pi(y):=\mathrm{span}_{\mathbb{R}}\{{y_{j}\mid 1\leq j\leq d}\}$ ($y_{j}$ denotes the $j$-th column vector of $y$) is a quotient map by the action of $\mathrm{GL}_{d}(\mathbb{R})$. Thus we get a diagram:

[TABLE]

Comparing to the Grassmannian, the vector space $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\times\mathrm{M}_{n,d}(\mathbb{R})$ is easier to handle. In particular, we introduce two basic relative invariants $\psi_{1}$ and $\psi_{2}$ on $(z,y)\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\times\mathrm{M}_{n,d}(\mathbb{R})$ with respect to the above linear action,

[TABLE]

Note that

[TABLE]

so that it is actually a polynomial. We consider two characters of $(h,m)\in\mathrm{GL}_{n}(\mathbb{R})\times\mathrm{GL}_{d}(\mathbb{R})$:

[TABLE]

Then it is easy to check that the relative invariants $\psi_{1},\psi_{2}$ are transformed under characters $\chi_{1}^{-1},\chi_{2}^{-1}$ respectively. Let us define

[TABLE]

The set $\widetilde{\Omega}$ is clearly open and is a union of open $\mathrm{GL}_{n}(\mathbb{R})\times\mathrm{GL}_{d}(\mathbb{R})$-orbits in $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\times\mathrm{M}_{n,d}(\mathbb{R})$.

Theorem 5.1**.**

The sets $\widetilde{\Omega}(p,q;s,t)$, where

[TABLE]

are open $\mathrm{GL}_{n}(\mathbb{R})\times\mathrm{GL}_{d}(\mathbb{R})$-orbits, and they exhaust all the open orbits in $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\times\mathrm{M}_{n,d}(\mathbb{R})$, i.e.,

[TABLE]

where the union is taken over $p,q,s,t$ which satisfies (5.2). Moreover, the quotient $\widetilde{\Omega}(p,q;s,t)/\mathrm{GL}_{d}(\mathbb{R})$ is isomorphic to $\Omega(p,q;s,t)$, an open $L$-orbit in the double flag variety $X=\Lambda\times\Xi_{d}$.

This theorem is just a paraphrase of Theorem 4.3.

Since relative invariants are polynomials, we can consider them on the complexified vector space $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{C})\times\mathrm{M}_{n,d}(\mathbb{C})$. In the rest of this section, we will study them on this complexified vector space, and we denote it simply by $\mathop{\mathrm{Sym}}\nolimits_{n}\times\mathrm{M}_{n,d}$ omitting the base field. Similarly, we use $\mathrm{GL}_{n}=\mathrm{GL}_{n}(\mathbb{C})$, etc., for algebraic groups over $\mathbb{C}$.

Recall the characters $\chi_{1},\chi_{2}$ of $\mathrm{GL}_{n}\times\mathrm{GL}_{d}$ in (5.1). The following theorem should be well-known to the experts, but we need the proof of it to get further results.

Theorem 5.2**.**

$\mathrm{GL}_{n}\times\mathrm{GL}_{d}$-module $\mathrm{Pol}(\mathop{\mathrm{Sym}}\nolimits_{n}\times\mathrm{M}_{n,d})$ contains a unique non-zero relative invariant $f(z,y)$ with character $\chi_{1}^{-m_{1}}\chi_{2}^{-m_{2}}\;\;(m_{1},m_{2}\geq 0)$ up to non-zero scalar multiple. This relative invariant is explicitly given by $f(z,y)=(\det z)^{m_{1}+m_{2}}(\det(\,{}^{t}{y}\,z^{-1}y))^{m_{2}}$.

Proof.

In this proof, to avoid notational complexity, we consider the dual action

[TABLE]

To translate the results here to the original action is easy.

First, we quote results on the structure of the polynomial rings over $\mathop{\mathrm{Sym}}\nolimits_{n}$ and $\mathrm{M}_{n,d}$. Let us denote the irreducible finite dimensional representation of $\mathrm{GL}_{n}$ with highest weight $\lambda$ by $V^{(n)}(\lambda)$ (if $n$ is to be well understood, we will simply write it as $V(\lambda)$).

Lemma 5.3**.**

• $(1)$

As a $\mathrm{GL}_{n}$-module, $\mathop{\mathrm{Sym}}\nolimits_{n}$ is multiplicity free, and the irreducible decomposition of the polynomial ring is given by

[TABLE]

• $(2)$

Assume that $n\geq d\geq 1$. As a $\mathrm{GL}_{n}\times\mathrm{GL}_{d}$-module, $\mathrm{M}_{n,d}$ is also multiplicity free, and the irreducible decomposition of the polynomial ring is given by

[TABLE]

Since we are looking for relative invariants for $\mathrm{GL}_{d}$, it must belong to one dimensional representation space ${\det}_{d}^{\ell}=V^{(d)}(\ell\varpi_{d})$, where $\varpi_{d}=(1,\dots,1,0,\dots,0)$ denotes the $d$-th fundamental weight. Thus it must be contained in the space

[TABLE]

Since a relative invariant is also contained in the one dimensional representation of $\mathrm{GL}_{n}$, say ${\det}_{n}^{k}=V^{(n)}(k\varpi_{n})$, $V^{(n)}(2\mu)\otimes V^{(n)}(\ell\varpi_{d})$ must contain $V^{(n)}(k\varpi_{n})$. We argue

[TABLE]

Thus, $\ell\geq 0$ and $k-\ell\geq 0$ are both even integers, which completely determine $\mu$. So the relative invariant is unique (up to a scalar multiple) if we fix the character ${\det_{n}}^{k}{\det_{d}}^{\ell}=\chi_{1}^{(k-\ell)/2}\chi_{2}^{\ell/2}$. ∎

Corollary 5.4**.**

Let us consider the relative invariant

[TABLE]

*in the above theorem.

• $(1)$

The space $\mathrm{span}_{\mathbb{C}}\{{f(z,y)\mid z\in\mathop{\mathrm{Sym}}\nolimits_{n}}\}\subset\mathrm{Pol}(\mathrm{M}_{n,d})$ is stable under $\mathrm{GL}_{n}$ and it is isomorphic to $V^{(n)}(2m_{2}\varpi_{d})^{\ast}\otimes V^{(d)}(2m_{2}\varpi_{d})^{\ast}$ as a $\mathrm{GL}_{n}\times\mathrm{GL}_{d}$-module.

• $(2)$

Similarly, the space $\mathrm{span}_{\mathbb{C}}\{{f(z,y)\mid y\in\mathrm{M}_{n,d}}\}\subset\mathrm{Pol}(\mathop{\mathrm{Sym}}\nolimits_{n})$ is stable under $\mathrm{GL}_{n}$ and it is isomorphic to $V^{(n)}(2m_{1}\varpi_{n}+2m_{2}\varpi_{n-d})^{\ast}$.

Proof.

It is proved that

[TABLE]

where $k=2m_{1}+2m_{2},\;\ell=2m_{2}$ and $\mu=(k-\ell)\varpi_{n}+\ell\varpi_{n-d}$. For any specialization of $y$, this space is mapped to $V^{(n)}(2\mu)$ (or possibly zero), and if we specialize $z$ to some symmetric matrix, it is mapped to $V^{(n)}(\ell\varpi_{d})\otimes V^{(d)}(\ell\varpi_{d})$. This shows the results. ∎

Although, we do not need the following lemma below, it will be helpful to know the explicit formula for $\det(\,{}^{t}{y}\,z^{-1}y)$. Note that we take $z$ instead of $z^{-1}$ in the lemma.

Lemma 5.5**.**

Let $[n]=\{1,2,\dots,n\}$ and put $\binom{[n]}{d}:=\{I\subset[n]\mid\#I=d\}$, the family of subsets in $[n]$ of $d$-elements. For $X\in\mathrm{M}_{n}$ and $I,J\in\binom{[n]}{d}$, we will denote $X_{I,J}:=(x_{i,j})_{i\in I,j\in J}$, a $d\times d$-submatrix of $X$. For $(z,y)\in\mathop{\mathrm{Sym}}\nolimits_{n}\times\mathrm{M}_{n,d}$, we have

[TABLE]

We observe that $\{\det(y_{I,[d]})\mid I\in\binom{[n]}{d}\}$ is the Plücker coordinates and also $\{\det(z_{IJ})\mid I,J\in\binom{[n]}{d}\}$ is the coordinates for the determinantal variety of rank $d$ (there are much abundance though).

6. Degenerate principal series representations

Let us return back to the situation over real numbers, and we introduce degenerate principal series for $G=\mathrm{Sp}_{2n}(\mathbb{R})$ and $L=\mathrm{GL}_{n}(\mathbb{R})$ respectively.

6.1. Degenerate principal series for $G/P_{S}$

Let us recall $G=\mathrm{Sp}_{2n}(\mathbb{R})$ and its maximal parabolic subgroup $P_{S}$. Take a character $\chi_{P_{S}}$ of $P_{S}$, and consider a degenerate principal series representation

[TABLE]

where $G$ acts by left translations: $\pi_{\nu}^{G}(g)f(x)=f(g^{-1}x)$. In the following, we will take

[TABLE]

(We can multiply the sign $\mathop{sgn}\nolimits(\det a)$ by $\chi_{P_{S}}$, if we prefer.)

Since $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ is openly embedded into $G/P_{S}$, a function $f\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ is determined by the restriction f\big{|}_{\Omega} where $\Omega$ is the embedded image of $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ in $G/P_{S}$. Explicitly, $\Omega$ is defined by

[TABLE]

where $w_{0}$ is the longest element in the Weyl group, and we give an open embedding by

[TABLE]

In the following we mainly identify $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ and $\Omega$. Let us give the fractional linear action of $G$ on $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ in our setting.

Lemma 6.1**.**

In the above identification, the linear fractional action ${g}\boldsymbol{.}{z}$ of g=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{b}\\ {c}&{d}\end{array}\Bigr{)}\in G on $z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})=\Omega$ is given by

[TABLE]

if $\det(cz-d)\neq 0$.

Proof.

By the identification, $w={g}\boldsymbol{.}{z}$ corresponds to gJ\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{0}\\ {z}&{1}\end{array}\Bigr{)}P_{S}/P_{S}. We can calculate it as

[TABLE]

This proves the desired formula. ∎

Lemma 6.2**.**

For $f\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$, the action of $\pi_{\nu}^{G}(g)$ on $f$ is given by

[TABLE]

where $\chi_{P_{S}}(p)$ is given in (6.1). In particular, for h=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{0}\\ {0}&{\,{}^{t}{a}\,^{-1}}\end{array}\Bigr{)}\in L, we get

[TABLE]

We want to discuss the completion of the $C^{\infty}$-version of the degenerate principal series $C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ to a representation on a Hilbert space. Usually, this is achieved by the compact picture, but here we use noncompact picture. To do so, we need an elementary decomposition theorem.

Here we write $P_{S}=LN_{S},\;\;L=MA$, where we wrote $N$ for $N_{S}$ which is the unipotent radical of $P_{S}$, and $M=\mathrm{SL}^{\pm}_{n}(\mathbb{R}),A=\mathbb{R}_{+}$. Further, we denote $M_{K}=M\cap K=\mathrm{O}(n)$. For the opposite Siegel parabolic subgroup $\overline{\rule{0.0pt}{5.16663pt}P_{S}}$, we denote a Langlands decomposition by $\overline{\rule{0.0pt}{5.16663pt}P_{S}}=MA\overline{\rule{0.0pt}{5.16663pt}N_{S}}$.

Thus we conclude $\overline{\rule{0.0pt}{5.16663pt}N_{S}}MAN_{S}\subset KMAN_{S}=G$ (open embedding). Every $g\in G$ can be written as $g=kman\in KMAN_{S}$, and we call this generalized Iwasawa decomposition by abuse of the terminology. Iwasawa decomposition $g=kman$ may not be unique, but if we require ma=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{h}&{0}\\ {0}&{\,{}^{t}{h}\,^{-1}}\end{array}\Bigr{)} for an $h\in\mathop{\mathrm{Sym}}\nolimits_{n}^{+}(\mathbb{R})$, it is indeed unique. This follows from the facts that the decomposition $M=\mathrm{O}(n)\cdot\mathop{\mathrm{Sym}}\nolimits_{n}^{+}(\mathbb{R})$ is unique (Cartan decomposition), and that $M_{K}=K\cap M=\mathrm{O}(n)$.

Now we describe an explicit Iwasawa decomposition of elements in $\overline{\rule{0.0pt}{5.16663pt}N_{S}}$.

Lemma 6.3**.**

Let v(z):=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{0}\\ {z}&{1}\end{array}\Bigr{)}\in\overline{\rule{0.0pt}{5.16663pt}N_{S}}\;(z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})) and denote $h:=\sqrt{{1_{n}+z^{2}}\,}\in\mathop{\mathrm{Sym}}\nolimits_{n}^{+}(\mathbb{R})$, a positive definite symmetric matrix. Then we have the Iwasawa decomposition

[TABLE]

Proof.

Since

[TABLE]

we get (putting $h=\sqrt{{1+z^{2}}\,}$)

[TABLE]

Notice that \dfrac{\,{1}\,}{\,{\sqrt{{1+z^{2}}\,}}\,}\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{1}&{z}\\ {-z}&{1}\end{array}\Bigr{)} is in $K$, and its inverse is given by $k$ in the statement of the lemma. The rest of the statements are easy to derive. ∎

Since $\overline{\rule{0.0pt}{5.16663pt}N_{S}}MAN_{S}$ is open dense in $G$, $f\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ is determined by f\big{|}_{\overline{\rule{0.0pt}{3.61664pt}N_{S}}}. We complete the space of functions on $\overline{\rule{0.0pt}{5.16663pt}N_{S}}$ or $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})$ by the measure $(\det(1+z^{2}))^{\nu_{0}-\frac{n+1}{2}}dz$, where $\nu_{0}=\mathop{Re}\nolimits\nu$ and $dz$ denotes the usual Lebesgue measure, in order to get a Hilbert representation. See [Kna86, § VII.1] for details (we use unnormalized induction, so that there is a shift of \rho_{P_{S}}(a)=|\det(\mathop{Ad}\nolimits(a)\big{|}_{N_{S}})|^{1/2}=|\det a|^{\frac{n+1}{2}}). Thus our Hilbert space is

[TABLE]

We denote an induced representation $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ on the Hilbert space $\mathcal{H}_{\nu}^{G}$ by $\pi_{\nu}^{G}$.

Remark 6.4*.*

The degenerate principal series $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ induced from the character $\chi_{P_{S}}(p)=|\det a|^{\nu}$ (cf. Eq. (6.1)) has the unitary axis at $\nu_{0}=\frac{n+1}{2}$. If $n$ is even, there exist complementary series for real $\nu$ which satisfies $\frac{n}{2}<\nu<\frac{n}{2}+1$ (see [Lee96, Th. 4.3]).

6.2. Degenerate principal series for $L/Q$

In this subsection, we fix the notations for degenerate principal series of $L=\mathrm{GL}_{n}(\mathbb{R})$ from its maximal parabolic subgroup $Q=P_{(d,n-d)}^{\mathrm{GL}}$. We will denote

[TABLE]

Then $\chi_{Q}$ is a character of $Q$, and we consider a degenerate principal series representation

[TABLE]

where $L$ acts by left translations: $\pi_{\mu}^{L}(a)F(Y)=f(a^{-1}Y)\;\;(a,Y\in L)$. We introduce an $L^{2}$-norm on this space just like usual integral over a maximal compact subgroup $K_{L}=K\cap L=\mathrm{O}(n)$:

[TABLE]

and take a completion with respect to this norm to get a Hilbert space $\mathcal{H}_{\mu}^{L}$. Note that the integration is in fact well-defined on $K_{L}/(K\cap Q)\simeq\mathrm{O}(n)/\mathrm{O}(d)\times\mathrm{O}(n-d)$, because of the right equivariance of $F$. Thus we get a representation $\pi_{\mu}^{L}=\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ on the Hilbert space $\mathcal{H}_{\mu}^{L}$.

To make the definition of intertwiners more easy to handle, we unfold the Grassmannian $L/Q\simeq\mathop{Gr}\nolimits_{d}(\mathbb{R}^{n})$. Recall $\mathrm{M}^{\circ}_{n,d}(\mathbb{R})=\{y\in\mathrm{M}_{n,d}(\mathbb{R})\mid\mathop{rank}\nolimits y=d\}$. Then, we get a map

[TABLE]

which induces an isomorphism $\Xi_{d}=L/Q\xrightarrow{\;\sim\;}\mathrm{M}^{\circ}_{n,d}(\mathbb{R})/\mathrm{GL}_{d}(\mathbb{R})$. Thus we can identify $C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ with the space of $C^{\infty}$ functions $F:\mathrm{M}^{\circ}_{n,d}(\mathbb{R})\to\mathbb{C}$ with the property $F(yk)=|\det k|^{-\mu}F(y)$. In this picture, the action of $L$ is just the left translation:

[TABLE]

To have the $L^{2}$-norm defined in (6.6), we restrict the projection map (6.7) from $L=\mathrm{GL}_{n}$ to $K_{L}=\mathrm{O}(n)$, the resulting space being the Stiefel manifold of orthonormal frames $S_{n,d}=\{y\in\mathrm{M}^{\circ}_{n,d}\mid\,{}^{t}{y}\,y=1_{d}\}$. Then $L/Q$ is isomorphic to $S_{n,d}/\mathrm{O}(d)$. The norm given in (6.6) is equal to

[TABLE]

where $d\sigma(v)$ is the uniquely determined $\mathrm{O}(n)$-invariant non-zero measure. Note that $S_{n,d}\simeq\mathrm{O}(n)/\mathrm{O}(n-d)$.

Remark 6.5*.*

The degenerate principal series $\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ induced from the character $\chi_{Q}(q)=|{\det k}|^{\mu}$ (cf. Eq. (6.5)) is never unitary as a representation of $\mathrm{GL}_{n}(\mathbb{R})$. However, if we restrict it to $\mathrm{SL}_{n}(\mathbb{R})$, it has the unitary axis at $\mu_{0}=\frac{n-d}{2}$. In addition, there exist complementary series for real $\mu$ in the interval of $\frac{n-d}{2}-1<\mu<\frac{n-d}{2}+1$ (see [HL99, § 3.5]).

Remark 6.6*.*

If you prefer the fractional linear action, we should make $y_{1}$ part to $1_{d}$. Thus we get

[TABLE]

and the fractional linear action is given by

[TABLE]

7. Intertwiners between degenerate principal series representations

In this section, we consider the following kernel function

[TABLE]

with complex parameters $\alpha,\beta\in\mathbb{C}$. Using this kernel, we aim at defining two integral kernel operators $\mathcal{P}$ and $\mathcal{Q}$, which intertwine degenerate principal series representations.

7.1. Kernel operator $\mathcal{P}$ from $\pi_{\nu}^{G}$ to $\pi_{\mu}^{L}$

In this subsection, we define an integral kernel operator $\mathcal{P}$ for $f\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ with compact support in $\Omega(p,q)$:

[TABLE]

where $d\omega(z)$ is an $L$-invariant measure on the open $L$-orbit $\Omega(p,q)\subset\Omega$. So the operator $\mathcal{P}$ depends on the parameters $p$ and $q$ as well as $\alpha$ and $\beta$.

For h=\Bigl{(}\begin{array}[]{@{\,}c@{\;\;}c@{\,}}{a}&{0}\\ {0}&{\,{}^{t}{a}\,^{-1}}\end{array}\Bigr{)}\in L and $f$ above, we have

[TABLE]

Thus, if $\nu=2\alpha$, we get an intertwiner. In this case, we have $\mathcal{P}f(yk)=|\det(k)|^{2\beta}\mathcal{P}f(y)$ so that

[TABLE]

if it is a $C^{\infty}$-function on $L/Q$. To get an intertwiner to $\pi_{\mu}^{L}$, we should have $2\beta=-\mu$.

As we observed

[TABLE]

For each $p,q$, the space $\mathcal{H}_{\mu}^{L}(p,q):=L^{2}(\Omega(p,q),(\det(1+z^{2}))^{\nu_{0}-\frac{n+1}{2}}dz)$ is a closed subspace of $\mathcal{H}_{\nu}^{G}$ and $L$-stable. From the decomposition of the base spaces, we get a direct sum decomposition of $L$-modules:

[TABLE]

Now we state one of the main theorem in this section.

Theorem 7.1**.**

Let $\nu_{0}:=\mathop{Re}\nolimits\nu,\;\mu_{0}:=\mathop{Re}\nolimits\mu$ and assume that they satisfy inequalities

[TABLE]

and

[TABLE]

Put $\alpha=\nu/2,\;\beta=-\mu/2$. Then the integral kernel operator $\mathcal{P}f$ defined in (7.2) converges and gives a bounded linear operator $\mathcal{P}:\mathcal{H}_{\nu}^{G}(p,q)\to\mathcal{H}_{\mu}^{L}$ which intertwines \pi_{\nu}^{G}\big{|}_{L} to $\pi_{\mu}^{L}$.

The rest of this subsection is devoted to prove the theorem above. Mostly we omit $p,q$ if there is no misunderstandings and we write $\nu,\mu$ instead of $\nu_{0},\mu_{0}$ in the following.

Let us evaluate the square of integral $|\mathcal{P}f(y)|^{2}$ point wise. The first evaluation is given by Cauchy-Schwartz inequality:

[TABLE]

where $dz$ is the Lebesgue measure on $\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\simeq\mathbb{R}^{\frac{n(n+1)}{2}}$, and we use $d\omega(z)=|\det z|^{-\frac{n+1}{2}}dz$. Since $\alpha=\nu/2$ and $\beta=-\mu/2$, the second integral becomes

[TABLE]

To evaluate the last integral, we use polar coordinates for $z$. Namely, we put $r:=\sqrt{\mathop{trace}\nolimits z^{2}}$ and write $z=r\Theta$. Then $\mathop{trace}\nolimits(\Theta^{2})=1$, and $\Omega^{\Theta}(p,q)=\overline{\Omega(p,q)}\cap\{\Theta\mid\mathop{trace}\nolimits(\Theta^{2})=1\}$ is compact. Using polar coordinates, we get $\det z=r^{n}\det\Theta$ and

[TABLE]

Also we note that $dz=r^{\frac{n(n+1)}{2}-1}drd\Theta$. Thus we get

[TABLE]

By the assumption (7.4), the integrand in the first integral over $\Omega^{\Theta}(p,q)$ is continuous, and converges. For the second, we separate it according as $r\downarrow 0$ or $r\uparrow\infty$.

If $r$ is near zero, the factor $\det(1+r^{2}\Theta^{2})$ is approximately $1$, so the integral converges if $\int_{0}^{1}r^{n\nu+d\mu-\frac{n(n+1)}{2}-1}dr$ converges. The first inequality in (7.3) guarantees the convergence.

On the other hand, if $r$ is large, the factor $\det(1+r^{2}\Theta^{2})$ is asymptotically $r^{2n}$, so the integral converges if $\int_{1}^{\infty}r^{n\nu+d\mu-\frac{n(n+1)}{2}-1-2n(\nu-\frac{n+1}{2})}dr$ converges. We use the second inequality in (7.3) to conclude the convergence.

Thus the integral (7.7) does converge, and the square root of it gives a bound for the operator norm of $\mathcal{P}$. We finished the proof of Theorem 7.1.

7.2. Kernel operator $\mathcal{Q}$ from $\pi_{\mu}^{L}$ to $\pi_{\nu}^{G}$

Similarly, we define $\mathcal{Q}F(z)$, for the moment, for $F(y)\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ by

[TABLE]

where $dy$ denotes the Lebesgue measure on $\mathrm{M}_{n,d}(\mathbb{R})$. We will update the definition of $\mathcal{Q}$ afterwards in (7.9), although we will check $L$-equivariance using this expression.

The integral (7.8) may diverge, but at least we can formally calculate as

[TABLE]

Thus, if $\chi_{P_{S}}(h)^{-1}=|\det a|^{2\alpha+d}$, we get an intertwiner. Here, we need a compatibility for the action, i.e., $F(yk)=|\det k|^{-\mu}F(y)\;\;(k\in\mathrm{GL}_{d}(\mathbb{R}))$ and we get

[TABLE]

From this we can see, if $\mu=2\beta+n$, the integrand (or measure) $F(y)K^{\alpha,\beta}(z,y)dy$ is defined over $\mathrm{M}^{\circ}_{n,d}(\mathbb{R})/\mathrm{GL}_{d}(\mathbb{R})\simeq\mathrm{O}(n)/\mathrm{O}(d){\times}\mathrm{O}(n-d)$. This last space is compact. Instead of this full quotient, we use the Stiefel manifold $S_{n,d}$ introduced in § 6.2 inside $\mathrm{M}_{n,d}(\mathbb{R})$. Thus, for $\alpha=-(\nu+d)/2$ and $\beta=(\mu-n)/2$, we redefine the intertwiner $\mathcal{Q}$ by

[TABLE]

where $d\sigma(y)$ denotes the $\mathrm{O}(n)$-invariant measure on $S_{n,d}$.

Theorem 7.2**.**

Let $\nu_{0}:=\mathop{Re}\nolimits\nu,\;\mu_{0}:=\mathop{Re}\nolimits\mu$ and assume that they satisfy inequalities

[TABLE]

and

[TABLE]

If $\alpha=-(\nu+d)/2$ and $\beta=(\mu-n)/2$, the integral kernel operator $\mathcal{Q}$ defined in (7.9) converges and gives an $L$-intertwiner $\mathcal{Q}:\mathcal{H}_{\mu}^{L}\to\mathcal{H}_{\nu}^{G}$.

Two remarks are in order. First, the inequalities (7.10) and (7.11) is “opposite” to the inequalities in Theorem 7.1. So $(\nu,\mu)$ does not share a common region for convergence. Second, the condition (7.11) in fact implies (7.10). However, we suspect the inequality (7.11) is too strong to ensure the convergence. So we leave them as they are.

Now let us prove the theorem. For brevity, we denote $\nu_{0},\mu_{0}$ by $\nu,\mu$ in the following.

Since $\alpha-\beta=-(\nu+d)/2-(\mu-n)/2=\frac{1}{2}(n-d-(\nu+\mu))\geq 0$ and $\beta=(\mu-n)/2\geq 0$, the kernel function $K^{\alpha,\beta}(z,y)$ is continuous. So the integral (7.9) converges. Let us check $\mathcal{Q}F(z)\in\mathcal{H}_{\nu}^{G}$ for $F\in\mathcal{H}_{\mu}^{L}$. By Cauchy-Schwarz inequality, we get

[TABLE]

Thus

[TABLE]

Since $\alpha=-(\nu+d)/2$ and $\beta=(\mu-n)/2$, the integral of square of the kernel is

[TABLE]

As in the proof of Theorem 7.1, we use polar coordinate $z=r\Theta$. Namely, we put $r:=\sqrt{\mathop{trace}\nolimits z^{2}}$ and write $z=r\Theta$. If we put $\Omega^{\Theta}=\{\Theta\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{R})\mid\mathop{trace}\nolimits(\Theta^{2})=1\}$, it is compact and $dz=r^{\frac{n(n+1)}{2}-1}drd\Theta$. Thus we get

[TABLE]

Since the integrand in the first integral over $\Omega^{\Theta}$ is continuous and hence converges. For the second, we separate it according as $r\downarrow 0$ or $r\uparrow\infty$ as in the proof of Theorem 7.1.

When $r$ is near zero, the integral converges if $\int_{0}^{1}r^{-(n\nu+d\mu)+\frac{n(n+1)}{2}-1}dr$ converges. The convergence follows from The first inequality in (7.10). When $r$ is large, the integral converges if $\int_{1}^{\infty}r^{-(n\nu+d\mu)+\frac{n(n+1)}{2}-1+2n(\nu-\frac{n+1}{2})}dr$ converges. We use the second inequality in (7.10) for the convergence.

This completes the proof of Theorem 7.2.

7.3. Finite dimensional representations

If $\alpha,\beta\in\mathbb{Z}$, we can naturally consider an algebraic kernel function

[TABLE]

without taking absolute value. By abuse of notation, we use the same symbol as before. Similarly we also consider algebraic characters

[TABLE]

if $\mu$ and $\nu$ are integers. In this setting the results in the above subsections are also valid.

We make use of Corollary 5.4 to deduce the facts on the image and kernels of integral kernel operators considered above.

Theorem 7.3**.**

*For nonnegative integers $m_{1}$ and $m_{2}$, we put $\alpha=m_{1}+m_{2},\beta=m_{2}$ and define $K^{\alpha,\beta}(z,y)$ as above.

• $(1)$

Put $\nu=-2(m_{1}+m_{2})-d$ and $\mu=2m_{2}+n$, and define the characters $\chi_{P_{S}}$ and $\chi_{Q}$ as above. Then $\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ contains the finite dimensional representation $V^{(n)}(2m_{1}\varpi_{n}+2m_{2}\varpi_{n-d})^{\ast}$ as an irreducible quotient. On the other hand, the representation $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ contains the same finite dimensional representation of $L$ as a subrepresentation, and $\mathcal{Q}$ intertwines these two representations. This subrepresentation is the same for any $p$ and $q$.

• $(2)$

Assume $2m_{1}\geq n+1$ and put $\nu=2(m_{1}+m_{2})$ and $\mu=-2m_{2}$. Define the characters $\chi_{P_{S}}$ and $\chi_{Q}$ as above. Then $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ contains the finite dimensional representation $V^{(n)}(2m_{2}\varpi_{d})^{\ast}$ of $L=\mathrm{GL}_{n}(\mathbb{R})$ as an irreducible quotient. On the other hand $\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ contains the same finite dimensional representation as a subrepresentation, and $\mathcal{P}$ intertwines these two representations. The intertwiners depend on $p$ and $q$, so there are at least $(n+1)$ different irreducible quotients which is isomorphic to $V^{(n)}(2m_{2}\varpi_{d})^{\ast}$, while the image in $\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$ is the same.

Proof.

This follows immediately from Corollary 5.4 and Theorems 7.1 and 7.2. Note that $2m_{1}\geq n+1$ is required for the convergence of the integral operator. ∎

The above result illustrates how knowledge about the geometry of a double flag variety and associated relative invariants may give information about the structure of parabolically induced representations, and in particular about some branching laws. Let us explain, that the branching laws in the above Theorem are consistent with other approaches to the structure of $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ in Theorem 7.3 (1).

Let us in the following remind about the connection between this induced representation, living on the Shilov boundary $S$ of the Hermitian symmetric space $G/K$, and the structure of holomorphic line bundles on this symmetric space. Let $\mathfrak{g}=\mathfrak{k}+\mathfrak{p}$ be a Cartan decomposition, and $\mathfrak{p}_{\mathbb{C}}=\mathfrak{p}^{+}\oplus\mathfrak{p}^{-}$ be a decomposition into irreducible representations of $K$. For holomorphic polynomials on the symmetric space we have the Schmid decomposition (see [FK94], XI.2.4) of the space of polynomials

[TABLE]

and the sum is over multi-indices ${\boldsymbol{a}}$ of integers with $\alpha_{1}\geq\alpha_{2}\geq\dots\geq\alpha_{n}\geq 0$, labeling (strictly speaking, here one chooses an order so that these are the negative of) $K$-highest weights $\alpha_{1}\gamma_{1}+\dots+\alpha_{n}\gamma_{n}$ with $\gamma_{1},\dots,\gamma_{n}$ Harish-Chandra strongly orthogonal non-compact roots. Now by restricting polynomials to the Shilov boundary $S$ we obtain an imbedding of the Harish-Chandra module corresponding to holomorphic sections of the line bundle with parameter $\nu$ in the parabolically induced representation on $S$ with the same parameter. For concreteness, recall: For $f\in C^{\infty}\!\!\text{-}\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$, the action of $\pi_{\nu}^{G}(g)$ on $f$ is given by

[TABLE]

When $\nu$ is an even integer, this is exactly the action in the (trivialized) holomorphic bundle, now valid for holomorphic functions of $z\in\mathop{\mathrm{Sym}}\nolimits_{n}(\mathbb{C})$. So if we can find parameters with a finite-dimensional invariant subspace in this Harish-Chandra module, then the same module will be an invariant subspace in $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$.

Recall that the maximal compact subgroup $K$ of $G$ has a complexification isomorphic to that of $L$, and the $G/K$ is a Hermitian symmetric space of tube type. Indeed, inside the complexified group $G_{\mathbb{C}}$ the two complexifications are conjugate. Hence if we consider a finite-dimensional representation of $G$ (or $G_{\mathbb{C}}$), then the branching law for each of these subgroups will be isomorphic.

For Hermitian symmetric spaces of tube type in general also recall the reproducing kernel (as in [FK94], especially Theorem XIII.2.4 and the notation there) for holomorphic sections of line bundles on $G/K$,

[TABLE]

and the sum is again over multi-indices ${\boldsymbol{a}}$ of integers with $\alpha_{1}\geq\alpha_{2}\geq\dots\geq\alpha_{n}\geq 0$. Here the functions $K^{\boldsymbol{a}}(z,w)$ are (suitably normalized) reproducing kernels of the $K$-representations $\mathcal{P}_{\boldsymbol{a}}(\mathfrak{p}^{+})$. The Pochhammer symbol is in terms of the scalar symbol in our case here

[TABLE]

Recall that for positive-definiteness of the above kernel, $\nu$ must belong to the so-called Wallach set; this means that the corresponding Harish-Chandra module is unitary and corresponds to a unitary reproducing-kernel representation of $G$ (or a double covering of $G$). Here the Wallach set is

[TABLE]

as in [FK94], XIII.2.7.

On the other hand, if $\nu$ is a negative integer, the Pochhammer symbols $(\nu)_{\boldsymbol{a}}$ vanishes when $\alpha_{1}>-\nu$. So this gives a finite sum in the formula (7.15) for the reproducing kernel corresponding to a finite-dimensional representation of $G$, and $\boldsymbol{a}$ labels the $K$-types occurring here as precisely those with $-\nu\geq\alpha_{1}$. By taking boundary values we obtain an imbedding of the $K$-finite holomorphic sections on $G/K$ to sections of the line bundle on $G/P_{S}$. Recalling that for our $G$ the Harish-Chandra strongly orthogonal non-compact roots are $2e_{j}$ in terms of the usual basis $e_{j}$, this means that the $L$-types in Theorem 7.3 (1) indeed occur. Namely, we may identify the parameters by the equation

[TABLE]

with the right-hand side of the form of a multi-index $\boldsymbol{a}$ satisfying $-\nu=2(m_{1}+m_{2})+d\geq\alpha_{1}$ as required above.

Thus we have seen, that there is consistency with the results about branching laws from $G$ to $K$ coming from considering finite-dimensional continuations of holomorphic discrete series representations, and on the other hand those branching laws from $G$ to $L$ coming from our study of relative invariants and intertwining operators from $\mathop{Ind}\nolimits_{P_{S}}^{G}\chi_{P_{S}}$ to $\mathop{Ind}\nolimits_{Q}^{L}\chi_{Q}$.