Characters of some unitary highest weight representations via the theta correspondence

TL;DR

This paper explicitly computes the characters of certain unitary highest weight representations in the context of the Howe correspondence for dual pairs in symplectic groups, focusing on their restrictions to regular points.

Contribution

It provides explicit character formulas for representations related by Howe correspondence in the setting of dual pairs with compact groups.

Findings

Explicit character restrictions on regular points of compact Cartan subgroups.

Detailed computations for representations in the Howe correspondence.

Enhanced understanding of unitary highest weight representations via theta correspondence.

Abstract

In this article, we consider a dual pair $(G, G')$ in the symplectic group $Sp(W)$ with $G$ compact and let $(\tilde{G}, \tilde{G}')$ be the preimages of $G$ and $G'$ in the metaplectic group $\widetilde{Sp(W)}$. For every irreducible representation $\Pi$ of $\tilde{G}$ appearing in Howe correspondence, we compute explicitly the restriction of the character $\Theta_{\Pi'}$ of the associated representation $\Pi'$ of $\tilde{G}'$ on the set of regular points on the compact Cartan subgroup $\tilde{H}'$ of $\tilde{G}'$.

Equations

$${\Theta }_{\Pi }(\exp (x))=\sum _{\sigma \in \mathcal{W}(\mathfrak{g},\mathfrak{h})}\epsilon (\sigma )\frac{e^{\sigma (\lambda +\rho )}}{\prod _{\alpha \in {\Phi }^{+}(\mathfrak{g},\mathfrak{h})}(e^{\frac{\alpha (x)}{2}}-e^{-\frac{\alpha (x)}{2}})}(x\in {\mathfrak{h}}_0),$$

$${\Theta }_{\Pi }:{\mathcal{C}}_c^{\infty }(G)\ni \Psi \mapsto \mathrm{tr}\Pi (\Psi )\in \mathbb{C}.$$

$${\Theta }_{\Pi }(\Psi )={\int }_G{\Theta }_{\Pi }(g)\Psi (g)d{\mu }_G(g).$$

$$\prod _{\alpha \in {\Phi }^{+}(\mathfrak{g},\mathfrak{h})}(e^{\frac{\alpha (x)}{2}}-e^{-\frac{\alpha (x)}{2}}){\Theta }_{\Pi }(\exp (x))=\sum _{\omega \in {\mathcal{W}}_{\lambda }^{\mathfrak{k}}}(-1{)}^{l_{\lambda }(\omega )}\Theta (K,\Lambda (\omega ,\lambda ))(\exp (x))(x\in {\mathfrak{h}}^{\mathrm{reg}}),$$

$$c_{-}^{\ast }{\Theta }_{{\Pi }^{\prime }}(\phi )=T(\overline{{\Theta }_{\Pi }})(\varphi )(\phi \in {\mathcal{C}}_c^{\infty }({\mathfrak{g}}^{\prime })),$$

$$\varphi (w)=\mathcal{F}(\phi \Theta \circ \tilde{c})\circ {\tau }_{{\mathfrak{g}}^{\prime }}$$

$${\Theta }_{{\Pi }^{\prime }}(h)=\sum _{\eta \in \mathcal{W}(G,\mathfrak{h},m)/\mathcal{W}(G,\mathfrak{h}{)}_m}\sum _{\sigma \in \mathcal{W}(K^{\prime },{\mathfrak{h}}^{\prime })}\frac{\mathrm{sgn}(\eta ){\mathrm{pr}}_m(\sigma h{)}^{-{\eta }^{-1}{\mu }^{\prime }}}{\prod _{\underset{{\beta }_{{|}_{{\mathfrak{h}}^{\prime }(m)}\ne 0}}{\beta \in {\Phi }^{+}({\mathfrak{g}}^{\prime },{\mathfrak{h}}^{\prime })}}((\sigma h{)}^{\frac{\beta }{2}}-(\sigma h{)}^{-\frac{\beta }{2}})}$$

$$\mathrm{Sp}(W)=\left\{g\in \mathrm{GL}(W),⟨g(w),g(w^{\prime })⟩=⟨w,w^{\prime }⟩,\forall w,w^{\prime }\in W\right\}$$

$$\mathfrak{sp}(W)=\left\{X\in \mathrm{End}(W),⟨X(w),w^{\prime }⟩+⟨w,X(w^{\prime })⟩=0,\forall w,w^{\prime }\in W\right\}.$$

$$\beta (J_{g}u,v)$$

$$\widetilde{\mathrm{Sp}(W)}=\left\{\tilde{g}=(g,\xi )\in \mathrm{Sp}(W)\times \mathbb{C},{\xi }^2=i^{{\dim }_{\mathbb{R}}(g-1)W}\det (J_g{)}_{J_g(W)}^{-1}\right\}.$$

$$(g_1,{\xi }_1).(g_2,{\xi }_2)=(g_1{g}_2,{\xi }_1{\xi }_{2}C(g_1,g_2)).$$

$$|C(g_1,g_2)|=\sqrt{\left|\frac{\det (J_{g_1}{)}_{J_{g_1}(W)}\det (J_{g_2}{)}_{J_{g_2}(W)}}{\det (J_{g_1{g}_2}{)}_{J_{g_1{g}_2}(W)}}\right|}(g_1,g_2\in \mathrm{Sp}(W)).$$

$${\chi }_{c(g)}(u)=\chi \left(\frac{1}{4}⟨c(g)u,u⟩\right)(u\in (g-1)W),$$

$$t(g)={\chi }_{c(g)}{\mu }_{(g-1)W},\Theta (\tilde{g})=\xi ,T(\tilde{g})=\Theta (\tilde{g})t(g).$$

$$\mathcal{K}(f)(x,x^{\prime })={\int }_{Y}f(x-x^{\prime }+y)\chi \left(\frac{1}{2}⟨y,x+x^{\prime }⟩\right)d{\mu }_Y(y)$$

$$\mathrm{Op}(K)v(x)={\int }_{X}K(x,x^{\prime })v(x^{\prime })d{\mu }_X(x^{\prime }).$$

$$\omega (\tilde{g})=\mathrm{Op}\circ \mathcal{K}\circ T(\tilde{g}).$$

$$\omega :\widetilde{\mathrm{Sp}(W)}\mapsto U(L^2(X))$$

$${\int }_{\widetilde{\mathrm{Sp}(W)}}\Theta (\tilde{g})\Psi (\tilde{g})d{\mu }_{\widetilde{\mathrm{Sp}(W)}}(\tilde{g})=\mathrm{tr}{\int }_{\widetilde{\mathrm{Sp}(W)}}\omega (\tilde{g})\Psi (\tilde{g})d{\mu }_{\widetilde{\mathrm{Sp}(W)}}(\tilde{g}).$$

$${\omega }^{\infty }=\underset{\Pi \in \mathcal{R}(\tilde{G},{\omega }^{\infty })}{⨁}\Pi \otimes {\Pi }^{\prime }.$$

$$\Pi \otimes {\Pi }^{\prime }\approx S(X)/\mathcal{N}.$$

$$(\Pi \otimes {\Pi }^{\prime }{)}^{\ast }\approx (S(X)/\mathcal{N}{)}^{\ast }\approx Ann(\mathcal{N})=\{f\in S^{\ast }(X),f(v)=0,(\forall v\in \mathcal{N})\}\subseteq S^{\ast }(X).$$

$${\Gamma }_{\Pi \otimes {\Pi }^{\prime }}\circ {\omega }^{\infty }(\tilde{g}{\tilde{g}}^{\prime })=\Pi (\tilde{g})\otimes {\Pi }^{\prime }({\tilde{g}}^{\prime })\circ {\Gamma }_{\Pi \otimes {\Pi }^{\prime }}(\tilde{g}\in \tilde{G},{\tilde{g}}^{\prime }\in {\tilde{G}}^{\prime }).$$

$${\mathcal{P}}_{\Pi \otimes {\Pi }^{\prime }}=\frac{1}{\dim (\Pi )}{\int }_{\tilde{G}}\overline{{\Theta }_{\Pi }(\tilde{g})}\omega (\tilde{g})d{\mu }_{\tilde{G}}(\tilde{g}).$$

$$f_{\Pi \otimes {\Pi }^{\prime }}={\int }_{\tilde{G}}\overline{{\Theta }_{\Pi }(\tilde{g})}T(\tilde{g})d{\mu }_{\tilde{G}}(\tilde{g})=T(\overline{{\Theta }_{\Pi }}),$$

$$\mathcal{F}(\Psi )(\eta )={\int }_{{\mathfrak{g}}^{\prime }}\Psi (\xi )\chi (\eta (\xi ))d\xi (\Psi \in S({\mathfrak{g}}^{\prime }),\eta \in {\mathfrak{g}}^{\prime \ast }),$$

$${\int }_{\widetilde{\mathrm{Sp}(W)}}\Psi (\tilde{g})d{\mu }_{\widetilde{\mathrm{Sp}(W)}}(\tilde{g})={\int }_{\mathfrak{sp}(W)}\Psi (\tilde{c}(x))j_{\mathfrak{sp}}(x)dx.$$

$${\tau }_{{\mathfrak{g}}^{\prime }}(w)(x^{\prime })=\frac{1}{4}⟨x^{\prime }(w),w⟩(w\in W,x^{\prime }\in {\mathfrak{g}}^{\prime \ast }).$$

$$({\tau }_{{\mathfrak{g}}^{\prime }}{)}_{\ast }(f)(\psi )=f(\psi \circ {\tau }_{{\mathfrak{g}}^{\prime }})(f\in S^{\ast }(W),\psi \in S({\mathfrak{g}}^{\prime \ast })).$$

Full text

1. 1 Introduction

2. 2 Weil representation and Howe’s duality theorem

3. 3 The intertwining distribution

4. 4 Fourier transform of a co-adjoint orbit

5. 5 Character formula for the pair $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{C}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm U}\nolimits(p,q,\mathbb{C}))$, $n\leq p+q$

6. 5.1 Harish-Chandra’s semisimple orbital integral

7. 5.2 The character $\Theta_{\Pi^{\prime}}$ on the compact diagonal Cartan subgroup

8. 6 The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2n,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2m,\mathbb{R}))$, $n\leq m$

9. 7 The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2n+1,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2m,\mathbb{R}))$, $n\leq m$

10. 8 The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{H}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm O}\nolimits^{*}(m,\mathbb{H}))$, $n\leq m$

11. A Dual pairs viewed as Lie supergroups

12. B Some comments concerning Theorem 5.11

13. B.1 How can we get the constant $C$?

14. B.2 A remark concerning the interval $[|m_{\min},m_{\max}|]$ for $\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(1,\mathbb{C})$

1. Introduction

One of the first results concerning characters of representations was established by Weyl around 1925. Let $\mathop{\rm G}\nolimits$ be a compact connected real Lie group and $\mathop{\rm H}\nolimits$ a maximal torus in $\mathop{\rm G}\nolimits$. We denote by $\mathfrak{h}_{0}$ and $\mathfrak{g}_{0}$ the Lie algebras of $\mathop{\rm H}\nolimits$ and $\mathop{\rm G}\nolimits$ respectively, and by $\mathfrak{h}$ and $\mathfrak{g}$ their complexifications. Let $\mathscr{W}(\mathfrak{g},\mathfrak{h})$ and $\Phi^{+}(\mathfrak{g},\mathfrak{h})$ be respectively the Weyl group and a choice of positive roots of $(\mathfrak{g},\mathfrak{h})$. We denote by $\varepsilon(\sigma)$ the signature of $\sigma\in\mathscr{W}(\mathfrak{g},\mathfrak{h})$. Weyl proved that, for an irreducible representation $(\Pi,V_{\Pi})$ of $\mathop{\rm G}\nolimits$, the character $\Theta_{\Pi}$ is given by

[TABLE]

where $\lambda\in\mathfrak{h}^{*}$ is the highest weight of $\Pi$ and $\rho=\frac{1}{2}\sum\limits_{\alpha\in\Phi^{+}(\mathfrak{g},\mathfrak{h})}\alpha$.

He also proved that the character determines the representation up to equivalence, i.e. two representations are equivalent if and only if their characters are equal.

In the early 1950’s, Harish-Chandra developed the notion of character for a certain class of representations of a real reductive group $\mathop{\rm G}\nolimits$, called quasi-simple [9, Section 1]. The character $\Theta_{\Pi}$ of a quasi-simple representation $(\Pi,\mathscr{H})$ of $\mathop{\rm G}\nolimits$ is a distribution on $\mathop{\rm G}\nolimits$, i.e. a well defined and continuous map

[TABLE]

He also proved that this distribution was given by a function, also denoted by $\Theta_{\Pi}$, on the open and dense subset of $\mathop{\rm G}\nolimits$ consisting of the so-called regular elements. More precisely, for all function $\Psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits)$, we get:

[TABLE]

Harish-Chandra also determined a character formula for the discrete series representations. This formula turns out to be formally very similar to Weyl’s formula (1); see [10] and [11].

Many other mathematicians have been interested in characters formulas. We can for example mention Kirillov [15], who proposed an integral formula for a general Lie group and Enright [8], who, using cohomological methods, established a formula for the character of a unitary highest weight module of a simple Lie group. Let us say some words on Enright’s work. Let $(\mathop{\rm G}\nolimits,\mathop{\rm K}\nolimits)$ be a hermitian symmetric pair with $\mathop{\rm G}\nolimits$ simple and such that $\mathop{\rm rk}\nolimits(\mathop{\rm K}\nolimits)=\mathop{\rm rk}\nolimits(\mathop{\rm G}\nolimits)$. Let $\mathop{\rm H}\nolimits$ be a maximal torus in $\mathop{\rm K}\nolimits$ and $(\Pi,\mathscr{H}_{\Pi})$ a representation of $\mathop{\rm G}\nolimits$ of highest weight $\lambda-\rho$. Then, the character $\Theta_{\Pi}$ is given by:

[TABLE]

where $\mathscr{W}^{\mathfrak{k}}_{\lambda}$ is defined in [8, Definition 2.1], and $\Theta(\mathop{\rm K}\nolimits,\Lambda(\omega,\lambda))(\mathop{\rm exp}\nolimits(x))$ is the character of a $\mathop{\rm K}\nolimits$-representation of highest weight $\Lambda(\omega,\lambda)$, with $\Lambda(\omega,\lambda)$ as in [8, Corollary 2.3].

In this article, we determine character formulas of unitary highest weight representations of non-compact reductive Lie groups, by using the theta correspondence. More precisely, we fix a dual pair $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})$ in the symplectic group $\mathop{\rm Sp}\nolimits(W)$ and we assume that $\mathop{\rm G}\nolimits$ is compact. Let $\tilde{\mathop{\rm G}\nolimits}$ and $\tilde{\mathop{\rm G}\nolimits}^{\prime}$ be the preimages of $\mathop{\rm G}\nolimits$ and $\mathop{\rm G}\nolimits^{\prime}$ in the metaplectic group $\mathop{\rm Sp}\nolimits(W)$ and let $\Pi\leftrightarrow\Pi^{\prime}$ be two representations appearing in the correspondence. By a result of Przebinda, we know that the pullback of the character $\Theta_{\Pi^{\prime}}$ via the Cayley transform is given by

[TABLE]

where $T:\widetilde{\mathop{\rm Sp}\nolimits(W)}\to\mathop{\rm S}\nolimits^{*}(W)$ is the embedding of $\widetilde{\mathop{\rm Sp}\nolimits(W)}$ in $\mathop{\rm S}\nolimits^{*}(W)$ given in section 2 and

[TABLE]

where $\mathscr{F}:\mathop{\rm S}\nolimits(\mathfrak{g}^{\prime})\mapsto\mathop{\rm S}\nolimits(\mathfrak{g}^{\prime*})$ is the Fourier transform defined in section 3.

In [19], the authors computed precisely the distribution $\mathop{\rm T}\nolimits(\overline{\Theta_{\Pi}})$ for the three compact dual pairs. With this distribution, they could compute the wave front set of the representation $\Pi^{\prime}$. Using these results, we determine in this paper the function $\Theta_{\Pi^{\prime}}$ on $\tilde{\mathop{\rm G}\nolimits}^{\prime\mathop{\rm reg}\nolimits}$, or more precisely on $\tilde{\mathop{\rm H}\nolimits}^{\prime\mathop{\rm reg}\nolimits}$, where $\mathop{\rm H}\nolimits^{\prime}$ is the compact Cartan subgroup of $\mathop{\rm G}\nolimits^{\prime}$.

This paper is organized as follows. In section 2, we recall the construction of the metaplectic representation as recently given by Aubert and Przebinda in [1], and quickly review Howe’s duality theorem. In section 3, we explain how the character of $\Pi^{\prime}$ can be obtained from the one of $\Pi$ using [20]. We then recall in section 4 the notion of Fourier transform of a coadjoint orbit, notion established by Rossmann for a regular parameter [25] and by Duflo and Vergne in the general case [7]. In section 5, we consider first the case $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{C}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm U}\nolimits(p,q,\mathbb{C}))$. The main result of this section is Theorem 5.11. In particular, the character $\Theta_{\Pi^{\prime}}$ is given by:

[TABLE]

We obtain similar formulas for the pair $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2m,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2n,\mathbb{R}))$ in section 6, $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2m+1,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2n,\mathbb{R}))$ in section 7 and for $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{H}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm O}\nolimits^{*}(m,\mathbb{H}))$ in section 8.

In this paper, we always work under the condition $\mathop{\rm rk}\nolimits(\mathop{\rm G}\nolimits)\leq\mathop{\rm rk}\nolimits(\mathop{\rm G}\nolimits^{\prime})$. Moreover, we assume that the character $\Theta_{\Pi^{\prime}}$ is supported in $\tilde{\mathop{\rm G}\nolimits}^{\prime}_{1}$, where $\mathop{\rm G}\nolimits^{\prime}_{1}$ is the Zariski identity component of $\mathop{\rm G}\nolimits^{\prime}$. In particular, this eliminates some representations of the even orthogonal group.

Acknowledgements: A part of this paper was done during my thesis at the University of Lorraine under the supervision of Angela Pasquale (University of Lorraine) and Tomasz Przebinda (University of Oklahoma). I would like to thank them for the ideas and time they shared with me. I also would like to thank the two referees of my thesis Hadi Salmasian (University of Ottawa) and Hung Yean Loke (National University of Singapore) for their suggestions. I finished this paper during my first months at the National University of Singapore, as a Research Fellow under the supervision of Hung Yean Loke.

2. Weil representation and Howe’s duality theorem

We start this section by recalling some main ideas of the construction of the metaplectic representation given in [1] (and by introducing some notation from [24]).

Fix a finite dimensional real vector space $W$ endowed with a symplectic non-degenerate form $\langle\cdot,\cdot\rangle$. We denote by $\mathop{\rm Sp}\nolimits(W)$ the corresponding symplectic group and by $\mathfrak{sp}(W)$ the Lie algebra, respectively defined as

[TABLE]

and

[TABLE]

Let $J\in\mathfrak{sp}(W)$ be an endomorphism satisfying $J^{2}=-\mathop{\rm Id}\nolimits_{W}$ and such that the symmetric bilinear form $\beta=\left<J\cdot,\cdot\right>$ is positive-definite (such an element is called a compatible positive complex structure on $W$). For all $g\in\mathop{\rm Sp}\nolimits(W)$, we denote by $J_{g}$ the automorphism of $W$ given by $J_{g}(w)=J^{-1}(g-1)$. For all $u,v\in W$, we have:

[TABLE]

Hence, the adjoint of $J_{g}$ with respect to the form $\beta$ is given by $J^{*}_{g}=Jg^{-1}(1-g)$. One can easily show that the restriction of $J_{g}$ to the space $\mathop{\rm Im}\nolimits(J_{g})=J_{g}(W)$ is well defined and invertible.

As in [1], let

[TABLE]

The multiplication on $\widetilde{\mathop{\rm Sp}\nolimits(W)}$ is given by

[TABLE]

where $C$ is a cocycle on $\mathop{\rm Sp}\nolimits(W)$ defined in [1, Proposition 4.13].

According to [24, equation (3)], the absolute value of $C$ is given by

[TABLE]

Fix a non trivial unitary character $\chi$ of $\mathbb{R}$ and let $W=X\oplus Y$ be a complete polarization of the symplectic space $W$. We denote by $\mathop{\rm S}\nolimits(X)$ and $\mathop{\rm S}\nolimits(W)$ the Schwartz space on $X$ and $W$ respectively and let $\mathop{\rm S}\nolimits^{*}(X)$ and $\mathop{\rm S}\nolimits^{*}(W)$ the corresponding spaces of tempered distributions.

For a subset $A$ of $\mathop{\rm End}\nolimits(W)$, we set $A^{c}=\{X\in A,\mathop{\rm det}\nolimits(X-1)\neq 0\}$. The Cayley transform is well defined on $\mathfrak{sp}^{c}(W)$ and $c(\mathfrak{sp}^{c}(W))=\mathop{\rm Sp}\nolimits^{c}(W)$. For all $g\in\mathop{\rm Sp}\nolimits(W)$, we define the function $\chi_{c(g)}:(g-1)W\mapsto\mathbb{C}$ by

[TABLE]

and let $t:\mathop{\rm Sp}\nolimits(W)\mapsto\mathop{\rm S}\nolimits^{*}(W)$, $\Theta:\widetilde{\mathop{\rm Sp}\nolimits(W)}\mapsto\mathbb{C}$ and $T:\widetilde{\mathop{\rm Sp}\nolimits(W)}\mapsto\mathop{\rm S}\nolimits^{*}(W)$ be the maps defined respectively by:

[TABLE]

where $\tilde{g}=(g,\xi)\in\widetilde{\mathop{\rm Sp}\nolimits(W)}$ and $\mu_{(g-1)W}\in\mathop{\rm S}\nolimits^{*}(W)$ is the Lebesgue measure on the space $(g-1)W$ normalized with respect to the form $\beta$ such that the volume of the associated unit cube is $1$.

The Weyl Transform $\mathscr{K}:\mathop{\rm S}\nolimits(W)\mapsto\mathop{\rm S}\nolimits(X\times X)$ defined by

[TABLE]

is an isomorphism of linear topological spaces. Its natural extension, also denoted by $\mathscr{K}$, to the space of tempered distributions is still an isomorphism. Recall the map $\mathop{\rm Op}\nolimits:\mathop{\rm S}\nolimits(X\times X)\mapsto\mathop{\rm Hom}\nolimits(\mathop{\rm S}\nolimits(X),\mathop{\rm S}\nolimits(X))$ given by

[TABLE]

By the Schwartz Kernel Theorem, its extension $\mathop{\rm Op}\nolimits:\mathop{\rm S}\nolimits^{*}(X\times X)\mapsto\mathop{\rm Hom}\nolimits(\mathop{\rm S}\nolimits(X),\mathop{\rm S}\nolimits^{*}(X))$ is an isomorphism of topological vector spaces.

Finally, we get a map $\omega:\widetilde{\mathop{\rm Sp}\nolimits(W)}\mapsto\mathop{\rm Hom}\nolimits(\mathop{\rm S}\nolimits(X),\mathop{\rm S}\nolimits^{*}(X))$ defined by

[TABLE]

As explained in [1, Section 4.8], the restriction $\mathop{\rm Op}\nolimits\circ\mathscr{K}:L^{2}(W)\to\mathop{\rm H.S}\nolimits(L^{2}(X))$, where $\mathop{\rm H.S}\nolimits(L^{2}(X))$ is the space of Hilbert Schmidt operators on $L^{2}(X)$ is an isometry and the map

[TABLE]

is a unitary representation of the metaplectic group, often called Weil representation, or metaplectic representation ([1, Section 4.8, Theorem 4.27]). Moreover, the function $\Theta$ defined as above coincides with the character of $\omega$; it means that for all $\Psi\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm Sp}\nolimits(W)})$, the following equality holds

[TABLE]

One can show that the Garding space of this representation is exactly the space of Schwartz functions on $X$.

By a reductive dual pair in $\mathop{\rm Sp}\nolimits(W)$, we mean a pair of subgroups $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})$ of $\mathop{\rm Sp}\nolimits(W)$ which act reductively on the symplectic space $W$ and are mutually centralizers in $\mathop{\rm Sp}\nolimits(W)$. The pair $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})$ is said irreducible if there is no orthogonal decomposition $W=W_{1}\oplus W_{2}$ with respect to the symplectic form $\langle\cdot,\cdot\rangle$ such that $W_{1}$ and $W_{2}$ are both $\mathop{\rm G}\nolimits.\mathop{\rm G}\nolimits^{\prime}$-invariants. A classification of such pairs was done by Howe. In the next sections of this article, we will be interested in irreducible reductive dual pairs with one member compact. We recall here the three cases in which we will be interested in:

• •

$(\mathop{\rm O}\nolimits(n,\mathbb{R}),\mathop{\rm Sp}\nolimits(2m,\mathbb{R}))\subseteq\mathop{\rm Sp}\nolimits(2nm,\mathbb{R})$,

• •

$(\mathop{\rm U}\nolimits(n,\mathbb{C}),\mathop{\rm U}\nolimits(p,q,\mathbb{C}))\subseteq\mathop{\rm Sp}\nolimits(2n(p+q),\mathbb{R})$,

• •

$(\mathop{\rm U}\nolimits(n,\mathbb{H}),\mathop{\rm O}\nolimits^{*}(m,\mathbb{H}))\subseteq\mathop{\rm Sp}\nolimits(4nm,\mathbb{R})$.

For a reductive dual pair $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})\in\mathop{\rm Sp}\nolimits(W)$, we denote by $\tilde{\mathop{\rm G}\nolimits}$ and $\tilde{\mathop{\rm G}\nolimits}^{\prime}$ the preimages of $\mathop{\rm G}\nolimits$ and $\mathop{\rm G}\nolimits^{\prime}$ respectively in $\widetilde{\mathop{\rm Sp}\nolimits(W)}$.

For any subgroup $\tilde{\mathop{\rm H}\nolimits}$ of $\widetilde{\mathop{\rm Sp}\nolimits(W)}$, we denote by $\mathscr{R}(\tilde{\mathop{\rm H}\nolimits})$ the set of infinitesimal equivalence classes of continuous irreducible admissible representations of $\tilde{\mathop{\rm H}\nolimits}$ on locally convex topological vector spaces. Let $\mathscr{R}(\tilde{\mathop{\rm H}\nolimits},\omega^{\infty})$ be the subset of $\mathscr{R}(\tilde{\mathop{\rm H}\nolimits})$ of representations which can be realized as a quotient of $\mathop{\rm S}\nolimits(X)$ by a closed $\omega^{\infty}(\tilde{\mathop{\rm H}\nolimits})$-invariant subspace (here, $\omega^{\infty}$ is the $\mathscr{C}^{\infty}$-representation of $(\omega,L^{2}(X))$ realized on the space $\mathop{\rm S}\nolimits(X)$).

According to [13, Theorem 1], for any reductive dual pair $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})\in\mathop{\rm Sp}\nolimits(W)$, there is a bijective correspondence between $\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ and $\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ having graph $\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}.\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$.

Remark 2.1*.*

Assume that $\mathop{\rm G}\nolimits$ is compact. Let $(\Pi,V_{\Pi})$ be an irreducible (unitary) representation of $\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$. Then, $(\Pi,V_{\Pi})$ is a subrepresentation of $(\omega^{\infty},\mathop{\rm S}\nolimits(X))$ and in particular, its isotypic component $V(\Pi)$ is a closed subspace of $\mathop{\rm S}\nolimits(X)$. Let $(\Pi^{\prime},V_{\Pi^{\prime}})$ be the associated representation in $\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$. Using Howe’s duality theorem, we get that $V(\Pi)=V(\Pi^{\prime})=V(\Pi\otimes\Pi^{\prime})\subseteq\mathop{\rm S}\nolimits(X)$, where $V(\Pi^{\prime})$ is the $\Pi^{\prime}$-isotypic component. In particular,

[TABLE]

where here the sum is not an algebraic sum but the closure of the algebraic sum with respect to the topology of $\mathop{\rm S}\nolimits(X)$.

3. The intertwining distribution

We start this section by recalling the notion of intertwining distribution introduced by Przebinda in [20].

Let $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})$ be an irreducible reductive dual pair in $\mathop{\rm Sp}\nolimits(W)$, $\Pi\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ and $\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ such that $\Pi\otimes\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}.\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$. There exists a closed $\omega^{\infty}(\tilde{\mathop{\rm G}\nolimits}.\tilde{\mathop{\rm G}\nolimits}^{\prime})$-invariant subspace $\mathscr{N}$ of $\mathop{\rm S}\nolimits(X)$ such that

[TABLE]

We denote by $(\Pi\otimes\Pi^{\prime})^{*}$ the contragredient representation of $\Pi\otimes\Pi^{\prime}$. We get:

[TABLE]

Choosing $\Pi^{*}$ and $\Pi^{\prime*}$ instead of $\Pi$ and $\Pi^{\prime}$, we obtain that the representation $\Pi\otimes\Pi^{\prime}$ can be realized in $\mathop{\rm S}\nolimits^{*}(X)$. Let $\mathop{\rm S}\nolimits^{*}(X)_{\Pi\otimes\Pi^{\prime}}$ be the $\Pi\otimes\Pi^{\prime}$-isotypic component. By Howe’s duality theorem, we get a unique, up to a constant, element $\Gamma_{\Pi\otimes\Pi^{\prime}}\in\mathop{\rm Hom}\nolimits_{\tilde{\mathop{\rm G}\nolimits}.\tilde{\mathop{\rm G}\nolimits}^{\prime}}(\mathop{\rm S}\nolimits(X),\mathop{\rm S}\nolimits^{*}(X))$ such that the following equation holds

[TABLE]

Using the isomorphisms $\mathscr{K}$ and $\mathop{\rm Op}\nolimits$ defined in the section 2, there exists a distribution $f_{\Pi\otimes\Pi^{\prime}}\in\mathop{\rm S}\nolimits^{*}(W)$ verifying $\Gamma_{\Pi\otimes\Pi^{\prime}}=\mathop{\rm Op}\nolimits\circ\mathscr{K}(f_{\Pi\otimes\Pi^{\prime}})$. The element $f_{\Pi\otimes\Pi^{\prime}}\in\mathop{\rm S}\nolimits^{*}(W)$ is called the intertwining distribution in [20, Section 5].

Remark 3.1*.*

If $\mathop{\rm G}\nolimits$ is compact, according to remark 20, we get that the $\Pi\otimes\Pi^{\prime}$-isotypic component is a closed subspace of $\mathop{\rm S}\nolimits(X)$. We denote by $\mathscr{P}_{\Pi\otimes\Pi^{\prime}}$ the projection onto the isotypic component, i.e. the map $\mathscr{P}_{\Pi\otimes\Pi^{\prime}}:\mathop{\rm S}\nolimits(X)\mapsto\mathop{\rm S}\nolimits(X)_{\Pi\otimes\Pi^{\prime}}$ given by:

[TABLE]

Then, $\Gamma_{\Pi\otimes\Pi^{\prime}}=\mathscr{P}_{\Pi\otimes\Pi^{\prime}}$.

For now on, we assume that the group $\mathop{\rm G}\nolimits$ is compact. Using [20, Lemma 5.4], we get that the distribution $f_{\Pi\otimes\Pi^{\prime}}$ is given by the formula

[TABLE]

where the function $\Theta_{\Pi}:\tilde{\mathop{\rm G}\nolimits}\to\mathbb{C}$ is the character of $\Pi$. Denote by $\Theta_{\Pi^{\prime}}$ the distribution character of $\Pi^{\prime}$ and the function defined on the regular points on $\tilde{\mathop{\rm G}\nolimits}^{\prime}$. This distribution can be expressed via $f_{\Pi\otimes\Pi^{\prime}}$ ([20, Theorem 6.7]). To explain this link, we’ll use the notations of [20]. Denote by $\mathscr{F}:\mathop{\rm S}\nolimits(\mathfrak{g}^{\prime})\to\mathop{\rm S}\nolimits(\mathfrak{g}^{\prime*})$ the Fourier transform defined as

[TABLE]

and let $\mathscr{F}^{*}:\mathop{\rm S}\nolimits^{*}(\mathfrak{g}^{\prime*})\to\mathop{\rm S}\nolimits^{*}(\mathfrak{g}^{\prime})$ be the dual map.

As explained in [20, Section 3], for a fixed element $\widetilde{(-1)}\in\widetilde{\mathop{\rm Sp}\nolimits(W)}$ in the preimage of $(-1)\in\mathop{\rm Sp}\nolimits(W)$, there exists a unique map $\tilde{c}:\mathfrak{sp}(W)\to\widetilde{\mathop{\rm Sp}\nolimits^{c}(W)}$ such that $\tilde{c}(0)=\widetilde{(-1)}$ and $\tilde{c}\circ\pi=c$.

We denote by $j_{\mathfrak{sp}}$ the function defined of the domain of $\tilde{c}$ which satisfies

[TABLE]

for all functions $\Psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm Sp}\nolimits^{c}(W))$ such that $\mathop{\rm supp}\nolimits(\Psi)\subseteq\mathop{\rm Im}\nolimits(\tilde{c})$. Denote by $j_{\mathfrak{g}}$ and $j_{\mathfrak{g}^{\prime}}$ the associated maps on $\mathfrak{g}$ and $\mathfrak{g}^{\prime}$ respectively, and let $\tau_{\mathfrak{g}^{\prime}}:W\to\mathfrak{g}^{\prime*}$ be the map defined by

[TABLE]

As proved in [20, Lemma 6.1], the pullback of $\tau_{\mathfrak{g}^{\prime}}$, from $\mathop{\rm S}\nolimits(\mathfrak{g}^{\prime*})$ onto $\mathop{\rm S}\nolimits(W)$, given by $\psi\to\psi\circ\tau_{\mathfrak{g}^{\prime}}$, is well defined and continuous. By dualization, we get a map $(\tau_{\mathfrak{g}^{\prime}})_{*}:\mathop{\rm S}\nolimits^{*}(W)\to\mathop{\rm S}\nolimits^{*}(\mathfrak{g}^{\prime*})$, defined by

[TABLE]

In [20, Theorem 6.7], Przebinda proved the following result:

[TABLE]

where $c_{-}(x)=\tilde{c}(x)\tilde{c}(0)^{-1}$, $\tilde{c}^{*}_{-}\Theta_{\Pi^{\prime}}$ is the pullback of $\Theta_{\Pi^{\prime}}$ via $\tilde{c}_{-}$ and $K_{\Pi}\in\mathbb{C}$ is a constant.

Remark 3.2*.*

Here we explain formally where this formula comes from. For every function $\Psi\in\mathscr{C}^{\infty}_{c}(\tilde{\mathop{\rm G}\nolimits}^{\prime})$, the distribution character is given by the following formula

[TABLE]

where $d_{\Pi}=\mathop{\rm dim}\nolimits(V_{\Pi})$. As shown in [1, Section 4.8], we have:

[TABLE]

where $\natural$ is the twisted convolution of distributions defined in [1, Section 4.5 and Lemma 4.24]. Using [1, Section 4.8], we get:

[TABLE]

Finally, for all $\Psi\in\mathscr{C}^{\infty}_{c}(\tilde{\mathop{\rm G}\nolimits}^{\prime})$, we get:

[TABLE]

Even though $\mathop{\rm Im}\nolimits(T)\subseteq\mathop{\rm S}\nolimits^{*}(W)$, $T(\Psi)$ is a Schwartz function on $W$. In [19], the authors computed $T(\overline{\Theta_{\Pi}})(\phi)$ for all compact dual pairs in $\mathop{\rm Sp}\nolimits(W)$, where $\phi$ is a Schwartz function on $W$. We will use their results for a particular function $\phi\in\mathop{\rm S}\nolimits(W)$ to get an explicit formula for the function $\Theta_{\Pi^{\prime}}$ on $\tilde{\mathop{\rm H}\nolimits}^{\prime\mathop{\rm reg}\nolimits}$, where $\mathop{\rm H}\nolimits^{\prime}$ is a compact Cartan subgroup of $\mathop{\rm G}\nolimits^{\prime}$.

4. Fourier transform of a co-adjoint orbit

To recall the concept of Fourier transform of a co-adjoint orbit, we use [2, Chapter 7, Section 5]. For a compact group, this result is essentially due to Harish-Chandra and Kirillov. For a general semi-simple Lie group, Rossmann [25] established a formula for a regular semi-simple element in $\mathfrak{g}^{*}$. This result had been generalized by M. Duflo and M. Vergne in [7] for every $\lambda\in\mathfrak{g}^{*}$.

For a semi-simple Lie group $\mathop{\rm G}\nolimits$ with maximal compact subgroup $\mathop{\rm K}\nolimits$ satisfying $\mathop{\rm rk}\nolimits(\mathop{\rm K}\nolimits)=\mathop{\rm rk}\nolimits(\mathop{\rm G}\nolimits)$, denote by $\mathop{\rm Ad}\nolimits:\mathop{\rm G}\nolimits\to\mathop{\rm GL}\nolimits(\mathfrak{g})$ the adjoint action and by $\mathop{\rm Ad}\nolimits^{*}$ the associated action of $\mathop{\rm G}\nolimits$ on $\mathfrak{g}^{*}$. For $\lambda\in\mathfrak{g}^{*}$, we note by $\mathop{\rm G}\nolimits_{\lambda}=\mathop{\rm Ad}\nolimits^{*}(\mathop{\rm G}\nolimits)(\lambda)$ the $\mathop{\rm G}\nolimits$-orbit associated to $\lambda$. Let $\mathop{\rm H}\nolimits$ be a maximal torus in $\mathop{\rm K}\nolimits$ (which is a maximal torus in $\mathop{\rm G}\nolimits$ because of the rank equality), and fix $\Phi(\mathfrak{k},\mathfrak{h})$ and $\Phi(\mathfrak{g},\mathfrak{h})$ the roots corresponding to $(\mathfrak{k},\mathfrak{h})$ and $(\mathfrak{g},\mathfrak{h})$ respectively. Let $(\cdot,\cdot)$ be a $\mathop{\rm G}\nolimits$-invariant symmetric non-degenerate form on $\mathfrak{g}$. More precisely, for each $\alpha\in\mathfrak{h}^{*}$, there exists a unique element $H_{\alpha}\in\mathfrak{h}$ such that $\alpha(h)=(H_{\alpha},h)$ for all $h\in\mathfrak{h}$.

For $\lambda\in\mathfrak{h}^{*}$, we denote by $\mathop{\rm P}\nolimits_{\lambda}$ the subset of $\Phi(\mathfrak{g},\mathfrak{h})$ defined by

[TABLE]

We denote by $d\beta_{\lambda}$ the Liouville measure on $\mathop{\rm G}\nolimits_{\lambda}$. According to [6, page 170], this measure is given by the following formula

[TABLE]

where $\mathop{\rm G}\nolimits^{\lambda}$ is the stabilizer of $\lambda$. Here, $g\lambda$ means $\mathop{\rm Ad}\nolimits^{*}(g)(\lambda)$. The Fourier transform of $\mathop{\rm G}\nolimits_{\lambda}$, is the generalized function on $\mathfrak{g}$

[TABLE]

Using [2, Theorem 7.29], for all $X\in\mathfrak{h}^{\mathop{\rm reg}\nolimits}$, we get:

[TABLE]

where $n(\lambda)$ is the number of non-compact roots of $\mathop{\rm P}\nolimits_{\lambda}$, $\mathscr{W}(\mathfrak{k},\mathfrak{h})=\left<s_{\alpha},\alpha\in\Phi(\mathfrak{k},\mathfrak{h})\right>$ is the compact Weyl group and $\mathscr{W}(\mathfrak{k},\mathfrak{h})^{\lambda}$ is the stabilizer of the $\lambda$ under the action of $\mathscr{W}(\mathfrak{k},\mathfrak{h})$:

[TABLE]

Using equations (35) and (37), we get:

[TABLE]

The equality (39) is what we call the Rossmann-Duflo-Vergne formula.

Remark 4.1*.*

1. (1)

Assume that $\mathop{\rm G}\nolimits$ is compact and connected, and let $(\Pi,V_{\Pi})$ be an irreducible representation of Harish-Chandra parameter $\lambda\in\mathfrak{h}^{*}$. Then, $\lambda$ is regular and using equation (37), we get the classical Weyl character formula. 2. (2)

We assume that $\mathop{\rm G}\nolimits$ is semi-simple. In particular, the Killing form on $\mathfrak{g}$ is non-degenerate, which allows us to identify $\mathfrak{g}$ with its dual $\mathfrak{g}^{*}$. We denote by $(\cdot,\cdot)$ the Killing form on $\mathfrak{g}$. Then, the equation (39) can be written as:

[TABLE]

where $s\in\mathfrak{h}^{\mathop{\rm reg}\nolimits}$, $t\in\mathfrak{h}$ and $\mathop{\rm P}\nolimits_{t}=\left\{\alpha\in\Phi(\mathfrak{g},\mathfrak{h})\thinspace|\thinspace i\alpha(t)>0\right\}$.

5. Character formula for the pair $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{C}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm U}\nolimits(p,q,\mathbb{C}))$, $n\leq p+q$

Let $V_{\overline{0}}$ be a complex vector space of dimension $n$ over $\mathbb{C}$ endowed with a definite-positive hermitian form $b_{0}$. We fix a basis $\mathscr{B}=\{v_{1},\ldots,v_{n}\}$ of $V_{\overline{0}}$ such that the matrix of $b_{0}$ with respect to $\mathscr{B}$ is $\mathop{\rm Mat}\nolimits(b_{0},\mathscr{B})=\mathop{\rm Id}\nolimits_{n}$, and let $\mathop{\rm U}\nolimits(V_{\overline{0}},b_{0})$ be the group of isometries of the form $b_{0}$, i.e.

[TABLE]

We denote by $\mathfrak{g}_{0}=\mathfrak{u}(V_{\overline{0}},b_{0})$ the Lie algebra of $\mathop{\rm U}\nolimits(V_{\overline{0}},b_{0})$ given by

[TABLE]

Writing the endomorphism $X$ in the basis $\mathscr{B}$, the Lie algebra can be realized as:

[TABLE]

Let $\mathfrak{h}_{0}$ the diagonal subalgebra of $\mathfrak{u}(V_{\overline{0}},b_{0})$ given by:

[TABLE]

and we denote by $\mathfrak{h}$ and $\mathfrak{g}$ their complexifications. The roots of $\mathfrak{g}$ with respect to $\mathfrak{h}$ are given by:

[TABLE]

where the linear form $e_{k}$ is given by

[TABLE]

We denote by $\pi_{\mathfrak{g}/\mathfrak{h}}$ the product of all positive roots:

[TABLE]

For all $h=\sum\limits_{k=1}^{n}ih_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}$, we get:

[TABLE]

Now, we consider a complex vector space $V_{\overline{1}}$ endowed with a non-degenerate hermitian form $b_{1}$ of signature $(p,q)$. Fix a basis $\mathscr{B}^{\prime}=\{w_{1},\ldots,w_{p+q}\}$ of $V_{\overline{1}}$ such that $\mathop{\rm Mat}\nolimits(b_{1},\mathscr{B}^{\prime})=i\mathop{\rm Id}\nolimits_{p,q}$, where $\mathop{\rm Id}\nolimits_{p,q}=\begin{pmatrix}\mathop{\rm Id}\nolimits_{p}&0\\ 0&-\mathop{\rm Id}\nolimits_{q}\end{pmatrix}$ and we denote by $\mathop{\rm U}\nolimits(V_{\overline{1}},b_{1})$ the group of isometries of the form $b_{1}$, i.e.

[TABLE]

By writing the endomorphisms $X$ in the basis $\mathscr{B}^{\prime}$, we get the following realization for the Lie algebra

[TABLE]

Again, we consider the diagonal subspace $\mathfrak{h}^{\prime}_{0}$ of $\mathfrak{u}(V_{\overline{1}},b_{1})$

[TABLE]

and let $\Phi(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})$ be the set of roots of $\mathfrak{g}^{\prime}$ given by:

[TABLE]

Let $\mathop{\rm K}\nolimits=\mathop{\rm U}\nolimits(p,\mathbb{C})\times\mathop{\rm U}\nolimits(q,\mathbb{C})$ be the maximal compact subgroup of $\mathop{\rm U}\nolimits(p,q,\mathbb{C})$, $\mathfrak{k}_{0}$ the Lie algebra of $\mathop{\rm K}\nolimits$ and $\mathfrak{k}$ its complexification. The set of compact roots of $\mathfrak{g}^{\prime}$ are given by:

[TABLE]

For all $h^{\prime}=\sum\limits_{k=1}^{p+q}ih_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}^{\prime}$, we have

[TABLE]

and

[TABLE]

We now use the correspondence between dual pairs and Lie supergroups. Let $V=V_{\overline{0}}\oplus V_{\overline{1}}$, $b=b_{0}\oplus b_{1}$ as defined in appendix A, and let $(\mathop{\rm S}\nolimits,\mathfrak{s}=\mathfrak{s}(V,b))$ be the associated supergroup.

Lemma 5.1**.**

An element $X=\begin{pmatrix}0&X_{1}\\ X_{2}&0\end{pmatrix}$ is in $\mathfrak{s}(V,b)_{\overline{1}}$ if and only if $X_{2}=-i\mathop{\rm Id}\nolimits_{p,q}\bar{X_{1}}^{t}$.

Proof.

As explained in appendix A, an element $X$ is in $\mathfrak{s}(V,b)_{\overline{1}}$ if and only if $b(X(u),v)-b(u,sX(v))=0$ for all $u=(u_{0},u_{1}),v=(v_{0},v_{1})$. We have:

[TABLE]

In particular,

[TABLE]

We get the result by taking $v_{0}=0$ and remarking that $\mathop{\rm Id}\nolimits^{-1}_{p,q}=\mathop{\rm Id}\nolimits_{p,q}$.

∎

We define $V^{j}_{\overline{0}}=\mathbb{C}u_{j}$ and $V^{j}_{\overline{1}}=\mathbb{C}w_{j}$. Then,

[TABLE]

and $V=\left(\bigoplus\limits_{k=1}^{n}V^{k}_{\overline{0}}\oplus V^{k}_{\overline{1}}\right)\oplus\bigoplus\limits_{k=n+1}^{p+q}V^{k}_{\overline{1}}$.

Now, we fix an integer $m\in[|\max(n-q,0),\min(p,n)|]$. We define the endomorphisms $u_{j}\in\mathfrak{s}(V,b)_{\overline{1}}$ by:

[TABLE]

[TABLE]

with $u_{j}(w_{m+1})=\ldots=u_{j}(w_{p+q-n+m})=0$. Let $\mathfrak{h}_{1,m}$ be the subspace of $\mathfrak{s}(V,b)_{\overline{1}}$ given by

[TABLE]

We define $\tau:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{g}$ and $\tau^{\prime}:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{g}^{\prime}$ by

[TABLE]

For all $w=\sum\limits_{k=1}^{n}h_{k}u_{k}\in\mathfrak{h}_{1,m}$, we have:

[TABLE]

In particular, we get an injection of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$, depending on $m$, given by:

[TABLE]

We denote by $\mathfrak{h}^{\prime}(m)$ the image of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ via the map (63). Let $\mathfrak{z}^{\prime}(m)=C_{\mathfrak{g}^{\prime}}\mathfrak{h}^{\prime}(m)$ and

[TABLE]

We denote by $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ the Weyl group of $(\mathfrak{g},\mathfrak{h})$. It acts on the elements $h=\sum\limits_{k=1}^{n}ih_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}$ by permutations of the components $\left\{h_{k},1\leq i\leq n\right\}$. We will use the same symbol to indicate the Weyl group element and the corresponding permutation in $\mathscr{S}_{n}$, i.e.

[TABLE]

Similarly, for $h^{\prime}=\sum\limits_{k=1}^{p+q}ih^{\prime}_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}^{\prime}$, the Weyl group $\mathscr{W}(\mathop{\rm G}\nolimits^{\prime},\mathfrak{h}^{\prime})$ of $(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})$ permutes the components $h^{\prime}_{1},\ldots,h^{\prime}_{p+q}$. Moreover, we let $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$ be the compact Weyl group, i.e. the subgroup of $\mathscr{W}(\mathop{\rm G}\nolimits^{\prime},\mathfrak{h}^{\prime})$ which acts on the sets $\left\{h^{\prime}_{i},1\leq i\leq p\right\}$ and $\left\{h^{\prime}_{i},p+1\leq i\leq p+q\right\}$.

For every $m\in[|\max(n-q,0),\min(p,n)|]$ and $h=\sum\limits_{i=1}^{n}ih_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}$, we denote by $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})_{m}$ the subgroup of $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ which acts on the sets $\left\{h_{i},1\leq i\leq m\right\}$ and $\left\{h_{i},m+1\leq i\leq n\right\}$, and by $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})_{m}$ the subgroup of $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$ which permutes the elements of $\left\{h^{\prime}_{i},1\leq i\leq m\right\}$ and $\left\{h^{\prime}_{p+q+m-n+i},1\leq i\leq n-m\right\}$ separately.

Finally, for all $x,y\in\mathfrak{g}$ or $\mathfrak{g}^{\prime}$, we denote by $B$ the bilinear form defined by

[TABLE]

Remark 5.2*.*

The form $B$ is $\mathop{\rm G}\nolimits$ (resp. $\mathop{\rm G}\nolimits^{\prime}$)-invariant and non-degenerate on $\mathfrak{g}$ (resp. $\mathfrak{g}^{\prime}$). More precisely, for all $x=\sum\limits_{k=1}^{n}ix_{k}\mathop{\rm E}\nolimits_{k,k}$ (resp. $x^{\prime}=\sum\limits_{k=1}^{p+q}ix^{\prime}_{k}\mathop{\rm E}\nolimits_{k,k}$) and $y=\sum\limits_{k=1}^{n}iy_{k}\mathop{\rm E}\nolimits_{k,k}$ (resp. $y^{\prime}=\sum\limits_{k=1}^{p+q}iy^{\prime}_{k}\mathop{\rm E}\nolimits_{k,k}$), the form $B$ is given by

[TABLE]

See [19, page 49-50].

5.1. Harish-Chandra’s semisimple orbital integral

As introduced in [19, Section 4, Definition 10], for every function $\phi\in\mathop{\rm S}\nolimits(W)$, we define the function $f_{\phi}$ on $\tau(\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1,m})$ by

[TABLE]

where $C_{\mathfrak{h}_{1,m}}$ is the constant of modulus 1 given in [19, Lemma 8, page 17], which, a priori, depends on $m$.

Lemma 5.3**.**

The constant $C_{\mathfrak{h}_{1,m}}$ does not depend on $m$.

Proof.

As explained in [19], the constant $C_{\mathfrak{h}_{1,m}}$ satisfies, for all $w=\sum\limits_{k=1}^{n}w_{k}u_{k}\in\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1,m}$, the following equality:

[TABLE]

In particular, we have:

[TABLE]

and

[TABLE]

Similarly, we have:

[TABLE]

[TABLE]

In particular,

[TABLE]

∎

As we have seen in section 3, the distribution character is defined, up to a constant, by the distribution $T(\overline{\Theta_{\Pi}})$. Before giving a formula for $T(\overline{\Theta_{\Pi}})(\phi),\phi\in\mathop{\rm S}\nolimits(W)$, we introduce some notations. Let $\mathop{\rm H}\nolimits$ be a Cartan subgroup of $\mathop{\rm G}\nolimits$ and let $\mathop{\rm G}\nolimits^{\sharp}$ be a double cover of $\tilde{\mathop{\rm G}\nolimits}$ as defined in [19, Section 2]. Let $\mathop{\rm H}\nolimits^{\sharp}$ be the preimage of $\mathop{\rm H}\nolimits$ in $\mathop{\rm G}\nolimits^{\sharp}$ and let $\mathop{\rm H}\nolimits^{\sharp}_{0}$ be the connected identity component of $\mathop{\rm H}\nolimits^{\sharp}$. For every roots $\alpha\in\Phi(\mathfrak{g},\mathfrak{h})$, $\frac{\alpha}{2}$ is analytic integral on $\mathop{\rm H}\nolimits^{\sharp}_{0}$ (see [16, Proposition 4.58]). It means that there exists a character $\xi^{\sharp}_{\frac{\alpha}{2}}:\mathop{\rm H}\nolimits^{\sharp}_{0}\mapsto\mathop{\rm S}\nolimits^{1}$ having the linear form $\frac{\alpha}{2}$ as derivative. More generally, for every $\mu\in\mathfrak{h}^{*}$ which is analytic integral on $\mathop{\rm G}\nolimits^{\sharp}$, we denote by $\xi^{\sharp}_{\mu}:\mathop{\rm H}\nolimits^{\sharp}_{0}\mapsto\mathop{\rm S}\nolimits^{1}$ the corresponding character.

Remark 5.4*.*

In particular, $\rho=\frac{1}{2}\sum\limits_{\alpha>0}\alpha$ is analytic integral on $\mathop{\rm H}\nolimits^{\sharp}_{0}$. Then, the Weyl denominator is well defined on $\mathop{\rm H}\nolimits^{\sharp}_{0}$. More precisely, this function is analytic on $\mathop{\rm H}\nolimits^{\sharp}$.

We denote by $c^{\sharp}_{-}:\mathfrak{h}\to\mathop{\rm H}\nolimits^{\sharp}_{0}$ the lift of the map $c_{-}:\mathfrak{h}\to\mathop{\rm H}\nolimits$ to $\mathop{\rm H}\nolimits^{\sharp}_{0}$. For every analytic integral form $\mu\in\mathfrak{h}^{*}$, we note by $c_{-}(\cdot)^{\mu}$ the function defined on $\mathop{\rm H}\nolimits^{\sharp}_{0}$ by

[TABLE]

Notation 5.5**.**

For all $n\in\mathbb{N}$, we indicate by $\Psi^{n}$ the embedding of $\mathop{\rm M}\nolimits(n,\mathbb{C})$ onto $\mathop{\rm M}\nolimits(2n,\mathbb{R})$ given by:

[TABLE]

For an endomorphism $X=A+iB$ of $\mathop{\rm M}\nolimits(n,\mathbb{C})$, we denote by $\mathop{\rm det}\nolimits_{\mathbb{R}}(X)$ the determinant of the matrix $\Psi^{n}(X)\in\mathop{\rm M}\nolimits(2n,\mathbb{R})$.

Proposition 5.6**.**

Fix $\Pi\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ of highest weight $\nu$ and let $\mu=\nu+\rho$. For every $\phi\in\mathop{\rm S}\nolimits(W)$, the following formula holds:

[TABLE]

where $\mathop{\rm ch}\nolimits(x)=|\mathop{\rm det}\nolimits_{\mathbb{R}}(x-1)|^{\frac{1}{2}}$ and $\mu^{\prime}\in\mathfrak{h}^{*}$ is defined by

[TABLE]

Proof.

The proof can be found in [19, Corollary 38, page 47].

∎

Remark 5.7*.*

1. (1)

By [14] (one can also consult the appendix of [21]), the weights of the representations of $\widetilde{\mathop{\rm U}\nolimits(n,\mathbb{C})}$ which appear in the Howe correspondence are the $\nu$’s of the form:

[TABLE]

where $0\leqslant r\leqslant p$, $0\leqslant s\leqslant q$, $r+s\leqslant n$, and where $\lambda_{1},\ldots,\lambda_{r},\mu_{1},\ldots,\mu_{s}$ are integers which satisfy $\lambda_{1}\geqslant\ldots\geqslant\lambda_{r}>0$ and $\mu_{1}\geqslant\ldots\geqslant\mu_{s}>0$. The weights $\nu$ can then be written as follows

[TABLE]

where $\nu_{i}\in\mathbb{Z}$, $\nu_{1}\geq\ldots\geq\nu_{n}$ with at most $q$ positives $\nu_{i}$ and $p$ negatives.

The parameter $\rho$ for $\widetilde{\mathop{\rm U}\nolimits(n,\mathbb{C})}$ is given by the following formula:

[TABLE]

Then,

[TABLE]

and

[TABLE] 2. (2)

One can easily show that, for every regular element $y^{\prime}\in\mathfrak{h}^{\prime}(m)\subseteq\mathfrak{h}^{\prime}$, we have:

[TABLE]

Now we want to simplify the equation (76). We first get the following theorem.

Theorem 5.8**.**

For all regular element $y\in\mathfrak{h}^{\prime}(m)^{\mathop{\rm reg}\nolimits}$ (where $\mathfrak{h}^{\prime}(m)$ was introduced in (63)) and $y^{\prime}\in\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$, we get:

[TABLE]

where $n(\mathfrak{h}^{\prime}(m))$ is the number of non-compact positive roots which do not vanish on $\mathfrak{h}^{\prime}(m)$.

Proof.

Let $\mathop{\rm P}\nolimits_{y}=\{\alpha\in\Phi(\mathfrak{g}^{\prime},\mathfrak{t}^{\prime})\thinspace;\thinspace i\alpha(y)>0\}$ and $n(y)$ the number of non-compact roots in $\mathop{\rm P}\nolimits_{y}$. Using the equation (40), we get

[TABLE]

For all $\eta\in\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$, we still denote by $\eta$ the element of $\mathscr{W}(\mathop{\rm G}\nolimits^{\prime},\mathfrak{h}^{\prime})$ obtained by identification of $\mathfrak{h}$ with $\mathfrak{h}^{\prime}(m)\subseteq\mathfrak{h}^{\prime}$ via the embedding (63). So, the action on the orthogonal complement of $\mathfrak{h}^{\prime}(m)$ in $\mathfrak{h}^{\prime}$ is trivial. We fix $\eta$ such that

[TABLE]

Then, we get

[TABLE]

Now,

[TABLE]

and

[TABLE]

We have

[TABLE]

As $y\in\mathfrak{h}^{\prime}(m)^{\mathop{\rm reg}\nolimits}$, we obtain

[TABLE]

Similarly, if $\alpha$ is such that $\alpha(y)\neq 0$, then $-\alpha(y)\neq 0$. Finally,

[TABLE]

So, $(-1)^{n(y)}=(-1)^{n(\mathfrak{h}^{\prime}(m))}$, and the theorem is proved.

∎

It is clear that for all $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$,

[TABLE]

is a Schwartz function on $W$. Using the Weyl integration formula, we get for all $y\in\tau(\mathfrak{h}_{1,m})$

[TABLE]

because the measure on $\mathop{\rm G}\nolimits^{\prime}/\mathop{\rm Z}\nolimits^{\prime}(m)$ is $\mathop{\rm G}\nolimits^{\prime}$-invariant and the function $\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}:\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}\to\mathbb{C}$ is given by:

[TABLE]

Then, the equation (84) can be written as

[TABLE]

Clearly, $\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}$ and the form $B$ are both $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$-invariant. Hence,

$\begin{aligned} \sum\limits_{\sigma\in\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})/\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})^{\mathfrak{h}^{\prime}(m)}}\displaystyle\int_{\mathfrak{h}^{\prime}}\cfrac{e^{iB(y^{\prime},\sigma y)}}{\pi_{\mathfrak{g}^{\prime}/\mathfrak{z}^{\prime}(m)}(\sigma^{-1}y^{\prime})}\psi^{\mathop{\rm G}\nolimits^{\prime}/\mathop{\rm H}\nolimits^{\prime}}(y^{\prime})|\pi_{\mathfrak{g}^{\prime}/\mathfrak{h}^{\prime}}(y^{\prime})|^{2}dy^{\prime}=\cfrac{|\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})|}{|\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})^{\mathfrak{h}^{\prime}(m)}|}\displaystyle\int_{\mathfrak{h}^{\prime}}\cfrac{e^{iB(y^{\prime},y)}}{\pi_{\mathfrak{g}^{\prime}/\mathfrak{z}^{\prime}(m)}(y^{\prime})}\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}(y^{\prime})|\pi_{\mathfrak{g}^{\prime}/\mathfrak{h}^{\prime}}(y^{\prime})|^{2}dy^{\prime}.\end{aligned}$

Using the following equality

[TABLE]

we finally get

[TABLE]

where the constant $C_{m}$ is given by

[TABLE]

Remark 5.9*.*

We denote by $\mathscr{A}_{\psi}$ the function on $\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$ defined by

[TABLE]

Using the equation (89), we get:

[TABLE]

where $(\mathscr{F}_{\mathfrak{h}^{\prime}})_{|_{\mathfrak{h}^{\prime}(m)}}$ is the restriction of the Fourier transform on $\mathfrak{h}^{\prime}$ to $\mathfrak{h}^{\prime}(m)$.

5.2. The character $\Theta_{\Pi^{\prime}}$ on the compact diagonal Cartan subgroup

Before giving the main theorem of this section, we prove a result concerning the right-hand side of (76). More particularly, we are interested in the support of the following distribution:

[TABLE]

We get the proposition.

Proposition 5.10**.**

Let $m_{\min}$ and $m_{\max}$ be the two positive integers defined by

[TABLE]

[TABLE]

Then,

[TABLE]

Proof.

According to [19, Lemma 33, page 42], we have:

[TABLE]

where $a_{k}=\mu^{\prime}_{k}-\frac{p+q-n-1}{2}$ and $b_{k}=-\mu^{\prime}_{k}-\frac{p+q-n-1}{2}$. Hence,

[TABLE]

Using [19, Appendix C], we get:

[TABLE]

where $P_{a_{k},b_{k}}$ and $Q_{a_{k},b_{k}}$ are the polynomials defined in [19, Appendix C, equations C.1-5]. In particular, using [19, Appendix C], one can easily verify that the support of the polynomials $P_{a_{k},b_{k}}$ are

[TABLE]

We have $a_{k}=-p+k+\nu_{n+1-k}$ and $b_{k}=-\nu_{n+1-k}-k-q+n+1$. In particular, $a_{k}\geq 1$ (resp. $b_{k}\geq 1$) implies $b_{k}\leq 0$ (resp. $a_{k}\leq 0$). Then, (98) can be rewritten as follow:

[TABLE]

The sequence $(\mu^{\prime}_{i})_{i}$ is increasing. So,

[TABLE]

Therefore, using [19, Lemma 9], we conclude:

[TABLE]

Thus, the equality (94) is satisfied.

∎

From now on, we fix $m_{\min}\leq m\leq m_{\max}$ and we consider the decomposition of $\mathfrak{h}^{\prime}$ as:

[TABLE]

where $\mathfrak{h}^{\prime}(m)^{\perp}$ is the orthogonal complement of $\mathfrak{h}^{\prime}(m)$ with respect to the bilinear form $B$. Let $\mathop{\rm pr}\nolimits_{m}$ be the orthogonal projection of $\mathfrak{h}^{\prime}$ on $\mathfrak{h}^{\prime}(m)$ that is:

[TABLE]

Moreover, let $\rho_{\mathfrak{z}^{\prime}(m)}$ be the linear form of $\mathfrak{h}^{\prime*}$ defined by

[TABLE]

Finally, let us consider the subgroup $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h},m)$ of $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ defined by:

[TABLE]

Theorem 5.11**.**

The character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ is given, up to a constant, by the following formula:

[TABLE]

Notation 5.12**.**

To simplify the notation, we note $d=p=q-n-1$.

Proof.

By (76) and (94) , for every $m\in[|m_{\min},m_{\max}|]$, we get

[TABLE]

Indeed, for all $m\in[|m_{\min},m_{\max}|]$:

[TABLE]

Then,

[TABLE]

Again, playing with the support of $f_{\phi}$, we get:

[TABLE]

We identify $\mathfrak{h}$ with its image $\mathfrak{h}^{\prime}(m)$ in $\mathfrak{h}^{\prime}$. Hence, (108) can be written as:

[TABLE]

By Remark 5.9, (109) can be written as:

[TABLE]

Using (6), we now choose a particular function $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$. We denote by $\mathop{\rm G}\nolimits^{\prime c}\subseteq\mathop{\rm G}\nolimits^{\prime}$ the domain of the Cayley transform in $\mathop{\rm G}\nolimits^{\prime}$. Fix $\Psi\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm G}\nolimits}^{\prime})$ such that $\mathop{\rm supp}\nolimits(\Psi)\in\widetilde{\mathop{\rm G}\nolimits^{\prime c}}\subseteq\widetilde{\mathop{\rm G}\nolimits}^{\prime}$ and let $j_{\mathfrak{g}^{\prime}}$ be the Jacobian of $c:\mathfrak{g}^{\prime c}\to\mathop{\rm G}\nolimits^{\prime c}$. We recall that $\tilde{c}_{-}$ is defined by $\tilde{c}_{-}(x^{\prime})=\tilde{c}(x^{\prime})\tilde{c}(0)^{-1}$ and let $\psi$ be the function given by

[TABLE]

With such a function $\psi$, the integral (110) becomes:

[TABLE]

where $\mathop{\rm pr}\nolimits_{m}$ is the projection given in the equation (103).

To get the character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}$, we would like to write the integral defined in (112) as an integral over $\widetilde{\mathop{\rm H}\nolimits}^{\prime}$. For all $h^{\prime}=\sum\limits_{k=1}^{p+q}ih_{k}\mathop{\rm E}\nolimits_{k,k}\in\mathfrak{h}^{\prime}$, we have:

[TABLE]

Moreover, for every $\mu=\sum\limits_{k=1}^{p+q}\mu_{k}e_{k}$ an analytic form on $\mathop{\rm H}\nolimits^{\prime\sharp}_{0}$, we have:

[TABLE]

The Cayley transform $c(h^{\prime})$ is given by

[TABLE]

where $c(ih_{k})=\frac{ih_{k}+1}{ih_{k}-1}=\frac{h^{2}_{k}-1}{h^{2}_{k}+1}+i\frac{-2}{h^{2}_{k}+1}=e^{im_{k}}$, with $m_{k}=\arccos\left(\frac{h^{2}_{k}-1}{h^{2}_{k}+1}\right)=\arcsin\left(\frac{-2}{h^{2}_{k}+1}\right)$. We denote by $M=\mathop{\rm diag}\nolimits(im_{1},\ldots,im_{p+q})$. Then,

[TABLE]

Then, we have:

[TABLE]

and the quantity $\prod\limits_{\underset{\alpha_{|_{\mathfrak{h}^{\prime}(m)}\neq 0}}{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}}\left(\xi^{\sharp}_{\frac{\alpha}{2}}(c^{\sharp}_{-}(h^{\prime}))-\xi^{\sharp}_{-\frac{\alpha}{2}}(c^{\sharp}_{-}(h^{\prime}))\right)$ is given by:

[TABLE]

Finally, we get:

[TABLE]

Using the fact that $j_{\mathfrak{g}^{\prime}}(h^{\prime})=\prod\limits_{k=1}^{p+q}(1+h^{2}_{k})^{-(p+q)}$ and $j_{\mathfrak{h}^{\prime}}(h^{\prime})=\prod\limits_{k=1}^{p+q}(1+h^{2}_{k})^{-1}$, we obtain, up to a constant, the following equality:

[TABLE]

Then, the equation (112) becomes:

[TABLE]

Finally,

[TABLE]

Using the invariance of $\left|\prod\limits_{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}(h^{\frac{\alpha}{2}}-h^{-\frac{\alpha}{2}})\right|^{2}\Psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}(h)$ under the action of $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$, we get:

[TABLE]

and

[TABLE]

Finally, using the Weyl integration formula, we get that the character $\Theta_{\Pi^{\prime}}(h)$ is given, up to a constant, by the formula:

[TABLE]

∎

Corollary 5.13**.**

For all $m\in[|m_{\min},m_{\max}|]$, the character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}$ is given by the following formula:

[TABLE]

where $C$ is a constant and $\rho_{\mathfrak{z}^{\prime}(m)}=\frac{1}{2}\sum\limits_{\underset{\alpha_{|_{\mathfrak{h}^{\prime}(m)}=0}}{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}}\alpha$.

Proof.

According to [16, Chapter V, Section 6, page 319], we get:

[TABLE]

∎

6. The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2n,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2m,\mathbb{R}))$, $n\leq m$

Let $V_{\overline{0}}$ be a real vector space of dimension $2n$ endowed with a positive-definite symmetric bilinear form $b_{0}$. We fix $\mathscr{B}=\{v_{1},v^{\prime}_{1},\ldots,v_{n},v^{\prime}_{n}\}$ a basis of $V_{\overline{0}}$ such that $\mathop{\rm Mat}\nolimits(b_{0},\mathscr{B})=\mathop{\rm Id}\nolimits_{2n}$ and let $\mathop{\rm O}\nolimits(V_{\overline{0}},b_{0})$ be the group of isometries of the form $b_{0}$, i.e.

[TABLE]

We denote by $\mathfrak{o}(V_{\overline{0}},b_{0})$ the Lie algebra of $\mathop{\rm O}\nolimits(V_{\overline{0}},b_{0})$ given by

[TABLE]

Writing the endomorphisms $X$ in the basis $\mathscr{B}$, the Lie algebra can be realized as:

[TABLE]

In particular, $\mathfrak{o}(V_{\overline{0}},b_{0})$ corresponds to the set of skew-symmetric matrices of $\mathop{\rm M}\nolimits(2n,\mathbb{R})$ and we get the following decomposition:

[TABLE]

Let $\mathfrak{h}_{0}$ be the subalgebra of $\mathfrak{g}_{0}$ defined by

[TABLE]

To simplify the notation, let $\mathop{\rm H}\nolimits_{k}$ be the matrix defined by $H_{k}=\mathop{\rm E}\nolimits_{-1+2k,2k}-\mathop{\rm E}\nolimits_{2k,-1+2k},1\leq k\leq n$. The complexifications of $\mathfrak{g}_{0}$ and $\mathfrak{h}_{0}$ are respectively denoted by $\mathfrak{g}$ and $\mathfrak{h}$. The roots of the Lie algebra $\mathfrak{g}$ with respect to $\mathfrak{h}$, denoted by $\Phi(\mathfrak{g},\mathfrak{h})$, are given by:

[TABLE]

where the linear form $e_{a}$, $1\leq a\leq n$, is given by

[TABLE]

We denote by $\pi_{\mathfrak{g}/\mathfrak{h}}$ the product of positive roots. For all $h=\sum\limits_{k=1}h_{k}\mathop{\rm H}\nolimits_{k}\in\mathfrak{h}_{0}$, we get:

[TABLE]

We consider $(V_{\overline{1}},b_{1})$ a symplectic vector space of dimension $2m$ over $\mathbb{R}$. Let $\mathscr{B}^{\prime}=\{w_{1},w^{\prime}_{1},\ldots,w_{m},w^{\prime}_{m}\}$ be a basis of $V_{\overline{1}}$ such that $\mathop{\rm Mat}\nolimits(b_{1},\mathscr{B}^{\prime})=\mathop{\rm diag}\nolimits\left(\begin{pmatrix}0&1\\ -1&0\end{pmatrix},\ldots,\begin{pmatrix}0&1\\ -1&0\end{pmatrix}\right)=\mathop{\rm J}\nolimits_{m,m}$, and we denote by $\mathop{\rm Sp}\nolimits(V_{\overline{1}},b_{1})$ the group of isometries of $b_{1}$, i.e.

[TABLE]

The Lie algebra $\mathfrak{sp}(V_{\overline{1}},b_{1})$ of $\mathop{\rm Sp}\nolimits(V_{\overline{1}},b_{1})$ is given by

[TABLE]

Writing the endomorphisms $X$ in the basis $\mathscr{B}^{\prime}$, we get:

[TABLE]

Let $\mathfrak{h}_{0}$ be the subalgebra of $\mathfrak{sp}(V_{\overline{1}},b_{1})$ given by:

[TABLE]

Similarly, we denote by $\mathfrak{h}$ and $\mathfrak{g}$ the complexifications of $\mathfrak{h}_{0}$ and $\mathfrak{g}_{0}$ respectively. The roots of $\mathfrak{g}$ with respect to $\mathfrak{h}$ are given by:

[TABLE]

where the linear form $e_{k}$ is defined by

[TABLE]

Let $\mathop{\rm K}\nolimits$ be a maximal compact subgroup of $\mathop{\rm Sp}\nolimits(V_{\overline{1}},b_{1})$, $\mathfrak{k}_{0}$ the Lie algebra of $\mathop{\rm K}\nolimits$ and $\mathfrak{k}$ its complexification. In particular, we have $\mathop{\rm K}\nolimits\approx\mathop{\rm U}\nolimits(m,\mathbb{C})$ and

[TABLE]

For all $h^{\prime}=\sum\limits_{k=1}^{m}h_{k}\mathop{\rm H}\nolimits_{k}\in\mathfrak{h}^{\prime}_{0}$, we get:

[TABLE]

Let $V=V_{\overline{0}}\oplus V_{\overline{1}}$, $b=b_{0}\oplus b_{1}$ as defined in Appendix A and let $(\mathop{\rm S}\nolimits,\mathfrak{s}(V,b)=\mathfrak{s}(V,b)_{\overline{0}}\oplus\mathfrak{s}(V,b)_{\overline{1}})$ be the corresponding Lie supergroup.

Lemma 6.1**.**

An element $X=\begin{pmatrix}0&X_{1}\\ X_{2}&0\end{pmatrix}$ is in $\mathfrak{s}(V,\tau)_{\overline{1}}$ if and only if $X_{2}=-J_{m,m}X^{t}_{1}$.

For all $j$, we denote by $V^{j}_{\overline{0}}$ and $V^{j}_{\overline{1}}$ the subspaces of $V_{\overline{0}}$ and $V_{\overline{1}}$ respectively given by:

[TABLE]

We now consider the endomorphisms $u_{j}\in\mathfrak{s}(V,b)_{\overline{1}},1\leq j\leq n$, defined on $V^{j}_{\overline{0}}$ (resp. $V^{j}_{\overline{1}}$) by

[TABLE]

[TABLE]

and let $\mathfrak{h}_{1}$ be the subspace of $\mathfrak{s}(V,\tau)_{\overline{1}}$ given by

[TABLE]

We define the moment maps $\tau:\mathfrak{s}(V,b)_{\overline{1}}\to\mathfrak{o}(2n,\mathbb{R})$ and $\tau^{\prime}:\mathfrak{s}(V,b)_{\overline{1}}\to\mathfrak{sp}(2m,\mathbb{R})$ by:

[TABLE]

More precisely, for all $w=\begin{pmatrix}0&X_{1}\\ \mathop{\rm J}\nolimits^{-1}_{m,m}X^{t}_{1}&0\end{pmatrix}\in\mathfrak{s}(V,b)_{\overline{1}}$, we have

[TABLE]

We consider the injection of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ given by:

[TABLE]

We denote by $\tilde{\mathfrak{h}}$ the image of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ via the map (142). Let $\mathfrak{z}^{\prime}$ (resp. $Z^{\prime}$) be the subalgebra (resp. subgroup) of $\mathfrak{g}^{\prime}$ (resp. $\mathop{\rm G}\nolimits^{\prime}$) defined by

[TABLE]

[TABLE]

We define $\Phi(\mathfrak{g}^{\prime},\mathfrak{z}^{\prime})=\{\alpha\in\Phi(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime}),\alpha_{|_{\tilde{\mathfrak{h}}}}\neq 0\}$ and $\Phi(\mathfrak{z}^{\prime},\mathfrak{h}^{\prime})=\Phi(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})\setminus\Phi(\mathfrak{g}^{\prime},\mathfrak{z}^{\prime})$, and let:

[TABLE]

Lemma 6.2**.**

For all $h^{\prime}=\sum\limits_{k=1}^{m}h_{k}\mathop{\rm H}\nolimits_{k}\in\mathfrak{h}^{\prime}_{0}$, we have:

[TABLE]

In particular, we have:

[TABLE]

Let $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$, $\mathscr{W}(\mathop{\rm G}\nolimits^{\prime},\mathfrak{h}^{\prime})$ and $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$ be the Weyl groups of $\mathop{\rm G}\nolimits$, $\mathop{\rm G}\nolimits^{\prime}$ and $\mathop{\rm K}\nolimits^{\prime}$ respectively.

For every function $\phi\in\mathop{\rm S}\nolimits(W)$, we define the function $f_{\phi}$ on $\tau(\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1})$ by

[TABLE]

where $C_{\mathfrak{h}_{1}}$ is a constant of modulus 1 defined in [19, Lemma 8, page 17].

As in the previous section, we denote by $\mathop{\rm H}\nolimits^{\sharp}$ the two fold cover of $\widetilde{\mathop{\rm H}\nolimits}$ such that the linear forms $\frac{\alpha}{2}$ are analytic integral for every roots $\alpha\in\Phi(\mathfrak{g},\mathfrak{h})$ and let $\mathop{\rm H}\nolimits^{\sharp}_{0}$ be its connected identity component. Let $\xi^{\sharp}_{\frac{\alpha}{2}}:\mathop{\rm H}\nolimits^{\sharp}_{0}\to\mathop{\rm S}\nolimits^{1}$ the multiplicative character having the linear form $\frac{\alpha}{2}$ as derivative and let $c^{\sharp}_{-}:\mathfrak{h}\to\mathop{\rm H}\nolimits^{\sharp}_{0}$ be the extension of $c_{-}$ on $\mathop{\rm H}\nolimits^{\sharp}_{0}$ (section 5.1).

Proposition 6.3**.**

Fix $\Pi\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ of highest weight $\nu$ and let $\mu=\nu+\rho$. For every $\phi\in\mathop{\rm S}\nolimits(W)$, the following formula holds:

[TABLE]

where $\mathop{\rm ch}\nolimits(x)=|\mathop{\rm det}\nolimits(x-1)|^{\frac{1}{2}}$ and $\mu^{\prime}\in\mathfrak{h}^{*}$ is defined by

[TABLE]

Proof.

For a proof, we refer the reader to [19, Corollary 38, page 47].

∎

Remark 6.4*.*

1. (1)

By [14], the weights $\nu$ and $\nu^{\prime}$ of the representations $\Pi$ and $\Pi^{\prime}$ respectively are given by:

[TABLE]

where $0\leq k\leq n$ and $\nu_{1}\geq\ldots\geq\nu_{k}>0$. By fixing $k=m$ and considering a decreasing sequence $\nu_{1}\geq\ldots\geq\nu_{m}\geq 0$ with at most $n$ non zero $\nu_{i}$, we get

[TABLE]

with $\tau_{a}=-n-\nu_{m+1-a}$, where $(\tau_{a})_{a\in[|1,m|]}$ is a decreasing sequence of negative numbers.

The linear form $\rho$ is given by:

[TABLE]

Then $\mu=\sum\limits_{k=1}^{n}(\nu+\rho)_{a}e_{a}$ with $(\nu+\rho)_{a}=\nu_{a}+n-a$ and

[TABLE] 2. (2)

For every $x,y\in\mathfrak{g}$ or $\mathfrak{g}^{\prime}$, we denote by $B$ the bilinear form defined by

[TABLE]

The form $B$ is $\mathop{\rm G}\nolimits$ (resp. $\mathop{\rm G}\nolimits^{\prime}$)-invariant and non-degenerate on $\mathfrak{g}$ and $\mathfrak{g}^{\prime}$. More precisely, for all $x=\sum\limits_{k=1}^{n}x_{k}\mathop{\rm H}\nolimits_{k,k}$ (resp. $x^{\prime}=\sum\limits_{k=1}^{m}x^{\prime}_{k}\mathop{\rm H}\nolimits_{k,k}$) and $y=\sum\limits_{k=1}^{n}y_{k}\mathop{\rm H}\nolimits_{k,k}$ (resp. $y^{\prime}=\sum\limits_{k=1}^{m}y^{\prime}_{k}\mathop{\rm H}\nolimits_{k,k}$), the form $B$ is given by

[TABLE]

See [19, page 49-50].

Theorem 6.5**.**

For all regular element $y\in\tilde{\mathfrak{h}}$ and $y^{\prime}\in\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$, we get:

[TABLE]

where $n(\tilde{\mathfrak{h}})$ is the number of non-compacts positives roots which do not vanish on $\tilde{\mathfrak{h}}$.

It is clear that for all $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$,

[TABLE]

is a Schwartz function on $W$. Using the Weyl integration formula, we get for all $y\in\tau(\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1})$

[TABLE]

where $\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}:\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}\to\mathbb{C}$ is given by:

[TABLE]

Using the same arguments from the previous section, we get the following equality:

[TABLE]

We denote by $\mathscr{A}_{\psi}$ the function on $\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$ defined by

[TABLE]

Using (159), we get:

[TABLE]

Before giving the main theorem, we prove a result concerning the right hand-side of (149). More particularly, we are interested in the support of the following distribution:

[TABLE]

Proposition 6.6**.**

Let $\beta_{1}$ and $\beta_{2}$ be the two integers defined by:

[TABLE]

[TABLE]

Then,

[TABLE]

Proof.

According to [19, Lemma 33, page 43], we have:

[TABLE]

where $a_{k}=\mu^{\prime}_{k}-m+n=\nu_{n+1-k}+k-m+n-1$ and $b_{k}=-\mu^{\prime}_{k}-m+n=-\nu_{n+1-k}-k-m+n+1$. In particular, $a_{k}\geq 1$ (resp. $b_{k}\geq 1$) implies $b_{k}\leq 0$ (resp. $a_{k}\leq 0$).

Using [19, Appendix C], we get

[TABLE]

where $P_{a_{k},b_{k}}$ and $Q_{a_{k},b_{k}}$ are the polynomials defined in [19, Appendix C]. In particular, using [19, Appendix C], the support of the polynomials $P_{a_{k},b_{k}}$ are

[TABLE]

Then,

[TABLE]

and finally

[TABLE]

∎

From now on, we denote by $\mathop{\rm supp}\nolimits(\beta_{1},\beta_{2})$ the support of the distribution given in (162). We denote by $\tilde{\mathfrak{h}}^{\perp}$ the orthogonal complement of $\tilde{\mathfrak{h}}$ with respect to the bilinear form $B$, and by $\mathop{\rm pr}\nolimits:\mathfrak{h}\to\tilde{\mathfrak{h}}$ the associated projection, i.e.

[TABLE]

Finally, let us consider the subgroup $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h},\beta)$ of $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ defined by:

[TABLE]

Theorem 6.7**.**

The character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ is given, up to a constant, by the following formula:

[TABLE]

Proof.

By (149) and (165), we get

[TABLE]

Then,

[TABLE]

Again, playing with the support of $f_{\phi}$, we get:

[TABLE]

We identify $\mathfrak{h}$ with its image $\tilde{\mathfrak{h}}$ in $\mathfrak{h}^{\prime}$. Hence, (175) can be written as:

[TABLE]

By (161), this can be written as:

[TABLE]

We now choose a particular $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$. We denote by $\mathop{\rm G}\nolimits^{\prime c}\subseteq\mathop{\rm G}\nolimits^{\prime}$ the domain of the Cayley transform in $\mathop{\rm G}\nolimits^{\prime}$. Fix $\Psi\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm G}\nolimits}^{\prime})$ such that $\mathop{\rm supp}\nolimits(\Psi)\in\widetilde{\mathop{\rm G}\nolimits^{\prime c}}\subseteq\widetilde{\mathop{\rm G}\nolimits}^{\prime}$ and let $j_{\mathfrak{g}^{\prime}}$ be the Jacobian of $c:\mathfrak{g}^{\prime c}\to\mathop{\rm G}\nolimits^{\prime c}$. We recall that $\tilde{c}_{-}$ is defined by $\tilde{c}_{-}(x^{\prime})=\tilde{c}(x^{\prime})\tilde{c}(0)^{-1}$ and let $\psi$ be the function given by

[TABLE]

With such a function $\psi$, the integral (177) becomes:

[TABLE]

where $\mathop{\rm pr}\nolimits$ is the projection given in (171). To get the character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}$, we would like to write the integral defined in (179) as an integral over $\widetilde{\mathop{\rm H}\nolimits}^{\prime}$.

For all $h^{\prime}=\sum\limits_{k=1}^{m}h_{k}\mathop{\rm H}\nolimits_{k}$, we get:

[TABLE]

Indeed, for all $h\in\mathbb{R}$, we denote by $\mathop{\rm A}\nolimits(h)$ the matrix defined by $\mathop{\rm A}\nolimits(h)=\begin{pmatrix}-1&h\\ -h&-1\end{pmatrix}$. Then,

[TABLE]

Moreover, we have $\Theta(\tilde{c}(h^{\prime}))=\mathop{\rm ch}\nolimits(h^{\prime})^{2n}=\prod\limits_{k=1}^{m}(1+h^{2}_{k})^{n}$.

Now, we focus our attention on the term $\prod\limits_{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}(\xi_{\frac{\alpha}{2}}(c^{\sharp}_{-}(h^{\prime}))-\xi_{-\frac{\alpha}{2}}(c^{\sharp}_{-}(h^{\prime})))$.

First, we remark that $c(h^{\prime})=\mathop{\rm diag}\nolimits\left(c\begin{pmatrix}0&h_{1}\\ -h_{1}&0\end{pmatrix},\ldots,c\begin{pmatrix}0&h_{m}\\ -h_{m}&0\end{pmatrix}\right)$. Moreover, for all $x\in\mathbb{R}$, we have:

[TABLE]

where $R(\theta)$ is the rotation matrix of angle $\theta$. Then,

[TABLE]

and

[TABLE]

Similarly, we obtain:

[TABLE]

and then, the following equality holds:

[TABLE]

Using the fact that $j_{\mathfrak{g}^{\prime}}(h^{\prime})=\prod\limits_{k=1}^{p+q}(1+h^{2}_{k})^{-(2m+1)}$ and $j_{\mathfrak{h}^{\prime}}(h^{\prime})=\prod\limits_{k=1}^{p+q}(1+h^{2}_{k})^{-1}$, we get:

[TABLE]

Then, the equation (179) becomes:

[TABLE]

Finally,

[TABLE]

Using the invariance of $\left|\prod\limits_{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}(h^{\frac{\alpha}{2}}-h^{-\frac{\alpha}{2}})\right|^{2}\Psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}(h)$ under the action of $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$, we have:

[TABLE]

and

[TABLE]

Finally, we obtain that the character $\Theta_{\Pi^{\prime}}(h)$ is given, up to a constant, by the formula:

[TABLE]

∎

7. The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm O}\nolimits(2n+1,\mathbb{R}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm Sp}\nolimits(2m,\mathbb{R}))$, $n\leq m$

Let $V_{\overline{0}}$ be a real vector space of dimension $2n+1$ endowed with a positive-definite symmetric bilinear form $b_{0}$. Let $\mathscr{B}=\{v_{1},v^{\prime}_{1},\ldots,v_{n},v^{\prime}_{n},v_{n+1}\}$ be a basis of $V_{\overline{0}}$ such that $\mathop{\rm Mat}\nolimits(b_{0},\mathscr{B})=\mathop{\rm Id}\nolimits_{2n+1}$ and let $\mathop{\rm O}\nolimits(V_{\overline{0}},b_{0})$ the group of isometries of the form $b_{0}$, i.e.

[TABLE]

We denote by $\mathfrak{o}(V_{\overline{0}},b_{0})$ the Lie algebra of $\mathop{\rm O}\nolimits(V_{\overline{0}},b_{0})$ given by:

[TABLE]

Writing the endomorphisms $X$ in the basis $\mathscr{B}$, the Lie algebra can be realized as:

[TABLE]

In particular, $\mathfrak{o}(V_{\overline{0}},b_{0})$ corresponds to the set of skew-symmetric matrices of $\mathop{\rm M}\nolimits(2n+1,\mathbb{R})$ and we get the following decomposition:

[TABLE]

Let $\mathfrak{h}_{0}$ be the subalgebra of $\mathfrak{g}_{0}$ defined by:

[TABLE]

To simplify the notation, let $H_{k}$ be the matrix defined by $\mathop{\rm H}\nolimits_{k}=\mathop{\rm E}\nolimits_{-1+2k,2k}-\mathop{\rm E}\nolimits_{2k,-1+2k},1\leq k\leq n$. The complexifications of $\mathfrak{g}_{0}$ and $\mathfrak{h}_{0}$ are respectively denoted by $\mathfrak{g}$ and $\mathfrak{h}$. The roots of $\mathfrak{g}$ with respect to $\mathfrak{h}$ are given by:

[TABLE]

where the linear forms $e_{a}$, $1\leq a\leq n$, are defined in equation (128).

Let $\pi_{\mathfrak{g}/\mathfrak{h}}$ be the product of positive roots. For all $h=\sum\limits_{k=1}^{n}h_{k}\mathop{\rm H}\nolimits_{k}\in\mathfrak{h}_{0}$, we have:

[TABLE]

Let $V=V_{\overline{0}}\oplus V_{\overline{1}}$, $b=b_{0}\oplus b_{1}$ as defined in Appendix A and let $(\mathop{\rm S}\nolimits,\mathfrak{s}(V,b)=\mathfrak{s}(V,b)_{\overline{0}}\oplus\mathfrak{s}(V,b)_{\overline{1}})$ be the corresponding Lie supergroup.

Lemma 7.1**.**

An element $X=\begin{pmatrix}0&X_{1}\\ X_{2}&0\end{pmatrix}$ is in $\mathfrak{s}(V,\tau)_{\overline{1}}$ if and only if $X_{2}=-\mathop{\rm J}\nolimits_{m,m}X^{t}_{1}$.

For all $j$, we denote by $V^{j}_{\overline{0}}$ and $V^{j}_{\overline{1}}$ the subspaces of $V_{\overline{0}}$ and $V_{\overline{1}}$ given by:

[TABLE]

We now consider the elements $u_{j}\in\mathfrak{s}(V)_{\overline{1}},1\leq j\leq n$, defined on $V_{\overline{0}}$ (resp. $V_{\overline{1}}$) by

[TABLE]

[TABLE]

and let $\mathfrak{h}_{1}$ be the subspace of $\mathfrak{s}(V,b)_{\overline{1}}$ given by

[TABLE]

We define the moment maps $\tau:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{o}(2n+1,\mathbb{R})$ and $\tau^{\prime}:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{sp}(2m,\mathbb{R})$ by

[TABLE]

More precisely, for all $w=\begin{pmatrix}0&X_{1}\\ \mathop{\rm J}\nolimits^{-1}_{m,m}X^{t}_{1}&0\end{pmatrix}$, we have:

[TABLE]

We get an injection of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ given by:

[TABLE]

We denote by $\tilde{\mathfrak{h}}$ the image of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$. We define $\mathfrak{z}^{\prime}$ and $\mathop{\rm Z}\nolimits^{\prime}$ by

[TABLE]

[TABLE]

We define $\Phi(\mathfrak{g}^{\prime},\mathfrak{z}^{\prime})=\{\alpha\in\Phi(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime}),\alpha_{|_{\tilde{\mathfrak{h}}}}\neq 0\}$ and $\Phi(\mathfrak{z}^{\prime},\mathfrak{h}^{\prime})=\Phi(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})\setminus\Phi(\mathfrak{g}^{\prime},\mathfrak{z}^{\prime})$, and let:

[TABLE]

For every $\phi\in\mathop{\rm S}\nolimits(W)$, we define the function $f_{\phi}$ on $\tau(\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1})$ by

[TABLE]

where $C_{\mathfrak{h}_{1}}$ is a constant of modulus 1 defined in [19, Lemma 8, page 17].

As in the previous section, we denote by $\mathop{\rm H}\nolimits^{\sharp}$ the two fold cover of $\widetilde{\mathop{\rm H}\nolimits}$ such that the linear forms $\frac{\alpha}{2}$ are analytic integral for every roots $\alpha\in\Phi(\mathfrak{g},\mathfrak{h})$ and let $\mathop{\rm H}\nolimits^{\sharp}_{0}$ be its connected identity component. Let $\xi^{\sharp}_{\frac{\alpha}{2}}:\mathop{\rm H}\nolimits^{\sharp}_{0}\to\mathop{\rm S}\nolimits^{1}$ be the multiplicative character having the linear form $\frac{\alpha}{2}$ as derivative and let $c^{\sharp}_{-}:\mathfrak{h}\to\mathop{\rm H}\nolimits^{\sharp}_{0}$ be the extension of $c_{-}$ on $\mathop{\rm H}\nolimits^{\sharp}_{0}$ (section 5.1)

Proposition 7.2**.**

Fix $\Pi\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ of highest weight $\nu$ and let $\mu=\nu+\rho$. For every $\phi\in\mathop{\rm S}\nolimits(W)$, the following formula holds:

[TABLE]

where $\mathop{\rm ch}\nolimits(x)=|\mathop{\rm det}\nolimits(x-1)|^{\frac{1}{2}}$ and $\mu^{\prime}\in\mathfrak{h}^{*}$ is defined by

[TABLE]

Proof.

For a proof of this proposition, we refer to [19, Corollary 38, page 47].

∎

Remark 7.3*.*

1. (1)

By [14], the weights $\nu$ and $\nu^{\prime}$ of the representations $\Pi$ and $\Pi^{\prime}$ respectively are given by:

[TABLE]

where $0\leq k\leq n$ and $\nu_{1}\geq\ldots\geq\nu_{k}>0$. By fixing $k=m$ and considering a decreasing sequence $\nu_{1}\geq\ldots\geq\nu_{m}\geq 0$ with at most $n$ non zero $\nu_{i}$, we get

[TABLE]

with $\tau_{a}=-\frac{2n+1}{2}-\nu_{m+1-a}$, where $(\tau_{a})_{a\in[|1,m|]}$ is a decreasing sequence of negative numbers.

The linear form $\rho$ is given by:

[TABLE]

Then $\mu=\sum\limits_{k=1}^{n}(\nu+\rho)_{a}e_{a}$ with $(\nu+\rho)_{a}=\nu_{a}+n-a+\frac{1}{2}$ and

[TABLE] 2. (2)

For every $x,y\in\mathfrak{g}$ or $\mathfrak{g}^{\prime}$, we denote by $B$ the bilinear form defined by

[TABLE]

The form $B$ is $\mathop{\rm G}\nolimits$ (resp. $\mathop{\rm G}\nolimits^{\prime}$)-invariant and non-degenerate on $\mathfrak{g}$ and $\mathfrak{g}^{\prime}$. More precisely, for all $x=\sum\limits_{k=1}^{n}x_{k}\mathop{\rm H}\nolimits_{k,k}$ (resp. $x^{\prime}=\sum\limits_{k=1}^{m}x^{\prime}_{k}\mathop{\rm H}\nolimits_{k,k}$) and $y=\sum\limits_{k=1}^{n}y_{k}\mathop{\rm H}\nolimits_{k,k}$ (resp. $y^{\prime}=\sum\limits_{k=1}^{m}y^{\prime}_{k}\mathop{\rm H}\nolimits_{k,k}$), the form $B$ is given by

[TABLE]

See [19, page 49-50].

Theorem 7.4**.**

For all regular element $y\in\tilde{\mathfrak{h}}$ and $y^{\prime}\in\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$, we get:

[TABLE]

where $n(\tilde{\mathfrak{h}})$ is the number of non-compacts positives roots which do not vanish on $\tilde{\mathfrak{h}}^{\prime}$.

It is clear that for all $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$,

[TABLE]

is a Schwartz function on $W$. Using the Weyl integration formula, we get for all $y\in\tau(\mathfrak{h}_{1})$

[TABLE]

where $\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}:\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}\to\mathbb{C}$ is given by:

[TABLE]

Using the same method than before, we get the following equality:

[TABLE]

We denote by $\mathscr{A}_{\psi}$ the function defined on $\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$ by

[TABLE]

Then, we get:

[TABLE]

Before giving the main theorem, we prove a result concerning the right hand-side of (204). More particularly, we are interested in the support of the following distribution:

[TABLE]

Proposition 7.5**.**

Let $\beta_{1}$ and $\beta_{2}$ the two integers defined by:

[TABLE]

[TABLE]

Then,

[TABLE]

Proof.

According to [19, Lemma 33, page 43], we get:

[TABLE]

where $a_{k}=\mu^{\prime}_{k}-m+n+\frac{1}{2}$ and $b_{k}=-\mu^{\prime}_{k}-m+n+\frac{1}{2}$. In particular, $a_{k}\geq 1$ (resp. $b_{k}\geq 1$) implies $b_{k}\leq 0$ (resp. $a_{k}\leq 0$).

Using [19, Appendix C], we get the following equality

[TABLE]

where $P_{a_{k},b_{k}}$ and $Q_{a_{k},b_{k}}$ are the polynomials defined in [19, Appendix C]. In particular, using [19, Appendix C], the support of the polynomial $P_{a_{k},b_{k}}$ are

[TABLE]

Then,

[TABLE]

and finally

[TABLE]

∎

From now on, we denote by $\mathop{\rm supp}\nolimits(\beta_{1},\beta_{2})$ the support of the distribution given in equation (217). We denote by $\tilde{\mathfrak{h}}^{\perp}$ the orthogonal complement of $\tilde{\mathfrak{h}}$ with respect to the bilinear form $B$, and by $\mathop{\rm pr}\nolimits:\mathfrak{h}\to\tilde{\mathfrak{h}}$ the associated projection, i.e.

[TABLE]

Finally,, let us consider the subgroup $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h},\beta)$ of $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ defined by:

[TABLE]

Theorem 7.6**.**

The character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ is given, up to a constant, by the following formula:

[TABLE]

Proof.

The proof of this theorem is similar to what we did in section 6 for $\mathop{\rm O}\nolimits(2n,\mathbb{R})$.

∎

8. The dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(n,\mathbb{H}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm O}\nolimits^{*}(m,\mathbb{H}))$, $n\leq m$

Let $\mathbb{H}$ be the fields of quaternions, i.e. $\mathbb{H}=\mathbb{R}+i\mathbb{R}+j\mathbb{R}+ij\mathbb{R}$ with $ij=-ji$. Let $z=a+ib+jc+ijd=(a+ib)+j(c-id)\in\mathbb{H}$. We have the following morphisms:

[TABLE]

and $\Psi^{2}:\mathop{\rm M}\nolimits(2,\mathbb{C})\to\mathop{\rm M}\nolimits(4,\mathbb{R})$ defined in equation (75). In particular, we associate to a quaternionic number $z=(a+ib)+j(c-id)$ a $4\times 4$-real matrix $\Psi^{2}\circ\Psi_{1}(z)$ given by:

[TABLE]

More generally, we denote by $\Psi_{d}$ the corresponding morphism from $\mathop{\rm M}\nolimits(d,\mathbb{H})$ into $\mathop{\rm M}\nolimits(2d,\mathbb{C})$ and then we get a morphism $\Psi^{d}\circ\Psi_{d}:\mathop{\rm M}\nolimits(d,\mathbb{H})\to\mathop{\rm M}\nolimits(4d,\mathbb{R})$. We view $\mathbb{H}^{k}$ as a left $\mathbb{H}$-vector space

[TABLE]

where $\bar{z}=a-ib-jc-ijd$ is the non-trivial conjugation on $\mathbb{H}$.

Let $V_{\overline{0}}$ be a left $n$-dimensional vector space over $\mathbb{H}$ endowed with a positive definite hermitian form $b_{0}$. Let $\mathscr{B}=\{v_{1},\ldots,v_{n}\}$ be a basis of $V_{\overline{0}}$ such that the matrix of $b_{0}$ with respect to $\mathscr{B}$ is $\mathop{\rm Mat}\nolimits(b_{0},\mathscr{B})=\mathop{\rm Id}\nolimits_{n}$ and we denote by $\mathop{\rm U}\nolimits(V_{\overline{0}},b_{0})$ the group of isometries of $b_{0}$, i.e.

[TABLE]

Writing the endomorphisms $g$ in the basis $\mathscr{B}$, the group $\mathop{\rm U}\nolimits(V_{\overline{0}},b_{0})$ can be realized as:

[TABLE]

where $g^{*}=\bar{g}^{t}$. From now on, we denote by $\mathop{\rm U}\nolimits(n,\mathbb{H})$ the group defined in (232).

Remark 8.1*.*

As mentioned in [16, Chapter 1, Section 17], the quaternionic unitary group $\mathop{\rm U}\nolimits(n,\mathbb{H})$ is isomorphic to $\mathop{\rm Sp}\nolimits(2n,\mathbb{C})\cap\mathop{\rm U}\nolimits(2n,\mathbb{C})$. In particular, the complexification of the Lie algebra of $\mathop{\rm U}\nolimits(n,\mathbb{H})$ is isomorphic to $\mathfrak{sp}(2n,\mathbb{C})$.

We denote by $\mathfrak{g}_{0}=\mathfrak{u}(n,\mathbb{H})$ the Lie Algebra of $\mathop{\rm U}\nolimits(n,\mathbb{H})$ and by $\mathfrak{g}$ its complexification. Let $\mathfrak{h}_{0}$ the subalgebra of $\mathfrak{g}_{0}$ given by:

[TABLE]

The corresponding subspace in $\mathop{\rm M}\nolimits(2n,\mathbb{C})$ is given by:

[TABLE]

We denote by $\mathfrak{h}$ (resp. $\Psi_{n}(\mathfrak{h})$) the complexification of $\mathfrak{h}_{0}$ (resp. $\Psi_{n}(\mathfrak{h}_{0})$). As recalled in [16], the roots are given by

[TABLE]

where the linear form $e_{a}$ is given by

[TABLE]

Without considering the embedding of $\mathfrak{u}(n,\mathbb{H})$ on $\mathop{\rm M}\nolimits(2n,\mathbb{C})$, one can define the forms $\widetilde{e_{a}}$ as

[TABLE]

and get the same root system. Again, we define $\pi_{\mathfrak{g}/\mathfrak{h}}=\prod\limits_{\alpha\in\Phi^{+}(\mathfrak{g},\mathfrak{h})}\alpha$.

Lemma 8.2**.**

For all $h=\sum\limits_{k=1}^{n}ih_{k}E_{k,k}$, we have:

[TABLE]

Now, let $V_{\overline{1}}$ be a quaternionic vector space of dimension $m$ endowed with a non-degenerate skew-hermitian form $b_{1}$. Let $\mathscr{B}^{\prime}=\{w_{1},\ldots,w_{m}\}$ be a basis of $V_{\overline{1}}$ such that $\mathop{\rm Mat}\nolimits(b_{1},\mathscr{B}^{\prime})=i\mathop{\rm Id}\nolimits_{m}$. We denote by $\mathop{\rm O}\nolimits^{*}(V_{\overline{1}},b_{1})$ the group of isometries of the form $b_{1}$, i.e.

[TABLE]

Writing the endomorphisms in the basis $\mathscr{B}^{\prime}$, we get the following realization

[TABLE]

We denote by $\mathop{\rm O}\nolimits^{*}(m,\mathbb{H})$ the subgroup of $\mathop{\rm GL}\nolimits(m,\mathbb{H})$ defined in equation (240).

Remark 8.3*.*

Using the embedding $\Psi_{m}$, the group $\mathop{\rm O}\nolimits^{*}(m,\mathbb{H})$ can be seen as a subgroup of $\mathop{\rm GL}\nolimits(2m,\mathbb{C})$. More precisely, one can check that $\mathop{\rm O}\nolimits^{*}(m,\mathbb{H})$ is isomorphic to

[TABLE]

where $M_{m,m}=\begin{pmatrix}0&\mathop{\rm Id}\nolimits_{m}\\ \mathop{\rm Id}\nolimits_{m}&0\end{pmatrix}$. Until the end, we will denote by $\mathop{\rm O}\nolimits^{*}(2m,\mathbb{C})$ the subgroup defined in equation (241).

Let $\mathfrak{g}^{\prime}_{0}=\mathfrak{o}^{*}(m,\mathbb{H})$ be the Lie algebra of $\mathop{\rm O}\nolimits^{*}(m,\mathbb{H})$ and let $\mathfrak{h}^{\prime}_{0}$ be the subalgebra of $\mathfrak{g}^{\prime}_{0}$ given by

[TABLE]

The complexification of $\mathfrak{g}^{\prime}_{0}$, denoted by $\mathfrak{g}^{\prime}$, is isomorphic to $\mathfrak{o}(2m,\mathbb{C})$. In particular, as recalled in [16], the roots of $\mathfrak{g}^{\prime}$ are given by

[TABLE]

where the linear form $e_{a},1\leq a\leq m$ is given by

[TABLE]

As before, let $\pi_{\mathfrak{g}^{\prime}/\mathfrak{h}^{\prime}}$ be the product of all positive roots.

Lemma 8.4**.**

For all $h^{\prime}=\sum\limits_{k=1}^{m}ih_{k}\mathop{\rm E}\nolimits_{k,k}$, we have:

[TABLE]

Remark 8.5*.*

The maximal compact subgroup of $\mathop{\rm O}\nolimits^{*}(m,\mathbb{H})$ is $\mathop{\rm K}\nolimits=\mathop{\rm U}\nolimits(m,\mathbb{H})$. Let $\mathfrak{k}_{0}$ be the Lie algebra of $\mathop{\rm K}\nolimits$ and $\mathfrak{k}$ its complexification. According to the previous section, the compact roots are given by

[TABLE]

Let $V=V_{\overline{0}}\oplus V_{\overline{1}}$, $b=b_{0}\oplus b_{1}$ as defined in Appendix A and let $(\mathop{\rm S}\nolimits,\mathfrak{s}(V,b)=\mathfrak{s}(V,b)_{\overline{0}}\oplus\mathfrak{s}(V,b)_{\overline{1}})$ be the corresponding Lie supergroup.

Lemma 8.6**.**

An element $X=\begin{pmatrix}0&X_{1}\\ X_{2}&0\end{pmatrix}$ is in $\mathfrak{s}(V,b)_{\overline{1}}$ if and only if $X_{2}=-iX^{*}_{1}$.

We consider the decompositions of $V_{\overline{0}}$ and $V_{\overline{1}}$ given by

[TABLE]

We define the odd endomorphisms $u_{j}\in\mathfrak{s}(V,b)_{\overline{1}},1\leq j\leq n$, by

[TABLE]

and let $\mathfrak{h}_{1}$ be the subspace of $\mathfrak{s}(V,b)_{\overline{1}}$ given by

[TABLE]

We define the moment maps $\tau:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{g}$ and $\tau^{\prime}:\mathfrak{s}(V,b)_{\overline{1}}\mapsto\mathfrak{g}^{\prime}$ by

[TABLE]

In particular, $\tau(\mathfrak{h}_{1})\subseteq\mathfrak{h}$ and $\tau^{\prime}(\mathfrak{h}_{1})\subseteq\mathfrak{h}^{\prime}$, and we consider the following embedding of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ given by

[TABLE]

We denote by $\tilde{\mathfrak{h}}$ the image of $\mathfrak{h}$ into $\mathfrak{h}^{\prime}$ given in (251), and let $\mathfrak{z}^{\prime}$ be the subalgebra of $\mathfrak{g}^{\prime}$ given by

[TABLE]

Lemma 8.7**.**

Set $\pi_{\mathfrak{g}^{\prime}/\mathfrak{z}^{\prime}}=\prod\limits_{\underset{\alpha_{|_{\tilde{\mathfrak{h}}}\neq 0}}{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}}\alpha$. For all $h^{\prime}=\sum\limits_{k=1}^{m}ih_{k}\mathop{\rm E}\nolimits_{k,k}$, we have:

[TABLE]

For every $\phi\in\mathop{\rm S}\nolimits(W)$, we define the function $f_{\phi}$ on $\tau(\mathfrak{h}^{\mathop{\rm reg}\nolimits}_{1})$ by

[TABLE]

where $C_{\mathfrak{h}_{1}}$ is a constant of modulus 1 defined in [19, Lemma 8, page 17].

As in the previous section, we denote by $\mathop{\rm H}\nolimits^{\sharp}$ the two fold cover of $\tilde{\mathop{\rm H}\nolimits}$ such that the linear forms $\frac{\alpha}{2}$ are analytic integral for every roots $\alpha\in\Phi(\mathfrak{g},\mathfrak{h})$. Let $\xi^{\sharp}_{\frac{\alpha}{2}}:\mathop{\rm H}\nolimits^{\sharp}_{0}\to\mathop{\rm S}\nolimits^{1}$ the multiplicative character having the linear form $\frac{\alpha}{2}$ as derivative and let $c^{\sharp}_{-}:\mathfrak{h}\to\mathop{\rm H}\nolimits^{\sharp}_{0}$ be the map defined in section 5.1.

Proposition 8.8**.**

Fix $\Pi\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits},\omega^{\infty})$ of highest weight $\nu$ and let $\mu=\nu+\rho$. For every $\phi\in\mathop{\rm S}\nolimits(W)$, the following formula holds:

[TABLE]

where $\mathop{\rm ch}\nolimits(x)=|\mathop{\rm det}\nolimits_{\mathbb{R}}(x-1)|^{\frac{1}{2}}=|\mathop{\rm det}\nolimits(\Psi^{2n}\circ\Psi_{n}(x)-\mathop{\rm Id}\nolimits_{4n})|$ and $\mu^{\prime}\in\mathfrak{h}^{*}$ is defined by

[TABLE]

Proof.

For a proof of this proposition, we refer to [19, Corollary 38, page 47].

∎

Remark 8.9*.*

1. (1)

By [14], the weights $\nu$ and $\nu^{\prime}$ of the representations $\Pi$ and $\Pi^{\prime}$ respectively given by:

[TABLE]

where $\nu_{1}\geq\ldots\geq\nu_{n}\geq 0$. By considering a decreasing sequence $\nu_{1}\geq\ldots\geq\nu_{m}\geq 0$ with at most $n$ non zero $\nu_{i}$, we get

[TABLE]

with $\tau_{a}=-n-\nu_{m+1-a}$, where $(\tau_{a})_{a\in[|1,m|]}$ is a decreasing sequence of negative numbers.

The linear form $\rho$ is given by:

[TABLE]

Then $\mu=\sum\limits_{k=1}^{n}(\nu+\rho)_{a}e_{a}$ with $(\nu+\rho)_{a}=\nu_{a}+n-a+1$ and

[TABLE] 2. (2)

For every $x,y\in\mathfrak{g}$ or $\mathfrak{g}^{\prime}$, we denote by $B$ the bilinear form defined by

[TABLE]

The form $B$ is $\mathop{\rm G}\nolimits$ (resp. $\mathop{\rm G}\nolimits^{\prime}$)-invariant and non-degenerate on $\mathfrak{g}$ and $\mathfrak{g}^{\prime}$. More precisely, for all $x=\sum\limits_{k=1}^{n}ix_{k}\mathop{\rm E}\nolimits_{k,k}$ (resp. $x^{\prime}=\sum\limits_{k=1}^{m}ix^{\prime}_{k}\mathop{\rm E}\nolimits_{k,k}$) and $y=\sum\limits_{k=1}^{n}iy_{k}\mathop{\rm E}\nolimits_{k,k}$ (resp. $y^{\prime}=\sum\limits_{k=1}^{m}iy^{\prime}_{k}\mathop{\rm E}\nolimits_{k,k}$), the form $B$ is given by

[TABLE]

Theorem 8.10**.**

For all regular element $y\in\tilde{\mathfrak{h}}$ and $y^{\prime}\in\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$, we get:

[TABLE]

where $n(\tilde{\mathfrak{h}}^{\prime})$ is the number of non-compacts positives roots which do not vanish on $\tilde{\mathfrak{h}}$.

It is clear that for all $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$,

[TABLE]

is a Schwartz function on $W$. Using the Weyl integration formula, we get for all $y\in\tau(\mathfrak{h}_{1})$

[TABLE]

where $\psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}:\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}\to\mathbb{C}$ is given by:

[TABLE]

Using the same method than before, we get the following equality:

[TABLE]

We denote by $\mathscr{A}_{\psi}$ the function on $\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits}$ defined by

[TABLE]

Then,

[TABLE]

Before giving the main theorem, we prove a result concerning the right hand-side of (255). More particularly, we are interested in the support of the following distribution:

[TABLE]

Proposition 8.11**.**

Let $\beta_{1}$ and $\beta_{2}$ be the two integers defined by:

[TABLE]

[TABLE]

Then,

[TABLE]

Proof.

According to [19, Lemma 33, page 43], we get:

[TABLE]

where $a_{k}=\mu^{\prime}_{k}-m+n+1$ and $b_{k}=-\mu^{\prime}_{k}-m+n+1$. In particular, $a_{k}\geq 1$ (resp. $b_{k}\geq 1$) implies $b_{k}\leq 0$ (resp. $a_{k}\leq 0$).

Using [19, Appendix C], we get the following equality

[TABLE]

where $P_{a_{k},b_{k}}$ and $Q_{a_{k},b_{k}}$ are the polynomials defined in [19, Appendix C]. In particular, using [19, Appendix C], the support of the polynomial $P_{a_{k},b_{k}}$ are

[TABLE]

Then, we obtain:

[TABLE]

and finally

[TABLE]

∎

From now on, we denote by $\mathop{\rm supp}\nolimits(\beta_{1},\beta_{2})$ the support of the distribution given in equation (268). We denote by $\tilde{\mathfrak{h}}^{\perp}$ the orthogonal complement of $\tilde{\mathfrak{h}}$ with respect to the bilinear form $B$, and by $\mathop{\rm pr}\nolimits:\mathfrak{h}\to\tilde{\mathfrak{h}}$ the associated projection, i.e.

[TABLE]

Finally, let us consider the subgroup $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h},\beta)$ of $\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})$ defined by:

[TABLE]

Theorem 8.12**.**

The character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}\in\mathscr{R}(\tilde{\mathop{\rm G}\nolimits}^{\prime},\omega^{\infty})$ is given, up to a constant, by the following formula:

[TABLE]

Proof.

By (255) and (268), we get

[TABLE]

Then,

[TABLE]

Again, playing with the support of $f_{\phi}$, we get:

[TABLE]

We identify $\mathfrak{h}$ with its image $\tilde{\mathfrak{h}}$ in $\mathfrak{h}^{\prime}$. Therefore, the equation (281) can be written as:

[TABLE]

By (267), this can be written as:

[TABLE]

We now choose a particular function $\psi\in\mathscr{C}^{\infty}_{c}(\mathop{\rm G}\nolimits^{\prime}.\mathfrak{h}^{\prime\mathop{\rm reg}\nolimits})$. We denote by $\mathop{\rm G}\nolimits^{\prime c}\subseteq\mathop{\rm G}\nolimits^{\prime}$ the domain of the Cayley transform in $\mathop{\rm G}\nolimits^{\prime}$. Fix $\Psi\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm G}\nolimits}^{\prime})$ such that $\mathop{\rm supp}\nolimits(\Psi)\in\widetilde{\mathop{\rm G}\nolimits^{\prime c}}\subseteq\widetilde{\mathop{\rm G}\nolimits}^{\prime}$ and let $j_{\mathfrak{g}^{\prime}}$ be the Jacobian of $c:\mathfrak{g}^{\prime c}\to\mathop{\rm G}\nolimits^{\prime c}$. We recall that $\tilde{c}_{-}$ is defined by $\tilde{c}_{-}(x^{\prime})=\tilde{c}(x^{\prime})\tilde{c}(0)^{-1}$ and let $\psi$ be the function given by

[TABLE]

With such a function $\psi$, the integral (283) becomes:

[TABLE]

where $\mathop{\rm pr}\nolimits$ is the projection given in the equation (277).

To get the character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}$, we would like to write the integral defined in (285) as an integral over $\widetilde{\mathop{\rm H}\nolimits}^{\prime}$. For all $h^{\prime}=\sum\limits_{k=1}^{m}ih_{k}\mathop{\rm E}\nolimits_{k,k}$, we have:

[TABLE]

Similarly, we obtain:

[TABLE]

In particular, we get:

[TABLE]

For all $h^{\prime}=\sum\limits_{k=1}^{m}ih_{k}\mathop{\rm E}\nolimits_{k,k}$, we get:

[TABLE]

[TABLE]

Then,

[TABLE]

and the equation (285) becomes:

[TABLE]

Finally,

[TABLE]

Using the invariance of $\left|\prod\limits_{\alpha\in\Phi^{+}(\mathfrak{g}^{\prime},\mathfrak{h}^{\prime})}(h^{\frac{\alpha}{2}}-h^{-\frac{\alpha}{2}})\right|^{2}\Psi^{\tilde{\mathop{\rm G}\nolimits}^{\prime}/\tilde{\mathop{\rm H}\nolimits}^{\prime}}(h)$ under the action of $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$, we get:

[TABLE]

and

[TABLE]

Finally, we obtain that the character $\Theta_{\Pi^{\prime}}$ is given, up to a constant, by the formula:

[TABLE]

∎

Appendix A Dual pairs viewed as Lie supergroups

For readers who are not familiar with the "correspondence" between irreducible reductive dual pairs and some Lie supergroups, we recall the main ideas in this section. This correspondence appears to be very useful when one wants to work with dual pairs. All of this section is based on the article [23].

By a Lie supergroup, here we mean a Harish-Chandra pair. We recall the definition here (more details can be found in [26]).

Definition A.1**.**

A Harish-Chandra pair is a triple $(\mathop{\rm G}\nolimits,\mathfrak{g}=\mathfrak{g}_{\overline{0}}\oplus\mathfrak{g}_{\overline{1}},\mathop{\rm Ad}\nolimits)$ where:

• •

$\mathop{\rm G}\nolimits$ is a Lie group with the Lie algebra $\mathfrak{g}_{\overline{0}}$,

• •

$\mathfrak{g}$ is a Lie superalgebra,

• •

$\mathop{\rm Ad}\nolimits:\mathop{\rm G}\nolimits\to\mathop{\rm GL}\nolimits(\mathfrak{g})$ is a representation of $\mathop{\rm G}\nolimits$ such that the differential is equal to the action of $\mathfrak{g}_{\overline{0}}$ on $\mathfrak{g}$ via the superbracket.

Let $(\mathop{\rm G}\nolimits,\mathop{\rm G}\nolimits^{\prime})$ be an irreducible reductive dual pair of type $I$ in $\mathop{\rm Sp}\nolimits(W)$. According to [17, Lecture 5, Theorem 5.3], there exists a division algebra $\mathbb{D}$ over $\mathbb{R}$ with an involution $\iota$, a hermitian space $(V,(\cdot,\cdot))$ and a skew-hermitian space $(V^{\prime},(\cdot,\cdot)^{\prime})$ such that $W=V\otimes_{\mathbb{D}}V^{\prime}$, and $\mathop{\rm G}\nolimits$ (resp. $\mathop{\rm G}\nolimits^{\prime}$) can be identified with the isometry group of the form $(V,(\cdot,\cdot)^{\prime})$ (resp. $(V^{\prime},(\cdot,\cdot)^{\prime})$). Moreover, the symplectic form is given by the following formula

[TABLE]

Denote by $E_{\overline{0}}=V$ and $E_{\overline{1}}=V^{\prime}$ and let $E=E_{\overline{0}}\oplus E_{\overline{1}}$. We denote by $\mathop{\rm End}\nolimits(E)$ the set of endomorphisms of $E$ with $\mathbb{Z}_{2}$-graduation given by:

[TABLE]

We consider the even bilinear form $\tau=(\cdot,\cdot)\oplus(\cdot,\cdot)^{\prime}$ defined on $E$ by

[TABLE]

Let $s\in\mathop{\rm End}\nolimits(E)_{\overline{0}}$ be defined by

[TABLE]

The following equality holds

[TABLE]

We consider

[TABLE]

[TABLE]

[TABLE]

As shown in [23], $(\mathop{\rm S}\nolimits,\mathfrak{s},\mathop{\rm Ad}\nolimits)=(\mathop{\rm GL}\nolimits(E,\tau)_{\overline{0}},\mathfrak{g}(E,\tau)=\mathfrak{g}(E,\tau)_{\overline{0}}\oplus\mathfrak{g}(E,\tau)_{\overline{1}},\mathop{\rm Ad}\nolimits)$ is a Lie supergroup (where $\mathop{\rm Ad}\nolimits$ is the natural action by conjugation). Moreover, the space $\mathop{\rm Hom}\nolimits(V_{\overline{0}},V_{\overline{1}})$ is identified with the symplectic vector space $W$ and the symplectic form is given by:

[TABLE]

where $\{\cdot,\cdot\}$ is the superbracket on $\mathfrak{g}(E,\tau)$.

In this context, the moment maps $\tau:W\to\mathfrak{g}$ and $\tau^{\prime}:W\to\mathfrak{g}^{\prime}$ are defined, for $X=\begin{pmatrix}0&B\\ B^{*}&0\end{pmatrix}$, by $\tau(X)=X^{2}_{|_{V_{\overline{0}}}}$ and $\tau^{\prime}(X)=X^{2}_{|_{V_{\overline{1}}}}$. In particular, we have:

[TABLE]

Appendix B Some comments concerning Theorem 5.11

B.1. How can we get the constant $C$?

To determine the constant appearing in Theorem 5.11, we use an idea developed by Harish-Chandra in [9, Chapter III, Section 41, page 97]. Fix $a\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm G}\nolimits}^{\prime})$ a $\widetilde{\mathop{\rm K}\nolimits}^{\prime}-$right invariant function such that:

[TABLE]

To simplify the notations, we will denote by $\mathscr{W}$ the compact Weyl group $\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$ of $\mathop{\rm K}\nolimits^{\prime}$. Fix

[TABLE]

In particular,

[TABLE]

For every function $\beta\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm H}\nolimits}^{\prime\mathop{\rm reg}\nolimits})$, we denote by $f_{\beta}$ the function of $\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm G}\nolimits}^{\prime}_{\widetilde{\mathop{\rm H}\nolimits}^{\prime}})$ defined as

[TABLE]

(here, according to [9], $\widetilde{\mathop{\rm G}\nolimits}^{\prime}_{\widetilde{\mathop{\rm H}\nolimits}^{\prime}}$ is the set of the $\widetilde{\mathop{\rm G}\nolimits}^{\prime}$-orbits of $\widetilde{\mathop{\rm H}\nolimits}^{\prime\mathop{\rm reg}\nolimits}$ and $h^{x}=xhx^{-1}$). Similarly, we define the function $g_{\beta}$ of $\mathscr{C}^{\infty}(\widetilde{\mathop{\rm K}\nolimits}^{\prime})$ as

[TABLE]

We denote by $f=\frac{1}{2}\mathop{\rm dim}\nolimits(\mathop{\rm G}\nolimits^{\prime}/\mathop{\rm H}\nolimits^{\prime})$, $f_{0}=\frac{1}{2}\mathop{\rm dim}\nolimits(\mathop{\rm K}\nolimits^{\prime}/\mathop{\rm H}\nolimits^{\prime})$ and $e=f-f_{0}$. From now on, we will identify a representation $\Pi_{\gamma}\in\widehat{\widetilde{\mathop{\rm K}\nolimits}^{\prime}}$ with its highest weight $\gamma$. According to [9, page 97], we have:

[TABLE]

where $m(\gamma)$ is the multiplicity of $\gamma$ in the decomposition of $\Theta_{\Pi^{\prime}}$. The last equality can be rewritten as follows:

[TABLE]

The equation (106) is of the form

[TABLE]

with $P(h)=\sum\limits_{\eta\in\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h},m)/\mathscr{W}(\mathop{\rm G}\nolimits,\mathfrak{h})_{m}}\sum\limits_{\sigma\in\mathscr{W}(\mathop{\rm Z}\nolimits^{\prime}(m),\mathfrak{h}^{\prime})}\sum\limits_{\tau\in\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})}\mathop{\rm sgn}\nolimits(\eta\sigma\tau)\mathop{\rm pr}\nolimits_{m}(\tau h)^{-\eta^{-1}\mu^{\prime}}(\tau h)^{\sigma\rho_{\mathfrak{z}^{\prime}(m)}}$.

On the other hand, we have

[TABLE]

So, we get the following equality

[TABLE]

This equality is valid for every function $\beta\in\mathscr{C}^{\infty}_{c}(\widetilde{\mathop{\rm H}\nolimits}^{\prime\mathop{\rm reg}\nolimits})$, in particular for the function $\beta$ given by $\beta(h)=\cfrac{h^{-(\lambda+\rho)}}{\Delta_{+}(h)}$, where $\lambda$ is the highest weight of the representation $\Pi^{\prime}$. For this particular function $\beta$, we get:

[TABLE]

Using [16, Chapter IV, Section 2, Corollary 4.10], we obtain:

[TABLE]

By the Weyl integration formula, we have:

[TABLE]

According to [12, Theorem 9, page 555], there exists a unique $\widetilde{\mathop{\rm K}\nolimits}^{\prime}$-type with multiplicity $1$ and of highest weight $\lambda$. Then,

[TABLE]

Finally, we obtain the following formula for the constant $C$:

[TABLE]

B.2. A remark concerning the interval $[|m_{\min},m_{\max}|]$ for $\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(1,\mathbb{C})$

In section 5, more particularly in Theorem 5.11, we get an explicit formula for the character of the representation $\Pi^{\prime}$ on the maximal compact Cartan subgroup of $\mathop{\rm U}\nolimits(p,q,\mathbb{C})$. In particular, we consider an embedding of $\mathfrak{h}$ in $\mathfrak{h}^{\prime}$, denoted by $\mathfrak{h}^{\prime}(m)$, where $m$ can be chosen arbitrarily in $[|m_{\min},m_{\max}|]$, and the surprising fact is that the character formula (106) does not depend of the choice of $m$. In this section, for the dual pair $(\mathop{\rm G}\nolimits=\mathop{\rm U}\nolimits(1,\mathbb{C}),\mathop{\rm G}\nolimits^{\prime}=\mathop{\rm U}\nolimits(p,q,\mathbb{C})),1\leq p\leq q$, we compute explicitly the formula (106) for two different values of $m$. For this particular dual pair, we get:

[TABLE]

[TABLE]

Then, the possible values for the parameter $m$ are:

[TABLE]

Using (106), we get that the character $\Theta_{\Pi^{\prime}}$ is given by:

[TABLE]

where $C$ is a constant. We start with $m=1$. For all $h=\mathop{\rm diag}\nolimits(h_{1},\ldots,h_{p+q})$, we have $\mathop{\rm pr}\nolimits_{1}(h)=h_{1}$. Moreover,

[TABLE]

So,

[TABLE]

According to (106),

[TABLE]

Lemma B.1**.**

The following equality holds

[TABLE]

Proof.

First,

[TABLE]

Then, for all $k\in[|1,p|]$ and $\sigma\in\mathscr{W}(\mathop{\rm K}\nolimits^{\prime},\mathfrak{h}^{\prime})$ such that $\sigma(1)=k$, we have $\sigma\left(\left\{2,\ldots,p+q\right\}\right)=\left\{1,\ldots,p+q\right\}\setminus\left\{k\right\}$, and

[TABLE]

Finally,

[TABLE]

∎

Up to a constant, the character $\Theta_{\Pi^{\prime}}(h)$ is given by the following formula:

[TABLE]

Now, we treat the case $m=0$. We have $\mathop{\rm pr}\nolimits_{0}(h)=h_{p+q}$ and

[TABLE]

Then,

[TABLE]

i.e.

[TABLE]

Lemma B.2**.**

[TABLE]

Finally, the character $\Theta_{\Pi^{\prime}}$ of $\Pi^{\prime}$ is given, up to a constant, by the formula:

[TABLE]

Now, we prove that the character formula given in Theorem 5.11 is independent of the choice of $m\in[|0,1|]$.

Proposition B.3**.**

For all $\lambda_{1}\in[|-q+1,p-1|]$, we have:

[TABLE]

Remark B.4*.*

The previous proposition can be reformulated as follows: for all $k\in[|0,p+q-2|]$, the following equality holds:

[TABLE]

Proof.

We first multiply both sides of (312) by $\prod\limits_{1\leq i<j\leq p+q}(h_{i}-h_{j})$. Then, (312) can be written as

[TABLE]

which is equivalent to

[TABLE]

For each $k\in[|0,p+q-2|]$, we denote by $M^{k}$ the matrix in $\mathop{\rm M}\nolimits(p+q,\mathbb{R})$ defined by the following equalities:

• •

$M^{k}_{i,j}=h^{j-1}_{i}$ if $j\leq k$

• •

$M^{k}_{i,k+1}=M_{i,k+2}=t^{k}_{i}$

• •

$M^{k}_{i,j}=t^{j-2}_{i}$ if $j>k+2$

i.e.

[TABLE]

The $(k+1)th$ et $(k+2)th$ columns are equals, so for every $k\in[|0,p+q-2|]$, the determinant of $M^{k}$ is zero. Developing the determinant of $M^{k}$ with respect to the $(k+1)th$-column, we conclude:

[TABLE]

∎