On the dual topology of the groups U(n)xH_n

TL;DR

This paper characterizes the topology of the space of admissible coadjoint orbits for a specific semi-direct product group involving unitary and Heisenberg groups, establishing a homeomorphism with the unitary dual.

Contribution

It explicitly determines the topology of the admissible coadjoint orbit space and proves the homeomorphism with the unitary dual for the group G_n.

Findings

The topology of the admissible coadjoint orbit space is explicitly determined.

The correspondence between the dual space and the orbit space is a homeomorphism.

Provides a detailed topological description of the dual space for G_n.

Abstract

Let $G_n=U(n)\ltimes {\mathbb H}_n $ be the semi-direct product of the unitary group acting by automorphisms on the Heisenberg group ${\mathbb H}_n$. According to Lipsman, the unitary dual $\widehat {G_n} $ of $G_n $ is in one to one correspondence with the space of admissible coadjoint orbits $\mathfrak g_n^\ddagger /G_n $ of $G_n $. In this paper, we determine the topology of the space $\mathfrak g_n^\ddagger /G_n $ and we show that the correspondence with $\widehat {G_n} $ is a homeomorphism.

Equations

$$⟨x,y{⟩}_{{\mathbb{C}}^n}=\sum _{j=1}^n{x}_j\overline{y_j}\forall x=(x_1,...,x_n),y=(y_1,...,y_n)\in {\mathbb{C}}^n.$$

$$⟨.,.{⟩}_{{\mathbb{C}}^n}=(.,.{)}_{{\mathbb{C}}^n}+i\omega (.,.{)}_{{\mathbb{C}}^n}.$$

$$(z,t)(z^{\prime},t^{\prime}):=\Big{(}z+z^{\prime},t+t^{\prime}-\frac{1}{2}\omega(z,z^{\prime})_{{\mathbb{C}}^{n}}\Big{)}\>\>\>\forall(z,t),(z^{\prime},t^{\prime})\in\mathbb{H}_{n}.$$

$$A.(z,t):=(Az,t)\forall A\in U(n)\forall (z,t)\in {\mathbb{H}}_n.$$

$$(A,z,t)(B,z^{\prime},t^{\prime}):=\Big{(}AB,z+Az^{\prime},t+t^{\prime}-\frac{1}{2}\omega(z,Az^{\prime})_{{\mathbb{C}}^{n}}\Big{)}\>\>\>\forall(A,z,t),(B,z^{\prime},t^{\prime})\in G_{n}.$$

$$\displaystyle\big{[}(z,t),(w,s)\big{]}:=\big{(}0,-\omega(z,w)_{{\mathbb{C}}^{n}}\big{)}\>\>\>\forall(z,t),(w,s)\in\mathfrak{h}_{n}$$

$$A.(z,t):=(Az,0)\forall A\in \mathfrak{u}(n)\forall (z,t)\in {\mathfrak{h}}_n.$$

$$\text{A}d(A,z,t)(B,w,s)$$

$$\text{A}d(A)(B,w,s)=(AB{A}^{\ast },Aw,s).$$

$$\displaystyle\big{[}(A,z,t),(B,w,s)\big{]}$$

$$⟨A,B{⟩}_{\mathfrak{u}(n)}:=\text{t}r(AB)\forall A,B\in \mathfrak{u}(n).$$

$$z^{\vee }(w):=\omega (z,w{)}_{{\mathbb{C}}^n}\forall w\in {\mathbb{C}}^n.$$

$$z\times w(B):=w^{\vee }(Bz)=\omega (w,Bz{)}_{{\mathbb{C}}^n}\forall B\in \mathfrak{u}(n).$$

$$A{z}^{\vee }:=z^{\vee }\circ A^{-1}$$

$$z^{\vee }\circ B$$

$$z\times w$$

$$A(z\times w)A^{\ast }$$

$$\displaystyle\big{\langle}(U,u,x),(B,w,s)\big{\rangle}_{\mathfrak{g}_{n}}=\langle U,B\rangle_{\mathfrak{u}(n)}+u^{\vee}(w)+xs\>\>\>\forall(B,w,s)\in\mathfrak{g}_{n}.$$

$${\text{A}d}^{\ast }(A)(U,u,x)=(AU{A}^{\ast },Au,x)\forall (U,u,x)\in \mathfrak{u}(n)\times {\mathbb{C}}^n\times \mathbb{R}$$

$${\text{A}d}^{*}(A,z,t)(U,u,x)=\Big{(}AUA^{*}+z\times(Au)+\frac{x}{2}z\times z,Au+xz,x\Big{)}.$$

$${\mathcal{O}}_{(U,u,x)}=\bigg{\{}\Big{(}AUA^{*}+z\times(Au)+\frac{x}{2}z\times z,Au+xz,x\Big{)}\Big{|}\>A\in U(n),z\in{\mathbb{C}}^{n}\bigg{\}}$$

$${\mathcal{O}}_{(U,u,x)}=\bigg{\{}{\text{A}d}^{*}(A)\Big{(}U+z\times u+\frac{x}{2}z\times z,u+xz,x\Big{)}\Big{|}\>A\in U(n),z\in{\mathbb{C}}^{n}\bigg{\}}.$$

$$\displaystyle\big{\langle}uz^{*}+zu^{*},B\big{\rangle}_{\mathfrak{u}(n)}={\text{t}r}\big{(}(uz^{*}+zu^{*})B\big{)}=\sum_{1\leq i,j\leq n}B_{ji}z_{i}\overline{u}_{j}-\sum_{1\leq i,j\leq n}u_{i}\overline{B}_{ij}\overline{z}_{j}=-2iz\times u(B).$$

$${\mathcal{F}}_{\alpha}(n)=\Bigg{\{}f:{\mathbb{C}}^{n}\longrightarrow{\mathbb{C}}\textrm{ entire }\Big{|}~{}\int\limits_{{\mathbb{C}}^{n}}|f(w)|^{2}e^{-\frac{|\alpha|}{2}|w|^{2}}dw<\infty\Bigg{\}}$$

$${\sigma }_{\alpha }(z,t)f(w):=e^{i\alpha t-\frac{\alpha }{4}|z{|}^2-\frac{\alpha }{2}⟨w,z{⟩}_{{\mathbb{C}}^n}}f(w+z)$$

$${\sigma }_{\alpha }(z,t)f(\overline{w}):=e^{i\alpha t+\frac{\alpha }{4}|z{|}^2+\frac{\alpha }{2}⟨\overline{w},\overline{z}{⟩}_{{\mathbb{C}}^n}}f(\overline{w}+\overline{z}).$$

$$W_{\alpha}(A):{\mathcal{F}}_{\alpha}(n)\rightarrow{\mathcal{F}}_{\alpha}(n),~{}W_{\alpha}(A)f(z):=f\big{(}A^{-1}z\big{)}\>\>\>\forall f\in{\mathcal{F}}_{\alpha}(n)~{}\forall z\in{\mathbb{C}}^{n}$$

$$\pi_{(\lambda,\alpha)}(A,z,t):=\tau_{\lambda}(A)\otimes\big{(}\sigma_{\alpha}(z,t)\circ W_{\alpha}(A)\big{)}\>\>\>\forall(A,z,t)\in G_{n}$$

$$J_{\lambda}:=\left(\begin{array}[]{ccc}i\lambda_{1}&\ldots&0\\ \vdots&\ddots&\vdots\\ 0&\ldots&i\lambda_{n}\\ \end{array}\right).$$

$$G_n[{\ell }_{\lambda ,\alpha }]$$

$$U(n)[{\ell }_{\lambda ,\alpha }]$$

$${\mathbb{H}}_n[{\ell }_{\lambda ,\alpha }]$$

$${\chi }_v(z,t):=e^{-i(v,z{)}_{{\mathbb{C}}^n}}\forall (z,t)\in {\mathbb{H}}_n.$$

Full text

2000 Mathematics Subject Classification. Primary 43A40; Secondary 22E45

1. 1 Introduction

2. 2 The space of admissible coadjoint orbits

3. 2.1 The coadjoint orbits of $G_{n}$

4. 3 The spectrum of $G_{n}$

5. 3.1 The spectrum and the admissible coadjoint orbits of $G_{n}$

6. 4 Convergence in the quotient space $\mathfrak{g}_{n}^{\ddagger}/G_{n}$

7. 5 The continuity of the inverse of the Kirillov-Lipsman map ${\mathcal{K}}$

8. 5.1 The representation $\pi_{(\mu,r)}$

9. 5.2 The representation $\tau_{\lambda}$

10. 5.3 The representation $\pi_{(\lambda,\alpha)}$

11. 6 The continuity of ${\mathcal{K}}$

12. 7 The final result

13. 8 Appendix

14. 9 References

1. Introduction

Let $G$ be a locally compact group and $\widehat{G}$ the unitary dual of $G$, i.e., the set of equivalence classes of irreducible unitary representations of $G$, endowed with the pullback of the hull-kernel topology on the primitive ideal space of $C^{*}(G)$, the $C^{*}$-algebra of $G$. Besides the fundamental problem of determining $\widehat{G}$ as a set, there is a genuine interest in a precise and neat description of the topology on $\widehat{G}$. For several classes of Lie groups, such as simply connected nilpotent Lie groups or, more generally, exponential solvable Lie groups, the Euclidean motion groups and also the extension groups $U(n)\ltimes\mathbb{H}_{n}$ considered in this paper, there is a nice geometric object parameterizing $\widehat{G}$, namely the space of admissible coadjoint orbits in the dual $\mathfrak{g}^{*}$ of the Lie algebra $\mathfrak{g}$ of $G$.

In such a situation, the natural and important question arises of whether the bijection between the orbit space, equipped with the quotient topology, and $\widehat{G}$ is a homeomorphism. In [23], H. Leptin and J. Ludwig have proved that for an exponential solvable Lie group $G=\hbox{exp}\mathfrak{g}$, the dual space $\widehat{G}$ is homeomorphic to the space of coadjoint orbits $\mathfrak{g}^{*}/G$ through the Kirillov mapping. On the other hand, it had been shown in [12] that the dual topology of the classical motion groups $SO(n)\ltimes{\mathbb{R}}^{n}$ for $n\geq 2$ can be linked to the topology of the quotient space of admissible coadjoint orbits.

In this paper, we consider the semi-direct product $G_{n}=U(n)\ltimes\mathbb{H}_{n}$ for $n\geq 1$ and we identify its dual space $\widehat{G_{n}}$ with the lattice of admissible coadjoint orbits. Lipsman showed in [25] that each irreducible unitary representation of $G_{n}$ can be constructed by holomorphic induction from an admissible linear functional $\ell$ of the Lie algebra $\mathfrak{g}_{n}$ of $G_{n}$. Furthermore, two irreducible representations in $\widehat{G_{n}}$ are equivalent if and only if their respective linear functionals are in the same $G_{n}$-orbit. We prove that this identification is a homeomorphism.

This paper is structured in the following way. Section 2 contains preliminary material and summarizes results from previous work concerning the dual space of $G_{n}$ which is identified with the space of its admissible coadjoint orbits. The representations attached to an admissible linear functional are obtained via Mackey’s little-group method and the dual space $\widehat{G_{n}}$ of $G_{n}$ is given by the parameter space {\mathcal{K}}_{n}=\Big{\{}\alpha\in{\mathbb{R}}^{*},r>0,\rho_{\mu}\in\widehat{U(n-1)},\tau_{\lambda}\in\widehat{U(n)}\Big{\}}. In Section 3, we shall link the convergence of sequences of admissible coadjoint orbits to the convergence in ${\mathcal{K}}_{n}$. Section 4 describes the dual topology of a second countable locally compact group. In the last two sections, we discuss the topology of the dual space of our groups $G_{n}$.

2. The space of admissible coadjoint orbits

Let ${\mathbb{C}}^{n}$ be the $n$-dimensional complex vector space equipped with the standard scalar product $\langle.,.\rangle_{{\mathbb{C}}^{n}}$ given by

[TABLE]

Moreover, let $(.,.)_{{\mathbb{C}}^{n}}$ and $\omega(.,.)_{{\mathbb{C}}^{n}}$ denote the real and the imaginary part of $\langle.,.\rangle_{{\mathbb{C}}^{n}}$, respectively, i.e.

[TABLE]

The bilinear forms $(.,.)_{{\mathbb{C}}^{n}}$ and $\omega(.,.)_{{\mathbb{C}}^{n}}$ define a positive definite inner product and a symplectic structure on the underlying real vector space ${\mathbb{R}}^{2n}$ of ${\mathbb{C}}^{n}$, respectively. The associated Heisenberg group $\mathbb{H}_{n}={\mathbb{C}}^{n}\times{\mathbb{R}}$ of dimension $2n+1$ over ${\mathbb{R}}$ is given by the group multiplication

[TABLE]

Furthermore, consider the unitary group $U(n)$ of automorphisms of $\mathbb{H}_{n}$ preserving $\langle.,.\rangle_{{\mathbb{C}}^{n}}$ on ${\mathbb{C}}^{n}$ which embeds into $\text{Aut}(\mathbb{H}_{n})$ via

[TABLE]

Then, $U(n)$ yields a maximal compact connected subgroup of $\text{Aut}(\mathbb{H}_{n})$ (see [14], Theorem 1.22 and [20], Chapter I.1). Moreover, $G_{n}=U(n)\ltimes\mathbb{H}_{n}$ denotes the semi-direct product of $U(n)$ with the Heisenberg group $\mathbb{H}_{n}$ equipped with the group law

[TABLE]

The Lie algebra $\mathfrak{h}_{n}$ of $\mathbb{H}_{n}$ will be identified with $\mathbb{H}_{n}$ itself via the exponential map. The Lie bracket of $\mathfrak{h}_{n}$ is defined as

[TABLE]

and the derived action of the Lie algebra $\mathfrak{u}(n)$ of $U(n)$ on $\mathfrak{h}_{n}$ is

[TABLE]

Denoting by $\mathfrak{g}_{n}=\mathfrak{u}(n)\ltimes\mathfrak{h}_{n}$ the Lie algebra of $G_{n}$, for all $(A,z,t)\in G_{n}$ and all $(B,w,s)\in\mathfrak{g}_{n}$, one gets

[TABLE]

where $A^{*}$ is the adjoint matrix of $A$. In particular

[TABLE]

From Identity (2), one can deduce the Lie bracket

[TABLE]

for all $(A,z,t),(B,w,s)\in\mathfrak{g}_{n}$.

2.1. The coadjoint orbits of $G_{n}$

In this subsection, the coadjoint orbit space of $G_{n}$ will be described according to [3], Section 2.5.

In the following, $\mathfrak{u}(n)$ will be identified with its vector dual space $\mathfrak{u}^{*}(n)$ with the help of the $U(n)$-invariant inner product

[TABLE]

For $z\in{\mathbb{C}}^{n}$, define the linear form $z^{\vee}$ in $({\mathbb{C}}^{n})^{*}$ by

[TABLE]

Furthermore, one defines a map $\times:{\mathbb{C}}^{n}\times{\mathbb{C}}^{n}\longrightarrow\mathfrak{u}^{*}(n)$, $(z,w)\mapsto z\times w$ by

[TABLE]

One can verify that for $A\in U(n)$, $B\in\mathfrak{u}(n)$ and $z,w\in{\mathbb{C}}^{n}$,

[TABLE]

Hence, the dual \mathfrak{g}^{*}_{n}=\big{(}\mathfrak{u}(n)\ltimes\mathfrak{h}_{n}\big{)}^{*} will be identified with $\mathfrak{u}(n)\oplus\mathfrak{h}_{n}$, i.e. each element $\ell\in\mathfrak{g}^{*}_{n}$ can be identified with an element $(U,u,x)\in\mathfrak{u}(n)\times{\mathbb{C}}^{n}\times{\mathbb{R}}$ such that

[TABLE]

From (2.2) and (2.3), one obtains for all $A\in U(n)$,

[TABLE]

and for all $(A,z,t)\in G_{n}$ and all $(U,u,x)\in\mathfrak{u}(n)\times{\mathbb{C}}^{n}\times{\mathbb{R}}$,

[TABLE]

Letting $A$ and $z$ vary over $U(n)$ and ${\mathbb{C}}^{n}$, respectively, the coadjoint orbit ${\mathcal{O}}_{(U,u,x)}$ of the linear form $(U,u,x)$ can then be written as

[TABLE]

or equivalently, by replacing $z$ by $Az$ and using Identity (2.4),

[TABLE]

Here, $z$ is regarded as a column vector $z=(z_{1},\ldots,z_{n})^{T}$ and $z^{*}:=\overline{z}^{t}$.

One can show as follows that $z\times u\in\mathfrak{u}^{*}(n)\cong\mathfrak{u}(n)$ is the $n\times n$ skew-Hermitian matrix $\frac{i}{2}(uz^{*}+zu^{*})$:

For all $B\in\mathfrak{u}(n)$,

[TABLE]

In particular, $z\times z$ is the skew-Hermitian matrix $izz^{*}$ whose entries are determined by $(izz^{*})_{lj}=iz_{l}\overline{z}_{j}$.

3. The spectrum of $G_{n}$

3.1. The spectrum and the admissible coadjoint orbits of $G_{n}$

The description of the spectrum of $G_{n}$ is based on a method by Mackey (see [28], Chapter 10), which states that one has to determine the irreducible unitary representations of the subgroup $\mathbb{H}_{n}$ in order to construct representations of $G_{n}$.

 

First, regard the infinite-dimensional irreducible representations of the Heisenberg group $\mathbb{H}_{n}$, which are parameterized by ${\mathbb{R}}^{*}$:

For each element $\alpha\in{\mathbb{R}}^{*}$, the coadjoint orbit ${\mathcal{O}}_{\alpha}$ of the irreducible representation $\sigma_{\alpha}$ is the hyperplane {\mathcal{O}}_{\alpha}=\big{\{}(z,\alpha)|\>z\in{\mathbb{C}}^{n}\big{\}}. Since for every $\alpha$, this orbit is invariant under the action of $U(n)$, the unitary group $U(n)$ preserves the equivalence class of $\sigma_{\alpha}$.

The representation $\sigma_{\alpha}$ can be realized for $\alpha>0$ in the Fock space

[TABLE]

as

[TABLE]

and for $\alpha<0$ on the space ${\mathcal{F}}_{\alpha}(n)=\overline{{\mathcal{F}}_{-\alpha}(n)}$ as

[TABLE]

See [14], Chapter 1.6 or [19], Section 1.7 for a discussion of the Fock space.

 

For each $A\in U(n)$, the operator $W_{\alpha}(A)$ defined by

[TABLE]

intertwines $\sigma_{\alpha}$ and $(\sigma_{\alpha})_{A}$ given by $(\sigma_{\alpha})_{A}(z,t):=\sigma_{\alpha}(Az,t)$. $W_{\alpha}$ is called the projective intertwining representation of $U(n)$ on the Fock space. Then, by [28], Chapter 10, for each $\alpha\in{\mathbb{R}}^{*}$ and each element $\tau_{\lambda}$ in $\widehat{U(n)}$,

[TABLE]

is an irreducible unitary representation of $G_{n}$ realized in ${\mathcal{H}}_{(\lambda,\alpha)}:={\mathcal{H}}_{\lambda}\otimes{\mathcal{F}}_{\alpha}(n)$, where ${\mathcal{H}}_{\lambda}$ is the Hilbert space of $\tau_{\lambda}$.

Associate to $\pi_{(\lambda,\alpha)}$ the linear functional $\ell_{\lambda,\alpha}:=(J_{\lambda},0,\alpha)\in\mathfrak{g}_{n}^{*}$ given by

[TABLE]

Denote by $G_{n}[\ell_{\lambda,\alpha}]$, $U(n)[\ell_{\lambda,\alpha}]$ and $\mathbb{H}_{n}[\ell_{\lambda,\alpha}]$ the stabilizers of $\ell_{\lambda,\alpha}$ in $G_{n}$, $U(n)$ and $\mathbb{H}_{n}$, respectively. By Formula (2.5),

[TABLE]

It follows that $G_{n}[\ell_{\lambda,\alpha}]=U(n)[\ell_{\lambda,\alpha}]\ltimes\mathbb{H}_{n}[\ell_{\lambda,\alpha}]$. Hence, $\ell_{\lambda,\alpha}$ is aligned in the sense of Lipsman (see [25], Lemma 4.2).

 

The finite-dimensional irreducible representations of $\mathbb{H}_{n}$ are the characters $\chi_{v}$ for $v\in{\mathbb{C}}^{n}$, defined by

[TABLE]

Denote by $U(n)_{v}$ the stabilizer of the character $\chi_{v}$, or equivalently of the vector $v$, under the action of $U(n)$. Then, for every irreducible unitary representation $\rho$ of $U(n)_{v}$, the tensor product $\rho\otimes\chi_{v}$ is an irreducible representation of $U(n)_{v}\ltimes\mathbb{H}_{n}$ whose restriction to $\mathbb{H}_{n}$ is a multiple of $\chi_{v}$, and the induced representation

[TABLE]

is an irreducible representation of $G_{n}$. Furthermore, the restriction of $\pi_{(\rho,v)}$ to $U(n)$ is equivalent to $\hbox{ind}_{U(n)_{v}}^{U(n)}\rho$.

For any $v^{\prime}=Av$ for $A\in U(n)$ (i.e. $v$ and $v^{\prime}$ belong to the same sphere centered at [math] and of radius $r=\|v\|_{{\mathbb{C}}^{n}}$), one has $U(n)_{v^{\prime}}=AU(n)_{v}A^{*}$ and thus, the representation $\pi_{(\rho,v)}$ is equivalent with $\pi_{(\rho^{\prime},v^{\prime})}$ for any $\rho^{\prime}\in\widehat{U(n)_{v^{\prime}}}$ such that $\rho^{\prime}(B)=\rho(A^{*}BA)$ for each $B\in U(n)_{v^{\prime}}$. Hence, one can regard the character $\chi_{r}$ associated to the linear form $v_{r}$ which is identified with the vector $(0,\ldots,0,r)^{T}$ in ${\mathbb{C}}^{n}$. Throughout this text, denote by $\rho_{\mu}$ the representation of the subgroup $U(n-1)=U(n)_{v_{r}}$ with highest weight $\mu$ and by $\pi_{(\mu,r)}$ the representation $\pi_{(\rho_{\mu},v_{r})}=\hbox{ind}_{U(n-1)\ltimes\mathbb{H}_{n}}^{G_{n}}\rho_{\mu}\otimes\chi_{r}$. Its Hilbert space ${\mathcal{H}}_{(\mu,r)}$ is given by

[TABLE]

Again, $\pi_{(\mu,r)}$ is linked to the linear functional $\ell_{\mu,r}:=(J_{\mu},v_{r},0)\in\mathfrak{g}_{n}^{*}$ for

[TABLE]

By (2.5), one can check that

[TABLE]

since $AJ_{\mu}A^{*}\in\mathfrak{u}(n-1)$ and

[TABLE]

In addition,

[TABLE]

Thus, similarly to $\ell_{\lambda,\alpha}$, the linear functional $\ell_{\mu,r}$ is aligned.

 

One obtains in this way all the irreducible unitary representations of $G_{n}$ which are not trivial on $\mathbb{H}_{n}$.

The trivial extension of each element $\tau_{\lambda}$ of $\widehat{U(n)}$ to the entire group $G_{n}$ is an irreducible representation which will also be denoted by $\tau_{\lambda}$.

First of all, as $U(n)$ is a compact group, one knows that its spectrum is discrete and that every representation of $U(n)$ is finite-dimensional.

Now, let

[TABLE]

be a maximal torus of the unitary group $U(n)$ and let $\mathfrak{t}_{n}$ be its Lie algebra. By complexification of $\mathfrak{u}(n)$ and $\mathfrak{t}_{n}$, one gets the complex Lie algebras $\mathfrak{u}^{\mathbb{C}}(n)=\mathfrak{g}\mathfrak{l}(n,{\mathbb{C}})=M(n,{\mathbb{C}})$ and

[TABLE]

respectively, which is a Cartan subalgebra of $\mathfrak{u}^{\mathbb{C}}(n)$. For $j\in\{1,...,n\}$, define a linear functional $e_{j}$ by

[TABLE]

Let $P_{n}$ be the set of all dominant integral forms $\lambda$ for $U(n)$ which may be written in the form $\sum\limits_{j=1}^{n}i\lambda_{j}e_{j}$, or simply as $\lambda=(\lambda_{1},\cdots,\lambda_{n})$, where $\lambda_{j}$ are integers for every $j\in\{1,...,n\}$ such that $\lambda_{1}\geq\cdots\geq\lambda_{n}$. $P_{n}$ is a lattice in the vector dual space $\mathfrak{t}_{n}^{*}$ of $\mathfrak{t}_{n}$.

Since each irreducible unitary representation $(\tau_{\lambda},{\mathcal{H}}_{\lambda})$ of $U(n)$ is determined by its highest weight $\lambda\in P_{n}$, the spectrum $\widehat{U(n)}$ of $U(n)$ is in bijection with the set $P_{n}$.

 

For each $\lambda$ in $P_{n}$, the highest vector $\phi^{\lambda}$ in the Hilbert space ${\mathcal{H}}_{\lambda}$ of $\tau_{\lambda}$ verifies $\tau_{\lambda}(T)\phi^{\lambda}=\chi_{\lambda}(T)\phi^{\lambda}$, where $\chi_{\lambda}$ is the character of ${\mathbb{T}}_{n}$ associated to the linear functional $\lambda$ and defined by

[TABLE]

Moreover, for two irreducible unitary representations $(\tau_{\lambda},\mathcal{H}_{\lambda})$ and $(\tau_{\lambda^{\prime}},\mathcal{H}_{\lambda^{\prime}})$, the Schur orthogonality relation states that for all $\xi,\eta\in\mathcal{H}_{\lambda}$ and all $\xi^{\prime},\eta^{\prime}\in\mathcal{H}_{\lambda^{\prime}}$,

[TABLE]

where $d_{\lambda}$ denotes the dimension of the representation $\tau_{\lambda}$.

The linear functional corresponding to the irreducible $G_{n}$-representation $\tau_{\lambda}$ for $\lambda\in P_{n}$ is given by $\ell_{\lambda}:=(J_{\lambda},0,0)$.

According to [18], Chapter 1, if $\rho_{\mu}$ is an irreducible representation of $U(n-1)$ with highest weight $\mu=(\mu_{1},...,\mu_{n-1})$, the induced representation $\pi_{\mu}:=\hbox{ind}_{U(n-1)}^{U(n)}\rho_{\mu}$ of $U(n)$ decomposes with multiplicity one, and the representations of $U(n)$ that appear in this decomposition are exactly those with the highest weight $\lambda=(\lambda_{1},...,\lambda_{n})$ such that

[TABLE]

Therefore, by Mackey’s theory, the spectrum $\widehat{G_{n}}$ consists of the following families of representations:

1. (i)

$\pi_{(\lambda,\alpha)}$ for $\lambda\in P_{n}$ and $\alpha\in{\mathbb{R}}^{*}={\mathbb{R}}\setminus\{0\}$, 2. (ii)

$\pi_{(\mu,r)}$ for $\mu\in P_{n-1}$ and $r\in{\mathbb{R}}_{>0}~{}~{}$ and 3. (iii)

$\tau_{\lambda}$ for $\lambda\in P_{n}$.

Hence, $\widehat{G_{n}}$ is in bijection with the set

[TABLE]

A linear functional $\ell$ in $\mathfrak{g}_{n}^{*}$ is defined to be admissible if there exists a unitary character $\chi$ of the connected component of $G_{n}[\ell]$ such that $d\chi=i\ell_{|\mathfrak{g}_{n}[\ell]}$. A calculation shows that all the linear functionals $\ell_{\lambda,\alpha}$, $\ell_{\mu,r}$ and $\ell_{\lambda}$ are admissible. Then, according to [25], the representations $\pi_{(\lambda,\alpha)}$, $\pi_{(\mu,r)}$ and $\tau_{\lambda}$ described above are equivalent to the representations of $G_{n}$ obtained by holomorphic induction from their respective linear functionals $\ell_{\lambda,\alpha}$, $\ell_{\mu,r}$ and $\ell_{\lambda}$.

Denote by ${\mathcal{O}}_{(\lambda,\alpha)}$, ${\mathcal{O}}_{(\mu,r)}$ and ${\mathcal{O}}_{\lambda}$ the coadjoint orbits associated to the linear forms $\ell_{\lambda,\alpha}$, $\ell_{\mu,r}$ and $\ell_{\lambda}$, respectively. Let $\mathfrak{g}_{n}^{\ddagger}\subset\mathfrak{g}_{n}^{*}$ be the union of all the elements in ${\mathcal{O}}_{(\lambda,\alpha)}$, ${\mathcal{O}}_{(\mu,r)}$ and ${\mathcal{O}}_{\lambda}$ and denote by $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ the corresponding set in the orbit space. Now, from [25] follows that $\mathfrak{g}_{n}^{\ddagger}$ is the set of all admissible linear functionals of $\mathfrak{g}_{n}$.

We obtain in this way the Kirillov-Lipsman bijection

[TABLE]

between the space of admissible coadjoint orbits and the space of equivalence classes of irreducible unitary representations of $G_{n}$.

4. Convergence in the quotient space $\mathfrak{g}_{n}^{\ddagger}/G_{n}$

According to the last subsection, the spectrum of $G_{n}$ is parameterized by the dominant integral forms $\lambda$ for $U(n)$ and $\mu$ for $U(n-1)$, the non-zero $\alpha\in{\mathbb{R}}$ attached to the generic orbits ${\mathcal{O}}_{\alpha}$ in $\mathfrak{h}_{n}^{*}$ and the positive real $r$ derived from the natural action of the unitary group $U(n)$ on the characters of the Heisenberg group $\mathbb{H}_{n}$.

Moreover, it has been elaborated that the quotient space $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ of admissible coadjoint orbits is in bijection with $\widehat{G_{n}}$.

Now, the convergence of the admissible coadjoint orbits will be linked to the convergence in the parameter space \Big{\{}\alpha\in{\mathbb{R}}^{*},r>0,\rho_{\mu}\in\widehat{U(n-1)},\tau_{\lambda}\in\widehat{U(n)}\Big{\}}.

 

Letting ${\mathcal{W}}$ be the subspace of $\mathfrak{u}(n)$ generated by the matrices $z\times v_{r}=\frac{i}{2}(v_{r}z^{*}+zv_{r}^{*})$ for $z\in{\mathbb{C}}^{n}$, the space $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ is the set of all orbits

[TABLE]

for $\alpha\in{\mathbb{R}}^{*}$, $r\in{\mathbb{R}}_{>0}$, $\mu\in P_{n-1}$ and $\lambda\in P_{n}$.

 

Before beginning the discussion on the convergence of the admissible coadjoint orbits, the following preliminary lemmas are needed:

Lemma 4.1**.**

For $n\in{\mathbb{N}}^{*}$ and for any scalars $X_{1},...,X_{n}$ and $Y_{1},...,Y_{n-1}$ fulfilling $Y_{i}\neq Y_{j}$ for $i\neq j$, one has

[TABLE]

for each $k\in\{1,...,n\}$.

Proof:

For $n=1$, the formula is trivial.

So, let $n>1$ and assume that the assertion is true for this $n$. Consider the relation at $n+1$.

For $k=n+1$, a simple calculation gives the result. If $k\neq n+1$, one gets

[TABLE]

and the claim is shown.  

∎

Lemma 4.2**.**

Let $\mu\in P_{n-1}$ and $\lambda\in P_{n}$. Then, $\lambda_{1}\geq\mu_{1}\geq\lambda_{2}\geq...\geq\mu_{n-1}\geq\lambda_{n}$ if and only if there is a skew-Hermitian matrix

[TABLE]

in ${\mathcal{W}}$ such that $A(J_{\mu}+B)A^{*}=J_{\lambda}$ for an element $A\in U(n)$.

Proof:

For $y\in{\mathbb{R}}$, a computation shows that $\det(J_{\mu}+B-iy{\mathbb{I}})=(-i)^{n}P(y)$, where

[TABLE]

Furthermore, one can observe that $P(y)\overset{y\to\infty}{\longrightarrow}\infty$ and that $P(\mu_{j})\leq 0$ if $j$ is odd and $P(\mu_{j})\geq 0$ if $j$ is even.

Now, if $A(J_{\mu}+B)A^{*}=J_{\lambda}$ for an element $A\in U(n)$, by the spectral theorem, $i\lambda_{1},i\lambda_{2},\cdots,i\lambda_{n}$ are all the elements of the spectrum of $J_{\mu}+B$ fulfilling $\lambda_{1}\geq\mu_{1}\geq\lambda_{2}\geq...\geq\mu_{n-1}\geq\lambda_{n}$.

Conversely, suppose first that all the $\mu_{j}$ for $j\in\{1,...,n-1\}$ are pairwise distinct. In this case, let $B$ be the skew-Hermitian matrix with the entries $z_{1},\cdots,z_{n-1},x$ satisfying

[TABLE]

From Lemma 4.1,

[TABLE]

Hence, the spectrum of the matrix $J_{\mu}+B$ is the set $\{i\lambda_{1},i\lambda_{2},\cdots,i\lambda_{n}\}$ and thus, the spectral theorem implies that $A(J_{\mu}+B)A^{*}=J_{\lambda}$ for an element $A\in U(n)$.

Now, if the $\mu_{j}$ for $j\in\{1,...,n-1\}$ are not pairwise distinct, there exist two families of integers $\{p_{l}|~{}1\leq l\leq s\}$ and $\{q_{l}|~{}1\leq l\leq s\}$ such that $1\leq p_{1}<q_{1}<p_{2}<q_{2}<\cdots<p_{s}<q_{s}\leq n-1$ and $\mu_{p_{l}}=\mu_{p_{l}+1}=\cdots=\mu_{q_{l}-1}=\mu_{q_{l}}$, $\mu_{q_{l}}\not=\mu_{q_{l}+1}$ and $\mu_{p_{l}-1}\not=\mu_{p_{l}}$ for all $l\in\{1,...,s\}$. Let

[TABLE]

[TABLE]

Then, for

[TABLE]

one gets $\det(J_{\mu}+B-iy{\mathbb{I}})=(-i)^{n}\prod\limits_{l=1}^{s}(y-\mu_{p_{l}})^{q_{l}-p_{l}}P(y)$.

Now, the entries $z_{j}$ of the skew-Hermitian matrix $B$ can be chosen as follows:

[TABLE]

for each $j\in\{1,...,p_{1}-1,q_{1}+1,...,p_{s}-1,q_{s}+1,...,n-1\}$ and

[TABLE]

for each $l\in\{1,...,s\}$. The entry $x$ can be defined as

[TABLE]

Then, if $\lambda_{k}=\mu_{p_{l}}$, one obviously has $P(\lambda_{k})=Q(\lambda_{k})=0$ and the multiplicity of the root $\lambda_{k}=\mu_{p_{l}}$ of $P$ is $q_{l}-p_{l}$. Otherwise, one gets

[TABLE]

[TABLE]

Hence, the spectrum of the matrix $J_{\mu}+B$ equals the set $\{i\lambda_{1},i\lambda_{2},\cdots,i\lambda_{n}\}$. As above, this completes the proof.  

∎

Lemma 4.3**.**

1. (1)

Let $\lambda\in P_{n}$, $\alpha\in{\mathbb{R}}^{*}$ and $z\in{\mathbb{C}}^{n}$. Then, the matrix $J_{\lambda}+\frac{i}{\alpha}zz^{*}$ admits $n$ eigenvalues $i\beta_{1},i\beta_{2},\ldots,i\beta_{n}$ such that $\beta_{1}\geq\lambda_{1}\geq\beta_{2}\geq\lambda_{2}\geq\cdots\geq\beta_{n}\geq\lambda_{n}$ if $\alpha>0$ and $\lambda_{1}\geq\beta_{1}\geq\lambda_{2}\geq\beta_{2}\geq\cdots\geq\lambda_{n}\geq\beta_{n}$ if $\alpha<0$. 2. (2)

Let $\lambda,\beta\in P_{n}$. If $\beta_{1}\geq\lambda_{1}\geq\beta_{2}\geq\lambda_{2}\geq...\geq\beta_{n}\geq\lambda_{n}$, there exists a number $z\in{\mathbb{C}}^{n}$ such that the matrix $J_{\lambda}+izz^{*}$ admits the $n$ eigenvalues $i\beta_{1},...,i\beta_{n}$. If $\lambda_{1}\geq\beta_{1}\geq\lambda_{2}\geq\beta_{2}\geq...\geq\lambda_{n}\geq\beta_{n}$, there exists $z\in{\mathbb{C}}^{n}$ such that the matrix $J_{\lambda}-izz^{*}$ admits the $n$ eigenvalues $i\beta_{1},...,i\beta_{n}$.

Proof:

1. One can prove by induction that the characteristic polynomial of the matrix $\frac{1}{i}J_{\lambda}+\frac{zz^{*}}{\alpha}$ is equal to $Q_{n}^{\lambda,z,\alpha}$ defined by

[TABLE]

Assume that $\alpha$ is negative. Then, $Q_{n}^{\lambda,z,\alpha}(x)\overset{x\to\infty}{\longrightarrow}\infty$ and $Q_{n}^{\lambda,z,\alpha}(\lambda_{j})\geq 0$ if $j$ is odd and $Q_{n}^{\lambda,z,\alpha}(\lambda_{j})\leq 0$ if $j$ is even. Furthermore, $Q_{n}^{\lambda,z,\alpha}(x)\overset{x\to\infty}{\longrightarrow}-\infty$ if $n$ is odd and $Q_{n}^{\lambda,z,\alpha}(x)\overset{x\to-\infty}{\longrightarrow}\infty$ if $n$ is even and therefore, one can deduce that the matrix $\frac{1}{i}J_{\lambda}+\frac{zz^{*}}{\alpha}$ admits $n$ eigenvalues $\beta_{1},\beta_{2},\ldots,\beta_{n}$ verifying $\lambda_{1}\geq\beta_{1}\geq\lambda_{2}\geq\beta_{2}\geq\cdots\geq\lambda_{n}\geq\beta_{n}$. Hence, $J_{\lambda}+\frac{i}{\alpha}zz^{*}$ admits the $n$ eigenvalues $i\beta_{1},i\beta_{2},\ldots,i\beta_{n}$ fulfilling $\lambda_{1}\geq\beta_{1}\geq\lambda_{2}\geq\beta_{2}\geq\cdots\geq\lambda_{n}\geq\beta_{n}$.

The same reasoning applies when $\alpha$ is positive.

2. Let $\beta_{1}\geq\lambda_{1}\geq\beta_{2}\geq\lambda_{2}\geq...\geq\beta_{n}\geq\lambda_{n}$.

For any $z\in{\mathbb{C}}^{n}$, the characteristic polynomial of $\frac{1}{i}J_{\lambda}+zz^{*}$ is equal to $Q_{n}^{\lambda,z,1}$ with $Q_{n}^{\lambda,z,1}=:Q_{n}^{\lambda,z}$ like above.

First, assume that $\beta_{1}>\lambda_{1}>...>\beta_{n}>\lambda_{n}$.

Let

[TABLE]

Then, as $\lambda_{j}<\beta_{i}$ for all $i\in\{1,...,j\}$, as $\lambda_{j}>\beta_{i}$ for all $i\in\{j+1,...,n\}$, as $\lambda_{j}<\lambda_{i}$ for all $i\in\{1,...,j-1\}$ and as $\lambda_{j}>\lambda_{i}$ for all $i\in\{j+1,...,n\}$, one gets \text{sgn}\big{(}|z_{j}|^{2}\big{)}=(-1)\frac{(-1)^{j}}{(-1)^{j-1}}=1 and thus, this definition is meaningful.

One now has to show that $Q_{n}^{\lambda,z}(\beta_{\ell})=0$ for all $\ell\in\{1,...,n\}$.

[TABLE]

as by [12], Lemma 5.3, one obtains $\sum\limits_{j=1}^{n}\frac{\prod\limits_{\begin{subarray}{c}i=1\\ i\not=\ell\end{subarray}}^{n}(\lambda_{j}-\beta_{i})}{\prod\limits_{\begin{subarray}{c}i=1\\ i\not=j\end{subarray}}^{n}(\lambda_{j}-\lambda_{i})}=1$.

 

Now, regard arbitrary $\beta_{1}\geq\lambda_{1}\geq\beta_{2}\geq\lambda_{2}\geq...\geq\beta_{n}\geq\lambda_{n}$.

For $n=1$, one can choose $|z_{1}|^{2}:=(\beta_{1}-\lambda_{1})\geq 0$ and the claim is shown.

Let $n>1$ and assume that the assertion is true for $n-1$.

If $\lambda_{\ell-1}\not=\beta_{\ell}\not=\lambda_{\ell}$ for all $\ell\in\{1,...,n\}$, the claim is already shown above. So, without restriction let $\ell\in\{1,...,n\}$ with $\beta_{\ell}=\lambda_{\ell}$. The case $\lambda_{\ell-1}=\beta_{\ell}$ is very similar.

Hence, for $\lambda^{\ell}:=(\lambda_{1},...,\lambda_{\ell-1},\lambda_{\ell+1},...,\lambda_{n})$ and $\beta^{\ell}:=(\beta_{1},...,\beta_{\ell-1},\beta_{\ell+1},...,\beta_{n})$,

[TABLE]

and thus, by the induction hypothesis, there exists ${\mathbb{C}}^{n-1}\ni z^{\ell}:=(z_{1},...,z_{\ell-1},z_{\ell+1},...,z_{n})$ such that $Q_{n,\ell}^{\lambda^{\ell},z^{\ell}}(\beta_{i})=0$ for all $i\in\{1,...,\ell-1,\ell+1,...,n\}$, where

[TABLE]

Now, let $z_{\ell}:=0$, i.e. $z:=(z_{1},...,z_{\ell-1},0,z_{\ell+1},...,z_{n})$. Then,

[TABLE]

If $i\in\{1,...,\ell-1,\ell+1,...,n\}$, then $Q_{n,\ell}^{\lambda^{\ell},z^{\ell}}(\beta_{i})=0$ and thus, $Q_{n}^{\lambda,z}(\beta_{i})=0$. Furthermore, $Q_{n}^{\lambda,z}(\beta_{\ell})=0$, as $\beta_{\ell}=\lambda_{\ell}$.

Therefore, $Q_{n}^{\lambda,z}(\beta_{i})=0$ for all $i\in\{1,...,n\}$ and the claim is shown.

 

Next, let $\lambda_{1}\geq\beta_{1}\geq\lambda_{2}\geq\beta_{2}\geq...\geq\lambda_{n}\geq\beta_{n}$.

Then, for any $z\in{\mathbb{C}}^{n}$, the characteristic polynomial of $\frac{1}{i}J_{\lambda}-zz^{*}$ is equal to $Q_{n}^{\lambda,z,-1}$.

If $\lambda_{1}>\beta_{1}>...>\lambda_{n}>\beta_{n}$, let

[TABLE]

Here, \text{sgn}\big{(}|z_{j}|^{2}\big{)}=\frac{(-1)^{j-1}}{(-1)^{j-1}}=1 and hence, this definition is meaningful.

The rest of the proof is the same as in the first part of (2).  

∎ 

 

With these lemmas, one can now prove the following theorem which describes the topology of the space of admissible coadjoint orbits of $G_{n}$.

Theorem 4.4**.**

Let $\alpha\in{\mathbb{R}}^{*}$, $r>0$, $\mu\in P_{n-1}$ and $\lambda\in P_{n}$. Then, the following holds:

1. (1)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} converges to the orbit $\mathcal{O}_{(\lambda,\alpha)}$ in $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ if and only if $\underset{k\to\infty}{\lim}\alpha_{k}=\alpha$ and $\lambda^{k}=\lambda$ for large $k$. 2. (2)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the orbit $\mathcal{O}_{(\mu,r)}$ in $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ if and only if $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} satisfies one of the following conditions:

1. (i)

For $k$ large enough, $\alpha_{k}>0$, $\lambda_{j}^{k}=\mu_{j}$ for all $j\in\{1,...,n-1\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$. 2. (ii)

For $k$ large enough, $\alpha_{k}<0$, $\lambda_{j}^{k}=\mu_{j-1}$ for all $j\in\{2,...,n\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=-\frac{r^{2}}{2}$. 3. (3)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the orbit $\mathcal{O}_{\lambda}$ in $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ if and only if $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} satisfies one of the following conditions:

1. (i)

For $k$ large enough, $\alpha_{k}>0$, $\lambda_{1}\geq\lambda_{1}^{k}\geq...\geq\lambda_{n-1}\geq\lambda_{n-1}^{k}\geq\lambda_{n}\geq\lambda_{n}^{k}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=0$. 2. (ii)

For $k$ large enough, $\alpha_{k}<0$, $\lambda_{1}^{k}\geq\lambda_{1}\geq\lambda_{2}^{k}\geq\lambda_{2}\geq...\geq\lambda_{n-1}\geq\lambda_{n}^{k}\geq\lambda_{n}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=0$. 4. (4)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} converges to the orbit $\mathcal{O}_{(\mu,r)}$ in $\mathfrak{g}^{\ddagger}_{n}/G_{n}$ if and only if $\underset{k\to\infty}{\lim}r_{k}=r$ and $\mu^{k}=\mu$ for large $k$. 5. (5)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} converges to $\mathcal{O}_{\lambda}$ in $\mathfrak{g}^{\ddagger}_{n}/G_{n}$ if and only if $(r_{k})_{k\in{\mathbb{N}}}$ tends to [math] and $\lambda_{1}\geq\mu^{k}_{1}\geq\lambda_{2}\geq\mu^{k}_{2}\geq...\geq\lambda_{n-1}\geq\mu^{k}_{n-1}\geq\lambda_{n}$ for $k$ large enough. 6. (6)

A sequence of coadjoint orbits \big{(}\mathcal{O}_{\lambda^{k}}\big{)}_{k\in{\mathbb{N}}} converges to the orbit $\mathcal{O}_{\lambda}$ in $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ if and only if $\lambda^{k}=\lambda$ for large $k$.

Proof:

Examining the shape of the coadjoint orbits listed at the beginning of this subsection, 1) and 6) are clear and Assertion 5) follows immediately from Lemma 4.2. Furthermore, the proof of 4) is similar to that of [12], Theorem 4.2.

2. Assume that \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the orbit $\mathcal{O}_{(\mu,r)}$. Then, there exist a sequence $(A_{k})_{k\in{\mathbb{N}}}$ in $U(n)$ and a sequence of vectors \big{(}z(k)\big{)}_{k\in{\mathbb{N}}} in ${\mathbb{C}}^{n}$ such that

[TABLE]

Let $A=(a_{mj})_{1\leq m,j\leq n}$ be the limit of a subsequence $(A_{s})_{s\in I}$ for $I\subset{\mathbb{N}}$. Then,

[TABLE]

On the other hand, one has $(A^{*}J_{\mu}A)_{mj}=i\sum\limits_{l=1}^{n-1}\mu_{l}\overline{a}_{lm}a_{lj}$ and

[TABLE]

Hence, for $m\neq j$, \underset{s\to\infty}{\lim}\big{|}\frac{z_{m}(s)\overline{z}_{j}(s)}{\alpha_{s}}\big{|}=\Big{|}\sum\limits_{l=1}^{n-1}\mu_{l}\overline{a}_{lm}a_{lj}\Big{|}<\infty, and since $\underset{s\to\infty}{\lim}\|z(s)\|_{{\mathbb{C}}^{n}}=\frac{r}{\sqrt{2}}\neq 0$, there is a unique $i_{0}\in\{1,...,n\}$ such that $\underset{s\to\infty}{\lim}z_{i_{0}}(s)=\frac{r}{\sqrt{2}}e^{i\theta}$ for a certain number $\theta\in{\mathbb{R}}$ and $\underset{s\to\infty}{\lim}z_{j}(s)=0$ for $j\neq i_{0}$. One obtains $a_{ni_{0}}=e^{-i\theta}$ and $a_{nj}=0$ for $j\neq i_{0}$, i.e. the matrices $A$ and $A^{*}J_{\mu}A$ can be written in the following way:

[TABLE]

[TABLE]

since $\overline{(A^{*}J_{\mu}A)}_{i_{0}j}=-(A^{*}J_{\mu}A)_{ji_{0}}=-i\sum\limits_{l=1}^{n-1}\mu_{l}\overline{a}_{lj}a_{li_{0}}=0$ for $j\in\{1,...,n\}$. As for each $j\neq i_{0}$,

[TABLE]

one gets $\underset{s\to\infty}{\lim}\lambda^{s}_{j}=\sum\limits_{l=1}^{n-1}\mu_{l}|a_{lj}|^{2}$. On the other hand, $\underset{s\to\infty}{\lim}\lambda^{s}_{i_{0}}+\frac{|z_{i_{0}}(s)|^{2}}{\alpha_{s}}=0$, which in turn implies that \underset{s\to\infty}{\lim}\big{|}\lambda^{s}_{i_{0}}\big{|}=\infty. This proves that $i_{0}$ can only take the value 1 if $\alpha_{s}<0$ and $n$ if $\alpha_{s}>0$. Otherwise, since $\lambda^{s}_{i_{0}-1}\geq\lambda^{s}_{i_{0}}\geq\lambda^{s}_{i_{0}+1}$, one gets $\underset{s\to\infty}{\lim}\lambda^{s}_{i_{0}-1}=\infty$ if $\alpha_{s}<0$ and $\underset{s\to\infty}{\lim}\lambda^{s}_{i_{0}+1}=-\infty$ if $\alpha_{s}>0$ which contradicts the fact that $\underset{s\to\infty}{\lim}\lambda^{s}_{j}$ is finite for all $j\neq i_{0}$.

 

Case $i_{0}=n$:

In this case, one has $\underset{s\to\infty}{\lim}\alpha_{s}\lambda^{s}_{n}=-\frac{r^{2}}{2}$ and $\underset{s\to\infty}{\lim}\lambda^{s}_{j}=\sum\limits_{l=1}^{n-1}\mu_{l}|a_{lj}|^{2}$ for all $j\in\{1,...,n-1\}$. Furthermore, the matrices $A$ and $A^{*}J_{\mu}A$ have the form

[TABLE]

where $\tilde{A}\in U(n-1)$. However, the limit matrix of the subsequence \big{(}J_{\lambda^{s}}+\frac{i}{\alpha_{s}}z(s)z(s)^{*}\big{)}_{s\in I} has to be diagonal because $\underset{s\to\infty}{\lim}\frac{z_{m}(s)\overline{z}_{j}(s)}{\alpha_{s}}=0$ for all $m\neq j$. This implies that

[TABLE]

and consequently, $\lambda_{j}^{s}=\mu_{j}$ for large $s$ and $j\in\{1,...,n-1\}$.

 

Case $i_{0}=1$:

In this case, $\underset{s\to\infty}{\lim}\alpha_{s}\lambda^{s}_{1}=-\frac{r^{2}}{2}$ and $\underset{s\to\infty}{\lim}\lambda^{s}_{j}=\sum\limits_{l=1}^{n-1}\mu_{l}|a_{lj}|^{2}$ for every $j\in\{2,...,n\}$. Moreover, there is an element $\tilde{A}\in U(n-1)$ such that the matrix $A$ is given by

[TABLE]

Using the same arguments as above, one has $\lambda_{j+1}^{s}=\mu_{j}$ for $s$ large enough and for every $j\in\{1,...,n-1\}$.

 

Conversely, suppose that $\underset{k\to\infty}{\lim}\alpha_{k}=0$. If the regarded sequence of orbits satisfies the first condition, one can take z(k):=\left(\begin{array}[]{c}0\\ \vdots\\ 0\\ \sqrt{-\alpha_{k}\lambda_{n}^{k}}\\ \end{array}\right) and $A_{k}:={\mathbb{I}}$ for $k\geq N$ and $N\in{\mathbb{N}}$ large enough. In the other case, one lets

[TABLE]

Thus, \underset{k\to\infty}{\lim}\Big{(}A_{k}\big{(}J_{\lambda^{k}}+\frac{i}{\alpha_{k}}z(k)z(k)^{*}\big{)}A_{k}^{*},\sqrt{2}A_{k}z(k),\alpha_{k}\Big{)}=(J_{\mu},v_{r},0).

 

3. Suppose that \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the orbit $\mathcal{O}_{\lambda}$. Then, there exist a sequence $(A_{k})_{k\in\mathbb{N}}$ in $U(n)$ and a sequence \big{(}z(k)\big{)}_{k\in\mathbb{N}} in ${\mathbb{C}}^{n}$ such that

[TABLE]

It follows that $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and that \big{(}z(k)\big{)}_{k\in{\mathbb{N}}} tends to [math] in ${\mathbb{C}}^{n}$. Denote by $A=(a_{mj})_{1\leq m,j\leq n}$ the limit matrix of a subsequence $(A_{s})_{s\in I}$ for an index set $I\subset{\mathbb{N}}$. Then,

[TABLE]

Since $\underset{k\to\infty}{\lim}\alpha_{k}=0$, one can assume that $\alpha_{s}$ is either strictly positive for all $s\in I$ or strictly negative for all $s\in I$.

Let $\sqrt{|\alpha_{s}|}$ be the square root of $|\alpha_{s}|$. The fact that $\underset{s\to\infty}{\lim}\frac{z_{m}(s)\overline{z}_{j}(s)}{\alpha_{s}}$ is finite for all $m,j\in\{1,...,n\}$ implies that there exists at most one integer $1\leq i_{0}\leq n$ such that $\underset{s\to\infty}{\lim}\frac{z_{i_{0}}(s)}{\sqrt{|\alpha_{s}|}}=\infty$. Therefore,

[TABLE]

exists for all $j$ distinct from $i_{0}$. Hence, for the same reasons as in the proof of 4), necessarily $i_{0}\in\{1,n\}$.

If there is no such $i_{0}$, then there exists for all $j\in\{1,...,n\}$ an integer $\lambda^{\prime}_{j}\in{\mathbb{Z}}$ such that $\lambda^{\prime}_{j}=\lambda^{s}_{j}$ for all $s\in I$ (by passing to a subsequence if necessary) and $\tilde{z}_{j}:=\underset{s\to\infty}{\lim}\frac{z_{j}(s)}{\sqrt{|\alpha_{s}|}}$ is finite for all $j\in\{1,\ldots,n\}$. Thus, one gets

[TABLE]

It follows by Lemma 4.3 applied to $\tilde{z}$ and $\alpha=1$ or $\alpha=-1$, respectively, that

[TABLE]

Case $i_{0}=n$:

Here, $\underset{s\to\infty}{\lim}\alpha_{s}\lambda^{s}_{n}=0$, as \underset{s\to\infty}{\lim}\big{|}\lambda^{s}_{n}+\frac{|z_{n}(s)|^{2}}{\alpha_{s}}\big{|}<\infty. Furthermore, $\underset{s\to\infty}{\lim}\lambda^{s}_{n}=-\infty$ and $\underset{s\to\infty}{\lim}\lambda^{s}_{j}=\sum\limits_{l=1}^{n}\lambda_{l}|a_{lj}|^{2}$ for all $j\in\{1,...,n-1\}$ and $\alpha_{s}$ has to be positive for large $s$. Since $\lim\limits_{s\to\infty}\frac{z_{j}(s)}{\alpha_{s}}$ exists and $\lim\limits_{s\to\infty}\frac{|z_{n}(s)|}{\alpha_{s}}=\infty$, it follows that $\lim\limits_{s\to\infty}\frac{z_{j}(s)}{\alpha_{s}}=0$ for every $j\in\{1,...,n-1\}$.

Now, choose

[TABLE]

Then, the limit matrix $A^{*}J_{\lambda}A$ of the sequence \big{(}J_{\lambda^{s}}+\frac{i}{\alpha_{s}}z(s)z(s)^{*}\big{)}_{s\in I} has the form

[TABLE]

By Lemma 4.2, one obtains $\lambda_{1}\geq\lambda^{\prime}_{1}\geq\lambda_{2}\geq\lambda^{\prime}_{2}\geq...\geq\lambda_{n-1}\geq\lambda^{\prime}_{n-1}\geq\lambda_{n}$, and therefore, one has $\lambda_{1}\geq\lambda^{s}_{1}\geq\lambda_{2}\geq\lambda^{s}_{2}\geq...\geq\lambda_{n-1}\geq\lambda^{s}_{n-1}\geq\lambda_{n}\geq\lambda^{s}_{n}$ for large $s$.

 

Case $i_{0}=1$:

In this case, $\underset{s\to\infty}{\lim}\alpha_{s}\lambda^{s}_{1}=0$, since \underset{s\to\infty}{\lim}\big{|}\lambda^{s}_{1}+\frac{|z_{1}(s)|^{2}}{\alpha_{s}}\big{|}<\infty. Moreover,

[TABLE]

Hence, $\alpha_{s}<0$ for $s$ large enough. If one sets

[TABLE]

the limit matrix $A^{*}J_{\lambda}A$ of \big{(}J_{\lambda^{s}}+\frac{i}{\alpha_{s}}z(s)z(s)\big{)}_{s\in I} can be written as follows:

[TABLE]

where

[TABLE]

This proves that $\lambda^{s}_{1}\geq\lambda_{1}\geq\lambda^{s}_{2}\geq\lambda_{2}\geq...\geq\lambda^{s}_{n-1}\geq\lambda_{n-1}\geq\lambda^{s}_{n}\geq\lambda_{n}$ for large $s$.

Conversely, suppose that the sequence \big{(}\mathcal{O}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} satisfies the first condition.

First, consider the case $\lambda^{k}_{n}\overset{k\to\infty}{\longrightarrow}-\infty$.

Then, there is a subsequence $(\lambda^{s})_{s\in I}$ for an index set $I\subset{\mathbb{N}}$ fulfilling $\lambda^{s}_{j}=\lambda^{\prime}_{j}$ for every $j\in\{1,...,n-1\}$ and all $s\in I$. By Lemma 4.2, there exist $w_{1},w_{2},\ldots,w_{n-1}\in{\mathbb{C}}$, $x\in{\mathbb{R}}$ and $A\in U(n)$ such that

[TABLE]

In this case, $\lambda^{k}\neq\lambda$ for large $k$, as $\lambda^{k}_{n}\overset{k\to\infty}{\longrightarrow}-\infty$. Choose $x:=\sum\limits_{j=1}^{n}\lambda_{j}-\sum\limits_{j=1}^{n-1}\lambda^{\prime}_{j}$. (Compare the proof of Lemma 4.2.) It follows that

[TABLE]

Furthermore, define the sequence \big{(}z(s)\big{)}_{s\in I} in ${\mathbb{C}}^{n}$ by

[TABLE]

Then, one gets

[TABLE]

Hence, \Big{(}A\big{(}J_{\lambda^{s}}+\frac{i}{\alpha_{s}}z(s)z(s)^{*}\big{)}A^{*}\Big{)}_{s\in I} converges to $J_{\lambda}$ and \big{(}z(s)\big{)}_{s\in I} to [math].

If $\lim\limits_{k\to\infty}\lambda^{k}_{n}\not=-\infty$, there is a subsequence $(\lambda^{s})_{s\in I}$ for an index set $I\subset{\mathbb{N}}$ fulfilling $\lambda_{j}^{s}=\lambda^{\prime}_{j}$ for all $j\in\{1,...,n\}$ and all $s\in I$. Therefore,

[TABLE]

and thus, by Lemma 4.3(2), there exists $\tilde{z}\in{\mathbb{C}}^{n}$ such that $i\lambda_{1},...,i\lambda_{n}$ are the eigenvalues of $J_{\lambda^{\prime}}+i\tilde{z}(\tilde{z})^{*}$.

Let $z(s):=\tilde{z}\sqrt{\alpha_{s}}$, which is reasonable since $\alpha_{s}>0$ in this case.

As the matrices $J_{\lambda^{\prime}}+i\tilde{z}(\tilde{z})^{*}$ and $J_{\lambda}$ are both skew-Hermitian and have the same eigenvalues, they are unitarily conjugated. Therefore, there exists an element $A\in U(n)$ such that $J_{\lambda^{\prime}}+i\tilde{z}(\tilde{z})^{*}=A^{*}J_{\lambda}A$. Hence,

[TABLE]

i.e. \Big{(}A\big{(}J_{\lambda^{\prime}}+i\frac{z(s)z(s)^{*}}{\alpha_{s}}\big{)}A^{*}\Big{)}_{s\in I} converges to $J_{\lambda}$. Furthermore,

[TABLE]

as $\alpha_{k}\overset{k\to\infty}{\longrightarrow}0$.

Thus, the claim is shown in this case.

Suppose now that for $k$ large $\alpha_{k}<0$, $\lambda_{1}^{k}\geq\lambda_{1}\geq...\geq\lambda^{k}_{n-1}\geq\lambda_{n-1}\geq\lambda_{n}^{k}\geq\lambda_{n}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=0$. First consider the case $\lambda_{1}^{k}\overset{k\to\infty}{\longrightarrow}\infty$.

In this case, there is a subsequence $(\lambda^{s})_{s\in I}$ for an index set $I\subset{\mathbb{N}}$ such that $\lambda^{s}_{j}=\lambda^{\prime}_{j-1}$ for all $j\in\{2,...,n\}$ and all $s\in I$. By Identity (4.7) and Lemma 4.2 , there exist $w_{1},w_{2},\ldots,w_{n-1}\in{\mathbb{C}}$, $x\in{\mathbb{R}}$ and $A\in U(n)$ such that

[TABLE]

Similarly to the last case, one takes $x:=\sum\limits_{j=1}^{n}\lambda_{j}-\sum\limits_{j=1}^{n-1}\lambda^{\prime}_{j}$ and thus gets

[TABLE]

Hence, one can define the sequence \big{(}z(s)\big{)}_{s\in I} in ${\mathbb{C}}^{n}$ by

[TABLE]

Here again, one gets

[TABLE]

Again, one can conclude that \bigg{(}\Big{(}A\big{(}J_{\lambda^{s}}+\frac{i}{\alpha_{s}}z(s)z(s)^{*}\big{)}A^{*},\sqrt{2}Az(s),\alpha_{s}\Big{)}\bigg{)}_{s\in I} converges to $(J_{\lambda},0,0)$.

If $\lim\limits_{k\to\infty}\lambda^{k}_{1}\not=\infty$, there is a subsequence $(\lambda^{s})_{s\in I}$ for an index set $I\subset{\mathbb{N}}$ fulfilling $\lambda_{j}^{s}=\lambda^{\prime}_{j}$ for all $j\in\{1,...,n\}$ and all $s\in I$. Hence,

[TABLE]

and therefore, by Lemma 4.3(2), there exists $\tilde{z}\in{\mathbb{C}}^{n}$ in such a way that $i\lambda_{1},...,i\lambda_{n}$ are the eigenvalues of $J_{\lambda^{\prime}}-i\tilde{z}(\tilde{z})^{*}$.

Let now $z(s):=\tilde{z}\sqrt{-\alpha_{s}}$, which is reasonable since this time $\alpha_{s}<0$.

As above, there exists an element $A\in U(n)$ such that $J_{\lambda^{\prime}}-i\tilde{z}(\tilde{z})^{*}=A^{*}J_{\lambda}A$ and thus,

[TABLE]

i.e. \Big{(}A\big{(}J_{\lambda^{\prime}}+i\frac{z(s)z(s)^{*}}{\alpha_{s}}\big{)}A^{*}\Big{)}_{s\in I} converges to $J_{\lambda}$. Furthermore,

[TABLE]

as $\alpha_{k}\overset{k\to\infty}{\longrightarrow}0$.

Therefore, the assertion is also shown in this case.  

∎

5. The continuity of the inverse of the Kirillov-Lipsman map ${\mathcal{K}}$

In the next two sections, the topology of the spectrum $\widehat{G_{n}}$ of the group $G_{n}=U(n)\ltimes\mathbb{H}_{n}$ will be analyzed and the aim is to show that it is determined by the topology of its admissible quotient space.

5.1. The representation $\pi_{(\mu,r)}$

First, examine the representation $\pi_{(\mu,r)}=\hbox{ind}_{U(n-1)\ltimes\mathbb{H}_{n}}^{G_{n}}\rho_{\mu}\otimes\chi_{r}$. Its Hilbert space ${\mathcal{H}}_{(\mu,r)}$ is given by the space

[TABLE]

Let $\xi$ be a unit vector in ${\mathcal{H}}_{(\mu,r)}$ and recall that $(z,w)_{{\mathbb{C}}^{n}}=\mathrm{\,Re\,}\langle z,w\rangle_{{\mathbb{C}}^{n}}$ for $z,w\in{\mathbb{C}}^{n}$. For all $(z,t)\in\mathbb{H}_{n}$ and all $A,B\in U(n)$,

[TABLE]

Therefore,

[TABLE]

By (3.4) in Section 3, one has

[TABLE]

Every irreducible representation $\tau_{\lambda}$ of $U(n)$ can be realized as a subrepresentation of the left regular representation on L^{2}\big{(}U(n)\big{)} via the intertwining operator

[TABLE]

for a fixed unit vector $\xi_{\lambda}\in{\mathcal{H}}_{\lambda}$.

For $\tau_{\lambda}\in\widehat{U(n)}$, consider the orthonormal basis {\mathcal{B}}^{\lambda}=\big{\{}\phi^{\lambda}_{j}|~{}j\in\{1,...,d_{\lambda}\}\big{\}} of ${\mathcal{H}}_{\lambda}$ consisting of eigenvectors for ${\mathbb{T}}_{n}$ of ${\mathcal{H}}_{\lambda}$.  

Moreover, as a basis of the Lie algebra $\mathfrak{h}_{n}$ of the Heisenberg group, one can take the left invariant vector fields \big{\{}Z_{1},Z_{2},\ldots,Z_{n}, \overline{Z}_{1},\overline{Z}_{2},\ldots,\overline{Z}_{n},T\big{\}}, where

[TABLE]

and gets the Lie brackets $[Z_{j},\overline{Z}_{j}]=-2iT$ for $j\in\{1,...,n\}$.

Now, regard the Heisenberg sub-Laplacian differential operator which is given by

[TABLE]

This operator is $U(n)$-invariant.

Lemma 5.1**.**

For every representation $\pi_{(\mu,r)}$ for $r>0$ and $\rho_{\mu}\in\widehat{U(n-1)}$,

[TABLE]

Proof:

Since the representation $\pi_{(\mu,r)}$ is trivial on the center of $\mathfrak{h}_{n}$, one has

[TABLE]

Let ${\mathcal{D}}=\{e_{1},\ldots,e_{n}\}$ be an orthonormal basis for ${\mathbb{C}}^{n}$. By writing

[TABLE]

one gets

[TABLE]

∎ 

 

In addition, the following theorem describes the convergence of sequences of representations \big{(}\pi_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}}:

Theorem 5.2**.**

Let $r>0$, $\rho_{\mu}\in\widehat{U(n-1)}$ and $\tau_{\lambda}\in\widehat{U(n)}$.

1. (1)

A sequence \big{(}\pi_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} of irreducible unitary representations of $G_{n}$ converges to $\pi_{(\mu,r)}$ in $\widehat{G_{n}}$ if and only if $\underset{k\to\infty}{\lim}r_{k}=r$ and $\mu^{k}=\mu$ for $k$ large enough. 2. (2)

A sequence \big{(}\pi_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} of irreducible unitary representations of $G_{n}$ converges to $\tau_{\lambda}$ in $\widehat{G_{n}}$ if and only if $\underset{k\to\infty}{\lim}r_{k}=0$ and $\tau_{\lambda}$ occurs in $\pi_{\mu^{k}}$ for $k$ large enough.

These are all possibilities for a sequence \big{(}\pi_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} of irreducible unitary representations of $G_{n}$ to converge.

The proof of 1) and 2) of this theorem can be found in [1], Theorem 6.2.A.

Furthermore, since the representations $\pi_{(\mu,r)}$ and $\tau_{\lambda}$ are trivial on \big{\{}({\mathbb{I}},0,t)|~{}t\in{\mathbb{R}}\big{\}}, the center of $G_{n}$, while the representations $\pi_{(\lambda,\alpha)}$ are non-trivial there, the possibilities of convergence of a sequence \big{(}\pi_{(\mu^{k},r_{k})}\big{)}_{k\in{\mathbb{N}}} listed above are the only ones that are possible.

5.2. The representation $\tau_{\lambda}$

As $\tau_{\lambda}$ only acts on $U(n)$ and $\widehat{U(n)}$ is discrete, every converging sequence $(\tau_{\lambda^{k}})_{k\in{\mathbb{N}}}$ has to be constant for large $k$. Hence,

[TABLE]

5.3. The representation $\pi_{(\lambda,\alpha)}$

Next, regard the representations $\pi_{(\lambda,\alpha)}$.

Consider the unit vector $\xi:=\sum\limits_{j=1}^{d_{\lambda}}\phi_{j}^{\lambda}\otimes f_{j}$ in the Hilbert space ${\mathcal{H}}_{(\lambda,\alpha)}={\mathcal{H}}_{\lambda}\otimes{\mathcal{F}}_{\alpha}(n)$ of $\pi_{(\lambda,\alpha)}$, where $f_{1},\ldots,f_{d_{\lambda}}$ belong to the Fock space ${\mathcal{F}}_{\alpha}(n)$. Then, for all $A\in U(n)$ and $(z,t)\in\mathbb{H}_{n}$,

[TABLE]

It follows that

[TABLE]

Lemma 5.3**.**

For each representation $\pi_{(\lambda,\alpha)}$ for $\alpha\in{\mathbb{R}}^{*}$ and $\tau_{\lambda}\in\widehat{U(n)}$, one has

[TABLE]

Proof:

Let $\xi=\sum\limits_{j=1}^{d_{\lambda}}\phi_{j}^{\lambda}\otimes f_{j}$ be a unit vector in ${\mathcal{H}}_{(\lambda,\alpha)}$. Then,

[TABLE]

∎ 

 

If $\alpha$ is positive, the polynomials ${\mathbb{C}}[{\mathbb{C}}^{n}]$ are dense in ${\mathcal{F}}_{\alpha}(n)$ and its multiplicity free decomposition is

[TABLE]

where ${\mathcal{P}}_{m}$ is the space of homogeneous polynomials of degree $m$. Therefore, $p_{m}(z)=z_{1}^{m}$ is the highest weight vector in ${\mathcal{P}}_{m}$ with weight $(m,0,...,0)=:[m]$. Applying the classical Pieri’s rule (see [15], Proposition 15.25), one obtains

[TABLE]

where the definition of the operator $W_{\alpha}(A)$ can be found in Section 3 in the description of $\pi_{(\lambda,\alpha)}$.

If $\alpha$ is negative, one gets

[TABLE]

Both of the sums again are multiplicity free. This follows from [20], Chapter IV.11, since $W_{\alpha}$ is multiplicity free.

Furthermore, let {\mathcal{R}}_{\alpha}:=\big{\{}h_{m,\alpha}|~{}\>m=(m_{1},\ldots,m_{n})\in{\mathbb{N}}^{n}\big{\}} be the orthonormal basis of the Fock space ${\mathcal{F}}_{\alpha}(n)$ defined by the Hermite functions

[TABLE]

with $|m|=m_{1}+...+m_{n}$, $m!=m_{1}!\cdots m_{n}!$ and $z^{m}=z_{1}^{m_{1}}\cdots z_{n}^{m_{n}}$ (see [14], Chapter 1.7).

 

Now, one obtains the following theorem about the convergence of sequences of representations \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}}:

Theorem 5.4**.**

Let $\alpha\in{\mathbb{R}}^{*}$ and $\tau_{\lambda}\in\widehat{U(n)}$. Then, a sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} of elements in $\widehat{G_{n}}$ converges to $\pi_{(\lambda,\alpha)}$ if and only if $\underset{k\to\infty}{\lim}\alpha_{k}=\alpha$ and $\lambda^{k}=\lambda$ for large $k$.

Proof:

First, consider the case where $\alpha$ is positive. Assume that $\alpha_{k}\overset{k\to\infty}{\longrightarrow}\alpha$ and that $\lambda^{k}=\lambda$ for $k$ large enough. Moreover, let $f\in C_{0}^{\infty}(G_{n})$ and let $\xi$ be a unit vector in ${\mathcal{H}}_{\lambda}$. Then,

[TABLE]

tends to \Big{\langle}C^{\pi_{(\lambda,\alpha)}}_{\xi\otimes h_{0,\alpha}},f\Big{\rangle}_{\big{(}L^{\infty}(G_{n}),L^{1}(G_{n})\big{)}}. Hence, \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} converges to $\pi_{(\lambda,\alpha)}$.

The same reasoning applies when $\alpha$ is negative.

Conversely, the fact that the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} converges to the representation $\pi_{(\lambda,\alpha)}$ implies that for $\xi\in{\mathcal{H}}^{\infty}_{(\lambda,\alpha)}$ of length 1, there is for every $k\in{\mathbb{N}}$ a unit vector $\xi_{k}\in{\mathcal{H}}^{\infty}_{(\lambda^{k},\alpha_{k})}$ such that \Big{(}\big{\langle}d\pi_{(\lambda^{k},\alpha_{k})}(T)\xi_{k},\xi_{k}\big{\rangle}_{{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}}\Big{)}_{k\in{\mathbb{N}}} converges to \big{\langle}d\pi_{(\lambda,\alpha)}(T)\xi,\xi\big{\rangle}_{{\mathcal{H}}_{(\lambda,\alpha)}}. Thus, by Lemma 5.3, we have $\underset{k\to\infty}{\lim}\alpha_{k}=\alpha$. Hence, it remains to show that $\lambda^{k}=\lambda$ for $k$ large enough.

Let $\xi$ be a unit vector in ${\mathcal{H}}_{\lambda}$. Then for every $k\in{\mathbb{N}}$, there exists a vector $\xi_{k}=\underset{m\in{\mathbb{N}}^{n}}{\sum}\zeta^{k}_{m}\otimes h_{m,\alpha_{k}}\in\mathcal{H}_{(\lambda^{k},\alpha_{k})}$ of length 1 such that \Big{(}C^{\pi_{(\lambda^{k},\alpha_{k})}}_{\xi_{k}}\Big{)}_{k\in{\mathbb{N}}} converges uniformly on compacta to $C^{\pi_{(\lambda,\alpha)}}_{\xi\otimes h_{0,\alpha}}$.

Now, take $\delta\in{\mathbb{R}}_{>0}$ such that $0\not\in I_{\alpha,\delta}=(\alpha-\delta,\alpha+\delta)$, as well as a Schwartz function $\varphi$ on ${\mathbb{R}}$ fulfilling $\varphi_{|I_{\alpha,\delta}}\equiv 1$ and $\varphi\equiv 0$ in a neighbourhood of [math]. Then, there is a Schwartz function $\psi$ on $\mathbb{H}_{n}$ with the property

[TABLE]

where $\sigma_{\beta}$ is the $\mathbb{H}_{n}$-representation defined in Section 3 and $P_{\beta}:{\mathcal{F}}_{\beta}(n)\to{\mathbb{C}}$ is the orthogonal projection onto the one-dimensional subspace ${\mathbb{C}}h_{0,\beta}$ of all constant functions in ${\mathcal{F}}_{\beta}(n)$. On the other hand, there exists $k_{\delta}\in{\mathbb{N}}$ such that $\alpha_{k}\in I_{\alpha,\delta}$ for all $k\geq k_{\delta}$. One obtains $\sigma_{\alpha}(\psi)h_{0,\alpha}=h_{0,\alpha}$ and $\sigma_{\alpha_{k}}(\psi)h_{0,\alpha_{k}}=h_{0,\alpha_{k}}$ for all $k\geq k_{\delta}$ and thus, it follows that

[TABLE]

Hence, one gets

[TABLE]

and one can deduce that

[TABLE]

uniformly in $A\in U(n)$. Therefore, for all $k\in{\mathbb{N}}$, one can take the unit vector $\phi_{k}=\frac{\zeta^{k}_{0}}{\|\zeta^{k}_{0}\|_{{\mathcal{H}}_{\lambda^{k}}}}$ in ${\mathcal{H}}_{\lambda^{k}}$ to finally obtain the uniform convergence on compacta of \Big{(}C^{\tau_{\lambda^{k}}}_{\phi_{k}}\Big{)}_{k\in{\mathbb{N}}} to $C^{\tau_{\lambda}}_{\xi}$. Thus, $\lambda^{k}=\lambda$ for $k$ large enough.  

∎

Lemma 5.5**.**

For each representation $\pi_{(\lambda,\alpha)}$ for $\alpha\in{\mathbb{R}}^{*}$ and $\tau_{\lambda}\in\widehat{U(n)}$,

[TABLE]

The proof follows from [4], Proposition 3.20 together with [5], Lemma 3.4.

Theorem 5.6**.**

Let $r>0$, $\rho_{\mu}\in\widehat{U(n-1)}$ and $\tau_{\lambda}\in\widehat{U(n)}$.

1. (1)

If a sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} of elements of $\widehat{G_{n}}$ converges to the representation $\pi_{(\mu,r)}$ in $\widehat{G_{n}}$, then $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} satisfies one of the following conditions:

1. (i)

For $k$ large enough, $\alpha_{k}>0$, $\lambda_{j}^{k}=\mu_{j}$ for all $j\in\{1,...,n-1\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$. 2. (ii)

For $k$ large enough, $\alpha_{k}<0$, $\lambda_{j}^{k}=\mu_{j-1}$ for all $j\in\{2,...,n\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=-\frac{r^{2}}{2}$. 2. (2)

If a sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} of elements of $\widehat{G_{n}}$ converges to the representation $\tau_{\lambda}$ in $\widehat{G_{n}}$, then $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} satisfies one of the following conditions:

1. (i)

For $k$ large enough, $\alpha_{k}>0$, $\lambda_{1}\geq\lambda_{1}^{k}\geq...\geq\lambda_{n-1}\geq\lambda_{n-1}^{k}\geq\lambda_{n}\geq\lambda_{n}^{k}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=0$. 2. (ii)

*For $k$ large enough, $\alpha_{k}<0$, $\lambda_{1}^{k}\geq\lambda_{1}\geq\lambda_{2}^{k}\geq\lambda_{2}\geq...\geq\lambda_{n-1}\geq\lambda_{n}^{k}\geq\lambda_{n}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=0$. *

Proof:

1. Let $\tilde{\mu}^{s}=(\mu_{1},\ldots,\mu_{s},\mu_{s},\mu_{s+1},\ldots,\mu_{n-1})$ for $s\in\{1,...,n-1\}$. By hypothesis, the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} converges to the representation $\pi_{(\mu,r)}$ in $\widehat{G_{n}}$. Thus, for the unit vector $\xi^{s}=\sqrt{d_{\tilde{\mu}^{s}}}\overline{C^{\tilde{\mu}^{s}}_{\phi_{1}^{\tilde{\mu}^{s}},\phi_{1}^{\tilde{\mu}^{s}}}}\in{\mathcal{H}}_{(\mu,r)}^{\infty}$, there is a sequence of unit vectors $(\xi_{k}^{s})_{k\in{\mathbb{N}}}\subset{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}^{\infty}$ such that

[TABLE]

[TABLE]

As by Lemma 5.3 one gets \big{\langle}d\pi_{(\lambda^{k},\alpha_{k})}(T)\xi_{k}^{s},\xi_{k}^{s}\big{\rangle}_{{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}}=\big{\langle}i\alpha_{k}\xi_{k}^{s},\xi_{k}^{s}\big{\rangle}_{{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}}, it follows that $\underset{k\to\infty}{\lim}\alpha_{k}=0$. Therefore, one can assume without restriction that $\alpha_{k}>0$ for large $k$ (by passing to a subsequence if necessary). The case $\alpha_{k}<0$ is very similar.

 

On the other hand, the sequence \Big{(}\big{\langle}\tau_{\lambda^{k}}\otimes W_{\alpha_{k}}(A)\xi_{k}^{s},\xi_{k}^{s}\big{\rangle}_{{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}}\Big{)}_{k\in{\mathbb{N}}} converges to the matrix coefficient $C^{\pi_{(\mu,r)}}_{\xi^{s}}(A,0,0)=C^{\tilde{\mu}^{s}}_{\phi_{1}^{\tilde{\mu}^{s}},\phi_{1}^{\tilde{\mu}^{s}}}(A)$ uniformly in each $A\in U(n)$. Hence, from this convergence, Orthogonality Relation (3.3) and the fact that \Big{\|}C^{\tilde{\mu}^{s}}_{\phi_{1}^{\tilde{\mu}^{s}},\phi_{1}^{\tilde{\mu}^{s}}}\Big{\|}_{L^{2}\big{(}U(n)\big{)}}=\frac{1}{\sqrt{d_{\tilde{\mu}^{s}}}}, follows

[TABLE]

By (5.1), one can write the expression \big{(}\tau_{\lambda^{k}}\otimes W_{\alpha_{k}}\big{)}_{|U(n)} as

[TABLE]

and, since for $k$ large enough the above integral is not [math], again by the orthogonality relation, there has to be one $\tilde{\lambda}^{k}\in P_{n}$ with $\tilde{\lambda}_{1}^{k}\geq\lambda_{1}^{k}\geq...\geq\tilde{\lambda}_{n}^{k}\geq\lambda_{n}^{k}$ such that $\tilde{\lambda}^{k}=\tilde{\mu}^{s}$. But as $\tilde{\lambda}^{k}_{s}=\tilde{\mu}^{s}_{s}=\tilde{\mu}^{s}_{s+1}=\tilde{\lambda}^{k}_{s+1}$, one obtains that $\lambda^{k}_{s}=\tilde{\lambda}^{k}_{s}=\tilde{\mu}^{s}_{s}=\mu_{s}$ for $k$ large enough. As this is true for all $s\in\{1,...,n-1\}$, one gets $\lambda^{k}_{j}=\mu_{j}$ for all $j\in\{1,...,n-1\}$.

 

So, it remains to show that $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$.

Again, by the decomposition of \big{(}\tau_{\lambda^{k}}\otimes W_{\alpha_{k}}\big{)}_{|U(n)} in (5.1), one can decompose ${\mathcal{H}}_{(\lambda^{k},\alpha_{k})}$ as follows

[TABLE]

and thus, for every $k\in{\mathbb{N}}$, the vector $\xi_{k}^{s}$ can be written as

[TABLE]

Let

[TABLE]

Then, with the Orthogonality Relation (3.3),

[TABLE]

From (5.2) it follows that

[TABLE]

As

[TABLE]

one can assume that $\xi_{k}^{s}=\phi_{1}^{\tilde{\mu}^{s}}$ for large $k\in{\mathbb{N}}$. Since $\lambda_{j}^{k}=\mu_{j}$ for all $k\in{\mathbb{N}}$ and for all $j\in\{1,...,n-1\}$, one gets for $s=n-1$

[TABLE]

From now on, consider only $k$ large enough such that $\xi_{k}^{n-1}=\phi_{1}^{\tilde{\mu}^{n-1}}$. Then, $\xi_{k}^{n-1}$ is the highest weight vector with weight $\tilde{\mu}^{n-1}$ of length 1. Moreover,

[TABLE]

Every weight in the decomposition on the left hand side has multiplicity one, as mentioned in (5.1), and therefore, this is the case for every weight appearing in the sum on the right hand side as well. From this, one can deduce that there exists one unique $M_{k}$ such that $\tilde{\mu}^{n-1}$, the weight of $\xi_{k}^{n-1}\in{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}$, appears in ${\mathcal{H}}_{\lambda^{k}}\otimes{\mathcal{P}}_{M_{k}}$.

By [20], Chapter IV.11, every highest weight appearing in ${\mathcal{H}}_{\lambda^{k}}\otimes{\mathcal{P}}_{M_{k}}$ is the sum of the highest weight of ${\mathcal{H}}_{\lambda^{k}}$ and a weight of ${\mathcal{P}}_{M_{k}}$. Hence, $\tilde{\mu}^{n-1}$ is the sum of $\lambda^{k}$ and a weight of ${\mathcal{P}}_{M_{k}}$. From this follows that the mentioned weight of ${\mathcal{P}}_{M_{k}}$ has the same length as $\tilde{\mu}^{n-1}-\lambda^{k}=m_{k}$. Therefore, $M_{k}=|m_{k}|$, i.e. ${\mathcal{P}}_{M_{k}}={\mathcal{P}}_{|m_{k}|}$.

Let \big{(}\phi_{j}^{\lambda^{k}}\big{)}_{j\in{\mathbb{N}}} be an orthogonal weight vector basis of ${\mathcal{H}}_{\tau_{\lambda^{k}}}$, the corresponding Hilbert space of $\tau_{\lambda^{k}}$, and let $\gamma_{j}^{k}$ denote the weight of $\phi_{j}^{\lambda^{k}}$. Then, one gets

[TABLE]

where $\phi^{\gamma^{k}}=\sum\limits_{j:\gamma_{j}^{k}=\gamma^{k}}~{}c_{j}^{k}\phi^{\gamma^{k}}_{j}$ is a uniquely determined eigenvector for ${\mathbb{T}}_{n}$ of the space ${\mathcal{H}}_{\lambda^{k}}$ with weight $\gamma^{k}$, $\Omega_{\lambda^{k}}^{\tilde{\mu}^{n-1}}$ is the set of all pairs $(\gamma^{k},\tilde{m}_{k})$ such that $\tilde{m}_{k}\in{\mathbb{N}}^{n}$ with $|\tilde{m}_{k}|=|m_{k}|$ and $\gamma^{k}$ is a weight that appears in the representation $\tau_{\lambda^{k}}$ fulfilling $\gamma^{k}+\tilde{m}_{k}=\tilde{\mu}^{n-1}$. Furthermore,

[TABLE]

Let

[TABLE]

Then, as seen above, $\lim\limits_{k\to\infty}c_{k}=-r^{2}$ and from Lemma 5.5 and the $U(n)$-invariance of ${\mathcal{L}}$, it follows that

[TABLE]

As $\alpha_{k}\overset{k\to\infty}{\longrightarrow}0$, also $\alpha_{k}(n+2\mu_{n-1})\overset{k\to\infty}{\longrightarrow}0$ and thus, $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$.

2. The fact that the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in{\mathbb{N}}} converges to $\tau_{\lambda}$ in $\widehat{G_{n}}$ implies that for the unit vector $\phi_{1}^{\lambda}\in{\mathcal{H}}_{\lambda}^{\infty}$, there is a sequence of unit vectors $(\xi_{k})_{k\in{\mathbb{N}}}\subset{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}^{\infty}$ such that

[TABLE]

[TABLE]

As above in the first part, by Lemma 5.3, from the first convergence it follows that $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and one can assume without restriction that $\alpha_{k}>0$ for large $k$.

On the other hand, \Big{(}\big{\langle}\tau_{\lambda^{k}}\otimes W_{\alpha_{k}}(A)\xi_{k},\xi_{k}\big{\rangle}_{{\mathcal{H}}_{(\lambda^{k},\alpha_{k})}}\Big{)}_{k\in{\mathbb{N}}} converges to $C^{\lambda}_{\phi_{1}^{\lambda},\phi_{1}^{\lambda}}(A)$ uniformly in $A\in U(n)$. Hence, as above one gets

[TABLE]

Again, like in the first part above, by (5.1) and the orthogonality relation, one can deduce that $\lambda_{1}\geq\lambda^{k}_{1}\geq....\geq\lambda_{n}\geq\lambda^{k}_{n}$ for large $k$.

 

So again, it remains to show that $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=0$.

In the same manner as above, by replacing $\tilde{\mu}^{n-1}$ by $\lambda$, one can now show that for large $k\in{\mathbb{N}}$, it is possible to assume $\xi_{k}=\phi_{1}^{\lambda}$. So consider $k$ large enough in order for this equality to be true. Then $\xi_{k}$ is the highest weight vector of length 1 with weight $\lambda$.

Now,

[TABLE]

where the sequences \big{(}\lambda_{1}-\lambda_{1}^{k}\big{)}_{k\in{\mathbb{N}}},…,\big{(}\lambda_{n-1}-\lambda_{n-1}^{k}\big{)}_{k\in{\mathbb{N}}} are bounded, because $\lambda_{1}\geq\lambda^{k}_{1}\geq....\geq\lambda_{n}\geq\lambda^{k}_{n}$ for large $k$.

Again, by replacing $\tilde{\mu}^{n-1}$ by $\lambda$ in the proof of the first part above, one can also write $\xi_{k}$ as

[TABLE]

where $\phi^{\gamma^{k}}$ is a uniquely determined eigenvector for ${\mathbb{T}}_{n}$ of ${\mathcal{H}}_{\lambda^{k}}$ with weight $\gamma^{k}$ and $\Omega_{\lambda^{k}}^{\lambda}$ is the set of all pairs $(\gamma^{k},\tilde{m}_{k})$ such that $\tilde{m}_{k}\in{\mathbb{N}}^{n}$ with $|\tilde{m}_{k}|=|m_{k}|$ and $\gamma^{k}$ is a weight that appears in the representation $\tau_{\lambda^{k}}$ fulfilling $\gamma^{k}+\tilde{m}_{k}=\lambda$. Furthermore, again

[TABLE]

Now, like in the first part above, by Lemma 5.5 and the $U(n)$-invariance of ${\mathcal{L}}$,

[TABLE]

By (5.3), as $\alpha_{k}\overset{k\to\infty}{\longrightarrow}0$, also \alpha_{k}\Big{(}n+2\big{(}\lambda_{1}-\lambda_{1}^{k}\big{)}+...+2\big{(}\lambda_{n-1}-\lambda_{n-1}^{k}\big{)}+2\lambda_{n}\Big{)}\overset{k\to\infty}{\longrightarrow}0 because of the boundedness of the sequences \big{(}\lambda_{1}-\lambda_{1}^{k}\big{)}_{k\in{\mathbb{N}}},…,\big{(}\lambda_{n-1}-\lambda_{n-1}^{k}\big{)}_{k\in{\mathbb{N}}}. Therefore, $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=0$.  

∎

6. The continuity of ${\mathcal{K}}$

In this section, it will be shown that the inverse of the Kirillov-Lipsman mapping ${\mathcal{K}}$ is also continuous. By Theorem 5.2, it suffices to consider converging sequences of orbits $({\mathcal{O}}_{(\lambda^{k},\alpha_{k})})_{k\in{\mathbb{N}}}$ and to show that the corresponding representations $(\pi_{(\lambda^{k},\alpha_{k})})_{k\in{\mathbb{N}}}$ converge in the same way.

Theorem 6.1**.**

*Let $r>0$, $\rho_{\mu}\in\widehat{U(n-1)}$ and $\tau_{\lambda}\in\widehat{U(n)}$.

If $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}{\mathcal{O}}_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} of elements of the admissible orbit space $\mathfrak{g}_{n}^{\ddagger}/G_{n}$ satisfies one of the following conditions:*

1. (i)

for $k$ large enough, $\alpha_{k}>0$, $\lambda_{j}^{k}=\mu_{j}$ for all $j\in\{1,...,n-1\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$, 2. (ii)

for $k$ large enough, $\alpha_{k}<0$, $\lambda_{j}^{k}=\mu_{j-1}$ for all $j\in\{2,...,n\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=-\frac{r^{2}}{2}$,

then the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the representation $\pi_{(\mu,r)}$ in $\widehat{G_{n}}$.

In order to prove this theorem, one needs the following proposition:

Proposition 6.2**.**

*Let $r>0$, $\rho_{\mu}\in\widehat{U(n-1)}$ and $\tau_{\lambda}\in\widehat{U(n)}$.

Furthermore, let $\underset{k\to\infty}{\lim}\alpha_{k}=0$, $\alpha_{k}>0$ for large $k$ and consider the sequence \big{(}\lambda^{k})_{k\in\mathbb{N}} in $P_{n}$ fulfilling $\lambda_{j}^{k}=\mu_{j}$ for all $j\in\{1,...,n-1\}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$.

Denote $\tilde{\mu}:=\tilde{\mu}^{n-1}=(\mu_{1},...,\mu_{n-1},\mu_{n-1})$, $N_{k}:=\mu_{n-1}-\lambda_{n}^{k}$ and let $\overline{{\mathcal{P}}_{N_{k}}}$ be the space of conjugated homogeneous polynomials of degree $N_{k}$.

Moreover, define the representation $\pi^{(\tilde{\mu},\alpha_{k})}$ of $G_{n}$ on the subspace ${\mathcal{H}}_{\tilde{\mu}}\otimes\overline{{\mathcal{P}}_{N_{k}}}\otimes{\mathcal{P}}_{N_{k}}$ of the Hilbert space ${\mathcal{H}}_{\tilde{\mu}}\otimes\overline{{\mathcal{F}}_{\alpha_{k}}(n)}\otimes{\mathcal{F}}_{\alpha_{k}}(n)$ by*

[TABLE]

Then, for any $\theta\in{\mathcal{H}}_{\tilde{\mu}}$ and for each $k\in{\mathbb{N}}$, there exist vectors $\xi_{k}\in{\mathcal{H}}_{\tilde{\mu}}\otimes\overline{{\mathcal{P}}_{N_{k}}}\otimes{\mathcal{P}}_{N_{k}}$ such that for all $(A,z,t)\in G_{n}$ and for $\xi^{\theta}:=\theta\otimes 1\in{\mathcal{H}}_{\tilde{\mu}}\otimes{\mathcal{H}}_{(0,r)}$,

[TABLE]

uniformly on compacta.

Proof:

Let $m_{k}:=(0,...,0,N_{k})$. Then, $\lambda^{k}=\tilde{\mu}+m_{k}$. Moreover, since $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=-\frac{r^{2}}{2}$, $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and $\alpha_{k}>0$, one gets $N_{k}\overset{k\to\infty}{\longrightarrow}\infty$.

Let $\phi\in{\mathcal{H}}_{\tilde{\mu}}$ and let

[TABLE]

Then, $R_{k}$ is the dimension of the space ${\mathcal{P}}_{N_{k}}(n)={\mathcal{P}}_{N_{k}}$ of complex polynomials of degree $N_{k}$ in $n$ variables. Now, define

[TABLE]

Since ${R_{k}}^{\frac{1}{2}}$ is the norm of $\sum\limits_{\begin{subarray}{c}q\in{\mathbb{N}}^{n}:\\ |q|=N_{k}\end{subarray}}\overline{h_{q,\alpha_{k}}}\otimes h_{q,\alpha_{k}}$, the vector $\frac{1}{{R_{k}}^{\frac{1}{2}}}~{}\sum\limits_{\begin{subarray}{c}q\in{\mathbb{N}}^{n}:\\ |q|=N_{k}\end{subarray}}\overline{h_{q,\alpha_{k}}}\otimes h_{q,\alpha_{k}}$ has norm $1$.

Let $(A,z,t)\in G_{n}$. One has

[TABLE]

Now, one can write

[TABLE]

with $w_{m,q}^{k}(A)\in{\mathbb{C}}$. Because of the unitarity of the matrix $W_{\alpha_{k}}(A)$, one gets for $m,m^{\prime}\in{\mathbb{N}}^{n}$ with $|m|=|m^{\prime}|=N_{k}$,

[TABLE]

Hence,

[TABLE]

Now, by the binomial theorem, letting $\binom{q}{l}:=\binom{q_{1}}{l_{1}}\cdots\binom{q_{n}}{l_{n}}$ for $q=(q_{1},...,q_{n})\in{\mathbb{N}}^{n}$ and $l=(l_{1},...,l_{n})\in{\mathbb{N}}^{n}$,

[TABLE]

Thus, one gets for $q\in{\mathbb{N}}^{n}$ with $|q|=N_{k}$,

[TABLE]

The integrals in $w_{m}$ for $m\in\{1,...,n\}$ can be written as follows:

[TABLE]

Therefore,

[TABLE]

Because of the orthogonality of the functions ${\mathbb{C}}^{n}\to{\mathbb{C}}^{n},x\mapsto x^{a}$ and ${\mathbb{C}}^{n}\to{\mathbb{C}}^{n},x\mapsto x^{b}$ for $a,b\in{\mathbb{N}}^{n}$ with respect to the scalar product of the Fock space, $j+l=q$, i.e. $l=q-j$. As $\|h_{q,\alpha_{k}}\|_{{\mathcal{F}}_{\alpha_{k}}(n)}=1$, the norm of the function $z\mapsto z^{q}$ is \sqrt{\frac{1}{\big{(}\frac{\alpha_{k}}{2\pi}\big{)}^{n}\frac{\alpha_{k}^{N_{k}}}{2^{N_{k}}q!}}} and hence,

[TABLE]

Thus,

[TABLE]

Now, regard

[TABLE]

Then, fixing large $k\in{\mathbb{N}}$, since $\underset{k\to\infty}{\lim}\alpha_{k}N_{k}=\frac{r^{2}}{2}$, one gets for $j=(j_{1},...,j_{n})\in{\mathbb{N}}^{n}$,

[TABLE]

The above expression does not depend on $k$ and

[TABLE]

So, by the theorem of Lebesgue, the sum in (6) converges and it suffices to regard the limit of each summand by itself. Hence, for $j=(j_{1},...,j_{n})\in{\mathbb{N}}^{n}$, one has

[TABLE]

Now, define for $k\in{\mathbb{N}}$ the function $F_{k}:[0,1]^{n-1}\to{\mathbb{R}}$ by

[TABLE]

Then, for $\varepsilon>0$ and large $k\in{\mathbb{N}}$,

[TABLE]

Since F_{k}\Big{(}\frac{q_{1}}{N_{k}},...,\frac{q_{n-1}}{N_{k}}\Big{)}=0, if $q_{1}<j_{1}$, $q_{2}<j_{2}$ or … or $q_{n-1}<j_{n-1}$ or $q_{1}+...+q_{n-1}>N_{k}-j_{n}$, it follows that

[TABLE]

Furthermore, $F_{k}$ converges pointwise to the function $F:[0,1]^{n-1}\to{\mathbb{R}}$ defined by

[TABLE]

and thus, by the theorem of Lebesgue for integrals,

[TABLE]

From these observations, one can now deduce that

[TABLE]

[TABLE]

Therefore,

[TABLE]

where the measure $d\sigma(v)$ is the invariant measure on the complex sphere $S^{n}$ in ${\mathbb{C}}^{n}$ defined in Corollary 8.2.

Choosing $\xi:=\phi\otimes 1\in{\mathcal{H}}_{\tilde{\mu}}\otimes{\mathcal{H}}_{(0,r)}$, the claim is shown.  

∎

Definition 6.3**.**

Let $\nu=(\nu_{1},\cdots,\nu_{n}),\mu=(\mu_{1},\cdots,\mu_{n})\in P_{n}$ and write

[TABLE]

if

[TABLE]

Let also for $\nu\in P_{n}$

[TABLE]

Lemma 6.4**.**

Let $\tilde{\mu}\in P_{n}$ and let $\chi_{\tilde{\nu}}$ be a weight of ${\mathbb{T}}^{n}$ appearing in ${\mathcal{H}}_{\tilde{\mu}}$ with $\tilde{\mu}\neq\tilde{\nu}$. Then

[TABLE]

and $\mu:=(\mu_{1},\cdots,\mu_{n-1})\neq\nu:=(\nu_{1},\cdots,\nu_{n-1})$.

Proof:

Since $\tilde{\nu}$ is a weight of ${\mathcal{H}}_{\tilde{\mu}}$, one has

[TABLE]

where $m_{i}\in{\mathbb{N}}$ for $i\in\{1,\cdots,n-1\}$ and $l_{i}$ is the fundamental weight of $\mathfrak{t}^{n}$ defined by

[TABLE]

Since $\tilde{\mu}\neq\tilde{\nu}$, there exists $j\in\{1,\cdots,n\}$ such that $\nu_{1}=\mu_{1},\cdots,\nu_{j-1}=\nu_{j-1}$ and $\nu_{j}<\mu_{j}$. In particular, $m_{1}=\cdots m_{j-1}=0$ and $m_{j}\neq 0$. But since $\sum\limits_{i=1}^{n}\mu_{i}=\sum\limits_{i=1}^{n}\nu_{i}$, one cannot have $\nu_{i}\leq\mu_{i}$ for all $i$. Hence, for some smallest $k>j$, one has $\nu_{k}>\mu_{k}$. Necessarily $k<n$ since otherwise $m_{i}=0$ for all $i\leq n-1$ and then $\tilde{\mu}=\tilde{\nu}$. Therefore, $\nu\neq\mu$ and $\tilde{\nu}\not\precsim\tilde{\mu}$.

∎

Proof of Theorem 6.1:

Without restriction, one can assume that the sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} fulfills Condition (i). The case of a sequence \big{(}\pi_{(\lambda^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} fulfilling the second condition is very similar.

 

For $\tilde{n}\in{\mathbb{N}}$ and $\nu\in P_{\tilde{n}}$, let $\phi^{\nu}$ be the highest weight vector of $\tau_{\nu}$ in the Hilbert space ${\mathcal{H}}_{\nu}$. Let $\tilde{\mu}:=(\mu_{1},...,\mu_{n-1},\mu_{n-1})$ and define the representation $\sigma_{(\tilde{\mu},r)}$ of $G_{n}$ by

[TABLE]

The Hilbert space ${\mathcal{H}}_{\sigma_{(\tilde{\mu},r)}}$ of the representation $\sigma_{(\tilde{\mu},r)}$ is the space

[TABLE]

and $G_{n}$ acts on ${\mathcal{H}}_{\sigma_{(\tilde{\mu},r)}}$ by

[TABLE]

One decomposes the representation ${\tau_{\tilde{\mu}}}_{|U(n-1)}$ into the direct sum of irreducible representations of the group $U(n-1)$ as follows:

[TABLE]

where $S(\tilde{\mu})$ denotes the support of ${\tau_{\tilde{\mu}}}_{|U(n-1)}$ in $\widehat{U(n-1)}$. Furthermore, let $p_{\nu}$ be the orthogonal projection of ${\mathcal{H}}_{\tilde{\mu}}$ onto its $U(n-1)$-invariant component ${\mathcal{H}}_{\nu}$.

The representation $\rho_{\mu}$ is one of the representations appearing in this sum, since the highest weight vector $\phi^{\tilde{\mu}}$ of $\tau_{\tilde{\mu}}$ is also the highest weight vector of the representation $\rho_{\mu}$.

Defining for $\tilde{\nu}\in P_{n}$ and the highest weight vector $\phi^{\tilde{\nu}}$ in ${\mathcal{H}}_{\tilde{\nu}}$ the function $c^{\tilde{\nu}}_{\eta,\phi^{\tilde{\nu}}}$ by

[TABLE]

one can identify for any $\tau_{\tilde{\nu}}\in\widehat{U(n)}$ the Hilbert space ${\mathcal{H}}_{\tilde{\nu}}$ with the subspace $L^{2}_{\tilde{\nu}}$ of L^{2}\big{(}U(n)\big{)} given by

[TABLE]

Now, it will be shown that

[TABLE]

In particular, one then gets

[TABLE]

For $\nu\in S(\tilde{\mu})$, $\xi\in L^{2}(U(n)/U(n-1),{\mathcal{H}}_{\tilde{\mu}})$, $A\in U(n)$ and $A^{\prime}\in U(n-1)$, let

[TABLE]

Moreover,

[TABLE]

i.e. one has a scalar product on the space ${\mathcal{H}}_{\nu}$. Furthermore, for all $\xi\in L^{2}(U(n)/U(n-1),{\mathcal{H}}_{\tilde{\mu}})$, $A\in U(n)$ and $A^{\prime}\in U(n-1)$,

[TABLE]

Hence, each vector $U_{\tilde{\mu}}^{\nu}(\xi)$ fulfills the covariance condition of the space L^{2}\big{(}U(n)/U(n-1),\rho_{\nu}\big{)}. Therefore, $U_{\tilde{\mu}}(\xi):=\sum\limits_{\nu\in S(\tilde{\mu})}U_{\tilde{\mu}}^{\nu}(\xi)$ is an element of the space \sum\limits_{\nu\in S(\tilde{\mu})}L^{2}\big{(}U(n)/U(n-1),\rho_{\nu}\big{)}:

[TABLE]

Furthermore,

[TABLE]

[TABLE]

Moreover, for all $(A,z,t)\in G_{n}$, all \xi\in L^{2}\big{(}U(n)/U(n-1),{\mathcal{H}}_{\tilde{\mu}}\big{)}, all $B\in U(n)$ and all $A^{\prime}\in U(n-1)$, one gets

[TABLE]

Therefore, (6.3) holds.

For \xi=\phi^{\tilde{\mu}}\otimes 1\in L^{2}\big{(}U(n)/U(n-1),{\mathcal{H}}_{\tilde{\mu}}\big{)}, such that $\xi(A)=\phi^{\tilde{\mu}}$ for all $A$, one has for all $B\in U(n)$ and all $A^{\prime}\in U(n-1)$,

[TABLE]

In particular, since $\phi^{\mu}=\phi^{\tilde{\mu}}$,

[TABLE]

From Theorem 5.2 follows that the subset \big{\{}\pi_{(\nu,r)}|~{}\mu\neq\nu\in P_{n-1}\big{\}} is closed in $\widehat{G_{n}}$. Hence, there exists $F_{\mu}=(F_{\mu})^{*}$ of norm 1 in $C^{*}(G_{n})$ whose Fourier transform at $\pi_{(\nu,r)}$ is 0 if $\mu\neq\nu\in P_{n-1}$ and for which

[TABLE]

is the orthogonal projection onto the space ${\mathbb{C}}\Phi_{\tilde{\mu}}^{\mu}\subset{\mathcal{H}}_{(\mu,r)}$. In particular,

[TABLE]

Define the coefficient $c_{\mu}$ of $L^{1}(G_{n})$ by

[TABLE]

for all $F\in L^{1}(G_{n})$.

Let $X(\tilde{\mu},\overline{\alpha_{k}})$ be the collection of all $\tilde{\nu}=(\nu_{1},...,\nu_{n})\in P_{n}$ such that $\chi_{\tilde{\nu}}$ is a character of ${\mathbb{T}}_{n}$ appearing in ${\mathcal{H}}_{\tilde{\mu}}$ and such that $\tau_{\tilde{\nu}_{k}}$ is contained in the representation $\tau_{\tilde{\mu}}\otimes\overline{W_{\alpha_{k}}}$ for $\tilde{\nu}_{k}:=(\nu_{1},\nu_{2},...,\nu_{n-1},\nu_{n}+\lambda_{n}^{k})$.

Then, for $\pi^{(\tilde{\mu},\alpha_{k})}$ defined as in Proposition 6.2, by [20], Chapter IV.11,

[TABLE]

Furthermore, decompose the vector

[TABLE]

for every $k\in{\mathbb{N}}$ into the orthogonal sum

[TABLE]

for $\xi_{k}^{\tilde{\nu}}\in{\mathcal{H}}_{(\tilde{\nu}_{k},\alpha_{k})}$. This gives a decomposition

[TABLE]

Let $c_{\xi^{\tilde{\nu}}}^{\tilde{\nu}}$ be the weak∗-limit of a subsequence of \big{(}c_{\xi_{k}^{\tilde{\nu}}}^{\tilde{\nu}}\big{)}_{k\in{\mathbb{N}}} and let for $c_{\xi^{\tilde{\nu}}}^{\tilde{\nu}}\neq 0$ the representation $\pi\in\widehat{G_{n}}$ be an element of the support of $c_{\xi^{\tilde{\nu}}}^{\tilde{\nu}}$. From Theorem 5.6 follows that $\pi=\lim\limits_{k\to\infty}\pi_{(\tilde{\nu}_{k},\alpha_{k})}=\pi_{(\nu,r)}$ for $\nu=(\nu_{1},...,\nu_{n-1})$. Furthermore, one observes that for $\tilde{\mu}\neq\tilde{\nu}:=(\nu_{1},...,\nu_{n-1},\nu_{n})\in X(\tilde{\mu},\overline{\alpha_{k}})$, one has $\nu\neq\mu$. Hence, $\pi_{(\nu,r)}(F_{\mu})=0$. Thus,

[TABLE]

and therefore,

[TABLE]

Now, $\tilde{\mu}_{k}=(\mu_{1},...,\mu_{n-1},\mu_{n-1}+\lambda_{n}^{k})$ and $\lambda^{k}=(\mu_{1},...,\mu_{n-1},\lambda_{n}^{k})$. Since $\lambda_{n}^{k}\overset{k\to\infty}{\longrightarrow}-\infty$, their behavior for $k\to\infty$ is the same. Hence, by Proposition 6.2 and its proof, for all $F\in C^{*}(G_{n})$,

[TABLE]

Choosing $\tilde{\xi}:=\Phi_{\tilde{\mu}}^{\mu}$ and $\tilde{\xi}_{k}:=\pi_{(\tilde{\mu}_{k},\alpha_{k})}(F_{\mu})\xi_{k}^{\tilde{\mu}}$, one has for any $F\in C^{*}(G_{n})$

[TABLE]

and hence,

[TABLE]

∎

Remark 6.5**.**

It follows from (6.7) that

[TABLE]

Theorem 6.6**.**

Let $\tau_{\tilde{\rho}}\in\widehat{U(n)}$.

If $\underset{k\to\infty}{\lim}\alpha_{k}=0$ and the sequence \big{(}\pi_{(\tilde{\lambda}^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} of elements of $\widehat{G_{n}}$ satisfies one of the following conditions:

1. (i)

for $k$ large enough, $\alpha_{k}>0$, $\rho_{1}\geq\lambda_{1}^{k}\geq...\geq\rho_{n-1}\geq\lambda_{n-1}^{k}\geq\rho_{n}\geq\lambda_{n}^{k}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{n}=0$, 2. (ii)

for $k$ large enough, $\alpha_{k}<0$, $\lambda_{1}^{k}\geq\rho_{1}\geq\lambda_{2}^{k}\geq\rho_{2}\geq...\geq\lambda_{n}^{k}\geq\rho_{n}$ and $\underset{k\to\infty}{\lim}\alpha_{k}\lambda^{k}_{1}=0$,

*then the sequence \big{(}\pi_{(\tilde{\lambda}^{k},\alpha_{k})}\big{)}_{k\in\mathbb{N}} converges to the representation $\tau_{\tilde{\rho}}$ in $\widehat{G_{n}}$.

Proof:

Again, only consider the case $\alpha_{k}>0$ for all $k\in{\mathbb{N}}$.

Let $\tilde{\rho}=(\rho_{1},\cdots,\rho_{n})\in P_{n}$ satisfy the conditions of the theorem, i.e. $\rho_{1}\geq\lambda_{1}^{k}\geq...\geq\rho_{n-1}\geq\lambda_{n-1}^{k}\geq\rho_{n}\geq\lambda_{n}^{k}$. Passing to a subsequence, one can assume that $\mu_{1}:=\lambda_{1}^{k},\cdots,\mu_{n-1}:=\lambda^{k}_{n-1}$ for all $k\in{\mathbb{N}}$.

Let

[TABLE]

Then

[TABLE]

for some $r=(r_{1},\cdots,r_{n})\in{\mathbb{N}}^{n}$ and $r_{k}=r+(0,\cdots,0,N_{k})$. Let

[TABLE]

Hence, by Pieri’s rule, one obtains

[TABLE]

for $k$ large enough. We take the highest weight vector $\phi^{\tilde{\rho}}_{k}$ of the representation $\tau_{\tilde{\rho}}$ considered as a subrepresentation of $U(n)$ on the Hilbert space ${\mathcal{H}}_{\tilde{\mu}_{k}}\otimes{\mathcal{P}}_{N_{k}+m,\alpha_{k}}(n)$, where ${\mathcal{P}}_{N_{k}+m,\alpha_{k}}(n)$ is the space of all polynomials of degree $N_{k}+m$ in the Fock space ${\mathcal{F}}_{\alpha_{k}}(n)$.

Recall also that the polynomials

[TABLE]

form a Hilbert space basis of ${\mathcal{P}}_{N_{k}+m,\alpha_{k}}(n)$. Hence, one can write

[TABLE]

where for any $q$ and $k$, the vector $a^{k}_{q}$ is contained in the ${\mathbb{T}}^{n}$-eigenspace of ${\mathcal{H}}_{\tilde{\mu}_{k}}$ for the weight $\chi_{\tilde{\rho}-q}$, since $h_{q,\alpha_{k}}$ is in the eigenspace for the weight $\chi_{q}$. In particular, one has

[TABLE]

Let (\xi_{k})_{k\in{\mathbb{N}}}=\Bigg{(}\sum\limits_{\begin{subarray}{c}q\in{\mathbb{N}}^{n}:\\ |q|=N_{k}\end{subarray}}a_{q}^{k}\otimes h_{q,\alpha_{k}}\Bigg{)}_{k\in{\mathbb{N}}} be a sequence of vectors of length 1 in \big{(}{\mathcal{H}}_{\tilde{\mu}_{k}}\otimes{\mathcal{P}}_{N_{k}}\big{)}_{k\in{\mathbb{N}}} such that

[TABLE]

It will now be shown that

[TABLE]

uniformly on compacta in $(z,t)\in\mathbb{H}_{n}$:

Since \sum\limits_{\begin{subarray}{c}q\in{\mathbb{N}}^{n}:\\ |q|=N_{k}\end{subarray}}\big{\|}a_{q}^{k}\big{\|}^{2}_{{\mathcal{H}}_{\tilde{\mu}_{k}}}=\|\xi_{k}\|_{{\mathcal{H}}_{(\tilde{\mu}_{k},\alpha_{k})}}^{2}=1 for all $k\in{\mathbb{N}}$, it suffices to prove that

[TABLE]

uniformly on compacta.

One gets

[TABLE]

since for $q\neq q^{\prime}$

[TABLE]

Now, for any $q\in{\mathbb{N}}^{n}$ and any $k\in{\mathbb{N}}$, by (6.1),

[TABLE]

and thus, for $k$ large enough and $(z,t)$ in some compact set (i.e. $|\frac{\alpha_{k}N_{k}}{2}|<e^{-|z|^{2}}$), one has

[TABLE]

Therefore,

[TABLE]

This proves Claim (6.8).

To finish the proof of Theorem 6.6, by (6.8) for any $(A,z,t)\in G_{n}$, one has uniformly on compacta

[TABLE]

for the highest weight vector $\phi^{\tilde{\rho}}$ of the representation $\tau_{\tilde{\rho}}$. This shows that

[TABLE]

∎

7. The final result

Together with the Theorems 5.2, 5.4, 5.6, 6.1 and 6.6 and the result in Subsection 5.2, we obtain the final result below:

Theorem 7.1**.**

For general $n\in{\mathbb{N}}^{*}$, the spectrum of the group $G_{n}=U(n)\ltimes\mathbb{H}_{n}$ is homeomorphic to its space of admissible coadjoint orbits $\mathfrak{g}_{n}^{\ddagger}/G_{n}$.

8. Appendix

Lemma 8.1**.**

Let $n\in{\mathbb{N}}^{*}$, let $B_{{\mathbb{R}}}^{2n}$ be the $2n$-dimensional real unit ball and define the mapping

[TABLE]

*where $s=\sum\limits_{i=1}^{n-1}s_{i}$.

Then, the absolute value of the determinant of the Jacobian of $\psi$ equals $\frac{1}{2^{n-1}}\cdot\rho^{2n-1}$.*

Proof:

Denote for $i\in\{1,...,2n\}$ by $C_{i}$ the $i$-th column and by $R_{i}$ the $i$-th row of the Jacobian of $\psi$.

For $i\in\{1,...,n-1\}$, one has

[TABLE]

and

[TABLE]

Now, in several steps, this matrix will be transformed into a new matrix whose determinant can easily be calculated. For simplicity, the columns and rows of the matrices appearing in each step will also be denoted by $C_{i}$ and $R_{i}$ for $i\in\{1,...,2n\}$.

First, one takes out the factor $\rho$ in the rows $R_{i}$ for $i\in\{1,...,2n-1\}$, the factor $\frac{1}{2}$ in the rows $R_{i}$ for $i\in\{1,...,n-1\}$, the factor $\sqrt{s_{i}}$ in the columns $C_{2i-1}$ and $C_{2i}$ for every $i\in\{1,...,n-1\}$ and the factor $\sqrt{1-s}$ in the columns $C_{2n-1}$ and $C_{2n}$. Hence, one has the prefactor $\rho^{2n-1}\frac{1}{2^{n-1}}s_{1}\cdots s_{n-1}(1-s)$ and the columns of the remaining matrix have the shape

[TABLE]

for all $i\in\{1,...,n-1\}$ and

[TABLE]

Next, for every $i\in\{1,...,n\}$, the column $C_{2i-1}$ shall be replaced by $\sin(t_{i})C_{2i-1}-\cos(t_{i})C_{2i}$. Then, the prefactor changes to $\rho^{2n-1}\frac{1}{2^{n-1}}\frac{s_{1}\cdots s_{n-1}(1-s)}{\sin(t_{1})\cdots\sin(t_{n})}$ and for every $i\in\{1,...,n\}$,

[TABLE]

The columns $C_{2i}$ for $i\in\{1,...,n\}$ stay the same.

Now, for all $i\in\{1,...,n-1\}$, the rows $R_{i}$ and $R_{n-1+i}$ will be interchanged. Therefore, the prefactor is multiplied by $(-1)^{n-1}$ and for every $i\in\{1,...,n-1\}$,

[TABLE]

and

[TABLE]

In the next step, for every $i\in\{1,...,n-1\}$, the matrix will be developed with respect to the $i$-th row, which has only one non-zero entry, namely the entry $-1$ in the $(2i-1)$-th column. One develops with respect to the $(2n-1)$-th row which also only consists of one non-zero entry, $-1$, in the $(2n-1)$-th column. The prefactor is then multiplied by $(-1)^{n}(-1)^{2n-1+2n-1}\prod\limits_{i=1}^{n-1}(-1)^{i+2i-1}=\prod\limits_{i=1}^{n-1}(-1)^{n+i-1}$, i.e. the prefactor now equals

[TABLE]

One has a $n\times n$-matrix left, whose columns have the shape

[TABLE]

for all $i\in\{1,...,n-1\}$. Now, in every column $C_{i}$ for $i\in\{1,...,n\}$, one can take out the factor $\sin(t_{i})$. Then, the prefactor changes to $\prod\limits_{i=1}^{n-1}(-1)^{i}\rho^{2n-1}\frac{1}{2^{n-1}}s_{1}\cdots s_{n-1}(1-s)$ and one has the following columns:

[TABLE]

for all $i\in\{1,...,n-1\}$. In the last step, the column $C_{n}$ will be replaced by $C_{n}+\frac{1}{1-s}\sum\limits_{i=1}^{n-1}s_{i}C_{i}$. Since

[TABLE]

one obtains the columns

[TABLE]

for $i\in\{1,...,n-1\}$ and the prefactor stays the same, i.e. $\prod\limits_{i=1}^{n-1}(-1)^{i}\rho^{2n-1}\frac{1}{2^{n-1}}s_{1}\cdots s_{n-1}(1-s)$.

Since the remaining matrix is a triangular matrix, one can easily calculate its determinant and gets

[TABLE]

∎

Corollary 8.2**.**

For the map $\psi$ defined in Lemma 8.1, the measure defined on the complex sphere $S^{n}$ in ${\mathbb{C}}^{n}$ by

[TABLE]

is the $U(n)$-invariant measure such that for each function $f$ which is continuous on the unit ball in ${\mathbb{C}}^{n}$,

[TABLE]

Proof:

(8.1) gives the decomposition of the Lebesgue measure on $B^{n}\simeq[0,1]\times S^{n}$ through $z=\rho v$. Thus, it defines the $U(n)$-invariant measure $d\sigma(v)$ on the unit sphere $S^{n}$. Moreover, by Lemma 8.1

[TABLE]

Since $\psi(s_{1},...,s_{n-1},t_{1},...,t_{n},\rho)=\rho\psi(s_{1},...,s_{n-1},t_{1},...,t_{n},1)$ and $\psi(s_{1},...,s_{n-1},t_{1},...,t_{n},1)$ is normed, this proves the corollary.  

∎ 

 

Acknowledgements:

The authors would like to thank the referee for his / her careful reading of our paper and many valuable suggestions.

Janne-Kathrin Günther was supported for this work by the Fonds National de la Recherche, Luxembourg (Project Code 3964572).

9. References

1. (1)

L.W.Baggett, A description of the topology on the dual spaces of certain locally compact groups, Transactions of the American Mathematical Society 132, pp.175-215, 1968. 2. (2)

D.Beltita, I.Beltita and J.Ludwig, Fourier transforms of $C^{*}$-algebras of nilpotent Lie groups, to appear in: International Mathematics Research Notices, doi:10.1093/imrn/rnw040. 3. (3)

C.Benson, J.Jenkins, R.Lipsman and G.Ratcliff, A geometric criterion for Gelfand pairs associated with the Heisenberg group, Pacific Journal of Mathematics 178, no.1, pp.1-36, 1997. 4. (4)

C.Benson, J.Jenkins and G.Ratcliff, Bounded $K$-spherical functions on Heisenberg groups, Journal of Functional Analysis 105, pp.409-443, 1992. 5. (5)

C.Benson, J.Jenkins, G.Ratcliff and T.Worku, Spectra for Gelfand pairs associated with the Heisenberg group, Colloquium Mathematicae 71, pp.305-328, 1996. 6. (6)

I.Brown, Dual topology of a nilpotent Lie group, Annales scientifiques de l’É.N.S. $4^{e}$ série, tome 6, no.3, pp.407-411, 1973. 7. (7)

L.Cohn, Analytic Theory of the Harish-Chandra C-Function, Springer-Verlag, Berlin-Heidelberg-New York, 1974. 8. (8)

L.Corwin and F.P.Greenleaf, Representations of nilpotent Lie groups and their applications. Part I. Basic theory and examples, Cambridge Studies in Advanced Mathematics, vol.18, Cambridge University Press, Cambridge, 1990. 9. (9)

J.Dixmier, $C^{*}$-algebras. Translated from French by Francis Jellett, North-Holland Mathematical Library, vol.15, North-Holland Publishing Company, Amsterdam-New York-Oxford, 1977. 10. (10)

J.Dixmier and P.Malliavin, Factorisations de fonctions et de vecteurs indéfiniment différentiables, Bulletin des Sciences Mathématiques (2) 102, no.4, pp.307-330, 1978. 11. (11)

M.Elloumi, Espaces duaux de certains produits semi-directs et noyaux associés aux orbites plates, Ph.D. thesis at the Université de Lorraine, 2009. 12. (12)

M.Elloumi and J.Ludwig, Dual topology of the motion groups $SO(n)\ltimes{\mathbb{R}}^{n}$, Forum Mathematicum 22, pp.397-410, 2010. 13. (13)

J.M.G.Fell, The structure of algebras of operator fields, Acta Mathematica 106, pp.233-280, 1961. 14. (14)

G.B.Folland, Harmonic Analysis in Phase Space, Princeton University Press, 1989. 15. (15)

W.Fulton and J.Harris, Representation theory, Readings in Mathematics, Springer-Verlag, 1991. 16. (16)

J.-K.Günther, The $C^{*}$-algebra of $SL(2,{\mathbb{R}})$, arXiv:1605.09256, 2016. 17. (17)

J.-K.Günther and J.Ludwig, The $C^{*}$-algebras of connected real two-step nilpotent Lie groups, Revista Matemática Complutense 29(1), pp.13-57, 10.1007/s13163-015-0177-7, 2016. 18. (18)

W.J.Holman III and L.C.Biedenharn, The Representations and Tensor Operators of the Unitary Groups $U(n)$, in: Group Theory and its Applications, Volume 2, edited by E.M.Loebl, Academic Press, Incorporation, London, 1971. 19. (19)

R.Howe, Quantum mechanics and partial differential equations, Journal of Functional Analysis 38, pp.188-255, 1980. 20. (20)

A.Knapp, Representation Theory of Semisimple Groups. An Overview Based on Examples, Princeton University Press, Princeton, New Jersey, 1986. 21. (21)

R.Lahiani, Analyse Harmonique sur certains groupes de Lie à croissance polynomiale, Ph.D. thesis at the University of Luxembourg and the Université Paul Verlaine-Metz, 2010. 22. (22)

S.Lang, $SL_{2}({\mathbb{R}})$, Graduate Texts in Mathematics 105, Springer-Verlag, New York, 1985. 23. (23)

H.Leptin and J.Ludwig, Unitary representation theory of exponential Lie groups, De Gruyter Expositions in Mathematics 18, 1994. 24. (24)

Y.-F.Lin and J.Ludwig, The $C^{*}$-algebras of $ax+b$-like groups, Journal of Functional Analysis 259, pp.104-130, 2010. 25. (25)

R.L.Lipsman, Orbit theory and harmonic analysis on Lie groups with co-compact nilradical, Journal de Mathématiques Pures et Appliquées, tome 59, pp.337-374, 1980. 26. (26)

J.Ludwig and L.Turowska, The $C^{*}$-algebras of the Heisenberg Group and of thread-like Lie groups, Mathematische Zeitschrift 268, no.3-4, pp.897-930, 2011. 27. (27)

J.Ludwig and H.Zahir, On the Nilpotent $*$-Fourier Transform, Letters in Mathematical Physics 30, pp.23-24, 1994. 28. (28)

G.W.Mackey, Unitary group representations in physics, probability and number theory, Benjamin-Cummings, 1978. 29. (29)

L.Pukanszky, Leçons sur les représentations des groupes, Dunod, Paris, 1967. 30. (30)

H.Regeiba, Les $C^{*}$-algèbres des groupes de Lie nilpotents de dimension $\leq 6$, Ph.D. thesis at the Université de Lorraine, 2014. 31. (31)

H.Regeiba and J.Ludwig, $C^{*}$-Algebras with Norm Controlled Dual Limits and Nilpotent Lie Groups, arXiv: 1309.6941, 2013. 32. (32)

N.Wallach, Real Reductive Groups I, Academic Press, Pure and Applied Mathematics, San Diego, 1988. 33. (33)

N.Wallach, Real Reductive Groups II, Academic Press, Pure and Applied Mathematics, San Diego, 1992. 34. (34)

A.Wassermann, Une démonstration de la conjecture de Connes-Kasparov pour les groupes de Lie linéaires connexes réductifs, Comptes Rendus de l’Académie des Sciences, Paris Series I Mathematics 304, no.18, pp.559-562, 1987.