Projectors separating spectra for $L^2$ on pseudounitary groups $U(p,q)$

TL;DR

This paper explicitly constructs orthogonal projectors in the $L^2$ space on pseudo-unitary groups $U(p,q)$ to separate spectra into distinct types, addressing a classical question by Gelfand and Gindikin.

Contribution

It provides explicit formulas for orthogonal projectors in $L^2(U(p,q))$ that isolate spectral components, including new finer spectral decompositions based on discrete series and tempered parameters.

Findings

Explicit orthogonal projectors for spectral separation in $L^2(U(p,q))$

New spectral decompositions based on discrete series representations

Addresses classical spectral separation question by Gelfand and Gindikin

Abstract

The spectrum of $L^2$ on a pseudo-unitary group $U(p,q)$ (we assume $p\ge q$ naturally splits into $q+1$ types. We write explicitly orthogonal projectors in $L^2$ to subspaces with uniform spectra (this is an old question formulated by Gelfand and Gindikin). We also write two finer separations of $L^2$. In the first case pieces are enumerated by $r=0$, 1,..., $q$ and representations of discrete series of $U(p-r,q-r)$, where $r=0$, \dots, $q$. In the second case pieces are enumerated by all discrete parameters of the tempered spectrum of $U(p,q)$.

Equations

$$L^2(G)=L_1\oplus \cdots \oplus L_m.$$

$$E={\Pi }_1+\cdots +{\Pi }_m,$$

$$G:=\mathrm{U}(p,q)$$

$$g\left(\begin{array}{cc}{1}_p& \\ & -1_q\end{array}\right)g^{\ast }=\left(\begin{array}{cc}{1}_p& \\ & -1_q\end{array}\right).$$

$$\mathcal{R}(h_1,h_2)f(g):=f(h_1^{-1}g{h}_2),\text{(h1,h2)∈U(p,q)×U(p,q), g∈U(p,q)}.$$

$$I_r:=\left(\begin{array}{cccc}0& & & J_r\\ & 1_{p-r}& & \\ & & 1_{q-r}& \\ J_r& & & 0\end{array}\right)$$

$$g{I}_r{g}^{\ast }=I_r.$$

$$\underset{\text{r times}}{\underbrace{1+\cdots +1}}+(p+q-2r)+\underset{\text{r times}}{\underbrace{1+\cdots +1}}$$

$$h=\left(\begin{array}{ccccccccc}{\overline{\zeta }}_1^{-1}& \ast & \dots & \ast & \ast & \ast & \dots & \ast & \ast \\ & {\overline{\zeta }}_2^{-1}& \dots & \ast & \ast & \ast & \dots & \ast & \ast \\ & & \ddots & \ast & \ast & \ast & \dots & \ast & \ast \\ & & & {\overline{\zeta }}_r^{-1}& \ast & \ast & \dots & \ast & \ast \\ & & & & Z& \ast & \dots & \ast & \ast \\ & & & & & {\zeta }_r& \dots & \ast & \ast \\ & & & & & & \ddots & \ast & \ast \\ & & & & & & & {\zeta }_2& \ast \\ & & & & & & & & {\zeta }_1\end{array}\right).$$

$${\mu }_{A;c,m,\rho }(h)=\prod _{j=1}^r|{\zeta }_j{|}^{i{\rho }_j}{\zeta }_j^{m_j}\cdot {\tau }_{A;c}(Z),$$

$$L^{2}\bigl{(}\mathrm{U}(p,q)\bigr{)}\simeq\bigoplus_{r=0}^{q}\int_{T_{A;c,m,\rho}\in\mathfrak{A}_{r}}(T_{A;c,m,\rho})^{*}\otimes T_{A;c,m,\rho}\,d\mathcal{P}_{r}(A;c,m,\rho),$$

$$\left(\begin{array}{cccccccccc}{1}_{p-k}& & & & & & & & & 0\\ & \cosh t_k& & & & & & & \sinh t_k& \\ & & \cosh t_{k-1}& & & & & \sinh t_{k-1}& & \\ & & & \ddots & & & {}_{\text{.}}{\cdot }^{\text{.}}& & & \\ & & & & \cosh t_1& \sinh t_1& & & & \\ & & & & \sinh t_1& \cosh t_1& & & & \\ & & & {}_{\text{.}}{\cdot }^{\text{.}}& & & \ddots & & & \\ & & \sinh t_{k-1}& & & & & \cosh t_{k-1}& & \\ & \sinh t_k& & & & & & & \cosh t_k& \\ 0& & & & & & & & & 1_{q-k}\end{array}\right)$$

$$\begin{gathered} \text{eiφ1, eiφ2, …, eiφp−k, eiθk, eiθk−1, …, eiθ1,} \\ \text{ eiθ1, …, eiθk−1, eiθk, eiψq−k, …, eiψ2, eiψ1. } \end{gathered}$$

$$H_k:=H_k^{+}\cdot H_k^{-}.$$

$$z_j:=t+i{\theta }_j.$$

$$\begin{gathered} \text{eiφ1, eiφ2, …, eiφp−k, ez1, ez2,…, ezk,} \\ \text{eiψ1, eiψ2, …, eiψq−k, e−z1, e−z2,…, e−zk.} \end{gathered}$$

$$d_{k}h=\prod _{\alpha =1}^{p-k}d{\phi }_{\alpha }\prod _{\beta =1}^{q-k}d{\psi }_{\beta }\prod _{\gamma =1}^{k}d{t}_{\gamma }d{\theta }_{\gamma }.$$

$$W_k\simeq S_{p-k}\times S_{q-k}\times (S_k⋉{\mathbb{Z}}_2^k),$$

$$R_{\gamma }:(t_1,\dots ,t_{\gamma -1},t_{\gamma },t_{\gamma +1},\dots ,t_k)\mapsto (t_1,\dots ,t_{\gamma -1},-t_{\gamma },t_{\gamma +1},\dots ,t_k)$$

$$\text{Updelta}(y)=\text{Updelta}(y_1,\dots ,y_n)=\prod _{1⩽j<l⩽n}^n(y_j-y_l).$$

$${\text{Updelta}}_k(h):=\text{Updelta}(e^{x_1},\dots ,e^{x_n}).$$

$$\frac{\partial}{i\partial\varphi_{1}},\dots,\frac{\partial}{i\partial\varphi_{p-k}},\,\frac{1}{2}\Bigl{(}\frac{\partial}{\partial t_{1}}+\frac{\partial}{i\partial\theta_{1}}\Bigr{)},\dots,\frac{1}{2}\Bigl{(}\frac{\partial}{\partial t_{k}}+\frac{\partial}{i\partial\theta_{k}}\Bigr{)}\\ \frac{\partial}{i\partial\psi_{1}},\dots,\frac{\partial}{i\partial\psi_{q-k}},\,\frac{1}{2}\Bigl{(}-\frac{\partial}{\partial t_{1}}+\frac{\partial}{i\partial\theta_{1}}\Bigr{)},\dots,\frac{1}{2}\Bigl{(}-\frac{\partial}{\partial t_{k}}+\frac{\partial}{i\partial\theta_{k}}\Bigr{)}.$$

$${\text{Updelta}}_k(\partial )=\text{Updelta}(X_1,\dots ,X_n).$$

$$I_{k}f(h)={\int }_{G/H_k}f(yh{y}^{-1})dy,h\in H_k,y\in G.$$

$${\int }_{G}f(g)dg=\sum _{k=0}^q{\omega }_k{\int }_{H_k}{I}_{k}f(h)|\text{Updelta}(h){|}^2{d}_k(h),$$

$${\omega }_k:=\frac{1}{\#W_k}=\frac{1}{(p-k)!(q-k)!k!2^k}$$

$${\Xi }_{k}f(h):=\prod \mathrm{sign}(t_k)\cdot \overline{\text{Updelta}(h)}{I}_{k}f(h).$$

$$Q_{\chi }f(h)=⟨⟨f(gh),\chi (g)⟩{⟩}_{(G)},$$

$$M_{*}\langle\!\langle f,\Theta_{r}\rangle\!\rangle_{(G)}=(-1)^{n(n-1)/2+pq+qr+r(r-1)/2}\sum_{k=r}^{q}\frac{2^{n-2k}\pi^{n-k}k!}{(k-r)!\,r!}\times\\ \times\omega_{k}\Bigl{\langle}\!\!\Bigl{\langle}\Updelta_{k}(\partial)\,\Xi_{k}\,f(\varphi,\psi,\theta,t),\\ \prod_{\alpha=1}^{p-k}\delta(\varphi_{\alpha})\prod_{\beta=1}^{q-k}\delta(\psi_{\beta})\prod_{\gamma=1}^{k}\bigl{(}\coth(t_{\gamma}/2)\,\delta(\theta_{\gamma})+\tanh(t_{\gamma}/2)\delta(\theta_{\gamma}-\pi)\bigr{)}\Bigr{\rangle}\!\!\Bigr{\rangle}_{(H_{k})},$$

$$⟨⟨F,\coth (t/2)⟩{⟩}_{(\mathbb{R})}:=p.v{\int }_{\mathbb{R}}\coth (t/2)F(t)dt$$

Full text

**Projectors separating spectra

for $L^{2}$ on pseudounitary groups $\mathrm{U}(p,q)$**

Yury A. Neretin111Supported by the grants FWF, P25142, P28421

The spectrum of $L^{2}$ on a pseudo-unitary group $\mathrm{U}(p,q)$ (we assume $p\geqslant q$) naturally splits into $q+1$ types. We write explicitly orthogonal projectors in $L^{2}$ to subspaces with uniform spectra (this is an old question formulated by Gelfand and Gindikin). We also write two finer separations of $L^{2}$. In the first case pieces are enumerated by $r=0$, 1, …, $q$ and representations of discrete series of $\mathrm{U}(p-r,q-r)$, where $r=0$, …, $q$. In the second case pieces are enumerated by all discrete parameters of the tempered spectrum of $\mathrm{U}(p,q)$.

1 Formulas for the projectors

1.1. Problem of separation of spectra. Recall a problem formulated in the paper [8] by I. M. Gelfand and S. G. Gindikin in 1977. Consider a real semisimple Lie group $G$, the left-right action of $G\times G$ on $G$ and the corresponding regular representation in $L^{2}(G)$ (the group is equipped with the Haar measure). The spectrum of the regular representation splits in a natural way into several pieces (according the number of non-conjugate Cartan subgroups). Therefore there is a natural decomposition of $L^{2}(G)$ into a direct sum of subrepresentations with uniform spectra,

[TABLE]

Respectively, we have a natural decomposition of the identity operator

[TABLE]

where $\Pi_{j}$ are orthogonal projectors to the subspaces $L_{j}$. There arises a question about explicit descriptions of such decompositions.

In [8] there was considered the case $G=\mathrm{SL}(2,{\mathbb{R}})$. The space L^{2}\bigl{(}\mathrm{SL}(2,{\mathbb{R}})\bigr{)} is a sum of highest weight representations, a sum of lowest weight representations, and a direct integral over the continuous series. It appears that the summands corresponding to highest weight and lowest weight representations can be regarded as certain Hardy spaces $H^{2}$.

The same question about separation of spectra arises for $L^{2}$ on semi-simple pseudo-Riemannian symmetric spaces and for some other problems of non-commutative harmonic analysis (a natural splitting of spectrum to different pieces is a usual phenomenon).

1.2. Known results. a) Transparent descriptions of decompositions are known for several problems related to $\mathrm{SL}(2)$:

— For $L^{2}(\mathrm{SL}(2,{\mathbb{R}}))$, see [8], [11], [12], [1].

— Consider the homogeneous space $\mathrm{SL}(2,{\mathbb{R}})/H$, where $H$ is the diagonal subgroup, this space can be identified with one-sheeted hyperboloid. The separation of spectrum in $L^{2}$ was discussed in [23], [26].

— The space L^{2}\bigl{(}\mathrm{SL}(2,{\mathbb{C}})/\mathrm{SL}(2,{\mathbb{R}})\bigr{)} was considered in [7].

b) G. I. Olshanski [29] proposed a way of splitting of holomorphic series using non-commutative ’Hardy spaces’, this approach was used in several works, see, e.g., [18], [3], [22], [21].

Also, boundary values of holomorphic functions allow to split off a part of a mostly continuous series for $L^{2}$ on some pseudo-Riemannian symmetric spaces, [13].

c) S. G. Gindikin [10] and V. F. Molchanov [24] in different ways solved a problem for multi-dimensional hyperboloids $\mathrm{O}(p,q)/\mathrm{O}(p,q-1)$; this covers also all $\mathrm{SL}(2)$-cases mentioned above.

d) In [25] there was proposed a way to split summands of complementary series using trace theorems, see more in [28], [27].

However up to now explicit separations of spectra on groups remain to be unknown except $\mathrm{SL}(2,{\mathbb{R}})$.

In the present paper we obtain such description for $L^{2}$ on pseudounitary groups $\mathrm{U}(p,q)$.

1.3. Pseudounitary groups and principal series. Let $p\geqslant q>0$. We consider the space ${\mathbb{C}}^{p}\oplus{\mathbb{C}}^{q}$ equipped with an Hermitian form with matrix $\begin{pmatrix}1_{p}&\\ &-1_{q}\end{pmatrix}$ (absent elements of matrices are 0 on default, $1_{j}$ denotes a unit matrix of size $j$). The group

[TABLE]

consists of matrices $g$ preserving this form, i.e.,

[TABLE]

Consider the left-right regular representation $R$ of the group $\mathrm{U}(p,q)\times\mathrm{U}(p,q)$ in L^{2}\bigl{(}\mathrm{U}(p,q)\bigr{)},

[TABLE]

Recall a decomposition of $\mathcal{R}$ into an integral of irreducible representations.

Denote by $J_{r}$ the $r\times r$-matrix with units on the secondary diagonal (other matrix elements are 0-s). For a given $r=0$, 1, …, $q$, consider an Hermitian form determined by a matrix

[TABLE]

For different $r$ these forms are equivalent. Therefore the group of all matrices $g$ satisfying

[TABLE]

is isomorphic to $\mathrm{U}(p,q)$. In this model, we consider subgroup $P_{r}\subset\mathrm{U}(p,q)$ of all block upper-triangular matrices $h\in\mathrm{U}(p,q)$ of size

[TABLE]

having the form

[TABLE]

Here $Z\in\mathrm{U}(p-r,q-r)$. We consider a representation $\mu$ of $P_{r}$ given by

[TABLE]

where $m_{j}\in{\mathbb{Z}}$, $\rho_{j}\in{\mathbb{R}}$, and $\tau_{A;c}$ is an irreducible representation of $\mathrm{U}(p-r,q-r)$ of discrete series222By the definition, a representation of a reductive group is contained in a discrete series if it is contained in $L^{2}$ on the group.. Below in Subsect. 2 we will explain the meaning of the parameters $(A;c)$. Until this, we can understand $\tau_{A;c}$ as a symbol denoting an arbitrary representation of a discrete series of $\mathrm{U}(p-r,q-r)$. We consider representations $T_{A;c,m,\rho}$ of $\mathrm{U}(p,q)$ unitary induced (see, e.g, [2], §16.1) from representations $\mu_{A;c,m,\rho}$. For $\rho$ being in a general position they are irreducible (see the Harish-Chandra completeness theorem, [20], Theorem 14.31). Thus we get $q+1$ family $\mathfrak{A}_{r}$ of representations numerated by $r=0$, 1, …, $q$. The regular left-right representation of $\mathrm{U}(p,q)\times\mathrm{U}(p,q)$ admits a decomposition in a multiplicity free direct integral of the form

[TABLE]

where $d\mathcal{P}_{r}(A;c,m,\rho)$ is the Plancherel measure.

1.4. Purpose of the paper. Our main purpose is to write projectors $\Pi_{r}$ corresponding to the orthogonal decomposition $\oplus_{r=0}^{q}$ in (1.3). The formula is given in Theorem 1 at the end of this section.

We also consider two finer decompositions. First, we fix a representation $\tau_{A;c}$ of a discrete series of $\mathrm{U}(p-r,q-r)$ and consider in (1.3) the integral of all representations having fixed $(A;c)$. Secondly, we fix $(A;c,m)$. Formulas for the corresponding orthogonal projectors are given in Theorems 2-3 in Section 2. Notice that for $r=0$ these formulas must coincide with characters of discrete series, general formulas have a similar degree of complexity (not too simple).

The problems are reduced to an integration of characters as functions of parameters with respect to the Plancherel measure333The Plancherel measure is supperted by a space with discrete and continuous coordinates, below we prefer to say ’summation’ of characters.. Characters of representations of real semisimple groups and the Plancherel formula were obtained by Harish-Chandra [15], [16]. His formulas contain some undetermined constants, for more explicit formulas, see [17].

We obtain formulas for the projectors $\Pi_{r}$ as a simple byproduct of Takeshi Hirai’s [19] derivation of the Plancherel formula for $\mathrm{U}(p,q)$.

1.5. Cartan subgroups. We realize $\mathrm{U}(p,q)$ as (1.1). Cartan subgroups $H_{k}$, where $k=0$, …, $q$, are defined in the following way. First, define a subgroup $H_{k}^{+}$ consisting of matrices

[TABLE]

Next, we define a subgroup $H_{k}^{-}$ consisting of diagonal matrices with entries

[TABLE]

Here $t_{\gamma}\in{\mathbb{R}}$, $\varphi_{\alpha}$, $\psi_{\beta}$, $\theta_{\gamma}\in{\mathbb{R}}/2\pi{\mathbb{Z}}$. We set

[TABLE]

Denote

[TABLE]

The eigenvalues of $h\in H_{k}$ are

[TABLE]

It is convenient to use two notations for systems of coordinates on $H_{k}$, the first is $\varphi_{\alpha},\psi_{\beta},\theta_{\gamma},t_{\gamma},$ the second is $\varphi_{\alpha},\psi_{\beta},z_{\gamma}$.

Define the canonical Lebesgue measure on $H_{k}$ by

[TABLE]

The Weyl group $W_{k}$ corresponding to a Cartan subgroup $H_{k}$ is

[TABLE]

the symmetric group $S_{p-k}$ acts on $H_{k}$ by permutations of coordinates $\varphi_{\alpha}$, the group $S_{q-k}$ by permutations of $\psi_{\beta}$, the $S_{k}$ by permutations of pairs $(t_{\gamma},\theta_{\gamma})$, and ${\mathbb{Z}}_{2}^{k}$ is generated by $k$ reflections

[TABLE]

(over coordinates $\varphi_{\alpha}$, $\psi_{\beta}$, $\theta_{\gamma}$ on $H_{k}$ remain fixed).

We say that a function $f$ on $H_{k}$ is $\varepsilon_{k}$-symmetric if it is invariant with respect to the subgroups $S_{p-k}$, $S_{q-k}$, $S_{k}$ and changes sign under each reflections $R_{\gamma}$. We say that a function $f$ is $\varepsilon_{k}$-skew-symmetric if it is skew-symmetric with respect to $S_{p-k}\times S_{q-k}$ and invariant with respect to $S_{k}\ltimes{\mathbb{Z}}_{2}$.

1.6. The Vandermonde expression. We denote by $\Updelta(y)$ the Vandermonde expression

[TABLE]

Denote the eigenvalues (1.4) of a matrix $h\in H_{k}$ by $e^{x_{1}}$, …, $e^{x_{n}}$, and set

[TABLE]

Next, consider the following differential operators $X_{1}$, …, $X_{n}$ on $H_{k}$:

[TABLE]

We set444Such operators were introduced Gelfand and Naimark [9] for complex classical groups and later were used by Harish-Chandra in a more general context.

[TABLE]

1.7. Average operator. Denote by $C_{c}^{\infty}(G)$ the space of compactly supported smooth functions on $\mathrm{U}(p,q)$. Recall that a space $G/H_{k}$ admits a unique up to a scalar factor $G$-invariant measure (since both $G$, $H$ are unimodular, see, e.g., [2],§4.3). For any $f\in C_{c}^{\infty}(G)$ we assign a function (Harish-Chandra transform of $f$) on $H_{k}$ by

[TABLE]

Notice that for $y\in G$ the expression $yhy^{-1}$ depends only on a coset of $G$ with respect to $H_{k}$, therefore actually we have an integration over $G/H_{k}$. By the definition, a function $I_{k}f$ is invariant with respect to the group $W_{k}$.

Under a certain normalization of the Haar measure $dg$ on $G$ and invariant measures on $H_{k}$ we have a Weyl integration formula

[TABLE]

where

[TABLE]

(this follows from the usual arguments establishing the Weyl integration formula).

Denote by $H_{k}^{\prime}\subset H_{k}$ the subset $\Updelta_{k}(h)\neq 0$. By $H_{k}^{\circ}\subset H_{k}$ we denote the complement to the union of all hypersurfaces $\varphi_{j}=\psi_{l}$. For any $f\in C_{c}^{\infty}(G)$ we define a function on $H_{k}^{\prime}$ by

[TABLE]

A function $\Phi=\Xi_{k}f$ satisfies the following properties (see [30], Corollary 8.5.1.2).

A) $\Phi$ is compactly supported;

B) $\Phi$ is $\varepsilon_{k}$-skew-symmetric;

C) $\Phi$ is $C^{\infty}$-smooth on each component $U$ of $H_{k}^{\circ}$ and all partial derivatives are bounded.

Moreover, an a operator $\Xi_{k}$ is bounded in a natural sense, i.e. a convergence of a sequence $f_{j}\in C_{c}^{\infty}(G)$ implies uniform convergence of all partial derivatives of $\Xi_{k}f$ on $H_{k}^{\circ}$.

A collection of functions $(\Xi_{0}f,\dots,\Xi_{q}f)$ satisfies some gluing conditions on hypersurfaces $\varphi_{\alpha}=\psi_{\beta}$ and $t_{\gamma}=0$, see Lemma 2.2 of [19]. These conditions are an important element of the story about characters and the Plancherel formula, but below we do not use them explicitly (these conditions are hidden in an integration by parts in formula (3.2)).

Next, consider a function $F:=\Updelta_{k}(\partial)\Xi_{k}f$. It satisfies the following conditions (Harish-Chandra [14], Lemma 40, in [19] it is formulated in a beginning of Sect.3, see also [30], Corollary 8.5.1.5):

A∘) $F$ is compactly supported;

B∘) $F$ is $\varepsilon_{k}$-symmetric;

C∘) $F$ admits a $C^{\infty}$-smooth extension to the closure of each component of $H_{k}^{\circ}$ ;

D∘) $F$ admits a continuous extension to the whole $H_{k}$.

1.8. Formula for projectors. Consider a distribution $\chi$ on $G$ invariant with respect to conjugations. It determines a convolution operator $Q_{\chi}:C^{\infty}_{c}(G)\to C^{\infty}(G)$ by555By $\langle\!\langle f,\chi\rangle\!\rangle_{(L)}$ we denote a pairing of a test function and a distribution on a manifold $L$.

[TABLE]

where brackets denote a pairing of a test function and a distribution.

Theorem 1

For any $r=0$, …, $q$ the projector $\Pi_{r}$ is a convolution operator determined by the following distribution $\Theta_{r}$:

[TABLE]

where $\delta(\cdot)$ denotes the delta-function and $M_{*}$ is a constant.

Remark. A formula

[TABLE]

determines a distribution on $C_{c}^{\infty}({\mathbb{R}})$. However test functions in (1.7) are odd with respect to the variables $T_{\gamma}$, for odd smooth $F(t)$ the integrand in (1.8) is smooth at 0.

Remark. In particular, the projector to the most continuous series (i.e., $r=q$) is determined by the distribution

[TABLE]

1.9. Further structure of the paper. Section 2 contains preliminaries from Hirai [19]. Theorem 1 is proved in Section 3. In Section 4 we write formula for projectors determining finer orthogonal decompositions of L^{2}\bigl{(}\mathrm{U}(p,q)\bigr{)}.

2 The Plancherel formula. Preliminaries

Here we present the formula for characters and the Plancherel formula from [19].

2.1. Formula for characters. Recall that a character $\pi$ of a unitary representation $T$ of a unimodular Lie group $G$ is a distribution on $G$ defined by

[TABLE]

According Harish-Chandra, a character of an irreducible representation of a reductive Lie group is a locally integrable function. Here we present a formula for characters of representations of $\mathrm{U}(p,q)$ from [19], Section 1.

Fix $r=0$, $1$, …, $q$. Consider three collections of parameters

[TABLE]

We denote

[TABLE]

and use an alternative notation (a signature) for the same collection of parameters

[TABLE]

Next, we split the set $\{1,2,\dots,n-2r\}$ into two disjoint subsets $A:=\{a_{1},\dots,a_{p-r}\}$ and $B:=\{b_{1},\dots,b_{q-r}\}$. Data $(A;c,d)$ determine a character $\varkappa_{A;c,d}$ of the group $\mathrm{U}(p,q)$, it is defined in this subsection.

Let $j$, $l\in{\mathbb{Z}}$, $z=t+i\theta$ . We set

[TABLE]

For given $r$ and $k\geqslant r$ we consider all possible diagrams $\sigma$ of the form given on Fig.1.

The elements of the upper row, -s, -s, -s correspond to elements of a signature (2.4). More precisely, -s correspond to $c_{a_{j}}$, where $a_{j}\in A$, -s correspond to $c_{b_{l}}$, where $b_{l}\in B$, and -s correspond to $d_{1}$, $\overline{d}_{1}$, …, $d_{r}$, $\overline{d}_{r}$. For a clarity, we connect $d_{\gamma}$ and $\overline{d}_{\gamma}$ by an arc.

The elements of the lower row, -s, -s, and -s correspond to the coordinates

[TABLE]

Namely, -s correspond to $\varphi_{\alpha}$, -s to $\psi_{\beta}$, and -s to $z_{\gamma}$, $-\overline{z}_{\gamma}$. We connect each pair $z_{\gamma}$ and $-\overline{z}_{\gamma}$ by an arc.

We connect elements of the upper row with elements of the lower row by arcs (each element is an end of a unique arc). Each diagram $\sigma$ establishes a one-to-one correspondence between elements of rows (2.4) and of rows (2.6).

We allow only diagrams that are unions of pieces of 4 types a)-d) presented on Fig. 2:

a) Arcs — or $c_{a_{j}}$—$\varphi_{\alpha}$, where $a_{j}\in A$.

b) Arcs — or $c_{b_{l}}$—$\psi_{\beta}$, where $b_{l}\in B$.

c) Chains ——— or $c_{a_{j}}$—$z_{\gamma}$—$(-\overline{z}_{\gamma})$—$c_{b_{l}}$. Un particular, this means that a left is connected with and a right is connected with .

d) Cycles –———– or –$d_{s}$—$z_{\gamma}$—$(-\overline{z}_{\gamma})$—$\overline{d}_{s}$–. Notice that a left is connected with left .

We use the following notation:

— in the case a) we write $[c_{a_{j}},\varphi_{\alpha}]\in\sigma$;

— in the case b): $[c_{b_{l}},\psi_{\beta}]\in\sigma$;

— in the case c): $[c_{a_{j}},z_{\gamma},c_{b_{l}}]\in\sigma$;

— in the case d): $[[d_{s},z_{\gamma}]]\in\sigma$.

Denote by $\Omega(A|r,k)$ the set of all admissible diagrams $\sigma$. Recall that $\sigma\in\Omega(A|r,k)$ determines a substitution of a set $\{1,\dots,n\}$, in particular it has a well-defined sign $(-1)^{\sigma}$.

For a fixed signature (2.4) and a fixed $A$ we define functions $\varkappa^{k}=\varkappa^{k}_{A;c,d}$ on $H_{k}$ in the following way:

[TABLE]

For $k\geqslant r$ we set

[TABLE]

The functions $\varkappa^{k}_{A;c,d}$ are $\varepsilon_{k}$-skew-symmetric.

There exists a unitary representation $T_{A;c,d}$ of $G=\mathrm{U}(p,q)$ such that for any $f\in C_{c}^{\infty}(G)$, we have

[TABLE]

Moreover, $T_{A;c,d}$ are the representations defined in Subsect. 1.

If $r=0$ (i.e., the parameters $d$ are absent), then $T_{A;c}$ is a representation of $\mathrm{U}(p,q)$ of discrete series.

Let $r>0$. Then $\varkappa_{A;c,\varnothing}$ determines a representation of a discrete series $\tau_{A;c}$ of $\mathrm{U}(p-r,q-r)$. The representation $T_{A;c,d}$ is induced from the representation (1.2) of the parabolic $P_{r}$.

2.2. The Plancherel formula. See Hirai [19], Theorem 3. For $m\in{\mathbb{Z}}$, we set

[TABLE]

We also define the Vandermonde expression in the parameters $(c,d)$,

[TABLE]

The Plancherel formula for $\mathrm{U}(p,q)$ is given by

[TABLE]

where $M_{*}$ is a constant.

Denote by $F_{1}*F_{2}$ the convolution of functions on $G$. For $F\in C_{c}^{\infty}(G)$ denote by $F^{\star}$ the function $F^{\star}(g)=\overline{F(g^{-1})}$. For any unitary representation $T$ of $G$ we have

[TABLE]

Therefore for $f=F*F^{\circ}$ the integrand in the right hand side of (2.8) is positive. By polarization arguments this implies absolute convergence of the integral and the series in (2.8) for functions of the form $f=F*F^{\prime}$, hence the absolute convergence holds on the subspace consisting of functions

[TABLE]

This subspace is dense in $C^{\infty}_{c}(G)$ and invariant with respect to left and right shifts. For arbitrary $f\in C^{\infty}_{c}(G)$ the identity (2.8) holds if to understand the right-hand side in the sense of a successive integration as below.

3 Evaluation of the projectors $\Pi_{r}$

Here we prove Theorem 1.

3.1. Preliminary remarks. Denote by $\widehat{G}$ the set of all possible parameters $\lambda=(A;c,d)$, see (2.1)–(2.3), so $\widehat{G}$ consists of pieces enumerated by $r=0$, 1, …, $q$, and each piece is a product of a discrete set and a simplicial cone $\rho_{1}>\dots>\rho_{r}>0$. We equip the discrete set with the counting measure and the simplicial cone with the Lebesgue measure, so we get a sigma-finite measure on $\widehat{G}$. Denote $d\mathcal{P}(A;c,d)=d\mathcal{P}(\lambda)$ the measure on $\widehat{G}$ with the positive density $\Updelta_{r}(c,d)\prod_{s}\mathfrak{e}_{m_{s}}(\rho_{s})$. Denote by $T_{\lambda}$ the irreducible representation with parameter $\lambda$ and by $H_{\lambda}$ the space of the representation.

Next, consider the space $\mathcal{L}^{2}(G)$ of functions $\Phi$ on $\widehat{G}$, which for each $\lambda$ assign a Hilbert–Schmidt operator $\Phi(\lambda):H_{\lambda}\to H_{\lambda}$ and satisfy the condition

[TABLE]

This space is a Hilbert space with respect to the inner product

[TABLE]

For any $f\in L^{2}(G)\cap L^{1}(G)$ the formula

[TABLE]

determines an element of $\mathcal{L}^{2}(\widehat{G})$. Moreover

[TABLE]

and the map $I:f\mapsto T_{\lambda}(f)$ (the Fourier transform) extends to a unitary operator $L^{2}(G)\to\mathcal{L}^{2}(\widehat{G})$. The inverse Fourier transform is given by

[TABLE]

Consider a subset $U\subset\widehat{G}$ and the subspace $\mathcal{L}^{2}(U)\subset\mathcal{L}^{2}(\widehat{G})$ consisting of functions supported by $U$. Let us write a formula for a projection operator $\Pi_{U}$ in $L^{2}(G)$ corresponding to $\mathcal{L}^{2}(U)$. According the kernel theorem any bounded operator in $L^{2}(G)$ is determined by a kernel, which is a distribution on $G\times G$. Let $f$ has the form (2.9). Then

[TABLE]

where $L_{g}f(h):=f(g^{-1}h)$ and the integral absolutely converges.

3.2. Transformations of the Plancherel formula. Denote by $W^{(r)}$ the group of all transformations of the set of all signatures (2.4) generated by permutations of parameters $c$, permutations of parameters $d$ and reflections $R^{s}$ changing $d_{s}$ and $\overline{d}_{s}$. Clearly,

[TABLE]

We say that a function $F$ in variables $c$, $d$ is $\varepsilon^{r}$-symmetric if it is invariant with respect to $S_{n-2r}$ and $S_{r}$ and changes a sign under each reflection $R^{s}$. A function $F$ is $\varepsilon^{r}$-skew-symmetric if it is skew-symmetric with respect to $S_{n-2r}$ and invariant with respect to $S_{r}\ltimes{\mathbb{Z}}_{2}^{r}$.

We write the right-hand side of (2.8) as

[TABLE]

Our purpose is to find summands $Z_{r}$. Following [19], we define a sum

[TABLE]

The expressions

[TABLE]

have form (2.7) but the summation

[TABLE]

now is taken over the set $\widetilde{\Omega}(A;r,k)$ of diagrams shown on Fig. 3, namely we forget a difference between -s and -s and allow to connect any $\varphi_{\alpha}$, $\psi_{\beta}$ with an arbitrary $c_{\tau}$. The number of elements of $\widetilde{\Omega}(A|r,k)$ is

[TABLE]

We defined $\varkappa^{k}_{A;c,d}$ and $\widetilde{\varkappa}^{k}_{c,d}$ under conditions (2.1) and (2.3). However, the expressions make sense for arbitrary $c_{1}$, …, $c_{n-2r}\in{\mathbb{Z}}$ and $\rho_{1}$, …$\rho_{r}\in{\mathbb{R}}$. Functions $\widetilde{\varkappa}^{k}_{c,d}$ are $\varepsilon^{r}$-skew-symmetric with respect to the parameters $c$, $d$, $\overline{d}$.

For any signature (2.4) we define functions $\eta^{k}_{c,d}$ on $H_{k}$ by

[TABLE]

This expression is $\varepsilon_{k}$-symmetric as a function in variables $\varphi_{\alpha}$, $\psi_{\beta}$ and $\varepsilon^{r}$- symmetric as a function in parameters $c_{a}$, $d_{s}$, $\overline{d}_{s}$. It is easy to verify that

[TABLE]

Remark. We can also define functions $\eta^{k}_{c,d|A}$ replacing a summation with respect to $\widetilde{\Omega}(r,k)$ by a summation with respect to $\Omega(A|r,k)$. These functions are $\varepsilon_{k}$-symmetric in coordinates, but the $\varepsilon^{r}$-symmetry with respect to the parameters does not hold. $\boxtimes$

Therefore, we can present $Z_{r}$ as

[TABLE]

Next, Theorem 2 of [19] allows to integrate by parts:

[TABLE]

Remark. This is a delicate point since functions $\Xi_{k}f(h)$ and $\eta^{k}_{c,d}(h)$ have singularities on hypersurfaces $\varphi_{\alpha}=\psi_{\beta}$, an integration by parts in each summand produces extra terms, however in the sum $\sum_{k=r}^{q}$ all such terms cancel. $\square$

3.3. Transformations of distributions $Z_{r}$. Next, both $\Updelta_{k}(\partial)\Xi_{k}f(h)$ and $\eta^{k}_{c,d}(h)$ are $\varepsilon^{r}$-symmetric as functions of parameters $c$, $d$, $\overline{d}$, Therefore, we can transform $Z_{r}$ to the form

[TABLE]

(we also use the property $\mathfrak{e}_{m}(\rho)=-\mathfrak{e}_{m}(-\rho)$).

At least for $f$ of the form (2.9) this expression converges as an integral over $\widehat{G}$ (the integrand is the expression in the big brackets). But we have also a finite summation and an integration over $H_{k}$ and we have no reasons to believe to the absolute convergence of the whole expression. For further manipulations we pass to a successive integration and after this change the order of the successive integration and a finite summation. In this way, we transform $Z_{r}$ to the form

[TABLE]

where

[TABLE]

We apply the definition (3.1) of $\eta^{k}_{c,d}$ and move a finite summation $\sum_{\widetilde{\Omega}(r,k)}$ in the front of our integral. Thus we get

[TABLE]

where

[TABLE]

3.4. Summation of distributions. Preliminaries. Lemmas below follow Hirai [19], but we need some additional details.

Lemma 1

Let $f(t)$ be a smooth compactly supported function on ${\mathbb{R}}$. Then

[TABLE]

We will apply this lemma for odd functions $f(t)$. In this case, the integrals in the right hand sides and the repeated integrals in the left hand sides are absolutely convergent.

Proof. We must evaluate Fourier transforms of tempered distributions $\tanh(\pi\rho/2)$, $\coth(\pi\rho/2)$. See Tables of Fourier transforms of distributions in [4], Table 1, lines (396), (397), in the second case we also must apply [5], formula (1.7.11). Also, it is possible to apply formulas [6], (2.9.7), (2.9.8) and continuity of the Fourier transform in the space of tempered distributions. $\square$

Lemma 2

Let $f(t,\theta)$ be a smooth function with compact support, satisfying $f(-t,\theta)=f(t,\theta)$. Then

[TABLE]

Proof. The first equality follows from $f(-t,\theta)=f(t,\theta)$. The convergence of the series is obvious, and we fulfill a formal calculation with distributions. We have

[TABLE]

and therefore

[TABLE]

Keeping in the mind Lemma 1, we evaluate

[TABLE]

Lemma 3

Let $f(t,\theta)$ satisfy the conditions of the previous lemma and be supported by the set $|t|\leqslant R$. Then

a)* For any $N>0$ there exists a constant $C(f,N)$ such that for any $a$, $b\in{\mathbb{Z}}$,*

[TABLE]

where $C(f,N)$ admits a uniform estimate in terms of $R$ and numbers

[TABLE]

b)* The following identity holds:*

[TABLE]

Proof. a) We integrate by parts $N$ times with respect to $\theta$ and one time with respect to $t$.

b) By a) the double series in the left-hand side of (3.10) absolutely converges. We change summation indices to $m=a$, $n=a-b$. Thus we must evaluate a sum of distributions

[TABLE]

to simplify notation we assume $t>0$, recall that we consider functional on the space functions that odd in $t$. By (3.8) we get

[TABLE]

The last pass is a formal manipulation with series, it is justified by the dominated convergence of the expression

[TABLE]

recall that $f(t,\theta)=0$. After a simple transformation we come to

[TABLE]

Lemma 4

Let $f(\varphi)$ be a continuous piece-wise smooth function on ${\mathbb{R}}/2\pi{\mathbb{Z}}$. Then there exist a constant $C(f)$ such that

[TABLE]

where $C(f)$ can be estimated in terms of

[TABLE]

and the number of singular points of $f$. Moreover

[TABLE]

Proof. To obtain (3.11) we integrate two times by parts (after the first integration boundary term do not appear). This implies the absolute convergence of the Fourier series and the second statement. $\square$

Lemma 5

Let $f(\varphi,t,\theta)$ be a continuous function piece-wise smooth function on ${\mathbb{R}}/2\pi{\mathbb{Z}}\times{\mathbb{R}}/2\pi{\mathbb{Z}}$, which is smooth for any fixed $\varphi$. Then the following double sum absolutely converges

[TABLE]

Proof. This follows from estimates (3.9), (3.11). $\square$

Lemma 6

Let $f(t_{1},t_{2},\theta_{1},\theta_{2})$ be a smooth function on ${\mathbb{R}}\times{\mathbb{R}}\times{\mathbb{R}}/2\pi{\mathbb{Z}}\times{\mathbb{R}}/2\pi{\mathbb{Z}}$ odd in $t_{1}$ and odd in $t_{2}$. Then the following series absolutely converges

[TABLE]

Proof. This follows from (3.9). $\square$

3.5. Summation of distributions. Formally transforming (3.4) we get

[TABLE]

Applying Lemmas 2–4 we come to

[TABLE]

and we observe that for all $\sigma$ the sums $Y^{\sigma}_{r,k}$ are equal. However, in the initial expression for $Y^{\sigma}_{r,k}$ we have a successive integration-summation, the calculation above assumes changings of the order of the summation.

Step 1. We can change a summation in $m_{r}$ with any integration in $d\varphi_{\alpha}$, $d\psi_{\beta}$, and $dt_{\gamma}$, $d\theta_{\gamma}$ except $\gamma$ linked with $r$ on the diagram $\sigma$. For this, we use two identities, the first is

[TABLE]

for a function $F$, which is piece-wise smooth on a torus. The second is

[TABLE]

for a smooth compactly supported $G$ on ${\mathbb{R}}/2\pi{\mathbb{Z}}\times{\mathbb{R}}$.

The same property holds for the integration in $d\rho_{r}$,

So we can start the integration-summation from the integral

[TABLE]

We apply Lemma 2 and continue the process. In this way, we perform successive integrations and summations in (3.4) one after another for all $\sum_{m_{s}}\int d\rho_{s}$.

Step 2. We start a successive summation with respect to $c_{n-2r}$, $c_{n-2r-1}$, …. As above, we can change one summation in $c_{j}$ with an integration in $d\varphi_{\alpha}$, $d\psi_{\beta}$, $dt_{\gamma}$, $d\theta_{\gamma}$ (if this variables are not linked with $c_{a}$ in the diagram $\sigma$.) If we meet $c_{a}$ linked with $\varphi_{\alpha}$ or $\psi_{\alpha}$, we apply Lemma 4. For the factors of the type

[TABLE]

the corresponding summations $\sum_{c_{a_{j}}}$, $\sum_{c_{b_{j}}}$ generally are not adjacent in (3.4) and we can not immediately apply Lemma 2. But Lemmas 5, 6 allow to change adjacent summations $\sum_{c_{j}}$, $\sum_{c_{j+1}}$ in two case

— if both $c_{j}$, $c_{j+1}$ are connected with black boxes;

— if precisely one of $c_{j}$, $c_{j+1}$ is connected with a black box.

This allows a consequent application of Lemmas 4 and 3.

In this way, we justify (3.14 and get

[TABLE]

This implies Theorem 1.

4 Refinements of the orthogonal decomposition

4.1. The decomposition with respect to the parameters $\mathbf{(A;c)}$. Fix $r$ and $(A;c)$. Denote by $L_{A;c}$ the subspace in the Plancherel decomposition (1.3), which is the integral of all representations with given $(A;c)$. To write the projector to $L_{A;c}$ we need some notations. Denote by $\Omega^{\circ}(A|r,k)$ the set of all diagrams of the form shown on Figure 4. These diagrams are obtained from elements of $\Omega(A|r,k)$ by forgetting black circles and adjacent arcs. These diagrams split into elements of the following 4 types

a) Arcs — or $c_{a_{j}}$—$\varphi_{\alpha}$, where $a_{j}\in A$.

b) Arcs — or $c_{b_{l}}$—$\psi_{\beta}$, where $b_{l}\in B$.

c) Chains ——— or $c_{a_{j}}$—$z_{\gamma}$—$(-\overline{z}_{\gamma})$—$c_{b_{l}}$.

d)∗ Arcs —, or $z_{\gamma}$—$(-\overline{z}_{\gamma})$.

Theorem 2

An invariant distribution $\Theta_{A;c}$ determining an orthogonal projector to $L_{A;c}$ is given by the formula

[TABLE]

where

[TABLE]

A calculation of the distributions $\zeta^{k}_{A;c}$ are the same as above. It is important that Theorem 2 from [19] allows the integration by parts for functions $\eta^{k}_{A;c,d}$ defined in Subsect. 3. We symmetrize with respect to $S_{r}\ltimes{\mathbb{Z}}_{2}^{r}$ instead of $W^{(r)}$ in (3.3). In the calculation described in Subsect. 3 we make only Step 1.

4.2. The decomposition with respect to the parameters $\mathbf{(A;c)}$ and $\mathbf{m}$. Take $r$, $A$, $(c_{1},\dots,c_{n-2r})$, and

[TABLE]

Consider a representation $T_{A;c,m,\rho}$ of $\mathrm{U}(p,q)$ unitary induced from a representation (1.2). The hyperoctahedral group $S_{r}\ltimes{\mathbb{Z}}_{2}^{n}$ acts on the set of collections (4.1) by permutations and complex cojugations $m_{s}+i\rho_{s}\mapsto m_{s}-i\rho_{s}$. Elements of this group send representations $T_{A;c,m,\rho}$ to equivalent representations. In particular, we can assume that

[TABLE]

Denote by $L_{A;c,m}$ the integral of all representations with fixed $A$, $c$, $m$ in $L^{2}(G)$. We intend to write a projector to $L_{A;c,m}$.

Sinse a collection (4.2) can contain repeating entries, we will use an alternative notation for the same collection,

[TABLE]

(here $u_{\iota}>0$, $\sum u_{\iota}=r$).

Define function $\mathfrak{f}_{m}(t)$, where $m\in{\mathbb{Z}}$, by

[TABLE]

Theorem 3

The invariant distribution $\Theta_{A;c,m}$ determining the orthogonal projector to $L_{A;c,m}$ is given by

[TABLE]

where

[TABLE]

Proof. In the Plancherel formula we have summation over the set $m_{1}$, …, $m_{r}\in{\mathbb{Z}}$, $\rho_{1}>\rho_{2}>\dots>\rho_{r}$. We can replace this domain by any fundamental domain of the hyperoctahedral group $S_{r}\ltimes{\mathbb{Z}}_{2}^{r}$. Denote the parameters $\rho$ corresponding to (4.3) by

[TABLE]

We choose a fundamental domain determined by (4.3) and

[TABLE]

Now problem is reduced to an evaluation of

[TABLE]

Using symmetry, we change this to

[TABLE]

We pass to the sum $\sum_{\sigma\in\Omega(A|r,k)}$ and integrate it termwise in $\rho_{r}$, …$\rho_{1}$ using (3.6), (3.5). $\square$