The smoothness of convolutions of orbital measures on complex Grassmannian symmetric spaces

TL;DR

This paper proves that convolutions of orbital measures on complex Grassmannian symmetric spaces become smooth in the sense of belonging to L^2 space after a small number of convolutions, extending known results to these spaces.

Contribution

It establishes the L^2 property for convolutions of orbital measures on all complex Grassmannian symmetric spaces, a case not previously covered.

Findings

Convolution powers of orbital measures are in L^2 for all complex Grassmannian symmetric spaces.

For a dense set of points, the second or third convolution of orbital measures is in L^2.

The results extend the understanding of measure smoothness in symmetric spaces.

Abstract

It is well known that if $G/K$ is any irreducible symmetric space and $\mu _{a}$ is a continuous orbital measure supported on the double coset $KaK,$ then the convolution product, $\mu _{a}^{k},$ is absolutely continuous for some suitably large $k\leq \dim G/K$. The minimal value of $k$ is known in some symmetric spaces and in the special case of groups or rank one symmetric spaces it has even been shown that $\mu _{a}^{k}$ belongs to the smaller space $L^{2}$ for some $k$. Here we prove that this $L^{2}$ property holds for all the compact, complex Grassmanian symmetric spaces, $% SU(p+q)/S(U(p)\times U(q))$. Moreover, for the orbital measures at a dense set of points $a$, we prove that $\mu _{a}^{2}\in L^{2}$ (or $\mu _{a}^{3}\in L^{2}$ if $p=q$).

Equations

$$k>\max (p,2(p-q)+3)\text{ if }p>q\text{ and }k>\max (2p,6)\text{ if }p=q.$$

$$\int fd{\mu }_a={\int }_K{\int }_{K}f(k_{1}a{k}_2)d{k}_{1}d{k}_2$$

$$X=X(t_1,...,t_q)=\left[\begin{array}{ccc}0& 0& M\\ 0& 0& 0\\ M^{\prime }& 0& 0\end{array}\right]$$

$$\{2e_k,e_k,e_i\pm e_j:1\le i<j\le q,1\le k\le q\}$$

$${\Lambda }_{sph}=\left\{\sum _{j=1}^{q}2m_j{e}_j:m_j\in \mathbb{N}\text{, }{m}_1\ge m_2\cdot \cdot \cdot \ge m_q\ge 0\right\}.$$

$$r$$

$$n_j$$

$$\widetilde{P_n}(x)=\frac{P_n^{(p-q,0)}(x)}{P_n^{(p-q,0)}(1)}$$

$${\varphi }_{\lambda }(\exp iX(t_1,...,t_q))=\frac{C_{p,q}\det {\left(\widetilde{P_{n_j}}(\cos 2t_k)\right)}_{j,k=1}^q}{\prod _{1\le j<k\le q}(\cos 2t_j-\cos 2t_k)(n_j(n_j+r)-n_k(n_k+r))}$$

$$C_{p,q}=2^{q(q-1)/2}\prod _{j=1}^{q-1}j!(j+p-q{)}^{q-j}$$

$$n_j(n_j+r)-n_k(n_k+r)\ge n_j+r\text{ when }j<k.$$

$$\left|\phi_{\lambda}(\exp iX)\right|\leq C\left\{\begin{array}[]{cc}\prod_{j=1}^{q-1}(n_{j}+1)^{-1}&\text{if }p>q\\ \prod_{j=1}^{q-1}(n_{j}+1)^{-1/2}&\text{if }p=q\end{array}\right.$$

$$\left|{\varphi }_{\lambda }(\exp iX)\right|\le C\prod _{j=1}^q(n_j+1{)}^{-p+j-1/2}.$$

$$\left|{\varphi }_{\lambda }(\exp iX)\right|\le C\prod _{j=1}^q(n_j+1{)}^{-p+q/2}.$$

$$\left|\widetilde{P_n}(\cos (2t_k))\right|=\left|\frac{P_n^{(p-q,0)}(\cos (2t_k))}{P_n^{(p-q,0)}(1)}\right|\le \frac{C}{{\left(n_j+1\right)}^{p-q+1/2}}.$$

$$\left|{\varphi }_{\lambda }(X)\right|$$

$$\left|{\varphi }_{\lambda }(X(t_1,...,t_q))\right|\le C\prod _{j=1}^{q-1}(n_j+1{)}^{-1}.$$

$$\left|{\varphi }_{\lambda }(X(t_1,t_2))\right|=\frac{C\left|\det M\right|}{\left|\cos 2t_1-\cos 2t_2\right|(n(n_1+r)-n_2(n_2+r))|}$$

$$\left|{\varphi }_{\lambda }(X(t_1,t_2))\right|\le \frac{C}{n_1+r}\le C(n_1+1{)}^{-1}$$

$$\left|\det {\left(\widetilde{P_{n_j}}(\cos 2t_k)\right)}_{j,k=1}^q\right|$$

$$\left|{\varphi }_{\lambda }(X(t_1,...,t_q))\right|\le C{I}_1\cdot I_2$$

$$I_1=\frac{\left|\tilde{P_{n_{j_0}}}(\cos 2t_{k_0})\right|}{\prod _{k\ne k_0}\left|\cos 2t_k-\cos 2t_{k_0}\right|\prod _{j\ne j_0}\left|n_j(n_j+r)-n_{j_0}(n_{j_0}+r)\right|}$$

$$I_2=\frac{\left|\det {\left(\widetilde{P_{n_j}}(\cos 2t_k)\right)}_{j\ne j_0,k\ne k_0}\right|}{\prod _{\begin{array}{c}1\le j<k\le q\\ j,k\ne k_0\end{array}}|\cos 2t_j-\cos 2t_k|\prod _{\begin{array}{c}1\le j<k\le q\\ j,k\ne j_0\end{array}}(n_j(n_j+r)-n_k(n_k+r))}$$

$$I_1$$

$$m_{i}^{\prime}=\left\{\begin{array}[]{cc}n_{i}-(q-1)+i&\text{for }i=1,...,j_{0}-1\\ n_{i+1}-(q-1)+i&\text{for }i=j_{0},...,q-1\end{array}\right..$$

$$\widetilde{P_n}(x)=\frac{P_n^{((p-1)-(q-1),0)}(x)}{P_n^{((p-1)-(q-1),0)}(1)},$$

$$\left|{\varphi }_{\lambda }(X(t_1,...,t_q))\right|\le C{I}_1\cdot I_2\le C\prod _{j=1}^{q-1}(n_j+r{)}^{-1}$$

$$\left|{\varphi }_{\lambda }(X)\right|\le C\prod _{j=1}^q(n_j+r{)}^{-1}.$$

$$\det {\left(\widetilde{P_{n_j}}(\cos 2t_k)\right)}_{j,k=1}^q=\sum _{\sigma }{c}_{\sigma }{A}_{\sigma }{B}_{\sigma }$$

$$\left|{\varphi }_{\lambda }(X)\right|\le C\sum _{\sigma }{I}_{\sigma }{J}_{\sigma }{K}_{\sigma },$$

$$I_{\sigma }=\frac{\left|A_{\sigma }\right|}{\prod _{1\le j<k\le L_1}|\cos 2t_j-\cos 2t_k|\prod _{1\le i<l\le L_1}(n_{j_i}(n_{j_i}+r)-n_{j_l}(n_{j_l}+r))}$$

Full text

This research is supported in part by NSERC 2016-03719 and by Sultan Qaboos University. The authors thank Acadia University for their hospitality when this research was done.

This paper is in final form and no version of it will be submitted for publication elsewhere.

1.

Introduction

Suppose $G$ is a Lie group with Cartan involution $\theta$ and $K$ is the compact subgroup of $G$ fixed by $\theta$. In a now classical paper, [19], Ragozin proved that if the symmetric space, $G/K,$ is irreducible, then any convolution product of $\dim G/K,$ continuous, $K$-bi-invariant measures on $G$ is absolutely continuous with respect to the Haar measure on $G$. In the case that $G$ is a non-compact group, Ragozin’s result was significantly improved by Graczyk and Sawyer in [5] who showed that $rank(G/K)+1$ convolutions would suffice. The building blocks for all $K$-bi-invariant measures on $G/K$ are the so-called orbital measures, $\mu_{a},$ supported on the double cosets, $KaK$. Building on the work of Graczyk and Sawyer in [5]-[8], the authors in [11] improved this result by characterizing the convolution products of orbital measures that are absolutely continuous, for any classical, non-compact symmetric space.

For the compact symmetric spaces $G_{c}/K$ for $G_{c}$ compact, the authors in [12] proved that any convolution product of $2rank(G_{c}/K)+1,$ continuous, $K$-bi-invariant measures is absolutely continuous. Any compact Lie group, $G_{c},$ can be regarded as a compact symmetric space, namely $G_{c}\times G_{c}/K$ with $K=\{(g,g):g\in G_{c}\}$. The $K$–bi-invariant, orbital measures are the $G_{c}$-invariant measures supported on the conjugacy classes of the group. In this special case, it is known that $\mu_{a}^{k}$ belongs to the (smaller) space $L^{2}(G_{c})$ if and only if $\mu_{a}^{k}$ is absolutely continuous (equivalently, belongs to $L^{1}(G_{c})$). Moreover, the minimal exponent, $k(a),$ such that $\mu_{a}^{k(a)}\in L^{2}$ has even been determined for each $a$; see [10, 14, 20]. This $L^{1}-L^{2}$ dichotomy was shown to be false in the compact symmetric space $SU(2)/SO(2)$, although it is still true that $\mu_{a}^{3}\in L^{2}$ for all continuous orbital measures in any rank one symmetric space; see [2], [13]. These $L^{2}$ results were all found by studying the decay properties of characters or spherical functions.

In [1], Al-Hashami and Anchouche studied the analogous problem for the compact, complex Grassmannian symmetric spaces $G_{c}/K,$ where $G_{c}=SU(p+q)$ and $K=S(U(p)\times U(q))$. They proved that for the dense set of ‘regular’ points $a\in G_{c}$, $\mu_{a}^{k}\in L^{2}$ for a choice of $k$ which is much smaller than $\dim G_{c}/K$; see (4.2). They did this by using the Berezin-Karpelevich formula for the spherical functions and obtaining rates of decay for these functions at the regular elements. Using these estimates and the Sobolev embedding theorem, they also proved that $\mu_{a}^{k}\in C^{s}(G_{c})$ for regular elements $a$ and suitably large choices of $k,$ depending on $s$.

In this note, we find estimates on the rates of decay for the spherical functions at all points $a\notin$ $N_{G_{c}}(K)$, these being the points which give rise to the continuous orbital measures $\mu_{a}$. Obtaining such estimates at non-regular points is quite delicate for then the singularities of the spherical functions must be understood. With these estimates we are able to prove that $\mu_{a}^{k}\in L^{2}$ for all $a\notin N_{G_{c}}(K)$ whenever

[TABLE]

For the special case of the regular elements $a,$ we show that $\mu_{a}^{2}\in L^{2}$ if $p>q$ and $\mu_{a}^{3}\in L^{2}$ otherwise. This significantly improves the work in [1] as their exponent $k$ was unbounded in $p$. We also show that $\mu_{a}^{k}\in C^{s}$ for all $a\notin N_{G_{c}}(K)$ for suitable $k=k(s,a)$.

Although smaller than $\dim G_{c}/K,$ our minimal exponent $k$ is typically much larger than $2rank(G_{c}/K)+1$. It would be interesting to know what the sharp exponent is for the $L^{2}$ problem and whether the $L^{1}-L^{2}$ dichotomy holds.

2. Notation and Preliminaries

2.1. Orbital measures and their smoothness

Suppose $G_{c}/K$ is an irreducible symmetric space, where $G_{c}$ is a compact Lie group with Cartan involution $\theta$ and $K=\{g\in G_{c}:\theta(g)=g\}$. Denote its non-compact dual space by $G/K$ and suppose the Lie algebra of $G$ is $\mathfrak{g}$. We fix the Cartan decomposition $\mathfrak{g=t\oplus p}$ and the maximal abelian subspace $\mathfrak{a}$ of $\mathfrak{p}$ as in [15]. We identify the Lie algebra $\mathfrak{g}_{c}$ of $G_{c}$ with the subspace $\mathfrak{g}_{c}\mathfrak{=t\oplus ip}$ of the complexified Lie algebra $\mathfrak{g}^{\mathbb{C}}$ and let $A=\exp i\mathfrak{a}\subseteq G$. Always $G_{c}=KAK$. For any $a\in G_{c},$ the set $KaK$ is called a double coset and has Haar measure zero.

Definition 1**.**

Given any $a=\exp(iX)\in A,$ we define the orbital measure $\mu_{a}$ on $G_{c}$, with support $KaK,$ by

[TABLE]

for all continuous functions $f$.

This probability measure is $K$-bi-invariant. It is continuous (meaning as a measure on $G_{c}/K$) precisely when $a\notin N_{G_{c}}(K)$ and is purely singular with respect to Haar measure.

In [19], Ragozin essentially proved the following geometric characterization.

Proposition 1**.**

Let $a_{i}\in A$, $i=1,...,k$. Then $\mu_{a_{1}}\ast\cdot\cdot\cdot\ast\mu_{a_{k}}$ is absolutely continuous if and only if the product of double cosets $Ka_{1}K\cdot\cdot\cdot Ka_{k}K$ has non-empty interior in $G_{c}$. Moreover, if $a_{i}\notin N_{G_{c}}(K)$ and $k\geq\dim(G_{c}/K),$ then $\mu_{a_{1}}\ast\cdot\cdot\cdot\ast\mu_{a_{k}}$ is absolutely continuous.

Ragozin used geometric methods to show that $Ka_{1}K\cdot\cdot\cdot Ka_{k}K$ has non-empty interior for $k=$ $\dim G_{c}/K$ if $a_{i}\notin N_{G_{c}}(K)$ for all $i$. Notice that if $a\in N_{G_{c}}(K),$ then $\left(KaK\right)^{k}=Ka^{k}K$ and hence has Haar measure zero. Thus $\mu_{a}^{k}$ is purely singular to Haar measure for all $k$. Ragozin’s geometric characterization was verified using algebraic methods by the authors in [11] when they improved upon his result, showing that $k=2rank(G_{c}/K)+1$ suffices. This exponent is close to sharp as there are continuous orbital measures (in some compact symmetric spaces) with $\mu_{a}^{2rank(G_{c}/K)-1}$ singular; [10].

A measure $\mu$ is absolutely continuous with respect to Haar measure if and only if its density function (or Radon Nikodym derivative) belongs to $L^{1}$. If the density function belongs to the properly smaller space $L^{2}$ (or to $C^{s}),$ we will write $\mu\in L^{2}$ (resp., $C^{s}).$ Ragozin’s geometric approach is not helpful in studying the problem of determining if $\mu_{a}^{k}\in L^{2}$. Instead, a harmonic analysis approach has been taken: estimates are made on the rate of decay of the Fourier transform of the measure and then the Peter Weyl theorem is invoked. This approach has been applied very successfully when the symmetric space is a compact group or a rank one compact symmetric space, and is the approach we will take in this paper to study the $L^{2}$ problem for the complex Grassmannian symmetric spaces.

2.2. The symmetric space $SU(p+q)/S(U(p)\times U(q))$

For the remainder of this paper we will focus on the case of the complex Grassmannian symmetric space, $G_{c}/K$, where $G_{c}=SU(p+q)$, $K=S(U(p)\times U(q))$ and $p\geq q\geq 2$. This compact symmetric space has rank $q$ and dimension $2pq$. The non-compact dual space is the symmetric space $G/K,$ where $G$ has Lie algebra $\mathfrak{g}=su(p,q)$. With Cartan decomposition $\mathfrak{g=t\oplus p}$, we can take as the maximal abelian subspace $\mathfrak{a}$ the $(p+q)\times(p+q)$ matrices of the form

[TABLE]

where $M,M^{\prime}$ are $q\times q$ anti-diagonal matrices, with $t_{1},...,t_{q}$ and $t_{q},...,t_{1}$ respectively, on the anti-diagonal. As above, we let $A=\exp$ $i\mathfrak{a}\subseteq G_{c}$. We can identify $X(t_{1},...,t_{q})\in\mathfrak{a}$ by $(t_{1},...,t_{q})\in\mathbb{R}^{q}$.

The restricted roots and the highest spherical weights will be very important in our work. The positive restricted roots can be taken to be

[TABLE]

(where $\{e_{j}\}$ are the usual basis vectors for $\mathbb{R}^{q})$ with multiplicities $m_{2e_{k}}=1$, $m_{e_{k}}=2(p-q)$ (so these are not present if $p=q$) and $m_{e_{i}\pm e_{j}}=2$. The restricted roots act on $\mathfrak{a}$ in the natural way. We also view the roots as acting on $A$ by the (well defined) rule $\alpha(\exp iX)\equiv\alpha(X)\mathop{\mathrm{m}od}\pi$. The reader is referred to [15] for further background information.

It is well known that the normalizer, $N_{G_{c}}(K)\cap A,$ is characterized by the property that $\exp iX\in N_{G_{c}}(K)\cap A$ if and only if $\alpha(X)\equiv 0\mathop{\mathrm{m}od}\pi$ for all restricted roots $\alpha$.

An element $X\in\mathfrak{a}$ or $\exp iX\in A$ is said to be** regular** if $\alpha(X)\neq 0\mathop{\mathrm{m}od}\pi$ for each restricted root $\alpha,$ thus $X=X(t_{1},..,t_{q})$ is regular precisely when $t_{j}\neq\pm t_{k}\mathop{\mathrm{m}od}\pi$ for all $j\neq k$ and $t_{k}\neq 0,\pi/2$ $\mathop{\mathrm{m}od}\pi$ for all $k$. Such elements are dense in $\mathfrak{a}$ or $A$ respectively. When $X$ is regular, $\mu_{\exp iX}^{2}$ is absolutely continuous (whatever the compact symmetric space); [9].

The highest spherical weights are given by

[TABLE]

We denote the spherical function corresponding to $\lambda\in\Lambda_{sph}$ by $\phi_{\lambda}$. Put

[TABLE]

so that $n_{1}>n_{2}>\cdot\cdot\cdot>n_{q}\geq 0$. We denote the normalized Jacobi polynomials by

[TABLE]

According to the Berezin-Karpelevich formula (see [3] or [4])

[TABLE]

where

[TABLE]

and the quotient should be understood in the limiting sense if some $\cos 2t_{j}=\cos 2t_{k}$. Of course, this situation occurs precisely if some $t_{j}\equiv\pm t_{k}$ mod $\pi$.

An elementary, but useful, observation is that

[TABLE]

3. Decay of Spherical Functions

The objective of this section is to obtain estimates on the decay of the spherical functions. We will find estimates that hold for all $a\notin N_{G_{c}}(K)$ and better estimates for the regular elements.

Theorem 1**.**

Suppose $\lambda=\sum_{j=1}^{q}2m_{j}e_{j}$ is a highest spherical weight and $n_{j}=m_{j}+q-j$.

(i) If $\exp iX\in A\diagdown N_{G_{c}}(K),$ then

[TABLE]

where $C$ is a constant that depends on $p,q$ and $X$, but not $\lambda$.

(ii) If $\exp iX$ is regular, then there is a constant $C=C(p,q,X)$ such that

[TABLE]

Remark 1**.**

Note that when $X=X(t_{1},...,t_{q})$ is regular, $\cos 2t_{j}\neq\cos 2t_{k}$ for any $j\neq k$. This is very significant as it means we do not have the complication of having to understand $\phi_{\lambda}(\exp iX)$ through the limiting process. In [1], the decay in $\phi_{\lambda}(\exp iX)$ was studied for this special case.111The authors do not make this assumption explicit in the statement of their theorem, but the properties of a regular element are used in their proof. They obtained the bound,

[TABLE]

In our proof, the constants $C$ which appear may vary from one occurrence to another, but will always be independent of $\lambda$. We will frequently write $\phi_{\lambda}(X)$ as shorthand for $\phi_{\lambda}(\exp iX)$. When we say $f\sim g$ for functions $f,g$ defined on $\mathbb{N}$, we mean there are constants $A,B>0$ such that $Ag(n)\leq f(n)\leq Bg(n)$ for all $n.$

We begin by collecting useful facts about Jacobi polynomials.

Lemma 1**.**

The following are well known facts about Jacobi polynomials:

(i) $P_{n}^{(a,b)}(1)=\binom{n+a}{n}\sim(n+1)^{a};$ $P_{n}^{(a,b)}(-1)=(-1)^{n}\binom{n+b}{n};$

(ii) $\left|P_{n}^{(a,b)}(x)\right|\leq\frac{C_{a,b,x}}{\sqrt{n+1}}$ when $x\neq\pm 1;$

*(iii) $\left|P_{n}^{(a,0)}(x)\right|\leq C_{x,a}\left|P_{n}^{(a,0)}(1)\right|$ so $\left|\widetilde{P_{n}}(x)\right|\leq C_{x,p,q};$ *

(iv) $P_{0}^{(a,b)}(x)=0$ for all $x;$

(v) $\frac{d}{dx}P_{n}^{(a,b)}(x)=\frac{1}{2}(n+a+b+1)P_{n-1}^{(a+1,b+1)}(x)$ if $n\neq 0.$

We will first prove part (ii) of the Theorem, the special case when $X$ is regular.

Proof.

[of Theorem(ii)] Assume $X=X(t_{1},...,t_{q})$ is regular. Then $e_{i}\pm e_{j}(X)=t_{j}\pm t_{k}\neq 0\mathop{\mathrm{m}od}\pi$ for any $j\neq k$ and that means $\cos 2t_{j}\neq\cos 2t_{k}$ for any $j\neq k$. Furthermore, $2e_{k}(X)=2t_{k}\neq 0\mathop{\mathrm{m}od}\pi$ and that implies $\cos(2t_{k})\neq\pm 1$ for any $k$. This latter fact ensures that

[TABLE]

Consequently, formula (2.1) implies

[TABLE]

as claimed.

For general $X$ we must consider the possibility that the formula (2.1) for $\phi_{\lambda}(X)$ has to be understood through the limiting process. The next several lemmas will help with this. The proof of the first lemma also introduces a reduction technique that will be frequently used throughout the remainder of the proof of the theorem.

Lemma 2**.**

Fix $X=X(t_{1},...,t_{q})$ and suppose there is an index $k_{0}$ such that $\cos 2t_{k_{0}}\neq$ $\cos 2t_{k}$ for any $k\neq k_{0}$. Then

[TABLE]

Proof.

First, suppose $q=2$. In this case,

[TABLE]

where from Lemma 1(iii) we see that $M$ is a $2\times 2$ matrix with entries bounded independent of the choice of $\lambda$. As $\cos 2t_{1}\neq\cos 2t_{2},$ it follows that

[TABLE]

as we desired to show.

Now assume $q>2$. Expand the determinant in (2.1) along column $k_{0}$ to obtain

[TABLE]

Assume the maximum occurs at index $i=j_{0}$. Then

[TABLE]

where

[TABLE]

and

[TABLE]

(with $I_{2}$ understood in the limiting sense if some $\cos 2t_{j}=\cos 2t_{k}).$

As $\cos 2t_{k}\neq\cos 2t_{k_{0}}$ when $k\neq k_{0}$, applying property (2.2) gives

[TABLE]

since $n_{1}>n_{2}>\cdot\cdot\cdot>n_{q}.$

In order to bound $I_{2},$ we will use a reduction argument. Consider the symmetric space $G^{\prime}/K^{\prime}=SU((p-1)+(q-1))/S(U(p-1)\times U(q-1))$ where $p-1\geq q-1\geq 2,$ and the spherical representation $\lambda^{\prime}=\sum_{i=1}^{q-1}2m_{i}^{\prime}e_{i}$ where $e_{i}$ are the standard basis vectors for $\mathbb{R}^{q-1}$ and

[TABLE]

This choice is made so that $n_{i}^{\prime}=m_{i}^{\prime}+(q-1)-i$ satisfies the conditions $n_{i}^{\prime}=n_{i}$ for $i=1,...,j_{0}-1$ and $n_{i}^{\prime}=n_{i+1}$ for $i=j_{0},...,q-1$. Let $t^{\prime}=(t_{1},...,\widehat{t_{k_{0}}},...,t_{q})$ belong to $\mathfrak{a}^{\prime}$ for $G^{\prime}/K^{\prime}$ (here the notation $\widehat{\cdot}$ means the element is not present) and $\exp iX^{\prime}\in A^{\prime}\subseteq G^{\prime}$ for $X^{\prime}=X^{\prime}(t^{\prime})$. Notice that for any $x,$

[TABLE]

so $I_{2}=C\left|\phi_{\lambda^{\prime}}(X^{\prime})\right|\leq C$ since all spherical functions are bounded by $1$. Thus

[TABLE]

Lemma 3**.**

Suppose (upon a suitable reordering of coefficients, if necessary) $X=X(b_{1},...,b_{s})$ where each $b_{j}=(t_{1}^{(j)},...,t_{L_{j}}^{(j)})$ with $\cos(2t_{i}^{(j)})=\cos(2t_{1}^{(j)})$ for all $i=1,...,L_{j},$ $\cos(2t_{1}^{(i)})\neq\cos(2t_{1}^{(j)})$ if $i\neq j$, $L_{j}\geq 2$ for all $j$ and $s\geq 2$. Then

[TABLE]

Proof.

We will give the details for $s=2,$ but it will be clear how the method generalizes. Thus assume $X=X(t_{1},...,t_{L_{1}},t_{L_{1}+1},...,t_{q})$ where $\cos(2t_{1})=\cos(2t_{j})$ for $j=2,...,L_{1},$ $\cos(2t_{q})=\cos(2t_{k})$ for $k=L_{1}+1,...,q-1$, $\cos(2t_{1})\neq\cos(2t_{q})$ and $L_{1},L_{2}=q-L_{1}\geq 2$.

A basic property of the determinant is that

[TABLE]

where the sum is over all choices, $\sigma,$ of $L_{1}$ indices, say $\sigma:j_{1}<\cdot\cdot\cdot<j_{L_{1}},$ $A_{\sigma}$ is the determinant of the $L_{1}\times L_{1}$ matrix $\left(\widetilde{P_{n_{j_{i}}}}(\cos 2t_{k})\right)_{i,k=1}^{L_{1}}$, $B_{\sigma}$ is the determinant of the $L_{2}\times L_{2}$ submatrix of $\left(\widetilde{P_{n_{j}}}(\cos 2t_{k})\right)$ formed by the remaining rows and columns (the remaining columns being columns $L_{1}+1,...,q$), and $c_{\sigma}$ is a suitable choice of $\pm 1$. This gives the bound

[TABLE]

where

[TABLE]

[TABLE]

and

[TABLE]

(As usual, $I_{\sigma},J_{\sigma}$ should be understood in the limiting sense.)

Since

[TABLE]

it follows that

[TABLE]

As $\cos 2t_{j}\neq\cos 2t_{k}$ whenever $j\leq L_{1}$ and $K\geq L_{1}+1$, we see that

[TABLE]

Similar to the previous lemma, $I_{\sigma}$ and $J_{\sigma}$ are, up to a constant, the modulus of spherical functions on Grassmanian symmetric spaces of rank $L_{1}$ and $L_{2}$ respectively. Indeed, $I_{\sigma}=\left|\phi_{\lambda^{\prime}}(X^{\prime})\right|$ on $SU(p+q)/S(U(p+q-L_{1})\times U(L_{1}))$ where $\lambda^{\prime}$ is identified with $(n_{j_{1}},...,n_{L_{1}})$ and $X^{\prime}=X^{\prime}(t_{1},...,t_{L_{1}}),$ and similarly for $J_{\sigma}$. Hence both are bounded and therefore $\phi_{\lambda}(X)$ has the claimed bound.

Lemma 4**.**

Suppose $X=X(t_{1},...,t_{q})$ where $\cos(2t_{j})=\cos(2t_{k})$ for all $j,k,$ but $\cos(2t_{j})\neq\pm 1$. Then

[TABLE]

Proof.

Here we must understand $\phi_{\lambda}(X)$ as

[TABLE]

where $x_{i}$ are distinct and not equal to $\cos(2t_{i})$. According to [17] (see also [16, Lemma 4.1]), this limit is equal to

[TABLE]

where $C=C(q)=(-1)^{q(q-1)/2}/(q-1)!$ and $\widetilde{P_{n_{j}}}^{(k-1)}$ means the $(k-1)^{\prime}st$ derivative of $\widetilde{P_{n_{j}}}$. Applying Lemma 1(v) repeatedly, we see that

[TABLE]

where we recall that $P_{m}^{(a,b)}(x)=0$ if $m\leq 0$. So the main task is to bound

[TABLE]

Using the permutation method for taking derivatives, one can easily see this determinant is bounded in modulus by

[TABLE]

where the maximum is taken over all permutations $\pi$ of $\{1,...,q\}$.

If all $n_{j}\leq q-1$, this expression is clearly bounded (independently of the choice $(n_{1},...,n_{q})$), so assume otherwise, say

[TABLE]

for some $s=1,...,q$. Put $s=q+1$ if all $n_{j}>q-1$.

Now, $j\leq s-1$ implies $n_{j}>q-1\geq\pi(j)-1$ and thus $n_{j}-(\pi(j)-1)\geq(n_{j}+1)/(q+1)$. Since $\cos(2t_{1})\neq\pm 1,$ Lemma 1(ii) together with this observation yields the bound

[TABLE]

for these $j$. Moreover, $n_{j}+p-q+\pi(j)-1\leq C_{p,q}(n_{j}+1)$ for these $j$.

Since always $P_{n}^{(a,b)}(\cos 2t_{1})$ is uniformly bounded over $n$ (with constant depending on $t_{1},a,b)$ it follows that the terms

[TABLE]

are uniformly bounded when $n_{j}\leq q-1$. Consequently, the expression in (3.2) for any fixed $\pi$ is bounded by

[TABLE]

where $C$ does not depend on the choice of $n_{j}$.

Since the terms $n_{j}$ are strictly increasing, this is maximized with the permutation $\pi$ that maps $j$ to $q-j+1$. Hence

[TABLE]

Recalling the formula (3.1), we have

[TABLE]

When $s=q+1$ (all $n_{j}>q-1$), this simplifies to $C\prod_{j=1}^{q}(n_{j}+1)^{-p+q-1/2},$ which is even smaller than the valued stated in the lemma. Otherwise, $s-1\leq q-1$ and for $j\leq q-1$, $q-j-1/2>0,$ thus

[TABLE]

We immediately deduce that $\phi_{\lambda}(X)$ is bounded as stated in the lemma.

Lemma 5**.**

Suppose $X=(t_{1},...,t_{q})$ where $\cos(2t_{j})=-1$ for all $j$. Then

[TABLE]

Remark 2**.**

We note that when $p=q,$ $\exp iX\in N_{G_{c}}(K)$ for such $X$.

Proof.

Similar arguments to the previous proof show that

[TABLE]

In this situation, bounding the determinant by a multiple of the largest term in the sum that arises from the permutation method for calculating the determinant will not give us a good enough estimate. We will actually directly compute the determinant, instead. As before, we have

[TABLE]

From Lemma 1(i),

[TABLE]

thus we need to evaluate the determinant of the matrix whose $j,k$ entry is given by

[TABLE]

In other words, we want to find

[TABLE]

where $M_{j,1}=1$ and for $k>1,$

[TABLE]

Put $M_{j,1}(x)=1$ and $M_{j,k}(x)=\prod_{\ell=0}^{k-2}(x_{j}-\ell(r+\ell))$ and consider the multi-variable polynomial

[TABLE]

Our desire is to compute $F(x)$ at $x=(x_{j})_{j=1}^{q}$ with $x_{j}=n_{j}(n_{j}+r)$. Notice that if some coordinates $x_{i}=x_{j}$ for $i\neq j,$ then the matrix $(M_{j,k}(x))_{j,k}$ has two identical rows and therefore $F(x)=0$. Thus the polynomial $P(x)=\prod_{1\leq i<j\leq q}(x_{i}-x_{j})$ divides $F(x)$. Since both $F$ and $P$ are degree $q(q-1)$ polynomials, it must be that there is a constant $c$ such that for all $x$,

[TABLE]

In particular,

[TABLE]

Hence,

[TABLE]

We are now ready to complete the proof of the theorem for arbitrary $X$.

Proof.

[of Theorem (i)] Let $\exp iX\in A\diagdown N_{G_{c}}(K)$ with $X=X(t_{1},...,t_{q})$.

First, suppose there is a pair $j,k$ such that $t_{j}\neq\pm t_{k}\mathop{\mathrm{m}od}\pi$. In this case, $\cos(2t_{j})\neq\cos(2t_{k})$. Lemmas 2 or 3 (depending on the situation) show that for such $X,$ $\phi_{\lambda}(X)$ is bounded in modulus by $C\prod_{j=1}^{q-1}(n_{j}+1)^{-1}$.

So we can assume that for every $j,k,$ either $t_{j}\equiv t_{k}\mathop{\mathrm{m}od}\pi$ or $t_{j}\equiv-t_{k}\mathop{\mathrm{m}od}\pi$ (or both). If there is some  $j$ such that $t_{j}\equiv 0\mathop{\mathrm{m}od}\pi,$ then this will be true for all $t_{k}$. Then $\alpha(X)\equiv 0\mathop{\mathrm{m}od}\pi$ for all restricted roots $\alpha$ and that contradicts the assumption that $\exp iX\notin N_{G_{c}}(K)$.

If some $t_{j}\equiv\pi/2\mathop{\mathrm{m}od}\pi,$ then the same is true for all $t_{k}$ and then $\cos(2t_{j})=-1$ for all $j$. In the case that $p=q$, as the positive restricted roots are only of the form $\alpha=e_{j}\pm e_{k}$ and $2e_{j}$, we again have $\alpha(X)\equiv 0\mathop{\mathrm{m}od}\pi$ for all $\alpha$, giving a contradiction. If $p>q,$ Lemma 5 gives the bound

[TABLE]

Otherwise we must have $t_{j}\neq 0,\pi/2\mathop{\mathrm{m}od}\pi$ for any $j$, but $\cos(2t_{j})=\cos(2t_{k})$ for all $j,k,$ and then Lemma 4 says that $\left|\phi_{\lambda}(X)\right|\leq C\prod_{j=1}^{q-1}(n_{j}+1)^{-1/2}$ when $p=q$ and

[TABLE]

for $p>q$. That completes the proof.

4. $L^{2}$ and other smoothness results

With these decay estimates we can determine when convolution powers of orbital measures have square integrable density functions.

Theorem 2**.**

Let $a\in A\diagdown$ $N_{G_{c}}(K)$.

(i) Then $\mu_{a}^{k}\in L^{2}$ provided $k>\max(p,2(p-q)+3)$ if $p>q$ or $k>\max(2p,6)$ if $p=q$.

(ii) If $a$ is regular, then $\mu_{a}^{2}\in L^{2}$ when $p>q$ and $\mu_{a}^{3}\in L^{2}$ when $p=q$.

Proof.

We will use the Weyl character formula, which in this setting tells that

[TABLE]

c.f., [2]. As we already have estimates on the rate of decay of the spherical functions, the key additional idea needed to prove this result is a bound on the growth in the degree of $\lambda$ as a function of $(n_{1},...,n_{q})$. This uses the Weyl degree formula which states

[TABLE]

where $\rho$ is half the sum of the positive roots. Here

[TABLE]

so

[TABLE]

Thus (a) $<e_{j}+e_{i},\lambda+\rho>$ $\geq n_{j}+n_{i}+r\geq n_{j}+1$ if $j<i$ (and there are $q-j$ such roots, each with multiplicity $2)$;

(b) $<e_{j}-e_{i},\lambda+\rho>$ $\geq n_{j}-n_{i}\geq 1$ if $j<i$;

(c) $<(2)e_{j},\lambda+\rho>$ $\geq n_{j}+1$ (with multiplicity $1$ for the roots $2e_{j}$ and multiplicity $2(p-q)$ for the roots $e_{j}$).

Consequently,

[TABLE]

Thus if $p>q,$ Theorem 1(i) and the degree formula (4.1) tells us that

[TABLE]

Since $\sum_{n_{q}<n_{q-1}}(n_{q}+1)^{t}\leq C(n_{q-1}+1)^{t+1}$ when $t\geq 0$, it follows that

[TABLE]

This sum is clearly finite if $2p-2j+1-2k<-1$ for all $j=1,...,q-2$ (equivalently, $p<k$) and $4p-4q-2k+5<-1$ (i.e., $k>2(p-q)+3$), as we desired to show.

Similar reasoning using the other formula from Theorem 1(i) for the case $p=q$ or the formula from part (ii) in the regular case, gives the other statements.

Remark 3**.**

We remark that the conclusion $\mu_{a}^{2}\in L^{2}$ for $a$ regular is sharp since $\mu_{a}$ is singular. Previously, it was shown in [1] that when $a$ is regular $\mu_{a}^{k}\in L^{2}$ if

[TABLE]

In [1], conditions are given under which the density function of $\mu_{a}^{k}$ belongs to $C^{d}(G_{c})$. This is done by noting that if $H^{s}(G_{c})$ is the Sobolev space of functions in $L^{2}$ with derivatives up to order $s$ in $L^{2},$ then the Sobolev embedding theorem [18, Theorem 1.2.1], implies that if $s>d+\dim G_{c}/2,$ then $H^{s}(G_{c})\subseteq C^{d}(G_{c})$. Moreover, the $H^{s}$ norm can be computed as

[TABLE]

where $\kappa_{\lambda}$ is the Casmir constant, $\kappa_{\lambda}=<\lambda+2\rho,\lambda>\sim\left\|\lambda\right\|^{2}\sim n_{1}^{2}$. They deduce that $\mu_{a}^{k}\in H^{s}$ for $k>\frac{1+\binom{p+q}{2}+2s}{2p-q}$ when $a$ is regular.

With our better estimates on the decay of the spherical functions, we can obtain the following stronger results.

Proposition 2**.**

(i) For any $a\notin N_{G_{c}}(K),$ $\mu_{a}^{k}\in H^{s}$ if $k>\max(s+p,2(p-q)+3)$ if $p>q$ and $k>\max(2s+2p,6)$ if $p=q$.

(ii) If $a$ is regular, then $\mu_{a}^{k}\in H^{s}$ if $k>(p+s)/(p-1/2)$.

The proof involves the same types of calculations as in the proof of the theorem above.

Remark 4**.**

It would be interesting to know if these results are sharp and also to determine for each $a,$ the minimal exponent $k$ such that $\mu_{a}^{k}\in L^{2}$ (or $C^{d}).$