On spherical unitary representations of groups of spheromorphisms of Bruhat--Tits trees

TL;DR

This paper investigates the structure of spherical unitary representations of the group of spheromorphisms of Bruhat--Tits trees, revealing new representations and their relation to classical groups like Thompson and PSL(2,Z).

Contribution

It introduces a combinatorial framework for understanding double cosets and constructs new spherical representations of the spheromorphism group.

Findings

Aut(T_n) is spherical in Hie(T_n)

Constructed a new family of spherical representations

Thompson group has PSL(2,Z)-spherical unitary representations

Abstract

Consider an infinite homogeneous tree $T_n$ of valence $n+1$, its group $Aut(T_n)$ of automorphisms, and the group $Hie(T_n)$ of its spheromorphisms (hierarchomorphisms), i.~e., the group of homeomorphisms of the boundary of $T_n$ that locally coincide with transformations defined by automorphisms. We show that the subgroup $Aut(T_n)$ is spherical in $Hie(T_n)$, i.~e., any irreducible unitary representation of $Hie(T_n)$ contains at most one $Aut(T_n)$-fixed vector. We present a combinatorial description of the space of double cosets of $Hie(T_n)$ with respect to $Aut(T_n)$ and construct a 'new' family of spherical representations of $Hie(T_n)$. We also show that the Thompson group has $PSL(2,\mathbb{Z})$-spherical unitary

Equations

$$\Phi (g)=⟨\rho (g)v,v{⟩}_H,\text{where ∥v∥2=1,}$$

$$\Phi (k_{1}g{k}_2)=\Phi (g)\text{for g∈G, h1, h2∈K.}$$

$$\mathrm{Aut}({\mathcal{T}}_n)\setminus \mathrm{Hier}({\mathcal{T}}_n)/\mathrm{Aut}({\mathcal{T}}_n).$$

$$B_1\supset B_2,\text{or}{B}_2\supset B_1,\text{or}{B}_1\cap B_2=\varnothing$$

$$P\rho (h)=\rho (h)P=P\text{for all h∈Aut+(Tn).}$$

$$\hat{\rho }(g):=P\rho (g)P.$$

$$\hat{\rho }(g_1)\hat{\rho }(g_2)=\hat{\rho }(g_2)\hat{\rho }(g_1).$$

$$\widehat{\rho}(g)+\widehat{\rho}(g^{-1}),\qquad i\bigl{(}\widehat{\rho}(g)-\widehat{\rho}(g^{-1})\bigr{)},$$

$$P\rho (g)v=P\rho (g)Pv=\hat{\rho }(g)v\in V,$$

$${\mathrm{Aut}}_{+}(T_n)\cdot G(B)\cdot {\mathrm{Aut}}_{+}(T_n)=\mathrm{Hier}(T_p),$$

$$U^k{g}_2{U}^{-k}\in G(U^k{B}_2)\subset G(B_2).$$

$$\rho (g_1)\rho (U^k)\rho (g_2)\rho (U^{-k})=\rho (U^k)\rho (g_2)\rho (U^{-k})\rho (g_1).$$

$$P\rho (g_1)\rho (U^k)\rho (g_2)P=P\rho (g_2)\rho (U^{-k})\rho (g_1)P.$$

$$P\rho (g_1)P\rho (g_2)P=P\rho (g_2)P\rho (g_1)P.$$

$$\left\langle\bigl{(}\rho_{1}(g)\otimes\rho_{2}(g)\bigr{)}v_{1}\otimes v_{2},\,v_{1}\otimes v_{2}\right\rangle_{W}=\langle\rho_{1}(g)v_{1},v_{1}\rangle_{H_{1}}\cdot\langle\rho_{2}(g)v_{2},v_{2}\rangle_{H_{2}}\\ =\Phi_{1}(g)\,\Phi_{2}(g).\qquad\square\!\!\!\!\!\!$$

$$a)\text{epsfbox}thorn.4b)\text{epsfbox}thorn.5$$

$$\mathrm{a})\mathrm{b})\text{epsfbox}thorn.9$$

$$\Omega :=B_1\bigsqcup \cdots \bigsqcup B_{\iota }.$$

$${\mathrm{Clop}}^{\circ }({\mathcal{T}}_n)=\coprod _{\iota =0}^{n-2}{\mathrm{Clop}}_{\iota }^{\circ }({\mathcal{T}}_n).$$

$$\mathrm{a})\text{epsfbox}thorn.7$$

$$\mathrm{b})\text{epsfbox}thorn.8$$

$$\mathrm{Abs}({\mathcal{T}}_n)=C_1\bigsqcup C_2\bigsqcup \cdots \bigsqcup C_N$$

$$\mathrm{Abs}({\mathcal{T}}_n)=g{C}_1\bigsqcup g{C}_2\bigsqcup \cdots \bigsqcup g{C}_N$$

$$\mathbf{\Pi }:={\Pi }_1\bigsqcup \cdots \bigsqcup {\Pi }_k.$$

$${\Phi }_S(\sigma )=\prod _{p,q=1}^k{s}_{pq}^{{\theta }_{p,q}(\sigma )},\sigma \in G$$

$$\bigotimes_{p=1}^{k}\biggl{(}\bigotimes_{\alpha\in\Pi_{p}}(V,e_{p})\biggr{)},$$

$$\mathcal{E}:={\otimes }_{p=1}^k{e}_p^{\otimes \infty }.$$

$${\mathrm{Clop}}_{\iota }^{\circ }({\mathcal{T}}_n)=\mathcal{P}\bigsqcup {\mathcal{O}}_1\bigsqcup \cdots \bigsqcup {\mathcal{O}}_N.$$

$$\mathcal{H}:=\bigotimes_{\Omega\in\mathop{\mathrm{Clop}}\nolimits_{\iota}^{\circ}}\bigl{(}V,e_{T(\Omega)}\bigr{)}.$$

$$\Phi (g)=\prod _{\Omega \in {\mathrm{Clop}}_{\iota }^{\circ }}⟨e_{T(g\Omega )},e_{T(\Omega )}{⟩}_V.$$

Full text

On spherical unitary representations of groups of spheromorphisms of Bruhat–Tits trees

Yury A. Neretin111The research was supported by the grants FWF, Projects P25142, P28421, P31591.

Consider an infinite homogeneous tree $\mathcal{T}_{n}$ of valence $n+1$, its group $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ of automorphisms, and the group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ of its spheromorphisms (hierarchomorphisms), i. e., the group of homeomorphisms of the boundary of $\mathcal{T}_{n}$ that locally coincide with transformations defined by automorphisms. We show that the subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is spherical in $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$, i. e., any irreducible unitary representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ contains at most one $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-fixed vector. We present a combinatorial description of the space of double cosets of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ with respect to $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ and construct a ’new’ family of spherical representations of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. We also show that the Thompson group $\mathrm{T\!h}$ has $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$-spherical unitary representations.

1. Introduction

1.1. Groups of spheromorphisms of trees. Fix an integer $n\geqslant 2$. The Bruhat–Tits tree $\mathcal{T}_{n}$ is the infinite tree such that each vertex belongs to $n+1$ edges, see Fig. 1. Denote by $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ the group of all automorphisms of $\mathcal{T}_{n}$. It is a totally disconnected locally compact group, its topology is defined from the condition: stabilizers of finite subtrees are open in $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$.

Recall that Bruhat and Tits in 1966-1967 (see [3]) invented simplicial complexes (Bruhat–Tits buildings), which are $p$-adic counterparts of noncompact Riemannian symmetric spaces. Analogs of rank one noncompact symmetric spaces (as the Lobachevsky plane) are Bruhat–Tits trees with $n$ being powers of prime $p$. In particular, $p$-adic $\mathop{\mathrm{PSL}}\nolimits(2)$ acts on the tree $\mathcal{T}_{p}$. This fact became an initial point for investigations of group acting on trees, see, e.g., Tits [58], Serre [55]. Cartier [5] observed that the groups $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ are interesting objects from the point of view of representation theory and non-commutative harmonic analysis, and these groups are relatives of $\mathop{\mathrm{SL}}\nolimits(2)$ over real and $p$-adic fields. G. Olshanski established that $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ are type $I$ groups [47] and obtained a pleasant classification [48] of irreducible unitary representations of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ (see an exposition in [10], see also the work [8] on tensor products).

The boundary $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ of $\mathcal{T}_{n}$ is a totally disconnected compact set, for a prime $n=p$ it can be identified with a $p$-adic projective line. The group $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ acts by homeomorphisms of the boundary. A spheromorphism (or hierarchomorphism) of $\mathcal{T}_{n}$ is a homeomorphism $q$ of $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ such that for each point $y\in\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ there is its neighborhood $\mathcal{N}(y)$, in which $q$ coincides with some $r_{y}\in\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{p})$. In other words, we cut a finite number of mid-edges of the tree and get a collection of finite pieces $W_{i}$ and infinite pieces $U_{j}$. We forget finite pieces and choose embeddings $\theta_{j}:U_{j}\to\mathcal{T}_{n}$ such that images $\theta_{j}$ are mutually disjoint and cover the whole tree (may be) without a finite piece, see Fig. 1.

The group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ of all spheromorphisms of the tree $\mathcal{T}_{n}$ is a locally compact topological group (see, [13]). The topology is defined by the condition: the subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is open and closed (clopen) in $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. The (countable) space of cosets $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ has a discrete topology222So we have a group $G=\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ and a subgroup $K=\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ such that $K$ is a continuous totally disconnected group and the homogeneous space $G/K$ is discrete. Topologies of this kind arise in representation theory of infinite symmetric groups, see [42], Subsect. 3.7; a group with such a topology is used below in Sect. 4..

There is a well-known discrete group $\mathrm{T\!h}$ consisting of spheromorphisms333We can imagine the Bruhat–Tits tree as drawn on the plane ${\mathbb{R}}^{2}$. Then we get a structure of a cyclically ordered set on the boundary $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$. The Thompson group $\mathrm{T\!h}$ is the group of all spheromorphisms preserving the cyclic order on $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{2})$. defined by 1965 R. Thompson in 1965. Initially it was proposed as a counterexample, and it really has strange properties but also it is an interesting positive object (see, e.g., [16], [6], [20], [51], [25], [4], [30], [11]).

The groups $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ were introduced in 1984, [37]–[38] with the following reasoning:

1. For prime $n=p$ the group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ contains the group of locally analytic diffeomorphisms of the $p$-adic projective line.

2. Unitary representations of the group of diffeomorphisms of the circle partially survive for the groups $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$.

3. The groups $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ have several families of unitary representations that are spherical (see below) with respect to (noncompact) subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$; in Addendum we explain why this property seems to be distinguished.

The topic of the present paper are unitary representations, we list some references on a wider context. The groups $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ are simple as abstract groups (Kapoudjian [22]), uniformly simple (Gal, Gismatullin, [12]), compactly generated (Caprace, De Medts [7]) compactly presentable (Le Boudec [29]), they have nontrivial ${\mathbb{Z}}_{2}$-central extensions444It is interesting to find unitary faithful unitary representations of this extension. constructed by Kapoudjian [23]. They have no property (T) (Navas, [36]555Notice that families of spherical representations of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ in the boson and fermion Fock spaces constructed in [38] approximate the trivial one-dimensional representation). These groups are simple locally compact groups that do not admit a lattice (Bader, Caprace, Gelander, and Mozes, [2], this is the first example of such kind). See Kapoudjian [24], Sauer, Thumann, [53] on action of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ on CW-complexes. These groups can be included to families of relatives [39], [31], [53]. It seems to the author that these groups being locally compact have various properties of infinite-dimensional (or ’large’) groups666For instance, constructions of spherical representations both in [37], [38] and below in Section 4 are distinctive construction for infinite-dimensional groups. On the other hand, a parallel with infinite-dimensional groups also is incomplete, apparently the groups $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ have no trains in the sense of [40]..

1.2. Sphericity. Let $G$ be a topological group, $K$ is its subgroup. Let $\rho$ be an irreducible unitary representation of the group $G$ in a Hilbert space $H$. We say that a representation $\rho$ is $K$-spherical if $H$ contains a unique upto a scalar factor nonzero $K$-fixed vector $v$ (the spherical vector). Its matrix element

[TABLE]

is called a spherical function. This function is automatically $K$-biinvariant, i.e.,

[TABLE]

In other words, a spherical function is defined on the double coset space $K\setminus G/K$.

Definition 1.1**.**

Let $G$ be a topological group, $K$ a closed subgroup. The subgroup $K$ is spherical if

A)* For any irreducible unitary representation of $G$ the subspace of $K$-fixed vectors has dimension $\leqslant 1$.*

B)* There is a faithful unitary representation of $G$ and a vector $v$ such that the stabilizer of $v$ is $K$.*

Remark. The second condition is necessary for the following reason. Quite often (if $K$ is not compact or ’heavy’ in the sense of [40]) a restriction of any nontrivial irreducible unitary representation of $G$ to $K$ has not $K$-fixed vectors at all. More generally, if a vector $v$ is fixed by $K$, then quite often $v$ is automatically is fixed by a certain larger group $\widetilde{K}\supset K$. Such phenomena were widely used in classical ergodic theory after Gelfand, Fomin [15] and Mautner [34]. A detailed investigation of such phenomena for Lie groups were done by Moore [35] and Wang [59], for $p$-adic groups by Wang [59]–[60]. Kaniuth, Lau [21] and Losert [32] discussed stabilizers of vectors in unitary representations of general locally compact groups777In their terminology subgroups that can be stabilizers of vectors ’satisfy separation property’.. $\square$

1.3. The purposes of the paper. We prove the following statements.

Theorem 1.2**.**

The subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is spherical in $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$.

Proposition 1.3**.**

Let $\Phi_{1}(g)$, $\Phi_{2}(g)$ be $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-spherical functions on $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Then $\Phi_{1}(g)\,\Phi_{2}(g)$ is a spherical function.

For known spherical pairs $G\supset K$ (finite-dimensional and infinite-dimensional) double coset spaces $K\setminus G/K$ admit explicit descriptions. In Section 3, we present such a description for the double coset space

[TABLE]

Double cosets correspond to $(n+1)$-valent graphs $\Gamma$ consisting of two disjoint trees $T_{+}$ and $T_{-}$ and a collection of edges connecting vertices of $T_{+}$ with vertices of $T_{-}$ (cf. ’tree pairs diagrams’ in [4]).

In Section 4 we apply Nessonov’s construction [46] of representations of infinite symmetric group to obtain a ’new’ family of spherical representations of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$.

Addendum contains some comments on problem of sphericity for locally compact groups. We also show that the Thompson group $\mathrm{T\!h}$ has $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$-spherical representations.

1.4. Some questions. Theorem 1.2 implies the following questions.

1. Is it possible to classify $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-spherical functions on $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$?

2. Is $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ a type $I$ group?

3. Is it possible a harmonic analysis on the space $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ in some sense888This is not a question about the decomposition of $\ell^{2}$ on this space, see Addendum, Proposition A.3..

4. Let $\rho$ be a spherical representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$, let $P$ be the operator of orthogonal projection to $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-fixed line. Consider the closure $\Gamma_{\rho}$ of $\rho(g)$, where $g$ ranges in $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$, in the weak operator topology. Obviously (see Lemma 2.4) the semigroup $\Gamma_{\rho}$ contains $P$, therefore $\Gamma_{\rho}$ contains operators of the form $\rho(g_{1})P\rho(g_{2})$ with $g_{1}$, $g_{2}\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Does it contain something else?

Remark. The analog of the group of spheromorphisms for $n=\infty$ and its unitary representations are topics of a separate paper [44]. $\square$

2. Sphericity

2.1. Notation. A way in the Bruhat–Tits tree is a sequence of vertices $a_{j}$ such that $a_{i}$ and $a_{i+1}$ are adjacent and $a_{i+2}\neq a_{i}$ for all $i$. We say that ways $a_{i}$ and $b_{j}$ are equivalent if $a_{i}=b_{i+k}$ starting some $i$. The boundary (the notation: $\mathop{\mathrm{Abs}}\nolimits(T_{n})$) of $\mathcal{T}_{n}$ is the space of classes of equivalent ways.

Let us cut the tree $\mathcal{T}_{n}$ at the middle of an edge. We call two pieces of the tree obtained in this way by branches. Each branch $U$ determines a subset $B=\mathop{\mathrm{Ba}}\nolimits[U]$ in the boundary corresponding to ways lying in $U$. We call such subsets by balls, see Fig. 2.

For a given ball $B$ denote by $\mathop{\mathrm{Br}}\nolimits[B]$ the corresponding branch of the tree. In particular, each mid-edge determines a partition of $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ into two disjoint balls. We define the topology on $\mathop{\mathrm{Abs}}\nolimits(T_{n})$ assuming that balls are clopen subsets in $\mathop{\mathrm{Abs}}\nolimits(T_{n})$, this defines on $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ a structure of totally disconnected compact set.

If $B_{1}$, $B_{2}$ are two balls, then

[TABLE]

Lemma 2.1**.**

Let $B_{1}\subset B_{2}\subset\dots$ be an increasing sequence of balls. Then it has a maximal element or $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})\setminus\cup_{j}B_{j}$ is one point.

Proof. Let a sequence of balls $B_{j}=\mathop{\mathrm{Ba}}\nolimits[U_{j}]$ strictly decrease. Let $u_{j}$ be the corresponding mid-edges, and $[p_{j}q_{j}]$ the corresponding edges, to definiteness assume $p_{j}\notin U_{j}$, $q_{j}\in U_{j}$. Then points $q_{1}$, $p_{1}$, $q_{2}$, $p_{2}$, …lye on a way. Let $a\in\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ be the limit of this way. Then $\cup B_{j}=\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})\setminus a$. $\square$

We say that $h\in\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{p})$ is hyperbolic if it has two fixed points $a$, $b$ on $\mathop{\mathrm{Abs}}\nolimits(T_{n})$ and induces a nontrivial shift on the two-side way $\dots x_{-1}$, $x_{0}$, $x_{1}$, …connecting $a$ and $b$. Let $c$ be a point of the boundary. The parabolic subgroup $P_{c}\subset\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is the group of transformations $g$ such that $g$ fixes $c$, and for any way $x_{1}$, $x_{2}$, …going to $c$ we have $gx_{N}=x_{N}$ for sufficiently large $N$.

2.2. Proof of Theorem 1.2. The group $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ has a normal subgroup $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ of index 2 defined as follows. Let us paint vertices of $\mathcal{T}_{n}$ black and white in such a way that each edge has edges of different colors. The $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ is the subgroup of the group preserving coloring. This defines a homomorphism of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ to the group ${\mathbb{Z}}_{2}$ and therefore a one-dimensional representation of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$. Other nontrivial irreducible representations of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ are infinite-dimensional. It is sufficient to prove the following statement:

Proposition 2.2**.**

Consider an irreducible unitary representation $\rho$ of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ in a Hilbert space $H$. Denote by $H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$ the subspace of all $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$-fixed vectors. It is sufficient to prove that $\dim H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$ is $\leqslant 1$.

Denote by $P$ the operator of orthogonal projection to $H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$. Clearly,

[TABLE]

For $g\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ we define an operator $\widehat{\rho}(g):H^{\mathop{\mathrm{Aut}}\nolimits_{+}}\to H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$ by

[TABLE]

Clearly, $\widehat{\rho}(g)$ depends only on a double coset $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})\cdot g\cdot\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$.

Lemma 2.3**.**

The operators $\widetilde{\rho}(g)$ commute, i.e., for any $g_{1}$, $g_{2}\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$

[TABLE]

Reduction of Theorem 1.2 to Lemma 2.3. Let the conclusion of the lemma hold. Assume that $\dim H^{\mathop{\mathrm{Aut}}\nolimits_{+}}>1$. Notice that $\widehat{\rho}(g^{-1})=\widehat{\rho}(g)^{*}$, therefore commuting bounded operators

[TABLE]

are self-adjoint. Therefore all operators $\widehat{\rho}(g)$ have a proper common invariant subspace $V\subset H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$. Then $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$-cyclic span of $V$ is a proper subspace in $H$. Indeed, let $v\in V$. Then

[TABLE]

and the projection of the cyclic span to $H^{\mathop{\mathrm{Aut}}\nolimits_{+}}$ is contained to $V$. $\square$

Lemma 2.4**.**

Let $h_{j}\in\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ tend to infinity999We say that $h_{j}$ tends to $\infty$ if any compact subset of $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ contains only a finite number of elements $q_{j}$. In other words $h_{j}$ tends to infinity in the Alexandroff compactification of a locally compact space $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$.. Then for any unitary representation $\rho$ of $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ the sequence $\rho(h_{j})$ converges to $P$ in the weak operator topology.

Equivalently for any nontrivial irreducible representation of $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ the sequence $\rho(h_{j})$ weakly converges to 0. For the group $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{p})$ this holds for any irreducible unitary representation of dimension $>1$. This is proved in [28]. On the other hand this can be easily verified case-by-case starting Olshanski’s classification theorem [48]. Notice also that this is a counterpart of the well-known Howe–Moore theorem [19] about real Lie groups.

In fact, we need the following corollary.

Corollary 2.5**.**

Let $h\in\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ be a hyperbolic element. Then for any irreducible unitary representation $\rho$ of $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ the sequence $\rho(h^{m})$ weakly converges to 0.

Proof of Lemma 2.3. Fix a ball $B\subset\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$. Denote by $G(B)$ the subgroup in $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ consisting of spheromorphisms trivial outside $B$. Clearly,

[TABLE]

i.e., any double coset has a representative in $G(B)$. Choose two disjoint balls $B_{1}$, $B_{2}$. For a verification of (2.3) we can assume $g_{1}\in G(B_{1})$, $g_{2}\in G(B_{2})$. Choose a hyperbolic element $U\in\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$ with an attractive fixed point $a\in B_{2}$. For $k>0$ we have

[TABLE]

Hence $g_{1}$ and $U^{k}g_{2}U^{-k}$ have disjoint supports, therefore they commute. Thus,

[TABLE]

Multiplying this from the left and the right by $P$ and keeping in the mind (2.2), we get

[TABLE]

Passing to the weak limit as $k\to+\infty$ and applying Lemma 2.4 we come to

[TABLE]

This is the equality (2.3). $\square$

2.3. Proof of Proposition 1.3.

Proposition 2.6**.**

Let $G\supset K$ be topological groups. Assume that $K$ does not admit nontrivial finite-dimensional unitary representations. Let $\Phi_{1}(g)$, $\Phi_{2}(g)$ be $K$-spherical functions on $G$. Then $\Phi_{1}(g)\,\Phi_{2}(g)$ is a spherical function.

Lemma 2.7**.**

Let $\nu_{1}$ $\nu_{2}$ be unitary representations of a group $\Gamma$. If the tensor product $\nu_{1}\otimes\nu_{2}$ contains a nonzero $\Gamma$-invariant vector, then both $\nu_{1}$ and $\nu_{2}$ have finite-dimensional subrepresentations.

Proof of the lemma. Assume that an invariant vector exists. Denote the spaces of representations by $V_{1}$, $V_{2}$. We identify $V_{1}\otimes V_{2}$ with the space of Hilbert–Schmidt operators $V_{1}^{\prime}\to V_{2}$, where $V_{1}^{\prime}$ is the dual space to $V_{1}$. An invariant vector corresponds to an intertwining operator $T:V_{1}^{\prime}\to V_{2}$. The operator $TT^{*}$ is an intertwining operator in $V_{2}$. Since $TT^{*}$ is compact and nonzero, it has a finite-dimensional eigenspace, and this subspace is $G$-invariant. $\square$

Proof of Proposition 2.6. Let $\rho_{1}$ and $\rho_{2}$ be $K$-spherical representations of $G$ in $H_{1}$ and $H_{2}$. Let $v_{1}$, $v_{2}$ be fixed vectors. By the lemma, $v_{1}\otimes v_{2}$ is a unique $K$-fixed vector in $H_{1}\otimes H_{2}$. The cyclic span $W$ of $v_{1}\otimes v_{2}$ is an irreducible subrepresentation. Indeed, let $W=W_{1}\oplus W_{2}$ be reducible. Then both projections of $v_{1}\otimes v_{2}$ to $W_{1}$, $W_{2}$ are $K$-fixed, therefore $v_{1}\otimes v_{2}$ must be contained in one of summands, say $W_{1}$, and thus the cyclic span of $v_{1}\otimes v_{2}$ is contained in $W_{1}$, i.e., $W=W_{1}$.

Now we consider the representation of $G$ in $W$,

[TABLE]

Proof of Proposition 1.3. Consider $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-spherical representations $\rho_{1}$, $\rho_{2}$ of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. They also are $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$-spherical. Therefore their tensor product has a unique $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$-fixed vector. This vector also is $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-fixed. $\square$

3. The space of double cosets

3.1. Terminology. Let $T$ be a tree, $A_{1}$, …, $A_{N}$ a collection of vertices. The subtree spanned by $A_{1}$, …, $A_{N}$ is the minimal subtree containing these points.

Let $S$ be a finite tree. The boundary $\partial S$ of $S$ is the set of vertices of valence 1.

We regard Bruhat-Tits trees as 1-dimensional complexes with 0-cells located at vertices of the tree and mid-edges. Respectively, 1-cells are half-edges, see Fig. 3.

Let $R$ be a tree such that valences of all vertices are $\leqslant(n+1)$ and number of vertices is $\geqslant 3$. A thorn $R$ is such a tree equipped with the following structure of an 1-dimensional simplicial complex. Consider the subtree $R^{\circ}$ (the skeleton of the thorn) of $R$ spanned by all vertices that are not contained in the boundary $\partial R$. Then 0-cells of the thorn are vertices of $R$ and mid-edges of $R^{\circ}$. Respectively, 1-cells are half-edges of $R^{\circ}$ and edges of $R\setminus R^{\circ}$. We call vertices of $R^{\circ}$ by vertices of thorn, and points of $\partial R$ by spikes of the thorn, see Fig. 4.a.

Additionally, we allow an empty thorn and a thorn having 1 vertex and one spike, see Fig. 6.

Denote by $\mathop{\mathrm{spike}}\nolimits(R)$ the set of spikes of a thorn $R$, $\mathop{\mathrm{vert}}\nolimits(R)$ the set of vertices of $R$.

Two thorns $R_{1}$, $R_{2}$ are isomorphic if there is an isomorphism $R\to R^{\prime}$ of complexes sending vertices to vertices and spikes to spikes.

Cutting a thorn in a mid-edge we get two branches.

We embed thorns $R$ to the Bruhat–Tits tree $\mathcal{T}_{n}$ isomorphically sending vertices to vertices and spikes to mid-edges. We call images of such embeddings by sub-thorns of the Bruhat–Tits tree, see 4.b.

Let $R$ be a thorn. We say a thorn is perfect if all its vertices have valence $(n+1)$, see Fig 5. We say that a vertex is perfect if it is contained in $\partial R^{\circ}$ and its valence is $(n+1)$, see 4.b. More generally, a branch of a thorn is perfect if all its vertices have valences $(n+1)$.

A thorn is reduced if it has no perfect vertices. Let $R$ be an arbitrary thorn. Cutting of all perfect branches off we come to a reduced thorn (in particular, if $R$ is perfect, then the corresponding reduced thorn is empty.)

3.2. Clopen sets. Denote by $\mathop{\mathrm{Clop}}\nolimits(\mathcal{T}_{n})$ the set of all nonempty clopen subsets of $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$, by $\mathop{\mathrm{Clop}}\nolimits^{\circ}(\mathcal{T}_{n})$ the subset consisting of proper clopen subsets (i.e., we remove the point of $\mathop{\mathrm{Clop}}\nolimits(\mathcal{T}_{n})$ corresponding the whole $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$).

Clearly, any clopen subset $\Omega$ can be represented as a union of a finite number of disjoint balls

[TABLE]

This representation is not unique, since any ball $B$ can be canonically represented as a disjoint union of $n$ smaller balls. It is easy to observe (see [55], Addendum ’Structure of $p$-adic varieties’, or [38]), that the remainder $\upsilon(\Omega)$ of $\iota$ modulo $n-1$ is uniquely defined by $\Omega$. According this, $\mathop{\mathrm{Clop}}\nolimits^{\circ}(\mathcal{T}_{n})$ splits as a disjoint union

[TABLE]

Proposition 3.1**.**

a)* Disjoint unions of balls $B_{1}\sqcup\dots\sqcup B_{\iota}$ are in one-to-one correspondence with sub-thorns of $\mathcal{T}_{n}$.*

b)* Partitions $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})=B_{1}\sqcup\dots\sqcup B_{N}$ are in one-to-one correspondence with perfect sub-thorns of $\mathcal{T}_{n}$.*

c)* Nonempty clopen sets in $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$ are in one-to-one correspondence with reduced sub-thorns of $\mathcal{T}_{n}$.*

d)* Orbits of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ on $\mathop{\mathrm{Clop}}\nolimits(\mathcal{T}_{n})$ are numerated by equivalence classes of reduced thorns.*

Description of the correspondence. Let $p$, $q$ be adjacent vertices of $\mathcal{T}_{n}$. Denote by $\overrightarrow{pq}$ the thorn having one vertex $p$ and one spike in the mid-edge $pq$. Cutting the edge $pq$ at the mid-point we get two branches. We choose the branch $U$ containing $q$ and the corresponding ball $B[\overrightarrow{pq}]$, see Fig. 7.

A sub-thorn $\longrightarrow$ a union of balls. Consider a sub-thorn in $\mathcal{T}_{n}$. Then each spike corresponds to a ball. Taking a union of these balls we get a clopen subset with a given partition into balls.

Notice, that starting a perfect thorn we get the whole boundary $\mathop{\mathrm{Abs}}\nolimits(\mathcal{T}_{n})$.

A union of balls $\longrightarrow$ a sub-thorn. Conversely, fix a representation of $\Omega$ as a disjoint union of balls $B_{1}\sqcup\dots\sqcup B_{m}$. Let $U_{1}$, …$U_{m}$ be the corresponding branches of $\mathcal{T}_{n}$. Let $u_{1}$, …, $u_{m}$ be mid-edges that cut these branches off. We consider the minimal sub-thorn $R$ of $\mathcal{T}_{n}$ containing $u_{1}$, …, $u_{m}$.

A clopen set $\longrightarrow$ a reduced sub-thorn. Let $\Omega$ be a proper clopen set. By Lemma 2.1, any sub-ball $B\subset\Omega$ is contained in a unique maximal sub-ball $\widetilde{B}\subset\Omega$. We take the partition of $\Omega$ into maximal sub-balls and take the corresponding thorn. Clearly, it is reduced. $\square$

3.3. Double cosets and bi-thorns. A bi-thorn is the following collection of data $\{R,Q;\theta\}$:

$\bullet$ an ordered pair of perfect thorns $R$, $Q$ with the same number of vertices;

$\bullet$ a bijection $\theta:\mathop{\mathrm{spike}}\nolimits(R)\to\mathop{\mathrm{spike}}\nolimits(Q)$.

We admit an empty bi-thorn.

Equivalently, we have an $(n+1)$-valent graph $[R,Q;\theta]$, which contains a pair of disjoint subtrees $R$, $Q$ and the remaining edges connect vertices of $P$ and vertices of $Q$ (we admit several edges between two vertices), see Fig. 8.

Consider a bi-thorn $\{R,Q;\theta\}$. Let $a$ be a vertex of $\partial(R^{\circ})$, $a^{\prime}$ be a unique adjacent vertex of $R^{\circ}$. Let $b$ a vertex of $\partial(Q^{\circ})$ and $b^{\prime}$ the adjacent vertex. We say that $a$, $b$ are *similar * if $\theta$ sends all spikes at $a$ to spikes at $b$, see Fig. 8. In this situation, we can cut the mid-edges of $a^{\prime}a$ and $b^{\prime}b$. The thorn splits into two pieces. We remove the piece with two vertices $a$ and $b$ and modify $\theta$ saying that it sends the mid-edge of $a^{\prime}a$ to the mid-edge of $b^{\prime}b$. In this way we get a new thorn.

We say that a bi-thorn is minimal if it has not a pair of similar vertices.

Proposition 3.2**.**

There is a canonical one-to-one correspondence between the double coset space $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})\setminus\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ and the set of minimal bi-thorns.

Let us construct the correspondence. Let $g\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Take a ball $B=\mathop{\mathrm{Ba}}\nolimits[U]$ and assume that $gB$ is a ball, $gB=\mathop{\mathrm{Ba}}\nolimits[V]$. We say that $g$ regards the ball $B$ if the map $g:B\to gB$ is induced by an isomorphism of the branches $U\to V$.

Let $g$ regard a ball $B$. Then there is a unique maximal ball $C=\widetilde{B}\supset B$ regarded by $g$. Thus we get a partition

[TABLE]

consisting of maximal balls regarded by $g$ and the corresponding partition

[TABLE]

consisting of balls regarded by $g^{-1}$. We take thorns $R$ and $Q$ corresponding to this partitions, by construction $g$ determines a bijection between their spikes. $\square$

Corollary 3.3**.**

Fix $g\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Fix an $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-orbit $\mathcal{O}$ in $\mathop{\mathrm{Clop}}\nolimits^{\circ}(\mathcal{T}_{n})$. Then for all but a finite number of elements $\Omega\in\mathcal{O}$ we have $g\Omega\in\mathcal{O}$.

Proof. According the previous proof, $g$ canonically determines a pair of sub-thorns $R$ and $Q$ of the Bruhat–Tits tree. The orbit $\mathcal{O}$ corresponds to a certain reduced thorn $T$. Elements $\Omega$ of the orbit correspond to sub-thorns $S$ in $\mathcal{T}_{n}$ isomorphic to $T$. Clearly, if $S\cap R=\varnothing$, then $g\Omega\in\mathcal{O}$. $\square$

4. A family of spherical representations

4.1. The infinite symmetric group with Young subgroup. Fix $k$. Consider $k$ countable sets $\Pi_{1}$, …, $\Pi_{k}$ and their disjoint union

[TABLE]

First, consider the group $G$ of all finitely supported permutations of ${\boldsymbol{\Pi}}$ and its (Young) subgroup $K$ preserving each $\Pi_{j}$. Then $G\supset K$ is a spherical pair and according Nessonov [46] (see also, [42], Sect. 8) all $K$-spherical functions on $G$ have the following form $\Phi_{S}$. Consider a positive (semi)definite matrix $S$ of size $k\times k$ with $s_{jj}=1$. Then

[TABLE]

where $\theta_{p,q}(\sigma)$ is the number of elements $\alpha\in\Pi_{p}$ such that $\sigma\alpha\in\Pi_{q}$.

To construct the corresponding unitary representations of $G$ we consider a Euclidean space $V$ and a collection of unit vectors $e_{1}$, …, $e_{k}$ such that $\langle e_{p},e_{q}\rangle_{V}=s_{p,q}$ (we can assume that $V$ is spanned by these vectors). Consider the tensor product101010Recall that a definition of a tensor product of an infinite family $H_{j}$ of Hilbert spaces requires a fixing of a distinguished unit vector $\xi_{j}\in H_{j}$ in each factor, a tensor product depends on a choice of $\xi_{j}$. For details, see, e. g., [18], Appendix A.

[TABLE]

we see that factors are enumerated by elements of the set ${\boldsymbol{\Pi}}$. The group $G$ acts by permutations of the factors. A unique $K$-fixed vector is

[TABLE]

The $G$-cyclic span of the vector $\mathcal{E}$ is an irreducible spherical representation of $G$.

Second, we notice that our representation can be extended by the continuity to a larger group $\mathbf{G}$. It consists of all permutations $\sigma$ of the set $\Pi$ such that for all $p$ for all but a finite number of $\alpha\in\Pi_{p}$, we have $\sigma\alpha\in\Pi_{p}$ (permutations of factors in the tensor product are well-defined for such $\sigma$). The spherical subgroup $\mathbf{K}$ consists of all permutations preserving each subset $\Pi_{p}$.

4.2. Embeddings of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$ to the group $\mathbf{G}$. Consider a collection of reduced thorns $T_{1}$, …, $T_{N}$, let they correspond to the same $\iota$ in the decomposition (3.1). Consider the corresponding $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$-orbits $\mathcal{O}_{1}$, …, $\mathcal{O}_{N}$ in $\mathop{\mathrm{Clop}}\nolimits_{\iota}^{\circ}(\mathcal{T}_{n})$ and the complement $\mathcal{P}$ to the union of these orbits. Thus we get a partition

[TABLE]

Consider the group $\mathbf{G}$ corresponding to this partition. By Corollary 3.3, the group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ is contained in $\mathbf{G}$. Obviously, $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})\subset\mathbf{K}$. So we can apply the Nessonov construction.

Remark. Fix $\iota=0,$ 1, …, $n-2$. Consider a Hilbert space $V$ and a countable set of unit vectors $e_{T}$ enumerated by reduced thorns whose number of spikes is $\iota$ modulo $n-1$. Let this set have a unique limit point $e$ (and hence a sequence composed of $e_{S}$ in any order converges to $e$. For a clopen subset $\Omega$ denote by $T(\Omega)$ the corresponding reduced thorn. Consider the following tensor product

[TABLE]

The action of the group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ in $\mathcal{H}$ by permutations of factors is well-defined iff the following product absolutely converges for all hierarchomorphisms $g$:

[TABLE]

Clearly, if the sequence $e_{T}$ converges fast enough, then this is the case. In this situation, we get a spherical representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ in $\mathcal{H}$ with the spherical vector $\otimes_{\Omega\in\mathop{\mathrm{Clop}}\nolimits_{\iota}^{\circ}}e_{T(\Omega)}$ and the spherical function $\Phi(g)$.

It can be interesting to find precise conditions for a family $e_{T}$ providing well-definiteness of this construction.

Addendum. Several comments on the sphericity phenomenon

A.1. General remarks on sphericity. Thus $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is a noncompact spherical subgroup in a locally compact group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{p})$. According [43], the subgroup $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{R}})$ is spherical in the group $\mathop{\mathrm{Diff}}\nolimits^{3}(S^{1})$ of $C^{3}$-diffeomorphisms of the circle $S^{1}$. We explain why this seems distinguished.

Phenomenon of sphericity was discovered by Gelfand in 1950, [14]. He showed that maximal compact subgroups $K$ in semisimple Lie groups $G\supset K$ are spherical (as $\mathop{\mathrm{GL}}\nolimits(n,{\mathbb{R}})\supset\mathop{\mathrm{O}}\nolimits(n)$ or $\mathop{\mathrm{Sp}}\nolimits(2n,{\mathbb{R}})\supset\mathop{\mathrm{U}}\nolimits(n)$). Also symmetric subgroups in semisimple compact Lie groups are spherical (as $\mathop{\mathrm{U}}\nolimits(n)\supset\mathop{\mathrm{O}}\nolimits(n)$ or $\mathop{\mathrm{O}}\nolimits(2n)\supset\mathop{\mathrm{U}}\nolimits(n)$). Related spherical representations played a distinguished role in theory of unitary representations, and spherical functions were an important standpoint for development of modern theory of multi-dimensional special functions.

In 1979 Krämer [27] observed that simple compact Lie groups have smaller spherical subgroups as $\mathop{\mathrm{O}}\nolimits(2n+1)\supset\mathop{\mathrm{U}}\nolimits(n)$ or $\mathop{\mathrm{Sp}}\nolimits(2n+2)\supset\mathop{\mathrm{Sp}}\nolimits(2n)\times\mathop{\mathrm{SO}}\nolimits(2)$, in the most of cases such pairs can be obtained from a Gelfand pair $G\supset K$ by a minor enlargement of $G$ or minor reduction of $K$. Mikityuk and Brion extended the Krämer classification to semisimple compact groups.

There is also a story with finite spherical pairs $G\supset K$, see, e, g., [9]

On the other hand infinite-dimensional limits of Gelfand pairs (as $\mathop{\mathrm{GL}}\nolimits(\infty,{\mathbb{R}})\supset\mathop{\mathrm{O}}\nolimits(\infty)$) are spherical. G. Olshanski [49], [50] understood that such pairs have a substantial representation theory, later there appeared related harmonic analysis. For infinite-dimensional (large) groups the phenomenon of sphericity is more usual than for Lie group, and at least representation theory can be developed in quite wide generality, see, e. g. [45], [46], [42], [41], in Subsection 4 we used a construction of this kind. In a known zoo, spherical subgroups are ’heavy groups’ in the sense of [40] (as the complete unitary group, the complete symmetric group, the group of all measure preserving transformations).

Two examples mentioned in the beginning of the present subsection are outside these two families. In one case a noncompact Lie group $\mathop{\mathrm{SL}}\nolimits(2,{\mathbb{R}})$ is a spherical subgroup in an infinite-dimensional group $\mathop{\mathrm{Diff}}\nolimits^{3}(S^{1})$, in another case a noncompact subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is spherical in a locally compact group $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$.

A.2. On compactness of stabilizers of vectors in unitary representations. In any case in substantial theory of unitary representations of Lie groups spherical subgroups (in the sense formulated in Introduction) must be compact. There is a theorem of Moore [35] about possible stabilizer of vectors in unitary representation, whose precise formulation is slightly sophisticated. We formulate a simpler statement.

Let $G$ be a connected Lie group, $Z$ the center; denote by $\mathfrak{g}\supset\mathfrak{z}$ their Lie algebras. Denote by $\mathop{\mathrm{Ad}}\nolimits_{\mathfrak{g}}(\cdot)$ the adjoint representation of $G$ in $\mathfrak{g}$, in fact we have a representation of the quotient group $G/Z$ in the group $\mathop{\mathrm{GL}}\nolimits[\mathfrak{g}]$ of all linear operators of the space $\mathfrak{g}$.

Let $\rho$ be a faithful irreducible unitary representation of $G$ in a Hilbert space $V$. An irreducible faithful representation determines an injective homomorphism from $Z$ to the unit circle $\mathbb{T}$ on the complex plane. For this reason $\dim\mathfrak{z}\leqslant 1$, and we have 3 possibilities: $Z=\mathbb{T}$, $Z$ is finite, $Z$ is a dense subgroup in $\mathbb{T}$.

Proposition A. 1**.**

Let $G$, $\rho$, $V$ be as above.

(i)* Let the image of $G/Z$ in the group $\mathop{\mathrm{GL}}\nolimits[\mathfrak{g}]$ be closed.*

(ii)* Let the center $Z$ be compact.*

Then

a)* The stabilizer $K_{v}$ of a nonzero vector $v$ is compact.*

b)* The stabilizer $L_{v}$ of the line ${\mathbb{C}}v$ is compact.*

Proof. It is sufficient to prove the statement for the group $L_{v}$. By definition $L_{v}$ contains $Z$. Since $Z$ is compact, the image of $L_{v}$ in $G/Z$ is closed. Since the $\mathop{\mathrm{Ad}}\nolimits$-image of $G$ in $\mathop{\mathrm{GL}}\nolimits[\mathfrak{g}]$ is closed, the $\mathop{\mathrm{Ad}}\nolimits$-image of $L_{v}$ in $\mathop{\mathrm{GL}}\nolimits[\mathfrak{g}]$ also is closed.

We use Theorem 1.2 of Wang [59] (which is a strong version of the result of Moore [35]). We say that an element $g\in\mathop{\mathrm{GL}}\nolimits(\mathfrak{g})$ is pre-periodic if it is semisimple111111i. e., it is diagonalizable after a pass to the complexification. and its eigenvalues $\theta_{j}$ satisfy $|\theta_{j}|=1$. Equivalently, the closure of the set $\{g^{m}\}$, where $g$ ranges in ${\mathbb{Z}}$, is compact. By [59], for any $g\in L_{v}$ there is a subgroup $M_{g}$ such that:

1. $M_{v}\subset K_{v}$;

2. denote by $\mathfrak{m}_{g}$ the Lie algebra of $M_{g}$; then the image of $\mathop{\mathrm{Ad}}\nolimits(g)$ in $\mathfrak{g}/\mathfrak{m}$ is pre-periodic.

However, if a normal subgroup fixes a vector $v$, then it acts trivially on the whole space. Indeed, let $r\in G$, $m\in M_{g}$. Then

[TABLE]

Our representation is faithful and therefore the subgroup $M_{g}$ is trivial. Thus the image $L_{v}/Z$ of $L_{v}$ in $\mathop{\mathrm{GL}}\nolimits[\mathfrak{g}]$ is closed and consists of pre-periodic elements. It is more or less clear that $L_{v}/Z$ is compact121212To avoid a proof, we can refer to Lemma 1.3 from [60] about a group with a dense set of pre-periodic elements.. Since $Z$ is compact, $L_{v}$ also is compact. $\square$

Remark 1. There are several reasons, for which we can not simply say: for unitary representations stabilizers of vectors (lines) are compact.

a) Obviously we must consider faithful representations, since any closed normal subgroup $H\subset G$ can be a kernel of a representation.

b) More serious sources of problems are twinnings. Consider the group $\mathop{\mathrm{Isom}}\nolimits(2)$ of orientation preserving isometries of the Euclidean plane. Denote $Q:=\mathop{\mathrm{Isom}}\nolimits(2)\times\mathop{\mathrm{Isom}}\nolimits(2)$, we can regard an element of this group as a pair of matrices of the form

[TABLE]

Denote by $S\subset Q$ the subgroup consisting of pairs of matrices with $z=w=0$, i. e., $S=\mathop{\mathrm{SO}}\nolimits(2)\times\mathop{\mathrm{SO}}\nolimits(2)$. It is more-or-less obvious that $Q\supset S$ is a spherical pair (the Wigner–Mackey trick, see, e.g., [26], 13.3, Theorem 1, immediately gives a classification of irreducible unitary representations of $Q$). Next, choose an irrational real $\theta$ and take the subgroup $G\subset Q$ consisting of pairs of matrices (A.1) satisfying the condition $s=\theta t$, consider the corresponding subgroup $K=S\cap G$. The group $G$ is the Mautner group (see, e. g. [1]). Clearly, restricting an $S$-spherical representation of $Q$ to $G$ we get a $K$-spherical representation of $G$. However, $K\simeq{\mathbb{R}}$ is not compact.

c) Consider the universal covering $G^{\sim}$ of the group $G=\mathop{\mathrm{SL}}\nolimits(2,{\mathbb{R}})$ and the universal covering $R^{\sim}$ of the subgroup of rotations, $R^{\sim}\simeq{\mathbb{R}}$. Let $\rho$ be a faithful irreducible representation of $G^{\sim}$ (see [52]). Then $R^{\sim}$ has a discrete spectrum. For an eigenvector $v$ we have $L_{v}=R^{\sim}$ and $K_{v}\simeq{\mathbb{Z}}$. Both subgroups are non-compact. However, this non-compactness again is artificial, in our case $L_{v}/Z$ is compact in $G^{\sim}/Z$. $\boxtimes$

Remark 2. If $G$ can be covered by a real algebraic group, then the conditions (i)-(ii) are fulfilled automatically. $\boxtimes$

Notice that for $p$-adic groups stabilizers of vectors in unitary representations in interesting cases are compact (such stabilizers were topic of works of Wang [59]–[60]).

A.2. The Mautner phenomenon for the groups for $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Let $\rho$ be a unitary representation of a group $G$. Assume that a subgroup $K$ fixes some vector $v$. Then quite often $v$ is automatically fixed by certain larger group $\widetilde{K}$. For $G=\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ we have the following statement:

Proposition A. 2**.**

Let $\rho$ be a unitary representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$, let $v$ be a vector in the space of the representation.

a)* Let $h\in\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ be a hyperbolic element and $\rho(h)v=v$. Then $v$ is fixed by the whole subgroup $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{n})$.*

b)* Let $v$ be fixed by a parabolic subgroup $P_{c}\subset\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$. Then $v$ is fixed by the whole subgroup $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$.*

This is obvious: nontrivial irreducible representations of $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ have no fixed vectors with respect to these subgroup (of course, this argument requires to look at Olshanski’s list [48]). $\square$

A.3. A trivial spherical representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$. Recall that the homogeneous space $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ is countable and is equipped with the discrete topology. Therefore we have a quasi-regular representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ in $\ell^{2}$ on this space, the natural orthogonal basis $\delta_{z}$ in $\ell_{2}$ is enumerated by points $z\in\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$, the vector $\delta_{z}$ is the delta-function supported by $z$.

Proposition A. 3**.**

a)* The representation of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})$ in \ell^{2}\bigl{(}\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})\bigr{)} is irreducible and spherical, the spherical vector is $\delta_{z_{0}}$, where $z_{0}$ is the initial point of the homogeneous space, the spherical function is 1 on $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ and 0 outside this subgroup.*

b)* Let $G$ be a topological group, $L$ a closed subgroup, let the homogeneous space $G/L$ be countable and discrete. Let $z_{0}$ be the initial point of $G/L$. Let all orbits of $L$ on $G/L$ except $\{z_{0}\}$ be infinite. Then the representation of $G$ in $\ell^{2}(G/L)$ is irreducible and spherical. The spherical vector is $\delta_{z_{0}}$ and the spherical function is zero outside $L$.*

Proof. b) An $L$-invariant function on $G/L$ must be constant on orbits of $L$. Since a vector in $\ell^{2}$ can not have infinite number of nonzero equal coordinates, we get that $\delta_{z_{0}}$ is the unique $L$-invariant vector. By the same argument as in the proof of Proposition 2.6, the $G$-cyclic span of $\delta_{z_{0}}$ is an irreducible subrepresentation in $\ell^{2}$. However, this cyclic span contains all basis vectors $\delta_{z}$.

a) Keeping in mind Proposition 3.2, for any element of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{n})/\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ we can assign a bi-thorn $\{R,Q;\theta\}$ and an embedding of the thorn $Q$ to $\mathcal{T}_{n}$. The group $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{n})$ acts preserving the bi-thorn and changing embeddings. Clearly, if the bi-thorn $\{R,Q;\theta\}$ is non-empty, then orbits are infinite. So, we can apply the statement b). $\square$

A.4. A question about unitary representations of discrete groups. It is well-know that questions about unitary representations of discrete groups quite often are dangerous. By the Thoma theorem [57], discrete groups are not type $I$ except groups that have Abelian normal subgroups of finite index. Absence of type $I$ property implies numerous unpleasant phenomena (see, at least, the Glimm theorem [17] about a bad Borel structure on the dual space). However, we formulate the following informal question.

Question A. 4**.**

Consider a pair of countable groups $\Gamma\supset\Delta$ and let all orbits of $\Delta$ on $\Gamma/\Delta$ be infinite (except the initial point). To find such pairs with ’interesting’ $\Delta$-spherical representations of $\Gamma$.

Apparently, interesting situations are rare. However, there is a famous example of such a pair, which was basically discovered by in 1964 by Thoma [56] (see [50]). We take the group $\mathrm{S}(\infty)$ of finitely supported permutations of ${\mathbb{N}}$, let $\Gamma$ be $\mathrm{S}(\infty)\times\mathrm{S}(\infty)$ and $\Delta\simeq\mathrm{S}(\infty)$ be the diagonal subgroup. This was a start of big story (representation theory of infinite symmetric groups), we only mention that in this case spherical representations can be extended by continuity to a larger (continual) group (see [50], [42]).

The pair of discrete groups $G\supset K$ from Subsect. 4 is spherical (and again we have a continuous extension to a larger group $\mathbf{G}$). A big zoo of examples of spherical representations in [42] has a similar nature.

Next, consider the Thompson group $\mathrm{T\!h}$ realized as the group of all continuous piece-wise $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$-transformations of the real projective line ${\mathbb{R}}\mathbb{P}^{1}$, see [51], [20], by this construction $\mathrm{T\!h}$ is embedded to $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{2})$ and $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$ is contained in $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{2})$.

Proposition A. 5**.**

Consider a unitary $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{2})$-spherical representation $\rho$ of $\mathop{\mathrm{Hier}}\nolimits(\mathcal{T}_{2})$ with spherical vector $v$. Then the $\mathrm{T\!h}$-cyclic span of $v$ is a $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$-spherical representation of $\mathrm{T\!h}$.

Proof. It sufficient to show that the restriction of $\rho$ to $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$ does not contain additional $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$-fixed vectors. We take an hyperbolic element $h$ of $\mathop{\mathrm{PSL}}\nolimits(2,{\mathbb{Z}})$, say, $h=\begin{pmatrix}2&1\\ 3&2\end{pmatrix}$. It is hyperbolic in $\mathop{\mathrm{Aut}}\nolimits(\mathcal{T}_{2})$. By Proposition A.2.a, vectors fixed by $h$ are fixed by the whole group $\mathop{\mathrm{Aut}}\nolimits_{+}(\mathcal{T}_{2})$, and a vector fixed by this subgrop is unique. $\square$