Degenerate principal series representations and their holomorphic extensions

TL;DR

This paper investigates degenerate principal series representations of certain symmetric groups, deriving explicit formulas for spherical functions, and constructs new complementary series by analyzing their holomorphic extensions and boundary behaviors.

Contribution

It introduces new formulas for spherical functions using hypergeometric functions and constructs novel complementary series representations through boundary analysis and integral operators.

Findings

Derived formulas for spherical functions in terms of hypergeometric functions.

Constructed new complementary series for specific classical groups.

Realized complementary series as discrete components in holomorphic discrete series branching.

Abstract

Let $X=H/L$ be an irreducible real bounded symmetric domain realized as a real form in an Hermitian symmetric domain $D=G/K$. The intersection $S$ of the Shilov boundary of $D$ with $X$ defines a distinguished subset of the topological boundary of $X$ and is invariant under $H$ and can also be realized as $S=H/P$ for certain parabolic subgroup $P$ of $H$. We study the spherical representations $Ind_P^H(\lam)$ of $H$ induced from $P$. We find formulas for the spherical functions in terms of the Macdonald ${}_2F_1$ hypergeometric function. This generalizes the earlier result of Faraut-Koranyi for Hermitian symmetric spaces $D$. We consider a class of $H$-invariant integral intertwining operators from the representations $Ind_P^H(\lam)$ on $L^2(S)$ to the holomorphic representations of $G$ on $D$ restricted to $H$. We construct a new class of complementary series for the groups $H=SO(n, m)$, $SU(n, m)$ (with $n-m >2$) and $Sp(n, m)$ (with $n-m>1$). We realize them as a discrete component in the branching rule of the analytic continuation of the holomorphic discrete series of $G=SU(n, m)$, $SU(n, m)\times SU(n, m)$ and $SU(2n, 2m)$ respectively.