Regularity of CR maps into uniformly pseudoconvex hypersurfaces and applications to proper holomorphic maps

TL;DR

This paper investigates the regularity of CR maps into pseudoconvex manifolds with Levi foliations, establishing conditions for smoothness and applying results to boundary regularity of proper holomorphic maps.

Contribution

It introduces an invariant that determines when CR maps are smooth or constrained, and proves generic smoothness of certain CR maps between pseudoconvex hypersurfaces.

Findings

CR maps are either generically smooth or highly constrained.

Sufficient regularity implies generic smoothness of CR transversal maps.

Boundary regularity of proper holomorphic maps into symmetric domains is established.

Abstract

We study regularity properties of CR maps in positive codimension valued in pseudoconvex manifolds which carry a nontrivial Levi foliation. We introduce an invariant which can be used to deduce that any sufficiently regular CR map from a minimal manifold into such a foliated target is either generically smooth or geometrically highly constrained, and to show generic smoothness of sufficiently regular CR transversal CR maps between pseudoconvex hypersurfaces. As an application, we discuss boundary regularity of proper holomorphic maps into bounded symmetric domains.

Equations

$${\nu }_p:=\underset{0\ne V_p\in T_p\eta }{\max }{\dim }_{\mathbb{C}}\ker \left({\bar{L}}_p\to {\mathbb{P}}_{T{\mathbb{C}}^{N^{\prime }}/T\eta }({\bar{L}}_{p}V)\right)-{\dim }_{\mathbb{C}}\eta .$$

$$[\frac{1}{2}(\bar{L}+L),\frac{1}{2}(\bar{\Gamma }+\Gamma )]=\frac{1}{4}[\bar{L},\bar{\Gamma }]+\frac{1}{4}[L,\Gamma ]+\frac{1}{4}[L,\bar{\Gamma }]+\frac{1}{4}[\bar{L},\Gamma ]=-\Im \left(\mathcal{L}(\bar{L},\bar{\Gamma })\right),$$

$$\left[[V,W],{\Gamma }_q(T^{c}M)\right]$$

$$\subseteq \left[V,{\Gamma }_q(T^{c}M)\right]-\left[W,{\Gamma }_q(T^{c}M)\right]\subseteq {\Gamma }_q(T^{c}M),$$

$$r_0(p)$$

$$r_k(p)$$

$${\mathcal{L}}_{\Theta }(\bar{\Gamma },\bar{L})=\sum _{j,k=1}^{N^{\prime }}\frac{{\partial }^2\rho }{\partial w_j\partial {\bar{w}}_k}(p^{\prime }){\Gamma }_j{\bar{L}}_k.$$

$$\Gamma \rho$$

$$R_{p^{\prime }}(h_{\ast }\bar{L}{|}_p,{\gamma }_t(p))={\mathbb{P}}_{T_{p^{\prime }}^{\perp }{\eta }_{p^{\prime }}}\left(h_{\ast }\bar{L}{|}_p{\tilde{\gamma }}_t\right)={\mathbb{P}}_{T_{p^{\prime }}^{\perp }{\eta }_{p^{\prime }}}\left(\bar{L}{|}_p({\tilde{\gamma }}_t\circ h)\right)={\mathbb{P}}_{T_{p^{\prime }}^{\perp }{\eta }_{p^{\prime }}}\left(\bar{L}{|}_p{\gamma }_t\right)=0,$$

$${\nu }_{p^{\prime }}=\underset{0\ne V\in T_{p^{\prime }}\eta }{\max }{\dim }_{\mathbb{C}}\ker R_{p^{\prime }}(\cdot ,V)-{\dim }_{\mathbb{C}}\eta .$$

$$\Re {\left(q_1+t\Re (q_1)\right)}^2+\cdots +\Re {\left(q_{N^{\prime }-1}+t\Re (q_{N^{\prime }-1})\right)}^2$$

$$=(1+\Re (t){)}^2\left(\Re (q_1{)}^2+\cdots +\Re (q_{N^{\prime }-1}{)}^2\right)$$

$$=(1+\Re (t){)}^2\Re (q_{N^{\prime }}{)}^2=\Re {\left(q_{N^{\prime }}+t\Re (q_{N^{\prime }})\right)}^2.$$

$$K+{\nu }_{q^{\prime }}$$

$$\ge {\dim }_{\mathbb{C}}\left(T_{q^{\prime }}^{0,1}\eta +h_{\ast }{T}_q^{0,1}S\right)=K+n_{+},$$

$$T_p^{0,1}{\mathbb{C}}^N=⟨\{{\bar{L}}_1{\bar{L}}_j{\rho }_w:\rho \in {\mathcal{I}}_M(p),{\bar{L}}_1,\dots ,{\bar{L}}_j\in {\mathcal{V}}_p(M),0\le j\le k\in \}⟩;$$

$${\left({\mathcal{T}}_{\bar{L}}\omega \right)}_j$$

$$=\bar{L}\omega (\frac{\partial }{\partial z_j})-\frac{\partial }{\partial z_j}\omega (\bar{L})-\omega ([\bar{L},\frac{\partial }{\partial z_j}])=\bar{L}{\omega }_j,$$

$$0$$

$$={\bar{L}}_{2}d\Theta ({\bar{L}}_1,V)-Vd\Theta ({\bar{L}}_1,{\bar{L}}_2)-d\Theta ({\bar{L}}_1,[{\bar{L}}_2,V])$$

$$=-{\bar{L}}_1\Theta ([{\bar{L}}_2,V])+[{\bar{L}}_2,V]\Theta ({\bar{L}}_1)+\Theta ([{\bar{L}}_1,[{\bar{L}}_2,V]])=\Theta ([{\bar{L}}_1,[{\bar{L}}_2,V]]).$$

$$D_I^{m,n}=\{Z\in {\mathbb{C}}^{m\times n}:{\mathbb{I}}_m-Z{Z}^{\ast }\text{ is strictly positive definite.}\}$$

$$\rho (Z+\mu Z)=\det ({\mathbb{I}}_m-|1+\mu {|}^{2}Z{Z}^{\ast })=|1+\mu {|}^{2m}P(|1+\mu {|}^{-2})$$

$$(z_1,\dots ,z_m,w_2,\dots ,w_n)\mapsto (z_1,\dots ,z_m{)}^T(1,w_2\dots ,w_n)$$

$$\rho (z_1,\dots ,z_m,w_2,\dots ,w_n)=(|z_1{|}^2+\cdots +|z_m{|}^2)(1+|w_2{|}^2+\cdots +|w_n{|}^2)-1=0,$$

$${\eta }_{u{v}^{\ast }}:=\{u{v}^{\ast }+B:B\in {\mathbb{C}}^{m\times n},u^{\ast }B=0,Bv=0\}$$

$$B(Z)=({\mathbb{I}}_m-Z{Z}^{\ast })B_0({\mathbb{I}}_n-Z^{\ast }Z),$$

$$u^{\ast }B(u{v}^{\ast })$$

$$B(u{v}^{\ast })v$$

$$B(a{b}^{\ast })$$

$$\bar{L}{|}_{a{b}^{\ast }}B$$

$$\displaystyle=\frac{d}{d\bar{t}}\Big{|}_{t=0}\left(\mathbb{I}_{m}-\lVert b+\bar{t}\beta\rVert^{2}(a+t\alpha)(a+t\alpha)^{\ast}\right)B_{0}\left(\mathbb{I}_{n}-\lVert a+t\alpha\rVert^{2}(b+\bar{t}\beta)(b+\bar{t}\beta)^{\ast}\right)$$

$$=-a{\alpha }^{\ast }{B}_0-B_0\beta b^{\ast }.$$

$$\mathrm{tr}\left((\bar{L}{|}_{a{b}^{\ast }}B{)}^{\ast }Z\right)$$

$$D_{II}^m=\left\{Z\in {\mathbb{C}}^{m\times m}:Z^T=-Z,{\mathbb{I}}_m-Z^{\ast }Z>0\right\}.$$

Full text

1. Introduction

This paper is devoted to the study of regularity of CR maps into smooth Levi-degenerate hypersurfaces foliated by complex manifolds and the application of these results to boundary regularity of proper holomorphic maps in positive codimension. The positive codimensional case is much more challenging than the equidimensional case in many regards, but also has some specific features which make natural answers to regularity problems for mappings a bit different. We refer the reader to the discussion in [19], where the authors point out some of the salient points, and summarize here those which are important for our approach. We consider a CR map $h\colon M\to M^{\prime}$ with $M\subset\mathbb{C}^{N}$ a CR submanifold, and $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ a hypersurface (we shall simply refer to $M$ as the source, and $M^{\prime}$ as the target). All structures and manifolds in this paper are assumed to be smooth unless explicitly stated otherwise; our technique and our results extend to other categories as well, as we will outline after stating and discussing our results in the smooth setting first.

For the purpose of this discussion, the following observations are important. First of all, the typical conclusion of a regularity statement in higher codimension is that of generic smoothness of the map given some a priori regularity, i.e. smoothness on a dense, open subset. We cannot drop the a priori regularity across a certain threshhold, as for example Low [22] and Stensones [26] showed. The typical a priori bound that we are going to use are linear in the codimension of the map considered, which is typical for all known results except for the notable exception of mappings between spheres of “small” codimension, see e.g. Huang [12].

Automatic generic smoothness of all CR maps with such an a priori regularity follows if the target is of D’Angelo finite type (that is, if it does not contain any formal holomorphic curves), as shown in [20]. However, the condition that $M^{\prime}$ is of finite type is definitely not necessary if one excludes certain typical examples of non-smooth CR maps. For example, since there always exist a nowhere smooth CR function $\varphi$ (actually, with arbitrary finite smoothness prescribed) near a strictly pseudoconvex points, if the target contains a complex curve $\gamma(\zeta)$, one obtains a nowhere smooth CR map $\gamma\circ\varphi$.

However, as was already observed in [20] in the case where the target is the tube over the light cone, in many geometrically interesting situations, this type of behavior is the only exceptional example. Our first main theorem describes one such situation. In order to formulate it, we need to introduce an invariant measuring the (non)degeneracy of a foliation by complex manifolds.

Consider a foliation $\eta$ of $M^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ by complex manifolds $\eta_{p}$, where $\eta_{p}$ denotes the leaf of $\eta$ containing $p$. To each $p\in M^{\prime}$, we associate

[TABLE]

Here $\bar{L}_{p}V$ denotes the componentwise derivative at $p$ of an (arbitrary) smooth extension $V$ of $V_{p}$, and we project onto the quotient space $\faktor{T\mathbb{C}^{N}}{T\eta}$, where $T\eta\subset T\mathbb{C}^{N^{\prime}}|_{M}$ denotes the tangent bundle of $\eta$, i.e. $T_{p}\eta=T_{p}\eta_{p}$. It turns out that this yields a well defined invariant $\nu_{p}$ because as we shall show in section 3.1 the map associating to $\bar{L}\in T^{0,1}M^{\prime}$ and $V\in\Gamma(T\eta)$ the section $\mathbb{P}_{T\mathbb{C}^{N^{\prime}}/T\eta}(\bar{L}V)$ of the quotient bundle $T\mathbb{C}^{N^{\prime}}/T\eta$ is tensorial.

Theorem 1**.**

Let $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ be a uniformly pseudoconvex hypersurface with Levi foliation $\eta$, satisfying $\nu_{p^{\prime}}=0$ for all $p^{\prime}\in M^{\prime}$, and let $M$ be a connected minimal CR submanifold. Then any $C^{N^{\prime}-1}$-regular CR map $h:M\rightarrow M^{\prime}$ is either generically smooth, or it maps $M$ entirely into a single leaf of the foliation, i.e. $h(M)\subseteq\eta_{h(p)}$ for any $p\in M$.

If $\nu$ is nonzero, our second main theorem still guarantees automatic regularity if the number of positive Levi eigenvalues of the source manifold $M$ is large enough and $h$ is assumed to be CR transversal (which is automatically satisfied in many applications).

Theorem 2**.**

Let $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ be a uniformly pseudoconvex hypersurface, and $M\subset\mathbb{C}^{N}$ a pseudoconvex hypersurface with at least $n_{+}$ positive Levi eigenvalues. Then any CR-transversal CR map $h:M\rightarrow M^{\prime}$ of regularity $C^{N^{\prime}-n_{+}}\cap C^{2}$ is generically smooth near any point $p$ which satisfies $\nu_{h(p)}<n_{+}$.

We note that under a different set of assumptions (which are, as we will see later, not directly related to our invariant $\nu$) Xiao [29, Thm. 1] obtains everywhere regularity: to be precise, in his case, he considers maps from strongly pseudoconvex hypersurfaces into uniformly pseudoconvex hypersurfaces of the same signature which are $2$-nondegenerate; the point is that such maps are automatically $2$-nondegenerate in the sense of [16], so that one can apply theorems from [15, 17]. We discuss the connection with our result later in section 4.1, and point out here that $\nu=0$ implies, in particular, $2$-nondegeneracy of the target; our assumptions, however, do not imply that that the maps we are considering are $2$-nondegenerate. Let us furthermore point out the low codimension results in the paper of Kossovskiy, Xiao, and the second author [14].

As an application of Theorem 2 we obtain the following boundary regularity result for proper holomorphic maps:

Corollary 1**.**

Let $\Omega\subseteq\mathbb{C}^{N}$ and $\Omega^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ be domains, and $M\subset\partial\Omega$, $M^{\prime}\subset\partial\Omega^{\prime}$ be two hypersurfaces contained in the respective domains’ smooth boundary part. Assume that $\Omega^{\prime}$ is uniformly pseudoconvex at $M^{\prime}$, and $M$ is pseudoconvex with at least $n_{+}$ positive Levi eigenvalues. Then every holomorphic map $H:\Omega\rightarrow\Omega^{\prime}$ which extends as a $C^{N^{\prime}-n_{+}}\cap C^{2}$-regular map to $M$ and maps $M$ into $M^{\prime}$, is generically smooth (on $\bar{\Omega}$) near any $p\in M$ satisfying $\nu_{H(p)}<n_{+}$.

In Section 5 we discuss in detail proper holomorphic maps into pseudoconvex domains as an application of 2, and prove a theorem which deals with maps into boundaries of classical symmetric domains. By a careful study of the geometry of the smooth boundary components, we can calculate the invariant $\nu$ in each case, and obtain the following theorem, which significantly extends the results obtained by Xiao in [29]: the source manifolds we can consider are not required to be strictly pseudoconvex any longer. The price we have to pay is that we have to assume higher a priori regularity and only obtain generic smoothness.

Theorem 3**.**

Let $M\subset\mathbb{C}^{N}$ be a $C^{\infty}$-smooth pseudoconvex hypersurface, and assume that its Levi form has exactly $n_{+}$ positive eigenvalues everywhere. Denote by $M^{\prime}$ the smooth part of the boundary of a classical symmetric domain $\Omega\subseteq\mathbb{C}^{N^{\prime}}$. Then every CR-transversal CR map $h:M\rightarrow M^{\prime}$ of regularity $C^{N^{\prime}-n_{+}}$ is smooth on a dense open subset of $M$, given that

1. (1)

$\Omega=D_{I}^{m,n}$* for $m,n\geq 2$ and $n_{+}\subseteq\{m+n-3,m+n-2\}$,* 2. (2)

$\Omega=D_{II}^{m}$* for $m\geq 4$ and $n_{+}\subseteq\{2m-7,\dots,2m-4\}$,* 3. (3)

$\Omega=D_{III}^{m}$* for $m\geq 2$ and $n_{+}=m-1$ or* 4. (4)

$\Omega=D_{IV}^{m}$* for $m\geq 2$, $M$ is minimal and $n_{+}\leq m-2$.*

Let us note that 3 in particular applies to the setting of (appropriate) boundary values of proper mappings between classical symmetric domains.

The conditions on the number of positive Levi eigenvalues given in 3 are sharp in the following sense: In the case of $D_{I}^{m,n}$, $D_{II}^{m}$ and $D_{III}^{m}$, there are pseudoconvex hypersurfaces $M$ satisfying $n_{+}=m+n-4$, $n_{+}=2m-8$ and $n_{+}=m-2$, respectively, such that there exist nowhere smooth, but arbitrarily often continuously differentiable CR-transversal CR embeddings $h:M\rightarrow M^{\prime}$.

On the other hand, there exists no CR transversal map $h:M\rightarrow M^{\prime}$ at all if the number of positive Levi eigenvalues of $M$ exceeds the upper limit given in 3, which is just the number of positive Levi eigenvalues of $M^{\prime}$.

Remark 1**.**

We first remark that one can obtain results in the real-analytic category with exactly the same assumptions. The conclusion in this setting is that the map $h$ extends to a holomorphic map in a full neighbourhood of an open, dense subset of the source manifold. For this, one uses the result of Mir [23] on real-analytic regularity. We also remark that if both source and target are real-algebraic, one can use the proof of the algebraicity result of Coupet, Meylan, and Sukhov [6] to conclude that the map $h$ is real-algebraic (this conclusion is global in nature).

Remark 2**.**

An interesting observation in the algebraic case is that 3 and 1 yield a complete list of pairs of classical symmetric domains $(\Omega_{1},\Omega_{2})$ such that every proper holomorphic map $H:\Omega_{1}\rightarrow\Omega_{2}$, which extends to $\partial\Omega_{1}$ with sufficient initial regularity, and does not map $\partial\Omega_{1}$ entirely into the non-smooth part of $\partial\Omega_{2}$, is necessarily algebraic.

Remark 3**.**

We finally note that all of our results above apply as well in the case where the source manifold is not embedded, but rather an “abstract” CR structure with the microlocal extension property; in that case, we have to apply the results of [21] instead of [20]. We tried to avoid this more technical aspect in the presentation here, the reader is invited to make the obvious changes to the formulations if needed.

This paper developed from the first author’s master’s thesis [11], where (slightly weaker) versions of 1, 2 and 3 are first proven.

2. Preliminaries

2.1. CR manifolds and the Levi foliation

In this section, we will recall some basic notions and fix notation. We will be considering smooth CR submanifolds of complex Euclidean space, which we will denote by $M\subset\mathbb{C}^{N}$ or $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$, respectively, and CR maps $h:M\rightarrow M^{\prime}$. We will write $T^{0,1}M=\mathbb{C}TM\cap T^{0,1}\mathbb{C}^{N}$, and denote by $J$ the standard complex structure operator.

A continuously differentiable map $h:M\rightarrow M^{\prime}$ is called a CR map if it preserves the CR structure of its domain, i.e. if $h_{\ast}T_{q}^{0,1}M\subseteq T_{h(q)}^{0,1}M^{\prime}$ for all $q\in M$. If we denote by $\iota$ the embedding of $M^{\prime}$ into $\mathbb{C}^{N^{\prime}}$, an equivalent characterization is that $\iota\circ h:M\rightarrow\mathbb{C}^{N^{\prime}}$ is a CR map, which just means that each coordinate component of $h$ is a CR function.

We recall that the Levi form of a hypersurface $M\subset\mathbb{C}^{N^{\prime}}$ is defined by $\mathcal{L}(\bar{L},\bar{\Gamma})=\frac{1}{2i}[\bar{L},\Gamma]\mod T^{0,1}M\oplus T^{1,0}M$; given a defining function $\rho$ for $M$, we define $\Theta=i(\partial\rho-\bar{\partial}\rho)$, and refer to $\mathcal{L}_{\Theta}(\bar{L},\bar{\Gamma})=\frac{1}{2i}\Theta([\bar{L},\Gamma])$ as a scalar Levi form for $M$.

Our target manifolds will be uniformly pseudoconvex hypersurfaces, i.e. real hypersurfaces of $\mathbb{C}^{N^{\prime}}$ with positive semidefinite Levi form, and a constant number of zero and positive eigenvalues everywhere, respectively. It will turn out that these are foliated by complex manifolds.

In this paper, a foliation $\eta$ of an $n$-dimensional (real) manifold $M$ is a collection $\{\eta_{q}:q\in M\}$ of $k$-dimensional immersed submanifolds, where $q\in\eta_{q}$, which partitions $M$, i.e. $\eta_{p}$ and $\eta_{q}$ are either disjoint or identical for any two $p,q\in M$, and such that for any $p\in M$, there exists a neighborhood $O$ of $p$ and coordinates $\phi:O\rightarrow\mathbb{R}^{n}$ such that for any $q\in O$, the connected component of $\eta_{q}\cap O$ containing $q$ is just given by the coordinate plane $(\phi_{1}(p),\dots,\phi_{n-k}(p),\cdot,\dots,\cdot)\cap\phi(O)$.

The bundle $T\eta:=\bigcup_{q\in M}T_{q}\eta_{q}$ of tangent spaces to leaves then forms a smooth integrable distribution on $TM$. We will also consider the bundle $T^{0,1}\eta:=\bigcup_{q\in M}T^{0,1}_{q}\eta_{q}$ of CR tangent spaces to leaves, and always write $T_{q}\eta:=T_{q}\eta_{q}$ and $T_{q}^{0,1}\eta:=T_{q}^{0,1}\eta_{q}$ for simplicity.

If the rank of the Levi form of a CR manifold $M$ is constant in a neighborhood $U$ of a point $p\in M$, there exists a foliation $\eta$ of $U$ by complex manifolds, such that the Levi null space at any $q\in U$ is precisely given by the CR tangent space at $q$ to the the leaf of the foliation through $q$, henceforth denoted by $T^{0,1}_{q}\eta$. This foliation, discovered in the hypersurface case by Sommer [25] and proven to exist in general CR submanifolds by Freeman [10] is thus called the Levi foliation.

Theorem 4**.**

Let $M\subset\mathbb{C}^{N}$ be a CR manifold, and suppose that its Levi form has constant rank. Then there is a foliation $\eta$ of $U$ by complex manifolds, such that the Levi null spaces $\mathcal{N}_{q}\subseteq T_{q}^{0,1}M$ for $q\in U$ are given by $T_{q}^{0,1}\eta$.

Proof.

Let $N_{q}=\{\frac{1}{2}(\bar{L}_{q}+L_{q}),\bar{L}_{q}\in\mathcal{N}_{q}\}$. Because the rank of the Levi null space is constant across $M$, the union $N=\bigcup_{q\in M}N_{q}$ yields a smooth real distribution on $M$. By Frobenius’ theorem, integrability of this distribution is equivalent to the submodule $\Gamma_{q}(N)$ of germs of sections of $N$ at $q$ being closed under taking Lie brackets, for every $q\in M$. Since for any $\bar{L},\bar{\Gamma}\in\Gamma(T^{0,1}M)$ we have

[TABLE]

a given germ of a vector field $\frac{1}{2}(\bar{L}+L)\in\Gamma_{q}(T^{c}M)$ is a section of $N$ if and only if $[\frac{1}{2}(\bar{L}+L),\Gamma_{q}(T^{c}M)]\subseteq\Gamma_{q}(T^{c}M)$. Taking two sections $V,W\in\Gamma_{q}(N)$, we see thus that $[V,W]\subseteq[\Gamma_{q}(N),\Gamma_{q}(T^{c}M)]\subseteq\Gamma_{q}(T^{c}M)$, and by the Jacobi identity,

[TABLE]

showing that $\Gamma_{q}(N)$ is indeed closed under taking Lie brackets. Therefore, the Levi foliation exists, and since $N_{q}\subset T_{q}\mathbb{C}^{N}$ is a complex subspace for any $q\in M$, the leaves of this foliation are complex manifolds. ∎

2.2. Irregular CR maps and formal holomorphic foliations

Even though we are interested in the regularity of mappings, our results are obtained in a contrapositive way: We show that the existence of irregular maps forces some geometric property (namely, the existence of complex varieties, see 5 below). As a guiding principle, we therefore review a couple of natural instances in which irregular maps exist. We begin by considering CR functions, in a slight adaptation of [4, Theorem 2.7].

Example 1**.**

Let $M\subset\mathbb{C}^{N}$ be a strongly pseudoconvex CR hypersurface and $p\in M$. Then there exists a neighborhood $O\subseteq\mathbb{C}^{N}$ of $p$ such that for each $k\in\mathbb{N}_{\geq 1}$ there is a $C^{k}$-smooth CR function $\phi:O\cap M\rightarrow\mathbb{C}$ which is nowhere smooth on $O\cap M$.

As an immediate conseqence, there exist nowhere smooth CR maps from $M$ into $M^{\prime}$ if the target manifold $M^{\prime}$ contains a complex curve $\Gamma$. Indeed, any parametrization $t\mapsto\gamma(t)$ of $\Gamma$ is a smooth CR immersion of $\mathbb{C}$ into $M^{\prime}$, hence $\gamma\circ\phi:M\rightarrow M^{\prime}$ provides a nowhere smooth CR function of regularity $C^{k}$. We obtain another, more general set of examples from targets of the form $M^{\prime}=\hat{M}\times\mathbb{C}\subset\mathbb{C}^{N+1}$ and CR functions $\hat{h}:M\rightarrow\hat{M}$. Here, the map $(\hat{h},\phi):M\rightarrow\hat{M}\times\mathbb{C}$ is a CR map, since each of its components is a CR map, and it is nowhere smooth because $\phi$ is. In [20], Lamel and Mir prove a result in the other direction, essentially stating that near a generic point, any nowhere smooth CR map formally exhibits the structure of these latter examples.

2.3. The formal foliation theorem

Before we state the main technical theorem that we are going to use, we introduce some necessary concepts. A formal holomorphic submanifold $\Gamma$ of dimension $r$ at a point $p\in\mathbb{C}^{N^{\prime}}$ is simply a formal power series $\Gamma\in\mathbb{C}\llbracket t_{1},\dots,t_{r}\rrbracket^{N^{\prime}}$, $\Gamma=\sum_{\alpha\in\mathbb{N}^{r}}\gamma_{\alpha}t^{\alpha}$ satisfying $\gamma_{0}=p$ and $\mathrm{rk}\left(\Gamma^{\prime}(0)\right)=r$. It is tangential to infinite order to a set $S\subseteq\mathbb{C}^{N^{\prime}}$ if for any germ of a $C^{\infty}$-smooth function $\rho$ vanishing on $S$, the composition of $\Gamma$ with the Taylor series of $\rho$ at $p$ vanishes to infinite order. If $M$ is a CR manifold and $(\Gamma_{q})_{q\in M}$ is a family of such formal holomorphic submanifolds, we call this family a CR family if each of its coefficients is a CR map $M\rightarrow\mathbb{C}^{N^{\prime}}$.

It turns out that the structural property of the target which forces smoothness of CR maps is the number of different directions into which successive CR derivatives of gradients of defining functions can point. This motivates the introduction of the following numerical invariants. For a CR map $h:M\rightarrow\mathbb{C}^{N^{\prime}}$, let

[TABLE]

where we write $\mathcal{V}_{p}(M)$ for the set of germs of CR vector fields at $p$, and $\mathscr{I}_{S}(p)$ for the ideal of germs of smooth functions at $p$ which vanish on a given set $S$. The complex gradient $\rho_{w}=\left(\frac{\partial\rho}{\partial w_{1}},\dots,\frac{\partial\rho}{\partial w_{N^{\prime}}}\right)$ is considered here as a vector in $\mathbb{C}^{N^{\prime}}$. The function $q\mapsto r_{k}(q)$ is integer valued and lower semicontinuous as it is given by the rank of a collection of continuously varying vectors. Of course, $r_{k}(p)$ is only defined if $h\in C^{k}$, since $\rho_{w}\circ h$ is only as regular as $h$ is. To extract a global invariant of $h$, let $r_{k}$ be the maximum value such that $r_{k}(p)\geq r_{k}$ on a dense open subset of $M$. We are now in a position to state the formal foliation theorem of Lamel and Mir (Theorem 2.2 in [20]).

Theorem 5**.**

Let $M\subset\mathbb{C}^{N}$ be a $C^{\infty}$-smooth minimal CR submanifold, $k,l\in\mathbb{N}$ with $0\leq k\leq l\leq N^{\prime}$ and $N^{\prime}-l+k\geq 1$ be given integers and $h:M\rightarrow\mathbb{C}^{N^{\prime}}$ be a CR map of class $C^{N^{\prime}-l+k}$. Assume that $r_{k}\geq l$ and that there exists a non-empty open subset $M_{1}$ of $M$ where $h$ is nowhere $C^{\infty}$. Then there exists a dense open subset $M_{2}\subseteq M_{1}$ such that for every $p\in M_{2}$, there exists a neighborhood $V\subseteq M_{2}$ of $p$, an integer $r\geq 1$ and a $C^{1}$-smooth CR family of formal complex submanifolds $(\Gamma_{\xi})_{\xi\in V}$ of dimension $r$ through $h(V)$ for which $\Gamma_{\xi}$ is tangential to infinite order to $h(M)$ at $h(\xi)$, for every $\xi\in V$.

The rank $r$ of the family of holomorphic manifolds in the statement of this theorem merely serves as a reminder that in concrete cases, one can hope for a rank of more than one. Since there is no condition given when this might occur, for black-box applications of this theorem we will have to be satisfied with CR families of holomorphic curves with nonvanishing derivative, which can always be obtained by simply restricting $\Gamma_{q}=\sum_{\alpha\in\mathbb{N}^{r}}\gamma_{\alpha}(q)t^{\alpha}$ to $t=(t_{1},0,\dots,0)$.

Let us remark that if $h$ is not $C^{\infty}$-smooth on a dense open subset of $M$, there exists an open subset $O\subseteq M$ such that $h$ is nowhere $C^{\infty}$-smooth on $O$. The reason is simply that the set of all points $p\in M$ such that $h$ is $C^{\infty}$-smooth on a neighborhood of $p$ is open. If this set is not dense, then the complement of its closure is a non-empty open subset of $M$, where, by definition, $h$ is nowhere $C^{\infty}$-smooth.

Another interesting point to note is that while the formal complex manifolds obtained from 5 are tangential to infinite order to the image $h(M)$, infinite tangency to a non-smooth set is not nearly as strong as one might think at first sight. As a toy example, take a nowhere smooth, but $C^{1}$ function $\phi:\mathbb{R}\rightarrow\mathbb{R}$ and consider its graph $S:=\{(x,\phi(x)),x\in\mathbb{R}\}\subset\mathbb{R}^{2}$. Then any function $\rho\in C^{\infty}(\mathbb{R}^{2})$ vanishing on $S$ must already vanish to infinite order there by the following argument: If either $\rho_{x}$ or $\rho_{y}$ did not vanish at a point $(x,\phi(x))$, the implicit function theorem would yield a smooth parametrization of $S$ near that point, which does not exist. Thus both $\rho_{x}$ and $\rho_{y}$ vanish on $S$, and the argument may proceed at infinitum. The $y$-Axis is therefore tangential to infinite order to $S$ in the sense of 5, while not even being tangential to first order in the usual sense. However, if $h(M)\subseteq M^{\prime}$ for some smooth manifold $M^{\prime}$, then tangency to infinite order to $h(M)$ clearly implies tangency to infinite order to $M^{\prime}$.

To apply 5, we need $0\leq k\leq l\leq N^{\prime}$ such that $r_{k}\geq l$. It is always possible to choose $k=l=0$, but if $h$ maps $M$ into a CR submanifold $M^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$, a slight improvement holds (Lemma 6.1 in [20]).

Lemma 1**.**

Let $M\subset\mathbb{C}^{N}$ be a $C^{\infty}$-smooth CR submanifold and $h:M\rightarrow\mathbb{C}^{N^{\prime}}$ be a continuous CR map. If there exists a $C^{\infty}$-smooth CR submanifold $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ such that $h(M)\subseteq M^{\prime}$, then $r_{0}\geq N^{\prime}-n^{\prime}$, where $n^{\prime}=\dim_{CR}M^{\prime}$. In particular, if $M^{\prime}$ is maximally real, then $r_{0}=N^{\prime}$.

If it is guaranteed that enough CR directions tangential to $h(M)$ exist along which $M^{\prime}$ behaves like a Levi nondegenerate manifold, we can say more about the first derivatives of gradients, yielding a bound on $r_{1}$. We record here for later use a result similar to Lemma 6.2. in [20].

Lemma 2**.**

Consider a $C^{\infty}$-smooth CR submanifold $M\subset\mathbb{C}^{N}$, a $C^{\infty}$-smooth real hypersurface $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ and a continuously differentiable CR map $h:M\rightarrow M^{\prime}$ mapping $p\in M$ to ${p^{\prime}}\in M^{\prime}$. If $h$ is immersive at $p$ and a scalar Levi form $\mathcal{L}_{\Theta}$ of $M^{\prime}$ restricts to a nondegenerate Hermitian form on $h_{\ast}T_{p}^{0,1}M$, then $r_{1}\geq\dim_{CR}M+1$ on a neighborhood of $p$.

Proof.

Since we are in a purely local setting, we may assume that $\mathcal{L}_{\Theta}$ arises from a defining function $\rho$ of $M^{\prime}$, such that for any two CR vectors $\bar{\Gamma}=\sum_{j=1}^{N^{\prime}}\bar{\Gamma}_{j}\frac{\partial}{\partial\bar{w}_{j}}|_{p^{\prime}}$ and $\bar{L}=\sum_{k=1}^{N^{\prime}}\bar{L}_{k}\frac{\partial}{\partial\bar{w}_{k}}|_{p^{\prime}}$ we have

[TABLE]

By definition $\bar{L}\rho_{w}=\sum_{j=1}^{N^{\prime}}\bar{L}_{k}\frac{\partial^{2}\rho}{\partial w_{j}\partial\bar{w}_{k}}({p^{\prime}})$, so using the standard scalar product on $\mathbb{C}^{N^{\prime}}$ we can express $\mathcal{L}_{\Theta}(\bar{\Gamma},\bar{L})=\left((\bar{\Gamma}_{1},\dots,\bar{\Gamma}_{N^{\prime}})|\bar{L}\rho_{w}\right)_{\mathbb{C}^{N^{\prime}}}$. Nondegeneracy of the restricted Levi form on $h_{\ast}T_{p}^{0,1}M$ precisely means that the map $h_{\ast}\bar{L}\mapsto\mathcal{L}_{\Theta}(\cdot,h_{\ast}\bar{L})$ is an isomorphism of $h_{\ast}T_{p}^{0,1}M$ and the space of antilinear functionals on $h_{\ast}T_{p}^{0,1}M$. Since $h$ is immersive, $h_{\ast}$ is an isomorphism between $T_{p}^{0,1}M$ and $h_{\ast}T_{p}^{0,1}M$. The map associating to each $\bar{L}\in T_{p}^{0,1}M$ the antilinear functional $\mathcal{L}_{\Theta}(\cdot,h_{\ast}\bar{L})=\left(\hskip 1.0pt\cdot\hskip 2.0pt|\bar{L}(\rho_{w}\circ h)\right)_{\mathbb{C}^{N^{\prime}}}$ is thus an isomorphism, in particular implying that $\dim_{\mathbb{C}}\{\bar{L}(\rho_{w}\circ h):\bar{L}\in T_{p}^{0,1}M\}=\dim_{CR}M$. Furthermore, the complex gradient $\rho_{w}({p^{\prime}})$ itself is linearly independent of $\bar{L}(\rho_{w}\circ h)$ for any nonzero $\bar{L}\in T_{p^{\prime}}^{0,1}M^{\prime}$ by the following argument. For any $\bar{\Gamma}=\sum_{j=1}^{N^{\prime}}\bar{\Gamma}_{j}\frac{\partial}{\partial\bar{w}_{j}}|_{p^{\prime}}\in T_{p^{\prime}}^{0,1}M^{\prime}$, tangency implies that

[TABLE]

Thus $\rho_{w}({p^{\prime}})$ lies in the orthogonal complement of $\left\{(\bar{\Gamma}_{1},\dots,\bar{\Gamma}_{N^{\prime}}):\bar{\Gamma}\in T_{p^{\prime}}^{0,1}M^{\prime}\right\}$ while $\bar{L}(\rho_{w}\circ h)$ does not, showing linear independence. This implies $r_{1}(p)\geq\dim_{CR}M+1$ and since $r_{1}$ is lower semicontinuous and integer valued, $r_{1}\geq\dim_{CR}M+1$ holds on a neighborhood of $p$ as claimed. ∎

As we will have to treat non-immersive maps as well, let us note the following simple, but slightly clunky consequence of the previous proof.

Corollary 2**.**

If for a $C^{\infty}$-smooth CR submanifold $M\subset\mathbb{C}^{N}$, a $C^{\infty}$-smooth real hypersurface $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ and a continuously differentiable CR map $h:M\rightarrow M^{\prime}$ mapping $p\in M$ to ${p^{\prime}}\in M^{\prime}$ there exists a CR submanifold $S\subset M$ containing $p$, such that the restricted map $h|_{S}$ satisfies the hypothesis of 2, then $r_{1}\geq\dim_{CR}S+1$ on a neighborhood of $p$ in $M$.

Proof.

Take a basis $(\bar{L}_{j})_{j=1}^{\dim_{CR}S}$ of $T^{0,1}_{p}S$. Retracing the proof of 2, we see that for any defining function $\rho$ of $M$, the vectors $\rho_{w}(p^{\prime})$ and $\bar{L}_{j}(\rho_{w}\circ h),1\leq j\leq\dim_{CR}S$ are linearly independent, hence $r_{1}(p)\geq\dim_{CR}S+1$. But $r_{1}(q)$ is lower semicontinuous in $q$ on $M$, hence $r_{1}\geq\dim_{CR}S+1$ in a neighborhood of $p$ as claimed. ∎

3. The invariant $\nu$ and the proof of 1

As an example of a hypersurface foliated by complex manifolds, where an unconditional regularity result must necessarily fail, Lamel and Mir consider the tube over the light cone $M^{\prime}:=\{(z_{1},\dots,z_{N^{\prime}-1},z_{N^{\prime}}):\Re(z_{1})^{2}+\dots+\Re(z_{N^{\prime}-1})^{2}=\Re(z_{N^{\prime}})^{2},\ z_{N^{\prime}}\neq 0\}$. They obtain the following result (Corollary 2.6 in [20]).

Theorem 6**.**

Let $M\subset\mathbb{C}^{N}$ be a $C^{\infty}$-smooth minimal CR submanifold and $M^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ be the tube over the light cone. Then every CR map $h:M\rightarrow M^{\prime}$, of class $C^{N^{\prime}-1}$ and of rank $\geq 3$, is $C^{\infty}$-smooth on a dense open subset of $M$.

The proof given in [20] and [19] makes quite ingenious use of the simple structure of $M^{\prime}$, and is thus not easily adaptable to more general settings. In this section, we shall carefully define the invariant $\nu$ mentioned in the introduction (5), and show how it can be used to generalize the observation of 6 to the more general situation of pseudoconvex hypersurfaces whose Levi form is of constant rank. We will later see that this class of examples covers not only the tube over the light cone, but also the smooth part of the boundary of all classical irreducible symmetric domains. Mappings into such targets will be discussed in section 6.

3.1. Maps into uniformly pseudoconvex hypersurfaces

In view of the Levi foliation (4), 5 might allow for nowhere smooth maps into a uniformly pseudoconvex hypersurface, since there at least exist complex manifolds tangential to infinite order to the target manifold, contrary to the simpler case of manifolds of D’Angelo finite type. Indeed, any formal complex manifold tangential to infinite order to $M^{\prime}$ is necessarily tangential to the Levi foliation.

Lemma 3**.**

Let $M^{\prime}\subset\mathbb{C}^{N^{\prime}}$ be a uniformly pseudoconvex hypersurface with its Levi foliation $\eta$, and let $p^{\prime}\in M^{\prime}$. Suppose there exists a formal complex curve $\Gamma={p^{\prime}}+t\gamma_{t}+\frac{t^{2}}{2}\gamma_{tt}+\dots$ tangential to second order to $M^{\prime}$ at ${p^{\prime}}$. Then $\gamma_{t}\in T_{p^{\prime}}\eta$.

Proof.

A formal complex curve $\Gamma={p^{\prime}}+t\gamma_{t}+\frac{t^{2}}{2}\gamma_{tt}+\dots$ is tangential to second order to $M^{\prime}$ if and only if the curve $\tilde{\gamma}(t)={p^{\prime}}+t\gamma_{t}+\frac{t^{2}}{2}\gamma_{tt}$ arising from the truncated power series is. Choosing a positive semidefinite scalar Levi form $\mathcal{L}_{\Theta}$ arising from a defining function $\rho$, we have that $\mathcal{L}_{\Theta}(\frac{1}{2}(\gamma_{t}+iJ\gamma_{t}),\frac{1}{2}(\gamma_{t}+iJ\gamma_{t}))=\frac{d^{2}}{dtd\bar{t}}|_{t=0}\rho\circ\tilde{\gamma}=0$, since $\rho\circ\tilde{\gamma}$ vanishes to second order. But by 4, the null space of $\mathcal{L}_{\Theta}|_{p^{\prime}}$ is given by $T^{0,1}_{p^{\prime}}\eta$, implying that $\gamma_{t}\in T_{p^{\prime}}\eta$. ∎

Our main technical tool will be a tensorial quantity measuring obstructions to the existence of CR sections of $T\eta$. We denote by $T^{\bot}\eta:=\bigcup_{p\in M^{\prime}}(T_{p}\eta_{p})^{\bot}$ the bundle of orthogonal complements in $T\mathbb{C}^{N^{\prime}}$ of tangent spaces to leaves.

Lemma 4**.**

Let $M^{\prime}$ be a manifold endowed with a foliation $\eta$ by complex manifolds. There exists a tensor field $R\in\mathcal{V}(M^{\prime})^{\ast}\otimes\Gamma(T\eta)^{\ast}\otimes\Gamma(T^{\bot}\eta)$ such that for every $\bar{L}\in\mathcal{V}(M^{\prime})$ and $\psi\in\Gamma(T\eta)$, we have $\mathbb{P}_{T^{\bot}_{p}\eta_{p}}(\bar{L}|_{p}\psi)=R_{p}(\bar{L}|_{p},\psi|_{p})$. For any $V_{p}\in T_{p}\eta_{p}$, the kernel of $R_{p}(\cdot,V_{p})$ contains $T^{0,1}_{p}\eta_{p}$.

Proof.

Define $R(\bar{L},\psi)=\mathbb{P}_{T^{\bot}\eta}(\bar{L}\psi)$. Evidently, $R$ is $C^{1}$-linear in the first slot, as directional derivatives always are. For two sections $V$ and $W$ of $T\eta$ and $f\in C^{1}(M)$, we have that $\bar{L}|_{p}(V+fW)=\bar{L}|_{p}V+f\bar{L}|_{p}W+(\bar{L}|_{p}f)W$. The last term is canceled by the projection onto $T^{\bot}_{p}\eta_{p}$, thus $R$ is also $C^{1}$-linear in the second slot, hence $R$ is a tensor.

Consider now $V_{p}\in T_{p}\eta_{p}$. We may construct a section $V\in\Gamma(T\eta)$ satisfying $V|_{p}=V_{p}$, which is holomorphic on $\eta_{p}$ and smooth on $M^{\prime}$. First, we choose a holomorphic parametrization $\phi$ for $\eta_{p}$, extend $\phi^{-1}_{\ast}V_{p}$ to a constant vector field $\tilde{V}$ and note that $\phi_{\ast}\tilde{V}$ is holomorphic, since $D\phi$ has holomorphic components and $\tilde{V}$ is constant. To obtain a vector field, we then simply extend the result smoothly to $M^{\prime}$. But now, $\bar{L}|_{p}V=0$ if $\bar{L}\in\Gamma(T^{0,1}\eta)$, since $V$ is holomorphic on $\eta_{p}$ and $\bar{L}|_{p}$ only takes derivatives along $\eta$. Therefore, $T_{p}^{0,1}\eta\subseteq\ker R_{p}(\cdot,V_{p})$. ∎

Next we need to deal with the issue that the CR map $h$ of interest is neither assumed to be immersive nor $C^{\infty}$-smooth. After carefully verifying that nothing goes wrong, we will obtain the following result on obstructions to the existence of nowhere smooth CR maps.

Proposition 1**.**

Consider a uniformly pseudoconvex hypersurface $M^{\prime}$ with its Levi foliation $\eta$, a CR manifold $M$ and a $C^{1}$-smooth CR map $h:M\rightarrow M^{\prime}$ mapping a point $p\in M$ to $p^{\prime}\in M^{\prime}$. Suppose there exists a $C^{1}$-smooth CR family of formal complex curves $\left(\Gamma_{q}\right)_{q\in O}$ defined on a neighborhood $O\subseteq M$ of $p$ such that $\Gamma_{q}$ is tangential to second order to $M^{\prime}$ at $h(q)$ for each $q\in O$. Then $\gamma_{t}(p)\in T_{p^{\prime}}\eta_{p^{\prime}}$, and $h_{\ast}T^{0,1}_{p}M\subseteq\ker R_{p^{\prime}}(\cdot,\gamma_{t}(p))$.

Proof.

By Lemma 3, we know that at each point $q\in O$, $\gamma_{t}(q)\in T_{h(q)}\eta_{h(q)}$, since $\Gamma_{q}$ is a formal complex curve tangential to second order to $M^{\prime}$ at $h(q)$.

Consider now $\bar{L}\in\mathcal{V}(M)$ such that $h_{\ast}\bar{L}|_{p}\neq 0$. Choosing a two-dimensional real submanifold $S\subseteq O$ such that $\bar{L}|_{p}$ is tangential to $S$, the derivative of $h|_{S}$ has full rank at $p$, and hence $h|_{S}$ is a local embedding around $p$. We may thus extend $\gamma_{t}\circ h^{-1}|_{h(S)}$, defined on $h(S)$, to a section $\tilde{\gamma}_{t}\in\Gamma(T\eta)$ defined on an open neighbourhood of ${p^{\prime}}$. Since $\gamma_{t}$ and $\tilde{\gamma}_{t}\circ h$ agree on $S$ and $\bar{L}_{p}$ only takes derivatives along $S$, it follows that

[TABLE]

implying that $h_{\ast}\bar{L}|_{p}\in\ker R_{p^{\prime}}(\cdot,\gamma_{t}(p))$. ∎

In order to apply 5 to our situation, we are going to use the numerical quantity $\nu$ already mentioned in the introduction, which measures the size of $\ker R(\cdot,V)$, as well as a method of computing it.

Lemma 5**.**

Let $M^{\prime}$ be a uniformly pseudoconvex hypersurface with Levi foliation $\eta$. For $p^{\prime}\in M^{\prime}$, we define

[TABLE]

Then $\nu$ is a biholomorphic invariant, and the function $q\rightarrow\nu_{q}$ is nonnegative, integer valued and upper semicontinous.

Proof.

For biholomorphism invariance, let $H:\mathbb{C}^{N^{\prime}}\rightarrow\mathbb{C}^{N^{\prime}}$ be a local biholomorphism near $p^{\prime}$, and consider $H(M^{\prime})$. Its Levi foliation is $H(\eta)$. For all $\bar{L}\in\mathcal{V}(M^{\prime})$ and $V\in\Gamma(T\eta)$, we have $(H_{\ast}\bar{L})(H_{\ast}V)=H_{\ast}(\bar{L}V)$ as the Jacobian of $H$ is holomorphic again and thus commutes with taking antiholomorphic derivatives. Furthermore, projecting $H_{\ast}(T\eta^{\bot})$ onto $TH(\eta)^{\bot}$ is an isomorphism as $H_{\ast}(T\eta^{\bot})$ is another complement to $H_{\ast}(T\eta)=TH(\eta)$. Therefore $R$ transforms under biholomorphism via pre- and postcomposition with linear isomorphisms, so $\nu$ is invariant. Here we chose $T\eta^{\bot}$ instead of the more natural, isomorphic bundle $T\mathbb{C}^{N^{\prime}}/T\eta$ because it simplifies later calculations.

Clearly $\nu_{q}$ is integer-valued, and since $T_{q}^{0,1}\eta\subseteq\ker R(\cdot,V)$, it is nonnegative. Let $\dim_{\mathbb{C}}\eta=:K$. To show upper semicontinuity, we need to show that for all ${p^{\prime}}$ in $M^{\prime}$ and $l\in\mathbb{N}$, $\nu_{p^{\prime}}\leq l$ implies $\nu_{q}\leq l$ for all $q$ in a neighborhood of ${p^{\prime}}$. We observe first that for $l\in\mathbb{N}$, $\dim_{\mathbb{C}}\ker R_{p^{\prime}}(\cdot,V)\leq K+l$ if and only if $\mathrm{rk}R(\cdot,V)\geq N^{\prime}-1-K-l$. This condition is equivalent to some $(N^{\prime}{-}1{-}K{-}l)$-minor of the matrix representation of $R(\cdot,V)$ with respect to a choice of smooth local frames of $T^{0,1}M^{\prime}$ and $T^{\bot}\eta$ being nonzero. By homogeneity, we have $\nu_{p^{\prime}}\leq l$ if and only if for each $V\in\mathbb{S}^{2K-1}\subseteq T_{p^{\prime}}$, a (possibly different) such minor of $R(\cdot,V)$ does not vanish. Hence, $\nu_{p^{\prime}}\leq K+l$ if and only if the square sum $\sum_{j}|m_{j}|^{2}$ over all such minors does not vanish on $\{{p^{\prime}}\}{\times}\mathbb{S}^{2K-1}$. By compactness of the sphere, there is a neighborhood $O$ of ${p^{\prime}}$ such that $\sum_{j}|m_{j}|^{2}$ does not vanish on $O{\times}\mathbb{S}^{2K-1}$, showing that $\nu_{q}\leq l$ on $O$, which implies upper semicontinuity. ∎

To calculate $\nu_{p^{\prime}}$, the following setup will be helpful.

Lemma 6**.**

Let $M^{\prime}$ be a pseudoconvex hypersurface foliated by complex manifolds of dimension $K$. Consider a point ${p^{\prime}}\in M^{\prime}$ and an $(N^{\prime}{-}K)$-dimensional complex manifold $\Sigma$ through ${p^{\prime}}$ such that $T_{p^{\prime}}\eta\oplus T_{p^{\prime}}\Sigma=T_{p^{\prime}}\mathbb{C}^{N}$. If $S:=M^{\prime}\cap\Sigma$ is strongly pseudoconvex, then the Levi form of $M^{\prime}$ has exactly $K$ zero eigenvalues on an open neighborhood of ${p^{\prime}}$, making $M^{\prime}$ a uniformly pseudoconvex hypersurface and $\eta$ its Levi foliation.

Furthermore, as in Lemma 4, the map $R^{S}\in\mathcal{V}(S)^{\ast}\otimes\Gamma(T\eta|_{S})^{\ast}\otimes\Gamma(T^{\bot}\eta|_{S})$ given by $R^{S}(\bar{L},V)=\mathbb{P}_{T^{\bot}\eta}(\bar{L}V)$ is a tensor, and $\max_{0\neq V\in T_{p^{\prime}}\eta}\dim_{\mathbb{C}}\ker R_{p^{\prime}}^{S}(\cdot,V)=\nu_{p^{\prime}}$.

Proof.

Let $\Theta$ be a characteristic form on $M^{\prime}$ such that the respective scalar Levi form $\mathcal{L}_{\Theta}(\bar{L}_{1},\bar{L}_{2})=\frac{1}{2i}\Theta([L_{1},\bar{L}_{2}])$ is positive semidefinite. If $S$ is strongly pseudoconvex at ${p^{\prime}}$, then $\mathcal{L}_{\Theta}|_{T^{0,1}_{p^{\prime}}S}$ is strictly positive definite, hence, by elementary linear algebra, $\mathcal{L}_{\Theta}$ has at least $N^{\prime}-1-K$ positive eigenvalues and as the $K$-dimensional leaf $\eta_{p^{\prime}}$ through ${p^{\prime}}$ is a complex manifold and therefore Levi-flat, the other $K$ eigenvalues have to be zero. Consider now $R_{p^{\prime}}(\bar{L},V)$ for $\bar{L}\in\mathcal{V}(M)$ and $V\in T_{p^{\prime}}\eta_{p^{\prime}}$. Decompose $\bar{L}|_{p^{\prime}}=U+W$ for $U\in T^{0,1}_{p^{\prime}}S_{p^{\prime}}$ and $W\in T^{0,1}_{p^{\prime}}\eta_{p^{\prime}}$. As proven in Lemma 4, $R_{p^{\prime}}(W,V)=0$, hence $R_{p^{\prime}}(\bar{L},V)=0$ iff $R_{p^{\prime}}(U,V)=R^{S}_{p^{\prime}}(U,V)=0$. This implies that $\ker R_{p^{\prime}}(\cdot,V)=\ker R^{S}_{p^{\prime}}(\cdot,V)\oplus T^{0,1}_{p^{\prime}}\eta_{p^{\prime}}$, proving the second claim. ∎

3.2. The case $\nu=0$

If the kernel of $R$ is of minimal dimension even at a single point, Proposition 1 implies 1, fully generalizing the result on the tube over the light cone; before we give the proof, we first show that the tube over the light cone satisfies $\nu=0$.

Example 2**.**

Let $M^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ be the tube over the light cone. It is foliated by complex lines, at any point $p^{\prime}\in M^{\prime}$ with $\Re({p^{\prime}})\neq 0$ the Levi form of $M^{\prime}$ has exactly one zero eigenvalue, and $\nu_{p^{\prime}}=0$.

Proof.

Recall that the tube over the light cone is defined as the set of points $z\in\mathbb{C}^{N^{\prime}}$ such that $\Re(z_{1})^{2}+\dots\Re(z_{N^{\prime}-1})^{2}=\Re(z_{N^{\prime}})^{2}$. It is a smooth real hypersurface where $\Re(z_{N^{\prime}})\neq 0$, and foliated by complex lines $q+t\left(\Re(q_{1}),\dots,\Re(q_{N^{\prime}-1}),\Re(q_{N^{\prime}})\right)$, $q\in M^{\prime}$. Indeed, let us check that

[TABLE]

The hypersurface $M^{\prime}$ is pseudoconvex, since the tube over the interior of the light cone is convex. The hypersurface $\Sigma=\{z\in\mathbb{C}^{N^{\prime}}:z_{N^{\prime}}=p^{\prime}_{N^{\prime}}\}$ through ${p^{\prime}}\in M^{\prime}$ is transversal to $\eta_{p^{\prime}}$ and intersects $M^{\prime}$ in $S=\{z\in\mathbb{C}^{N^{\prime}}:z_{N^{\prime}}=p^{\prime}_{N^{\prime}},\Re(z_{1})^{2}+\dots+\Re(z_{N^{\prime}-1})^{2}=\Re(p^{\prime}_{N^{\prime}})^{2}\}$, which is a strongly pseudoconvex CR submanifold of $\Sigma$ because it is a tube over a strongly convex real manifold. To obtain the setup of Lemma 6, it now suffices to calculate $R^{S}_{p^{\prime}}(\bar{L}|_{p^{\prime}},V|_{p^{\prime}})$ for a single section $V$ of $T\eta$ (since $T\eta$ is one-dimensional). Take $V(q)=\left(\Re(q_{1}),\dots,\Re(q_{N^{\prime}-1}),\Re(q_{N^{\prime}})\right)$ for $q\in S$, and consider a CR vector $\bar{L}|_{p^{\prime}}\in T^{0,1}_{p^{\prime}}S$. Since $\bar{L}|_{p^{\prime}}\Re(q_{N^{\prime}})=0$, $L|_{p^{\prime}}V\in T_{p^{\prime}}\Sigma$, and because $T_{p^{\prime}}\Sigma$ and $T_{p^{\prime}}\eta$ lie in general position, $R_{p^{\prime}}^{S}(\bar{L}|_{p^{\prime}},V|_{p^{\prime}})=0$ if and only if $\bar{L}|_{p^{\prime}}V=0$, i.e. $\bar{L}|_{p^{\prime}}(\Re(q_{j}))=0$ for $j=1,\dots,N^{\prime}$. But since $\bar{L}_{p^{\prime}}q_{j}=0$, this is the case if and only if $\bar{L}_{p^{\prime}}\bar{q}_{j}=0$ as well, hence $\bar{L}|_{p^{\prime}}\in T_{p^{\prime}}^{0,1}S\cap T_{p^{\prime}}^{1,0}S=\{0\}$, which proves that $\nu_{p^{\prime}}=0$. ∎

Proof of 1.

If $h$ is not generically smooth, there exists an open set $U_{0}\subseteq M$ where $h$ is nowhere smooth. 1 always yields $r_{0}\geq 1$, thus we may apply 5, with $k=0$ and $l=1$, to obtain a point $p\in M$, a neighborhood $U\subseteq U_{0}$ of $p$ and a continuously differentiable CR family of formal complex curves $(\Gamma_{\xi})_{\xi\in U}$ such that $\Gamma_{\xi}$ is tangential to $M^{\prime}$ to infinite order at $h(\xi)$. For any $q\in U$ we have $\nu_{h(q)}=0$, i.e. $\ker R(\cdot,\gamma_{t}(q))=T^{0,1}_{h(q)}\eta$, hence by 1 we find $h_{\ast}T^{0,1}_{q}M\subseteq T^{0,1}_{h(q)}\eta$.

Near points $q\in U$ where $h$ is regular enough, this means that $h^{-1}(\eta_{h(q)})$ integrates the complex tangent bundle and thus, by minimality of $M$, has to contain an open neighborhood of $q$. First, take a small open set $O^{\prime}\subseteq h(U)$ where coordinates adapted to the foliation may be chosen, and hence the restricted foliation $\eta|_{O^{\prime}}$ is equipped with a manifold structure. Let $\pi:O^{\prime}\rightarrow\eta|_{O^{\prime}}$ denote the projection onto the foliation, given by $\pi(q)=\eta_{q}\cap O^{\prime}$. Since the rank of a continuously differentiable map is lower semicontinuous, we can find an open subset $\tilde{U}\subseteq h^{-1}(O^{\prime})$ where $\pi\circ h$ is of constant rank. By the rank theorem, $h^{-1}\circ\pi^{-1}(\eta_{h(q)})=h^{-1}(\eta_{h(q)})=:E_{q}$ is a submanifold of $\tilde{U}$, and $V\in T_{q}M$ satisfies $(\pi\circ h)_{\ast}V=0$ if and only if $V\in T_{q}E_{q}$. But as $h_{\ast}T_{q}^{0,1}M\subseteq T^{0,1}_{h(q)}\eta$, we infer that $T_{q}^{0,1}M\subseteq\mathbb{C}T_{q}E_{q}$ for any $q\in\tilde{U}$, which by minimality implies that $E_{q}$ is an open neighborhood of $q$ in $\tilde{U}$ already.

We have thus shown that an nonempty open subset of $U$ is mapped into a single leaf $\eta_{h(q)}$ of the foliation of $M^{\prime}$. It is left to prove that actually, all of $M$ will be mapped into $\eta_{h(q)}$. Let $\mathcal{K}$ denote the closure of the set of all points $q\in M$ which possess an open neighborhood that is mapped entirely into $\eta_{h(q)}$. We will show that $\mathcal{K}$ is open. For $p\in\mathcal{K}$, take a neighborhood $O^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ of $h(p)$ where the connected component of $\eta_{h(q)}\cap O^{\prime}$ containing $h(p)$ is given as the vanishing set of a holomorphic map $\pi:O^{\prime}\rightarrow\mathbb{C}^{K}$. Then $\pi\circ h:h^{-1}(O^{\prime})\rightarrow\mathbb{C}^{K}$ is a CR map, and since $p\in\mathcal{K}$, $\pi\circ h$ must vanish on open sets arbitrarily close to $p$. But as any CR function on a connected minimal submanifold which vanishes on an open subset already vanishes identically (a consequence e.g. of [3, Theorem III.3.13]), all of $h^{-1}(O^{\prime})$ must be mapped entirely into $\eta_{h(q)}$ and thus $p$ lies in the interior of $\mathcal{K}$. Now, $\mathcal{K}$ is both open and closed, and thus connectedness of $M$ implies $\mathcal{K}=M$. ∎

4. CR transversal maps and

proof of 2

If we want to treat positive $\nu_{p^{\prime}}$, we have to assume more about the map and the source manifold; for example, if $M^{\prime}=M\times\mathbb{C}$ for a strictly pseudoconvex $M\subset\mathbb{C}^{N}$, then $\nu=N-1$ everywhere, and our usual example $z\mapsto(z,\varphi(z))$ for some finitely smooth but nonsmooth CR function $\varphi$ will yield nonsmooth CR maps.

The approach we take here is based on the fact that if $h_{\ast}T_{p}^{0,1}M$ and $T_{p^{\prime}}\eta_{p^{\prime}}$ intersect trivially, then we can allow $\nu_{p^{\prime}}$ to be greater than zero provided that $h_{\ast}T_{p}^{0,1}M$ has enough dimensions. In particular, this occurs if $M$ is a uniformly pseudoconvex hypersurface with sufficiently many positive Levi eigenvalues, and $h$ satisfies a commonly considered nondegeneracy condition, that of CR-transversality.

Definition 1** (CR-transversality).**

A CR map $h:M\rightarrow M^{\prime}$ between hypersurfaces $M$ and $M^{\prime}$ is called CR-transversal at $p\in M$ if $T_{h(p)}^{0,1}M^{\prime}+T_{h(p)}^{1,0}M^{\prime}+h_{\ast}\mathbb{C}T_{p}M=\mathbb{C}T_{h(p)}M^{\prime}$.

If $M$ is actually strongly pseudoconvex and $h:M\rightarrow M^{\prime}$ is CR-transversal, then $h_{\ast}T_{p}^{0,1}M$ has maximal dimensions and intersects $T^{0,1}_{p^{\prime}}\eta_{p^{\prime}}$ trivially. This is very well known and a key component in the proof of many regularity results (see e.g. [29]. We summarize for later use the following statement:

Lemma 7**.**

Consider a pseudoconvex hypersurface $M^{\prime}$, a strongly pseudoconvex hypersurface $M$ and a $C^{2}$-smooth CR-transversal CR map $h:M\rightarrow M^{\prime}$ mapping $p\in M$ to $p^{\prime}\in M^{\prime}$. Then $h$ is an immersion, $\dim h_{\ast}T_{p}^{0,1}M=\dim_{CR}M$, and $h_{\ast}T_{p}^{0,1}M\cap\mathcal{N}_{p^{\prime}}=\{0\}$, where $\mathcal{N}_{p^{\prime}}\subseteq T_{p^{\prime}}^{0,1}M^{\prime}$ denotes the null space of the Levi form of $M^{\prime}$ at ${p^{\prime}}$.

With this fact in mind, it is clear that strict pseudoconvexity of $M$ and CR-transversality of $h$ come together to imply that there are many CR directions available along $h(M)$ where some obstructions to the existence of CR families of infinitely tangential formal complex curves, encoded in $R$, might exist. If $M$ is just pseudoconvex, these CR directions may be obtained by considering a strongly pseudoconvex slice of $M$ of maximal dimension; this observation is the basis of 2.

Proof of 2.

Take an open neighborhood $U\subseteq M^{\prime}$ of ${p^{\prime}}$ where $\nu_{q^{\prime}}<n_{+}$ for all $q^{\prime}\in U$. Let $\Sigma\subseteq\mathbb{C}^{N}$ be a complex manifold of dimension $1+n_{+}$ such that $T_{p}\Sigma+T^{c}_{p}M=T_{p}\mathbb{C}^{N}$, and such that the Levi form of $M$ is positive definite when restricted to $T_{p}^{0,1}M\cap T_{p}^{0,1}\Sigma$. Since both transversality and positive definiteness are open conditions, after possibly shrinking $U$, the intersection $S:=\Sigma\cap M\cap h^{-1}(U)$ is a strongly pseudoconvex CR submanifold of $M$ with $n_{+}=\dim_{CR}S$. Note that $T_{q}S$ for $q\in S$ always contains a transversal tangent vector, hence the restricted map $h|_{S}$ is CR transversal if and only if $h$ is.

By 7, $h|_{S}$ is an immersion, $\dim h_{\ast}T_{q}^{0,1}S=n_{+}$ and $h_{\ast}T_{q}^{0,1}S\cap T_{h(q)}^{0,1}\eta=\{0\}$ for any $q\in S$, where $\eta$ denotes the Levi foliation of $M^{\prime}$, and thus $T^{0,1}\eta$ is the Levi null space. This is precisely the situation of 2, hence $r_{1}\geq 1+\dim_{CR}S=1+n_{+}$, after possibly restricting $U$ again.

Assume now that $h$ was nowhere smooth on an open set $\tilde{O}\subset h^{-1}(U)$. Then 5, with $k=1$, $l=1+n_{+}$, would yield $q\in\tilde{O}$ (mapped to $q^{\prime}\in M^{\prime}$) and a continuously differentiable CR family of formal complex curves $(\Gamma_{\xi})_{\xi\in\tilde{O}_{1}}$ defined on a neighborhood $\tilde{O}_{1}\subseteq\tilde{O}$ of $q$. But now 1 implies $h_{\ast}T_{q}^{0,1}M\subseteq\ker R_{q^{\prime}}(\cdot,\gamma_{t}(q))$, thus we find that

[TABLE]

which contradicts the assumption $\nu_{q^{\prime}}<n_{+}$. Therefore $h$ must be $C^{\infty}$-smooth on a dense open subset of $O:=h^{-1}(U)$. ∎

4.1. Connection to $2$-nondegeneracy

In this section, we discuss the relationship of the invariant $\nu$ to finite nondegeneracy.

We first recall that a CR submanifold $M\in\mathbb{C}^{N}$ is called $k$-nondegenerate, for $k\in\mathbb{N}$, at a point $p\in M$ if

[TABLE]

equivalently, its identity map satisfies $r_{k}(p)=N$, where the $r_{k}$ are defined in section 2.3.

It turns out that $1$-nondegeneracy is equivalent to Levi-nondegeneracy, by essentially the same argument as in the proof of 2 in section 3.1.

For a pseudoconvex hypersurface which is not strictly pseudoconvex, the next step is $2$-nondegeneracy.

Proposition 2**.**

Let $M\subset\mathbb{C}^{N}$ be a uniformly Levi-degenerate hypersurface with $N-K-1$ nonzero Levi eigenvalues and Levi foliation $\eta$, and for $p\in M$, consider $\nu_{p}$ as in 5. Then the following are equivalent:

• •

$M$* is $2$-nondegenerate at $p$,*

• •

There exists no germ of a section $V\in\Gamma_{p}(T^{1,0}\eta)$ such that $[\bar{L}_{1},[\bar{L}_{2},V]]\rvert_{p}\in T_{p}^{1,0}M\oplus T_{p}^{0,1}M$ for all $\bar{L}_{1},\bar{L}_{2}\in\Gamma_{p}(T^{0,1}M)$,

• •

$\nu_{p}<N-K-1$, i.e. $\nu_{p}$ is not maximal.

Proof.

First, we note that by the product rule and the fact that two smooth defining functions of hypersurfaces differ by a smooth nonzero factor, it is sufficient to consider a single defining function $\rho$. To work in a covariant setting, we consider $T^{\prime}M=\left<dz_{1}|_{M},\dots,dz_{N}|_{M}\right>\subset\mathbb{C}T^{\ast}M$, the space of $(1,0)$-forms on $M$.

Let the Lie derivative of a $(1,0)$-form $\omega$ with respect to a CR vector field $\bar{L}$ be denoted by $\mathcal{T}_{\bar{L}}\omega$. By suitably extending $\bar{L}$ and $\omega$ to a neighborhood of $p$ in $\mathbb{C}^{N}$, we compute

[TABLE]

using Cartan’s magic formula and the fact that $\omega$ annihilates the $0,1$-vector fields $\bar{L}$ and $[\bar{L},\frac{\partial}{\partial z_{j}}]$. Hence, in this setting, taking Lie derivatives means just taking derivatives component-wise.

For a defining function $\rho$, consider now $i\partial\rho=i\sum_{j}\rho_{z_{j}}dz_{j}$. This form differs from the real contact form $\Theta=i(\partial\rho-\bar{\partial}\rho)$ only by a multiple of $d\rho$, which vanishes along $M$. By the previous calculation, $M$ is $2$-nondegenerate if and only if $\partial\rho$, $\mathcal{T}_{\bar{L}_{1}}\rho$ and $\mathcal{T}_{\bar{L}_{2}}\mathcal{T}_{\bar{L}_{1}}\rho$ together span all of $T_{p}^{\prime}M$, where $\bar{L}_{1}$ and $\bar{L}_{2}$ range across all germs of CR vector fields at $p$.

The hypersurface $M$ is $2$-degenerate at $p$ if and only if there exists a nonzero vector $V\in\mathbb{C}T_{p}M$ such that $\partial\rho(V)=0$, $(\mathcal{T}_{\bar{L}_{1}}\partial\rho)(V)=0$ and $(\mathcal{T}_{\bar{L}_{2}}\mathcal{T}_{\bar{L}_{1}}\partial\rho)(V)=0$. The first condition ensures that $V\in T_{p}^{1,0}M+T_{p}^{0,1}M$, and since $T^{\prime}M$ annihilates $T^{0,1}M$, we may assume without loss of generality that $V\in T_{p}^{1,0}M$. Next, we calculate

[TABLE]

for all $\bar{L}_{1}\in T^{0,1}M$, hence $V\in T^{1,0}\eta$. We extend $V$ to a local section of $T^{1,0}\eta$. Then, the third condition yields

[TABLE]

Thus, $M$ is $2$-degenerate at $p$ if and only if there exists a section $V\in\Gamma_{p}(T^{1,0}\eta)$ such that $[\bar{L}_{1},[\bar{L}_{2},V]]\rvert_{p}\in T_{p}^{1,0}M\oplus T_{p}^{0,1}M$, for all $\bar{L}_{1},\bar{L}_{2}\in\Gamma_{p}(T^{0,1}M)$.

Writing $V=\sum_{j}V^{j}\frac{\partial}{\partial z_{j}}$ and $\bar{L}_{2}=\sum_{j}\bar{L}_{2}^{j}\frac{\partial}{\partial\bar{z}_{j}}$, we find that $[\bar{L}_{2},V]=\sum_{j}(\bar{L}_{2}V^{j})\frac{\partial}{\partial z_{j}}-\sum_{j}(V\bar{L}_{2}^{j})\frac{\partial}{\partial\bar{z}_{j}}$. As $[\bar{L}_{1},[\bar{L}_{2},V]]\in T_{p}^{1,0}M\oplus T_{p}^{0,1}M$ for any $\bar{L}_{1}\in\mathcal{V}_{p}$, the $(1,0)$-part of $[\bar{L}_{2},V]$ must lie in the Levi null space of $M$, i.e. $\sum_{j}(\bar{L}_{2}V^{j})\frac{\partial}{\partial z_{j}}|_{p}\in T_{p}^{1,0}\eta$. Almost tautologically, this is the case if and only if $V_{r}:=(V^{1},\dots,V^{j})\in T\eta\subseteq T\mathbb{C}^{N}$ satisfies $\bar{L}_{2}V_{r}\in T\eta$ for all CR vector fields $\bar{L}_{2}$, i.e. if and only if $\nu_{p}=N-K-1$. ∎

The calculation of $\nu$ for boundaries of classical symmetric domains in section 6 thus also provides a way of concluding that these hypersurfaces are $2$-nondegenerate. In [29], Xiao proves that a merely $C^{2}$-smooth CR map from a strongly pseudoconvex hypersurface with $n_{+}$ positive Levi eigenvalues into a $2$-nondegenerate uniformly pseudoconvex hypersurface of precisely $n_{+}$ positive Levi eigenvalues must be $C^{\infty}$-smooth everywhere. The point is that, under these conditions on the Levi eigenvalues, the image of the source manifold already contains all relevant vector fields to conclude that the map itself is $2$-nondegenerate in the sense of Lamel [18], and thus as regular as the source and target manifolds themselves.

The invariant $\nu_{p}$ contains more subtle information than mere $2$-nondegeneracy of the target manifold, as is evinced by the sharp bounds on the number of Levi eigenvalues of the source manifold achieved in 3. One could hope that the condition $\nu_{p^{\prime}}<n_{+}$ from 2 already suffices to conclude that the CR map $h$ itself is $2$-nondegenerate (at least on a dense open subset), but this very likely true only in Xiao’s special case. It is not at all easy to find interesting examples of this behaviour, as everywhere finitely nondegenerate, pseudoconvex hypersurfaces of Levi number exceeding $2$ are extremely scarce and notoriously hard to construct; we refer the reader to the discussion in Baouendi, Ebenfelt, and Zaitsev’s paper [2].

5. Applications to holomorphic maps

In this section, we give the proof of 1 in which we apply our results on CR transversal CR maps between smooth hypersurfaces in complex Euclidean space to holomorphic maps which extend to CR maps between the smooth part of their source and target domains’ boundaries. Before presenting the proof, we need to collect some preliminary results. It is a well known fact that such holomorphic maps give rise to CR transversal boundary maps, as long as the target domain satisfies a suitable convexity condition (cf. [9] for strongly pseudoconvex or [29] for convex target domains). Indeed, mere pseudoconvexity of the target suffices to guarantee CR transversality of the boundary map.

Proposition 3**.**

Let $\Omega\subseteq\mathbb{C}^{N}$, $\Omega^{\prime}\subseteq\mathbb{C}^{N^{\prime}}$ be domains and let $M\subseteq\partial\Omega$, $M^{\prime}\subseteq\partial\Omega^{\prime}$ be smooth real hypersurfaces contained in the smooth parts of $\partial\Omega$ and $\partial\Omega^{\prime}$, respectively. Suppose that $\Omega^{\prime}$ is pseudoconvex at $M^{\prime}$.

Then any holomorphic map $F:\Omega\rightarrow\Omega^{\prime}$ which extends to a map of regularity $C^{1,\epsilon}$ on $\Omega\cup M$ and maps $M$ into $M^{\prime}$ is CR transversal along $M$.

This proposition as well as its proof parallel Proposition 9.10.5. in [1], but as the latter result is only stated for equidimensional mappings with smooth boundary extension, a proof for 3 shall nevertheless be presented. The proof hinges on the following observation by Diederich & Fornaess [7, p. 133, Remark b].

Lemma 8**.**

Let $\Omega\subseteq\mathbb{C}^{N}$ be a pseudoconvex domain and $p\in\partial\Omega$ be a point in the smooth part of its boundary. Take any $\eta\in(0,1)$. Then there exists a neighborhood $U$ of $p$ and a defining function $\rho$ of $\Omega$ on $U$ such that $-(-\rho)^{\eta}$ is strictly plurisubharmonic on $\Omega\cap U$.

As paper [7] is mainly interested in global properties of pseudoconvex domains, the proof of 8 is merely hinted at in a remark. For a full proof, see [1, Thm. 2.2.17].

Proof of 3.

Suppose that $F:\Omega\rightarrow\Omega^{\prime}$ extends to a CR map that is not CR-transversal at a point $p\in M$. Choose $0<\delta<\epsilon$, and let $\eta=\frac{1+\delta}{1+\epsilon}$. By 8, there exists a neighborhood $U$ of $F(p)$ and a defining function $\rho\in C^{\infty}(U,\mathbb{R})$ for $M^{\prime}$ such that $U\cap\Omega^{\prime}=\rho^{-1}(-\infty,0)$ and such that $-(-\rho)^{\eta}$ is strictly plurisubharmonic on $U\cap\Omega^{\prime}$. By assumption, the normal derivative of $\rho\circ F$ vanishes at $p$, hence $\rho\circ F$ has a critical point at $p$. By Hölder continuity of the derivative, $\rho\circ F(z)=\mathcal{O}(|z-p|^{1+\epsilon})$ near $p$. But this implies $-(-\rho)^{\eta}\circ F(z)=\mathcal{O}(|z-p|^{1+\delta})$ on $F^{-1}(U\cap\Omega^{\prime})$, and thus the normal derivative of $-(-\rho)^{\eta}\circ F$ at $p$ vanishes as well. Since $-(-\rho)^{\eta}\circ F$ as a pull-back of a subharmonic function along a holomorphic map is subharmonic as well, and since $-(-\rho)^{\eta}\circ F$ clearly has a local maximum at $p$, the normal derivative of $\rho^{\eta}\circ F$ at $p$ is nonzero by the Hopf lemma, a contradiction. ∎

A holomorphic map inherits the regularity of its induced boundary map, immediately allowing the transferral of 2 to holomorphic maps.

Proof of 1.

As $\Omega^{\prime}$ is pseudoconvex at $M^{\prime}$, 3 yields CR transversality of the boundary map $h:=H|_{M^{\prime}}$. Thus, the hypothesis of 2 is met, and $h$ is $C^{\infty}$-smooth on a dense open subset $O\subseteq M$ of a neighborhood of $p$ in $M$. By Theorem 7.5.1. in [1], $H$ then extends to a $C^{\infty}$-smooth map on $O\cup\Omega$. ∎

Remark 4**.**

In particular, if $H:\Omega\rightarrow\Omega^{\prime}$ is a proper holomorphic map and extends to a $C^{N^{\prime}-n_{+}}$-smooth map on $\bar{\Omega}$, it maps $M$ into the topological boundary of $\Omega^{\prime}$. If a point $p\in M$ is known to be mapped to some $p^{\prime}\in M^{\prime}$, an open neighborhood $U\subseteq M$ of $p$ is then also mapped into $M^{\prime}$, and the hypothesis of 1 is satisfied.

6. Maps into boundaries of classical symmetric domains

Before we discuss the CR geometry of the boundaries of the classical symmetric domains that we need to apply our results, let us recall some basic facts. We call a bounded domain $\Omega\subset\mathbb{C}^{N}$ a bounded symmetric domain if it exhibits a biholomorphic involution $h_{p}:\Omega\rightarrow\Omega$ for every point $p\in\Omega$ which has $p$ as an isolated fixed point and which satisfies $Dh(p)=-\mathbb{I}_{N}$ (cf. [27]).

A bounded domain $\Omega$ may be equipped with the Bergman metric, a Hermitian metric with the property that each biholomorphism on $\Omega$ is an isometry. Considered together with this metric, a bounded symmetric domain $\Omega$ becomes a special case of a Hermitian symmetric space, i.e. a manifold equipped with a Hermitian metric such that each point is an isolated fixed point of some involutive isometry. It can be shown that the group of isometries of such manifolds acts transitively, therefore they can be expressed as the coset space of the the stabilizer group of $\Omega$, defined as the group of isometries leaving a chosen point fixed, in the full isometry group of $\Omega$ (cf. [8]). This allows the classification of bounded symmetric domains by Lie group techniques.

According to [27], any bounded symmetric domain is biholomorphic to a direct product of irreducible bounded symmetric domains. Irreducible bounded symmetric domains fall into four series of classical symmetric domains as well as two exceptional cases (as classified by Cartan, cf. [5]). The study of regularity of proper holomorphic maps into classical symmetric domains, and consequently of CR maps into their boundaries, has been taken up by Xiao in [29]; for important applications of maps between classical symmetric domains, we refer the reader to e.g. Kim and Zaitsev’s paper on rigidity of these maps [13]. We will adopt Xiao’s naming convention for the classical symmetric domains, which differs from Cartan’s original numbering only in swapping domains of the third and fourth kind.

Finally, let us briefly recall the singular value decomposition from linear algebra. A matrix $A\in\mathbb{C}^{m\times n}$, $m\leq n$ may always be decomposed as $A=U\Sigma V^{\ast}$, where

1. (1)

$U\in\mathbb{C}^{m\times m}$ is a unitary matrix, forming a basis of eigenvectors for $AA^{\ast}$, 2. (2)

$\Sigma\in\mathbb{C}^{m\times n}$ is a diagonal matrix with nonnegative entries, and 3. (3)

$V\in\mathbb{C}^{n\times n}$ is another unitary matrix, forming a basis of eigenvectors for $A^{\ast}A$.

The diagonal entries of $\Sigma$, $0\leq\sigma_{1}\leq\dots\leq\sigma_{m}$ are called the singular values of $A$. They are given by the square roots of the eigenvalues of the (Hermitian, positive semidefinite) matrix $AA^{\ast}$, or equivalently by the square roots of the $m$ largest eigenvalues of $A^{\ast}A$. The largest singular value of $A$ yields the operator norm of $A$ with respect to the standard scalar product on $\mathbb{C}^{m}$ and $\mathbb{C}^{n}$. The matrix $V$ of right singular vectors may be freely chosen among the orthonormal eigenvector bases of $A^{\ast}A$, which then fixes $\Sigma U=AV$, and therefore those columns of $U$ corresponding to nonzero singular values, the left singular vectors.

6.0.1. Classical domains of the first kind

We will denote the examples in the first series by $D_{I}^{m,n}$ for $1\leq m\leq n$. According to Cartan [5], they may be realized as

[TABLE]

The condition $\mathbb{I}_{m}-ZZ^{\ast}>0$ is equivalent to the largest singular value of $Z$ being strictly bounded by one, i.e. $\lVert Z\rVert<1$, where $\lVert\cdot\rVert$ always denotes the usual Euclidean matrix norm (or vector norm, respectively). The boundary of $D_{I}^{m,n}$ is thus given by the set of matrices of norm $1$, equivalently, by those matrices which have $1$ as their largest singular value. This set is a smooth manifold where exactly one singular value is $1$. To see this, consider the characteristic polynomial $P(\lambda)=\det(\lambda\mathbb{I}_{m}-ZZ^{\ast})$ of $ZZ^{\ast}$, which has a simple zero at $1$ by assumption. Now $\rho(Z):=\det(\mathbb{I}_{m}-ZZ^{\ast})$ has nonvanishing gradient, since

[TABLE]

has nonvanishing derivative, providing us with a defining equation.

Let us denote this smooth piece of the boundary by $M_{I}^{m,n}$. Because $M_{I}^{m,n}$ bounds the convex region $D_{I}^{m,n}$, it is a pseudoconvex real hypersurface. The singular value decomposition will translate to a foliation of $M_{I}^{m,n}$ by complex (in fact, complex linear) manifolds, setting $M_{I}^{m,n}$ up as an interesting example case for applying 2. The following result should be compared to Proposition 1.2 in [29], where only strongly pseudoconvex hypersurfaces in $\mathbb{C}^{m{+}n{-}1}$ are considered.

Proposition 4**.**

Let $m\geq n\geq 2$ and $M$ be a pseudoconvex hypersurface with at least $n_{+}\geq m{+}n{-}3$ positive Levi eigenvalues. Then every CR-transversal CR map $h$ of regularity $C^{mn-n_{+}}$ from $M$ into $M_{I}^{m,n}$ is generically smooth.

In the course of our proof, we will be utilizing the boundary orbit theorem, which states that the Lie group of biholomorphic automorphisms of $D_{I}^{m,n}$ also acts transitively on $M_{I}^{m,n}$ by ambient biholomorphisms (see e.g. [28] or [24, proof of Lemma 2.2.3]). This allows us to analyze $M_{I}^{m,n}$ around points which are particularly easy to understand from the matrix model alone, namely the rank one matrices in $M_{I}^{m,n}$.

Indeed, suppose $h:M\rightarrow M_{I}^{m,n}$ is nowhere smooth on a neighborhood $O\subset M$ of a point $p\in M$. Any matrix $ab^{\ast}$ for vectors $a\in\mathbb{C}^{m}$, $b\in\mathbb{C}^{n}$ of unit norm is contained in $M_{I}^{m,n}$, since it has a lone singular value $1$. By the boundary orbit theorem, there exists a biholomorphic map $F_{h(p)}$ defined on a neighborhood of $h(p)$ mapping $h(p)$ to $ab^{\ast}\in M_{I}^{m,n}$ and $M_{I}^{m,n}$ into itself. Then $\tilde{h}:=F_{h(p)}\circ h$ is a CR-transversal CR map taking $p$ to $ab^{\ast}$, which is nowhere smooth on $O$ as well. At $ab^{\ast}$, we check directly that the prerequisites to apply 2 are fulfilled.

Lemma 9**.**

Let $a\in\mathbb{C}^{n},b\in\mathbb{C}^{m}$ be unit vectors. Around $ab^{\ast}$, the pseudoconvex hypersurface $M_{I}^{m,n}$ is foliated by $(n{-}1){\times}(m{-}1)$-dimensional complex (linear) manifolds. Its Levi form has exactly $m+n-2$ positive eigenvalues, and $\nu_{ab^{\ast}}=m+n-4$.

If $\tilde{h}$ was nowhere smooth around $p$, this would contradict 2, as $\nu_{ab^{\ast}}=m+n-4<n_{+}$. This proves 4.

Proof of 9.

Let $\Sigma\subset\mathbb{C}^{m\times n}$ be the set of $m\times n$ matrices of rank (exactly) $1$, which is an $(m{+}n{-}1)$-dimensional holomorphic manifold containing $ab^{\ast}$. In linear coordinates such that $a=(1,0,\dots,0)^{T}\in\mathbb{C}^{n}$ and $b=(1,0,\dots,0)^{T}\in\mathbb{C}^{m}$, $\Sigma$ is parametrized holomorphically by

[TABLE]

around $ab^{\ast}=(1,0,\dots,0)^{T}(1,0,\dots,0)$. To see explicitly that this map is one-to-one near $ab^{\ast}$, for a matrix $Z\in\Sigma$, let $w$ be the (unique) intersection of $(\ker Z)^{\bot}$ and $b+\left<b\right>^{\bot}$. Then $w^{\ast}=(1,w_{2},\dots,w_{n})$ and $Z(w)/\lVert w\rVert^{2}=(z_{1},\dots,z_{m})^{T}$.

The hypersurface $S:=\Sigma\cap M_{I}^{m,n}$ of rank one matrices with norm $1$ is strongly pseudoconvex. Indeed, because $\lVert uv^{\ast}\rVert=\lVert u\rVert\lVert v\rVert$, a defining equation for $S$ is given by

[TABLE]

with (real) Hessian $2\mathbb{I}_{2(m+n-1)}$ at $ab^{\ast}$, implying that $S$ is actually strongly convex.

The singular value decomposition expresses any matrix $A\in M_{I}^{m,n}$ as $uv^{\ast}+B$, where $u$ and $v$ are unit singular vectors (unique up to simultaneous multiplication by $\lambda\in\mathbb{S}^{1}$) corresponding to the lone singular value $1$, and the uniquely determined matrix $B\in\mathbb{C}^{m\times n}$ satisfies $Bv=0$, $u^{\ast}B=0$ and $\lVert B\rVert<1$. Conversely, every matrix $uv^{\ast}+B$ of this type lies in $M_{I}^{m,n}$. The set of all $B\in\mathbb{C}^{m\times n}$ with $u^{\ast}B=0$ and $Bv=0$ is an $(m-1)\times(n-1)$-dimensional vector space, and thus the affine planes

[TABLE]

for $uv^{\ast}\in S$ provide the desired foliation $\eta$ of $M_{I}^{m,n}$ near $ab^{\ast}$. The tangent bundle $T\eta$ at $S$ is just given by $T_{uv^{\ast}}\eta=\{B\in\mathbb{C}^{m\times n}:Bv=0,u^{\ast}B=0\}$.

Having established the setup from 6, all that remains is to compute the tensor $R^{S}$ at $ab^{\ast}$. Take $B_{0}\in T_{ab^{\ast}}\eta$, $B_{0}\neq 0$. If we define $B(Z)$ for $Z\in\mathbb{C}^{m\times n}$ by

[TABLE]

then $B(uv^{\ast})$ provides a section of $T\eta$ along $S$ satisfying $B(ab^{\ast})=B_{0}$, since

[TABLE]

Returning to $S$, we work out that $\mathfrak{V}:=\{a\beta^{\ast}+\alpha b^{\ast}:\alpha\in\left<a\right>^{\bot}\subset\mathbb{C}^{m},\beta\in\left<b\right>^{\bot}\subset\mathbb{C}^{n}\}$ is the complex tangent space of $S$ at $ab^{\ast}$. To show that $\mathfrak{V}$ is tangential, we take two curves $\gamma_{1}:(-\varepsilon,\varepsilon)\rightarrow\mathbb{S}^{2m-1}$ and $\gamma_{2}:(-\varepsilon,\varepsilon)\rightarrow\mathbb{S}^{2n-1}$ through $a$ and $b$, respectively, satisfying $\dot{\gamma}_{1}(0)=\alpha$ and $\dot{\gamma}_{2}(0)=\beta$. Now $\gamma_{1}\gamma_{2}^{\ast}$ defines a curve in $S$, and $\frac{d}{dt}|_{t=0}\left(\gamma_{1}(t)\gamma_{2}(t)^{\ast}\right)=a\beta^{\ast}+\alpha b^{\ast}$. The space $\mathfrak{V}$ is parametrized in a complex linear way by $(\alpha,\bar{\beta})\mapsto a\bar{\beta}^{T}+\alpha b^{\ast}$, where $(\alpha,\bar{\beta})$ lies in the $(m{+}n{-}2)$-dimensional complex subspace of $\mathbb{C}^{m+n}$ defined by $a^{\ast}\alpha=0$, $\bar{\beta}^{T}b=\beta^{\ast}b=0$. To check that this map is indeed injective, test $\alpha b^{\ast}+a\beta^{\ast}$ from right and left with $b$ and $a^{\ast}$, respectively, to obtain $\alpha$ and $\bar{\beta}$ again. Since the complex tangent space of $S$ has only $\dim_{\mathbb{C}}\Sigma-1=m+n-2$ dimensions, $T_{ab^{\ast}}^{c}S=\mathfrak{V}$ follows.

Consider a CR vector $\bar{L}|_{ab^{\ast}}=\frac{1}{2}(X+iJX)$ for $X\in T^{c}_{ab^{\ast}}S$ and write $X=a\beta^{\ast}+\alpha b^{\ast}$. Then the holomorphic curve $\gamma(t)=(a+t\alpha)(b+\bar{t}\beta)^{\ast}$ is tangential to $\bar{L}|_{ab^{\ast}}$ at $t=0$. Observing that both $\lVert a+t\alpha\rVert^{2}=1+|t|^{2}\lVert\alpha\rVert^{2}$ and $\lVert b+\bar{t}\beta\rVert^{2}$ are constant to first order, we obtain

[TABLE]

Recall that the scalar product in $\mathbb{C}^{m\times n}$ may be written as $(A|B)=\mathrm{tr}(A^{\ast}B)$. By commuting matrices inside the trace we see that for any $Z$ in $T_{ab^{\ast}}\eta$,

[TABLE]

since $a^{\ast}Z=Zb=0$. This means that $R^{S}_{ab^{\ast}}(\bar{L}|_{ab^{\ast}},B_{0})=-a\alpha^{\ast}B_{0}-B_{0}\beta b^{\ast}$, because the projection onto $T^{\bot}\eta$ is already taken care of, and $\bar{L}|_{ab^{\ast}}\in\ker R^{S}_{ab^{\ast}}(\cdot,B_{0})$ if and only if $a\alpha^{\ast}B_{0}+B_{0}\beta b^{\ast}=0$. Testing this with $a^{\ast}$ and $b$ from left and right, respectively, we obtain $\alpha\in\ker B_{0}^{\ast}$ and $\beta\in\ker B_{0}$. Since $B_{0}^{\ast}a=0$ and $B_{0}b=0$ already, both kernels have codimension at least one in $\left<a\right>^{\bot}$ and $\left<b\right>^{\bot}$, respectively, thus $\dim_{\mathbb{C}}\ker R^{S}_{ab^{\ast}}(\cdot,B_{0})\leq m+n-4$, implying $\nu_{ab^{\ast}}=m+n-4$. ∎

4 gives all dimensions where a statement this simple is meaningful and possible. If $M$ has more than $m+n-2$ positive Levi eigenvalues, there is no CR-transversal map from $M$ to $M_{I}^{m,n}$, since, by 7, the target manifold would need to have at least as many positive Levi eigenvalues as the source. If $M$ has less than $m+n-3$ positive Levi eigenvalues, there are nowhere smooth CR-transversal CR maps into $M_{I}^{m,n}$ of arbitrarily high regularity.

Example 3**.**

Let $\hat{S}$ be the strongly pseudoconvex hypersurface given by the $(m-1)\times(n-1)$ matrices of rank one and norm $1$. Then $\hat{S}$ has $m+n-4$ positive Levi eigenvalues. Take a $C^{k}$, but nowhere $C^{\infty}$-smooth CR function $\phi$ on $\hat{S}$ with $|\phi|<1$. Then $h(Z)=\begin{pmatrix}Z&0\\ 0&\phi\end{pmatrix}$ gives a nowhere smooth CR-transversal CR map $h:\hat{S}\rightarrow M_{I}^{m,n}$ of regularity $C^{k}$.

Proof.

Regularity is obvious from the component-wise definition. CR-transversality always holds for the graph map of a CR function, i.e. $h:M\rightarrow M\times\mathbb{C}$, $h(p)=(p,\phi(p))$, since $T^{c}(M\times\mathbb{C})\cong T^{c}M\times\mathbb{C}$, $T(M\times\mathbb{C})\cong TM\times\mathbb{C}$ and $\mathbb{P}_{TM}\circ h_{\ast}\cong\mathrm{id}$ together imply that any transversal vector $v\in TM\setminus T^{c}M$ maps into a transversal vector again. That $h(Z)\in M_{I}^{m,n}$ follows from the singular value computations in the proof of 9. ∎

6.0.2. Classical domains of the second kind

These classical symmetric domains, denoted by $D_{II}^{m}$, $m\geq 2$, are given as the sets of skew symmetric complex $m{\times}m$ matrices with norm less than $1$. Equivalently,

[TABLE]

Every nonzero singular value of a skew symmetric matrix $Z$ occurs with even multiplicity. Suppose $u$ is a right singular vector corresponding to a singular value $\sigma$, which is equivalent to $Z^{\ast}Zu=\sigma^{2}u$. Then $v:=\sigma^{-1}\overline{Zu}$ is another right singular vector corresponding to $\sigma$, since it follows from $Z^{\ast}=\bar{Z}^{T}=-\bar{Z}$ that $Z^{\ast}Zv=-\sigma^{-1}\bar{Z}Z\bar{Z}\bar{u}=-\sigma^{-1}\bar{Z}\overline{\bar{Z}Zu}=\sigma^{-1}\bar{Z}\overline{Z^{\ast}Zu}=\sigma\bar{Z}\bar{u}=\sigma^{2}v$, and $v^{\ast}v=\sigma^{-2}u^{T}Z^{T}\bar{Z}\bar{u}=\sigma^{-2}\overline{u^{\ast}Z^{\ast}Zu}=1$. Furthermore, $v$ and $u$ are orthogonal, and $u=-\sigma^{-1}\overline{Zv}$:

[TABLE]

The boundary of $D_{II}^{m}$ is given by those skew symmetric matrices with norm $1$. It is a smooth manifold where exactly the largest two singular values are $1$. We will denote this smooth piece of the boundary by $M_{II}^{m}$. Let us postpone checking that $M_{II}^{m}$ is a manifold to the proof of 10.

Proposition 5**.**

Let $m\geq 4$ and $M$ be a pseudoconvex hypersurface with at least $n_{+}\geq 2m-7$ positive Levi eigenvalues. Then any CR-transversal CR map $h$ of regularity $C^{\frac{m(m-1)}{2}-n_{+}}$ from $M$ into $M_{II}^{m}$ is generically smooth.

Completely analogously to the situation of 4, this follows from the boundary orbit theorem for $D_{II}^{m}$, which allows us to map each point in $M_{II}^{m}$ to ${p^{\prime}}:=ab^{T}-ba^{T}$ for orthonormal $a,b\in\mathbb{C}^{m}$ by an automorphism of $M_{II}^{m}$, and from the following structural properties.

Lemma 10**.**

Let $a,b\in\mathbb{C}^{m}$ be orthonormal vectors. Around $p^{\prime}:=ab^{T}-ba^{T}\in M_{II}^{m}$, the pseudoconvex hypersurface $M_{II}^{m}$ is foliated by $\frac{(m-2)(m-3)}{2}$-dimensional complex (linear) manifolds. Its Levi form has exactly $2m-4$ positive eigenvalues, and $\nu_{p^{\prime}}=2m-8$.

Proof.

As the intersection of the linear subspace of skew symmetric matrices with the convex matrix norm unit ball, $D_{II}^{m}$ is convex and $M_{II}^{m}$ is a pseudoconvex hypersurface.

The set $\Sigma$ of skew symmetric matrices of rank two is a $(2m{-}3)$-dimensional complex manifold around $p^{\prime}$. In coordinates where $a=(1,0,\dots,0)^{T}$ and $b=(0,1,0,\dots,0)^{T}$, it is parametrized around $p^{\prime}$ by

[TABLE]

To check surjectivity, let $\bar{u}$ and $\bar{v}$ be two right singular vectors corresponding to the only nonzero singular value $\sigma$, chosen such that $Z\bar{u}=-\sigma v$ and $Z\bar{v}=\sigma u$. Then $Z=u(\sigma v)^{T}-(\sigma v)u^{T}$. Since $a^{\ast}(ab^{T}-ba^{T})\bar{b}=1$, $a^{\ast}Z\bar{b}\neq 0$ near $p^{\prime}$, implying that at least one of $a^{\ast}u$ or $a^{\ast}v$ is nonzero. By substituting $(-v,u)$ for $(u,v)$ if necessary, we can arrange $a^{\ast}u\neq 0$. Let $\tilde{u}=u$, $\tilde{v}=\sigma(v-\frac{a^{\ast}v}{a^{\ast}u}u)$, then $a^{\ast}\tilde{v}=0$ and $Z=\tilde{u}\tilde{v}^{T}-\tilde{v}\tilde{u}^{T}$. Note that $a^{\ast}Z\bar{b}\neq 0$ now implies $b^{\ast}\tilde{v}\neq 0$. Let $z=\frac{1}{a^{\ast}\tilde{u}}\tilde{u}-\frac{b^{\ast}\tilde{u}}{(a^{\ast}\tilde{u})(b^{\ast}\tilde{v})}\tilde{v}$ and $w=(a^{\ast}\tilde{u})\tilde{v}$. Then we have $a^{\ast}z=1$, $b^{\ast}z=0$, $a^{\ast}w=0$ and $Z=zw^{T}-wz^{T}$, proving that $Z$ is in the range of our parametrization. To check that it is an immersion, it suffices to calculate $\frac{\partial}{\partial z_{j}}(zw^{T}-wz^{T})=e_{j}e_{2}^{T}-e_{2}e_{j}^{T}$, $3\leq j\leq m$ and $\frac{\partial}{\partial w_{k}}(zw^{T}-wz^{T})=e_{1}e_{k}^{T}-e_{k}e_{1}^{T}$, $2\leq k\leq m$, since these are evidently $\mathbb{C}$-linearly independent matrices.

The set $S=\Sigma\cap M_{II}^{m}$ of skew symmetric rank two matrices with norm $1$ is a strictly pseudoconvex hypersurface in $\Sigma$. To show this, first note that for orthogonal vectors $\alpha,\beta\in\mathbb{C}^{m}$, we have

[TABLE]

The standard Euclidean scalar product on $\mathbb{C}^{m\times m}$ coincides with the Frobenius scalar product $(A|B)=\mathrm{tr}(A^{\ast}B)$. For a matrix $Z=\alpha\beta^{T}-\beta\alpha^{T}$ with orthogonal $\alpha,\beta\in\mathbb{C}^{m}$, the Frobenius norm works out to

[TABLE]

Therefore, the Frobenius norm and the matrix norm agree up to a constant on $\Sigma$, and $S=\Sigma\cap M_{II}^{m}=\Sigma\cap\sqrt{2}\mathbb{S}^{2m^{2}-1}$ is strongly pseudoconvex, as it is given by the intersection of a complex manifold with a strongly convex hypersurface.

The singular value decomposition expresses $Z\in M_{II}^{m}$ as $uv^{T}-vu^{T}+B$, where $\bar{u}$ and $\bar{v}$ are right singular vectors corresponding to the double singular value $1$ satisfying $Z\bar{u}=-v$ and $Z\bar{v}=u$, and $B$ satisfies $B\bar{u}=B\bar{v}=0$, $u^{\ast}B=v^{\ast}B=0$ and $\lVert B\rVert<1$. By linearity, we have $B^{T}=-B$, implying that $B\bar{u}=B\bar{v}=0$ and $u^{\ast}B=v^{\ast}B=0$ are equivalent. In coordinates where $u=(1,0,0,\dots,0)$ and $v=(0,1,0,\dots,0)$, the conditions $B=-B^{T}$ and $B\bar{u}=B\bar{v}=0$ simply mean that $B$ is a skew symmetric matrix with the first two rows and columns empty. We conclude that the affine planes $\eta_{uv^{T}-vu^{T}}=uv^{T}-vu^{T}+\{B\in\mathbb{C}^{m\times m}:B=-B^{T},B\bar{u}=B\bar{v}=0\}$ for $uv^{T}-vu^{T}\in S$ provide a foliation of $M_{II}^{m}$ by $\frac{(m-2)(m-3)}{2}$-dimensional complex manifolds, and that $M_{II}^{m}$, as an embedded piece of a vector bundle over $S$, is indeed a manifold.

The complex tangent space $T_{p^{\prime}}^{c}S$ at ${p^{\prime}}=ab^{T}-ba^{T}$ will be given by the complex vector space $\mathfrak{V}:=\{a\beta^{T}-\beta a^{T}+\alpha b^{T}-b\alpha^{T}:\alpha,\beta\in\left<a,b\right>^{\bot}\subset\mathbb{C}^{m}\}$. To show tangency, consider the complex curve $\gamma(t)=(a+t\alpha)(b+t\beta)^{T}-(b+t\beta)(a+t\alpha)^{T}$, with tangent vector $\gamma_{t}(0)=a\beta^{T}-\beta a^{T}+\alpha b^{T}-b\alpha^{T}$. It is contained in $\Sigma$ and tangential to $M_{II}^{m}$, the latter because $\lVert\gamma(t)\rVert^{2}=\lVert a+t\alpha\rVert^{2}\lVert b+t\beta\rVert^{2}-\left|(a+t\alpha)^{\ast}(b+t\beta)\right|^{2}=\lVert a\rVert^{2}\lVert b\rVert^{2}+\mathcal{O}(|t|^{2})$, hence $\gamma_{t}(0)\in T_{p^{\prime}}^{c}S$. Since $\mathfrak{V}$ is isomorphic to $\left<a,b\right>^{\bot}$ by the map $\gamma_{t}(0)\mapsto\left(\gamma_{t}(0)\bar{b},-\gamma_{t}(0)\bar{a}\right)$, it has $2m-4=\dim_{CR}S$ dimensions, and $T_{p^{\prime}}^{c}S=\mathfrak{V}$.

Given $B_{0}\in T_{p^{\prime}}\eta$, the map $B(Z)=(\mathbb{I}_{m}-ZZ^{\ast})B_{0}(\mathbb{I}_{m}-Z^{\ast}Z)$ again provides a section of $T\eta$ along $S$, since for orthonormal $u,v\in\mathbb{C}^{m}$,

[TABLE]

Taking a CR vector $\bar{L}|_{p^{\prime}}\in T_{p^{\prime}}^{0,1}S$ with real part $\frac{1}{2}\left(a\beta^{T}-\beta a^{T}+\alpha b^{T}-b\alpha^{T}\right)$ and the curve $\gamma(t)=(a+t\alpha)(b+t\beta)^{T}-(b+t\beta)(a+t\alpha)^{T}$, we first obtain

[TABLE]

which simplifies the calculations for $R_{p^{\prime}}^{S}$ significantly. We obtain

[TABLE]

By the same calculations as in the proof of 9, we find that this already gives $R_{p^{\prime}}^{S}(\bar{L}|_{p^{\prime}},B_{0})=-b\beta^{\ast}B_{0}-a\alpha^{\ast}B_{0}-B_{0}\bar{\beta}b^{T}-B_{0}\bar{\alpha}a^{T}$, and that $\bar{L}|_{p^{\prime}}\in\ker R_{p^{\prime}}^{S}(\cdot,B_{0})$ if and only if $\alpha,\beta\in\ker\bar{B}_{0}$. As a nonzero skew symmetric matrix, $\bar{B}_{0}$ has at least two nonzero singular values, hence $\textrm{codim}_{\mathbb{C}}\ker\bar{B}_{0}\geq 2$. Since $\bar{B}_{0}a=\bar{B}_{0}b=0$, and $\alpha,\beta\in\left<a,b\right>^{\bot}$, we obtain $\textrm{codim}_{\mathbb{C}}\ker R^{S}_{p^{\prime}}(\cdot,B_{0})\geq 4$ and thus $\nu_{p^{\prime}}=2m-8$. ∎

As in 4, there are counterexamples to regularity if $M$ has exactly $2m-8$ positive Levi eigenvalues.

Example 4**.**

Let $\hat{S}\subset M_{II}^{m-2}$ be the strongly pseudoconvex hypersurface of antisymmetric $(m-2)\times(m-2)$ matrices of rank two and norm $1$. It has $2m-8$ positive Levi eigenvalues. Given a $C^{k}$-smooth, but nowhere $C^{\infty}$-smooth CR function $\phi$ on $\hat{S}$ strictly bounded by $1$, the map $h:\hat{S}\rightarrow M_{II}^{m}$ given by

[TABLE]

is a $C^{k}$-smooth, but nowhere $C^{\infty}$-smooth CR-transversal CR function.

6.0.3. Classical domains of the third kind

Domains of the third kind $D_{III}^{m}$ are given by the sets of symmetric complex $m{\times}m$ matrices with norm less than $1$. Equivalently,

[TABLE]

Here the regularity result obtained from 2 only holds for $M\subset\mathbb{C}^{m}$.

Proposition 6**.**

Let $m\hskip 2.0pt{\geq}\hskip 2.0pt2$ and $M$ be a pseudoconvex hypersurface with at at least $n_{+}\geq m-1$ positive Levi eigenvalues. Then every CR-transversal CR map $h$ of regularity $C^{\frac{m(m+1)}{2}-n_{+}}$ from $M$ into $M_{III}^{m}$ is generically smooth.

Let us note in passing that a nontrivial CR transversal CR map from $M$ into $M_{III}^{m}$ can only exist if the number of positive Levi eigenvalues of $M$ does not exceed $m-1$, thus this result only truly concerns uniformly pseudoconvex hypersurfaces.

6 is a consequence of the boundary orbit theorem for $D_{III}^{m}$, which tells us that every point $Z\in M_{III}^{m}$ may be mapped to $aa^{T}$ for a unit vector $a\in\mathbb{C}^{m}$ by an ambient biholomorphism mapping $M_{III}^{m}$ into itself. Almost completely analogously to the case of $M_{I}^{m,n}$, the following structural properties hold.

Lemma 11**.**

Let $a\in\mathbb{C}^{m}$ be a unit vector. Around $aa^{T}\in M_{III}^{m}$, the pseudoconvex hypersurface $M_{III}^{m}$ is foliated by $\frac{m(m-1)}{2}$-dimensional complex (linear) manifolds. Its Levi form has exactly $m-1$ positive eigenvalues, and $\nu_{aa^{T}}=m-2$.

Proof.

As the intersection of the convex set of matrices of norm less than $1$ with the linear subspace of symmetric matrices, $D_{III}^{m}$ is convex, and thus $M_{III}^{m}$ is pseudoconvex.

Let $\Sigma$ be the $m$-dimensional complex manifold of symmetric matrices of rank $1$. Near $aa^{T}$, it is parametrized by $z\mapsto zz^{T}$ for $z\in\mathbb{C}^{m}$ with $\Re(a^{\ast}z)>0$. To check bijectivity, write $Z=\sigma uv^{\ast}$ for singular vectors $u,v\in\mathbb{C}^{m}$ and the nonzero singular value $\sigma$. Since $u$ and $v$ lie in the one-dimensional kernels of $ZZ^{\ast}-\sigma^{2}\mathbb{I}_{m}=Z\bar{Z}-\sigma^{2}\mathbb{I}_{m}$ and $Z^{\ast}Z-\sigma^{2}\mathbb{I}_{m}=\bar{Z}Z-\sigma^{2}\mathbb{I}_{m}$, respectively, we infer by Cramer’s rule that $\lambda u=\bar{v}$ for some $\lambda\in\mathbb{S}^{1}$. Letting $z:=\sigma^{\frac{1}{2}}\lambda^{-\frac{1}{2}}\bar{v}=\sigma^{\frac{1}{2}}\lambda^{\frac{1}{2}}u$, we find that $Z=zz^{T}$. The only indeterminacy here - the choice of sign for the root $\lambda^{\frac{1}{2}}$ - is fixed by requiring $\Re(z^{\ast}a)>0$.

The real hypersurface $S\subset\Sigma$ of rank one matrices with norm $1$ is strongly pseudoconvex. Indeed, as $\lVert zz^{T}\rVert=\lVert z\rVert^{2}$, we have that $z\in\mathbb{S}^{2m-1}$ iff $zz^{T}\in S$, and the map $z\mapsto zz^{T}$ provides a holomorphic double cover of $S$ by $\mathbb{S}^{2m-1}$, showing that $S\cong\mathbb{R}P^{2m-1}$.

The complex affine planes $\eta_{ww^{T}}:=\{ww^{T}+B:B\bar{w}=0,B^{T}=B\}$ for $w\in\mathbb{S}^{2m-1}$ provide a foliation of $M_{III}^{m}$ near $aa^{T}$. As in the proof of 9, the singular value decomposition expresses $Z\in M_{III}^{m}$ as $uv^{\ast}+B$, where $u$, $v$ are unit vectors (unique up to simultaneous multiplication by $\lambda\in\mathbb{S}^{1}$), and $B$ satisfies $B^{\ast}u=Bv=0$ and $\lVert B\rVert<1$. Since as before, $u$ and $v$ lie in the one-dimensional kernels of $Z\bar{Z}-\mathbb{I}_{m}$ and $\bar{Z}Z-\mathbb{I}_{m}$, respectively, we may express $Z=ww^{T}+B$ for $w\in\mathbb{S}^{2m-1}$, implying $B^{T}=B$ by linearity. The condition $Bv=B^{\ast}u=0$ simplifies to $B\bar{w}=0$. In coordinates where $w=(1,0,\dots,0)^{T}\in\mathbb{C}^{m}$, $B\bar{w}=0$ just means that the first column is empty, a condition that is clearly linearly independent of $B^{T}=B$. Therefore, the space defined by $B^{T}=B$, $B\bar{w}=0$ is a complex vector space of $\frac{m(m-1)}{2}$ dimensions for $w$ near $a$.

Given $B_{0}\in T_{aa^{T}}\eta$, we prove that $B(Z)=(\mathbb{I}_{m}-ZZ^{\ast})B_{0}(\mathbb{I}_{m}-Z^{\ast}Z)$ provides a section of $T\eta$ along $S$. For $ww^{T}\in S$,

[TABLE]

Consider a CR vector $\bar{L}|_{aa^{T}}\in T_{aa^{T}}^{0,1}S$. Complex tangent vectors $\alpha\in T_{a}^{c}\mathbb{S}^{2m-1}$ are characterized by $\alpha^{\ast}a=0$. Since $z\mapsto zz^{T}$ is holomorphic and onto, we can just plug a suitable complex tangent $t\mapsto a+t\alpha$ into this map to obtain a curve $\gamma(t)=(a+t\alpha)(a+t\alpha)^{T}$ such that $\bar{L}|_{aa^{T}}=\gamma_{\ast}\frac{d}{d\bar{t}}|_{t=0}$. Then, after rewriting $\gamma(t)=(a+t\alpha)(\bar{a}+\bar{t}\bar{\alpha})^{\ast}$, we obtain by the exact same calculation as in 9 that $R_{aa^{T}}^{S}(\bar{L}|_{aa^{T}},B_{0})=-a\alpha^{\ast}B_{0}-B_{0}\bar{\alpha}a^{T}$. By multiplying from the right with $\bar{a}$, we find that $\bar{L}|_{aa^{T}}\in\ker R_{aa^{T}}^{S}(\cdot,B_{0})$ if and only if $B_{0}\bar{\alpha}=0$. Since $\bar{a}\in\ker B_{0}$, the codimension of the kernel of $B_{0}$ in $\left<\bar{a}\right>^{\bot}$ equals the codimension of the full kernel of $B_{0}$, hence $\dim_{\mathbb{C}}\ker R_{aa^{T}}^{S}(\cdot,B_{0})=m-1-\textrm{codim}_{\mathbb{C}}\ker B_{0}\leq m-2$, implying $\nu_{aa^{T}}=m-2$. ∎

Here a counterexample for regularity of CR-transversal maps from source manifolds with less than $m-1$ positive Levi eigenvalues may be constructed in the exact same fashion as in the case of $M_{I}^{m,n}$. Let us instead consider a slightly different example map into $M_{III}^{m}$.

Example 5**.**

Let $\phi$ be a nowhere smooth CR function of regularity $C^{k}$ on $\mathbb{S}^{2m-3}$ strictly bounded by $1$. Then the map $h:\mathbb{S}^{2m-3}\rightarrow M_{III}^{m}$ given by

[TABLE]

is a nowhere smooth CR-transversal CR embedding of regularity $C^{k}$.

Proof.

We first consider the map $H:\mathbb{C}^{m-1}_{z}\times\mathbb{C}_{w}$ given by

[TABLE]

It is a holomorphic immersion on $\mathbb{C}^{m-1}\times\mathbb{B}^{1}\subset\mathbb{C}^{m-1}\times\mathbb{C}$, since

[TABLE]

where $e_{j}$ denotes the $j^{th}$ standard unit vector. The matrices $\frac{\partial}{\partial z_{j}}H(z,w)$ for $1\leq j\leq m-1$ are linearly independent, since their last columns - given by $\frac{1-w}{2}e_{j}$ - are. Observing that $\frac{\partial}{\partial z_{j}}H(z,w)_{m,m}=0$, but $\frac{\partial}{\partial w}H(z,w)_{m,m}=\frac{1}{2}\neq 0$, we conclude that all partial derivatives of $H$ are linearly independent, hence $H$ is immersive. From the adapted singular value decomposition used in the proof of 11, we see that $H$ maps $\mathbb{S}^{2m-3}\times\mathbb{B}^{1}$ injectively into $M_{III}^{m}$. Considering the graph map $\Phi:\mathbb{S}^{2m-3}\rightarrow\mathbb{S}^{2m-3}\times\mathbb{C}$, $\Phi(z)=(z,\phi(z))$, which clearly is a $C^{k}$, but nowhere $C^{\infty}$-smooth CR embedding of $\mathbb{S}^{2m-3}$, we may write $h=H\circ\Phi$, showing that $h$ is a $C^{k}$, but nowhere $C^{\infty}$-smooth CR immersion of $\mathbb{S}^{2m-3}$ into $M_{III}^{m}$. Note that it is CR-transversal, since $H$ is transversal to $M_{III}^{m}$, and $\Phi$ was CR-transversal. Since $h$ is an injective immersion of the compact sphere, it is an embedding. ∎

6.0.4. Classical domains of the fourth kind

Somewhat different from the first three series of classical symmetric domains, the models for these domains, denoted by $D_{IV}^{m}$ for $m\geq 2$, are defined by simple quartic inequalities, first given in [5].

[TABLE]

The binding inequality is the second one. Indeed, a point $z\in\partial D_{IV}^{m}$ satisfying $z^{\ast}z=1$ also satisfies $|z^{T}z|\leq 1$ by Cauchy’s inequality, thus $1+|z^{T}z|^{2}-2z^{\ast}z\leq 0$. A low-dimensional toy image to have in mind is that of a lens-shaped region defined by $y^{2}-\frac{1}{4}\left(1-x^{2}\right)^{2}<0$, where we discard the unbounded region by requiring $x^{2}+y^{2}<1$. The smooth part of the boundary of $D_{IV}^{m}$, which we will denote by $M_{IV}^{m}$, is given by those $z\in\mathbb{C}^{m}$ satisfying $1+|z^{T}z|^{2}-2z^{\ast}z=0$ and $z^{\ast}z<1$.

In fact, $D_{IV}^{m}$ is biholomorphic to the tube domain over the light cone from 2. The tube domain over the future light cone is given by $\{(z_{1},\dots,z_{m-1},z_{m})\in\mathbb{C}^{m}:\Re(z_{1})^{2}+\dots+\Re(z_{m-1})^{2}<\Re(z_{m})^{2},\ \Re(z_{m})>0\}$. An explicit biholomorphism between the tube domain over the future light cone and $D_{IV}^{m}$ is given in [29] as

[TABLE]

where $\mathbf{i}$ denotes the vector $(0,\dots,0,i)\in\mathbb{C}^{m}$ and where $\mathcal{F}(z):=z_{m}^{2}-z_{1}^{2}-\dots-z_{m-1}^{2}$ for any $z\in\mathbb{C}^{m}$.

Let us nevertheless reprove the regularity result for $D_{IV}^{m}$ by computing the necessary quantities directly from Cartan’s representation. As an example point in $M_{IV}^{m}$ to base our calculations on, take $a:=(\frac{1}{2},\frac{i}{2},0,\dots,0)^{T}$. Here, $a^{T}a=0$ and $a^{\ast}a=\frac{1}{2}$. Contrary to the first three kinds of classical domains, $M_{IV}^{m}$ will necessarily behave exactly like the tube over the light cone.

Proposition 7**.**

Let $m\geq 2$ and $M$ be a minimal CR manifold. Then every CR map $h$ of regularity $C^{m-1}$ from $M$ into $M_{IV}^{m}$ which is of (real) rank $\geq 3$ is $C^{\infty}$-smooth on a dense open subset of $M$. If $h$ is CR transversal and for some $n_{+}\geq 1$, $M$ has at least $n_{+}$ positive Levi eigenvalues almost everywhere, then initial regularity may be dropped to $C^{m-n_{+}}$.

This is an immediate consequence of the boundary orbit theorem for $M_{IV}^{m}$, which allows us to take any point in $M_{IV}^{m}$ to $(\frac{1}{2},\frac{i}{2},0,\dots,0)$ by an ambient biholomorphism, and of 1 (2 for CR transversal $h$ and $n_{+}\geq 1$). The relevant structural properties of $M_{IV}^{m}$ of course do not differ at all from those of the tube over the light cone (2).

Lemma 12**.**

Let $a\in\mathbb{C}^{m}$ be such that $a^{T}a=0$ and $a^{\ast}a=\frac{1}{2}$. Around $a\in M_{IV}^{m}$, the pseudoconvex hypersurface $M_{IV}^{m}$ is foliated by complex lines. Its Levi form has exactly $m-2$ positive eigenvalues, and $\nu_{a}=0$.

Proof.

The complex quadric $\Sigma$ defined by $z^{T}z=0$ is a manifold where $z\neq 0$. Its intersection $S$ with $M_{IV}^{m}$ is given by $S=\{w\in\mathbb{C}^{m},w^{T}w=0,w^{\ast}w=\frac{1}{2}\}$. As it is the intersection of a complex manifold with the strongly pseudoconvex sphere given by $w^{\ast}w=\frac{1}{2}$, it is strongly pseudoconvex itself.

Near a point $w\in S$, the complex line given by $\eta_{w}(t)=w+t\bar{w}$ is contained in $S$. This is proven by straightforward calculation. Since $w^{\ast}w=\frac{1}{2}$ and $w^{T}w=0$, we observe $(w^{\ast}+\bar{t}w^{T})(\bar{w}+\bar{t}w)=\bar{t}$ and similar cancellations, and arrive at

[TABLE]

It remains to calculate the tensor $R^{S}(\cdot,\bar{w})$ at $a\in S$, since the section $\bar{w}$ already spans $T\eta$ along $S$. A vector $v\in T_{a}^{c}S$ is characterized by $(a+tv)^{T}(a+tv)=\mathcal{O}(|t|^{2})$ and $(a+tv)^{\ast}(a+tv)=\frac{1}{2}+\mathcal{O}(|t|^{2})$, which is equivalent to $a^{T}v=a^{\ast}v=0$. Take a CR vector $\bar{L}|_{a}\in T_{a}^{0,1}S$ with real part $\frac{1}{2}v$, and consider the holomorphic curve $\gamma(t)=a+tv$. Then $\bar{L}|_{a}\bar{w}=\frac{d}{d\bar{t}}|_{t=0}\overline{(a+tv)}=\bar{v}$, and we find that $\bar{v}\in T_{a}^{\bot}\eta_{a}=\left<\bar{a}\right>^{\bot}$ already, since $a^{\ast}v=0$. Therefore $R^{S}_{a}(\bar{L}|_{a},\bar{a})=\bar{v}$ only vanishes if $v$ and thus $\bar{L}|_{a}$ vanish, implying $\nu_{a}=0$. ∎