Variation and rigidity of quasi-local mass

TL;DR

This paper derives an evolution formula for Wang-Yau quasi-local energy, linking it to Bartnik mass, and uses it to prove a rigidity theorem characterizing equality cases in a localized Penrose inequality for 3-manifolds.

Contribution

It introduces a new evolution formula for quasi-local energy and establishes a rigidity theorem connecting quasi-local mass concepts with geometric inequalities.

Findings

Derivative of Wang-Yau energy equals Bartnik mass derivative under certain conditions

Rigidity theorem for 3-manifolds with nonnegative scalar curvature

Characterization of equality cases in localized Penrose inequality

Abstract

Inspired by the work of Chen-Zhang \cite{Chen-Zhang}, we derive an evolution formula for the Wang-Yau quasi-local energy in reference to a static space, introduced by Chen-Wang-Wang-Yau \cite{CWWY}. If the reference static space represents a mass minimizing, static extension of the initial surface $\Sigma$, we observe that the derivative of the Wang-Yau quasi-local energy is equal to the derivative of the Bartnik quasi-local mass at $\Sigma$. Combining the evolution formula for the quasi-local energy with a localized Penrose inequality proved in \cite{Lu-Miao}, we prove a rigidity theorem for compact $3$-manifolds with nonnegative scalar curvature, with boundary. This rigidity theorem in turn gives a characterization of the equality case of the localized Penrose inequality in $3$-dimension.

Equations

$${\int }_{{\Sigma }_t}N(\bar{H}-H)d\sigma$$

$$m+\frac{1}{8\pi }{\int }_{\Sigma }N(H_m-H)d\sigma \ge \sqrt{\frac{|{\Sigma }_{{}_H}|}{16\pi }}.$$

$$H=H_m,\sqrt{\frac{|{\Sigma }_{{}_H}|}{16\pi }}=m.$$

$$F:\Sigma \times I⟶M,\frac{\partial F}{\partial t}=\eta \nu .$$

$$(\bar{\Delta }N)\bar{g}-{\bar{D}}^{2}N+N\overline{Ric}=0,$$

$$\bar{F}:\Sigma \times I⟶\mathbb{N}$$

$${\bar{F}}_t^{\ast }(\bar{g})=F_t^{\ast }(g),\forall t\in I.$$

$$\frac{d}{dt}{\int }_{\Sigma }{N}_t({\bar{H}}_t-H_t)d{\sigma }_t,$$

$$\begin{array}{rl}& \frac{d}{dt}{\int }_{\Sigma }N(\bar{H}-H)d\sigma \\ =& {\int }_{\Sigma }N\left[\frac{1}{2}|A-\bar{A}{|}^2-\frac{1}{2}|H-\bar{H}{|}^2+\frac{1}{2}(R-\bar{R})\right]\eta d\sigma \\ & +{\int }_{\Sigma }\left[(f-\eta )\frac{\partial N}{\partial \bar{\nu }}+⟨\nabla N,Y⟩\right](\bar{H}-H)d\sigma .\end{array}$$

$$\begin{array}{r}\frac{d}{dt}{|}_{t=0}{\int }_{\Sigma }N(\bar{H}-H)d\sigma =\frac{1}{2}{\int }_{\Sigma }N|A-\bar{A}{|}^2\eta d\sigma .\end{array}$$

$$\begin{array}{rl}& \frac{d}{dt}{\int }_{\Sigma }N(\bar{H}-H)d\sigma \\ =& {\int }_{\Sigma }N\left[\frac{1}{2}|A-\bar{A}{|}^2-\frac{1}{2}|H-\bar{H}{|}^2\right]\eta d\sigma +{\int }_{\Sigma }(f-\eta )\frac{\partial N}{\partial \bar{\nu }}(\bar{H}-H)d\sigma \\ =& {\int }_{\Sigma }{\eta }^{-1}(f-\eta {)}^2\left(-N{\sigma }_2-\bar{H}\frac{\partial N}{\partial \bar{\nu }}\right)d\sigma .\end{array}$$

$$E_{{}_{WY}}^S(\Sigma ,\mathcal{S},X)=\frac{1}{8\pi }{\int }_{\Sigma }N(\bar{H}-H)d\sigma .$$

$$\frac{d}{dt}{E}_{{}_{WY}}^S({\Sigma }_t,\mathcal{S},X_t),$$

$$\frac{d}{dt}{|}_{t=0}{\mathfrak{m}}_{{}_B}({\Sigma }_t)=\frac{1}{16\pi }{\int }_{\Sigma }N\left(|A-\bar{A}{|}^2+R\right)\eta d\sigma .$$

$$\begin{array}{rl}& \frac{d}{dt}{|}_{t=0}{E}_{{}_{WY}}^S({\Sigma }_t,\mathcal{S},X_t)\\ =& \frac{1}{16\pi }{\int }_{\Sigma }N\left[|A-\bar{A}{|}^2+(R-\bar{R})\right]\eta d\sigma \\ =& \frac{d}{dt}{|}_{t=0}{\mathfrak{m}}_{{}_B}({\Sigma }_t).\end{array}$$

$${\gamma }^{\prime }=2\eta A,{\partial }_{t}d\sigma =\eta Hd\sigma$$

$${\gamma }^{\prime }=2f\bar{A}+L_Y\gamma ,{\partial }_{t}d\sigma =(f\bar{H}+\text{mbox}divY)d\sigma ,$$

$$2\eta A=2f\bar{A}+L_Y\gamma ,\eta H=f\bar{H}+\text{mbox}divY.$$

$$\begin{array}{rl}\frac{d}{dt}{\int }_{\Sigma }N\bar{H}d\sigma =& {\int }_{\Sigma }(N^{\prime }\bar{H}+N{\bar{H}}^{\prime })d\sigma +N\bar{H}{\partial }_{t}d\sigma .\end{array}$$

$$\begin{array}{r}{N}^{\prime }=⟨\bar{\nabla }N,\frac{\partial \bar{F}}{\partial t}⟩=⟨\bar{\nabla }N,f\bar{\nu }+Y⟩=f\frac{\partial N}{\partial \bar{\nu }}+⟨\nabla N,Y⟩.\end{array}$$

$$\begin{array}{r}{\int }_{\Sigma }{N}^{\prime }\bar{H}d\sigma ={\int }_{\Sigma }\left(f\frac{\partial N}{\partial \bar{\nu }}+⟨\nabla N,Y⟩\right)\bar{H}d\sigma .\end{array}$$

$${\bar{A}}_{\alpha \beta }^{\prime }=f{\bar{A}}_{\alpha \delta }{\bar{A}}_{\beta }^{\delta }+(L_Y\bar{A}{)}_{\alpha \beta }-({\nabla }^{2}f{)}_{\alpha \beta }+f⟨\bar{R}(\bar{\nu },{\partial }_{\alpha })\bar{\nu },{\partial }_{\beta }⟩,$$

$$\begin{array}{rl}{\bar{H}}^{\prime }=& ({\gamma }^{\alpha \beta }{)}^{\prime }{\bar{A}}_{\alpha \beta }+{\gamma }^{\alpha \beta }{\bar{A}}_{\alpha \beta }^{\prime }\\ =& -⟨{\gamma }^{\prime },\bar{A}⟩+f|\bar{A}{|}^2+⟨\gamma ,L_Y\bar{A}⟩-\Delta f-\bar{R}ic(\bar{\nu },\bar{\nu })f.\end{array}$$

$$\begin{array}{r}⟨{\gamma }^{\prime },\bar{A}⟩=⟨2f\bar{A}+L_Y\gamma ,\bar{A}⟩=2f|\bar{A}{|}^2+⟨L_Y\gamma ,\bar{A}⟩.\end{array}$$

$$\begin{array}{rl}{\bar{H}}^{\prime }=& -⟨L_Y\gamma ,\bar{A}⟩+⟨\gamma ,L_Y\bar{A}⟩-\Delta f-f|\bar{A}{|}^2-\bar{R}ic(\bar{\nu },\bar{\nu })f.\end{array}$$

$$-⟨L_Y\gamma ,\bar{A}⟩+⟨\gamma ,L_Y\bar{A}⟩=⟨Y,\nabla \bar{H}⟩.$$

$${\bar{H}}^{\prime }=-\Delta f-f|\bar{A}{|}^2-\bar{R}ic(\bar{\nu },\bar{\nu })f+⟨Y,\nabla \bar{H}⟩.$$

$$\begin{array}{rl}{\int }_{\Sigma }N{\bar{H}}^{\prime }d\sigma =& {\int }_{\Sigma }(-\Delta N-\bar{R}ic(\bar{\nu },\bar{\nu })N)f+N\left[-f|\bar{A}{|}^2+⟨Y,\nabla \bar{H}⟩\right]d\sigma \\ =& {\int }_{\Sigma }\bar{H}\frac{\partial N}{\partial \bar{\nu }}f-Nf|\bar{A}{|}^2+N⟨Y,\nabla \bar{H}⟩d\sigma .\end{array}$$

$$\Delta N+\bar{R}ic(\bar{\nu },\bar{\nu })N=-\bar{H}\frac{\partial N}{\partial \bar{\nu }},$$

$$\begin{array}{rl}& {\int }_{\Sigma }{N}^{\prime }\bar{H}+N{\bar{H}}^{\prime }d\sigma \\ =& {\int }_{\Sigma }\left(f\frac{\partial N}{\partial \bar{\nu }}+⟨\nabla N,Y⟩\right)\bar{H}+\bar{H}\frac{\partial N}{\partial \bar{\nu }}f-Nf|\bar{A}{|}^2+N⟨Y,\nabla \bar{H}⟩d\sigma \\ =& {\int }_{\Sigma }2f\frac{\partial N}{\partial \bar{\nu }}\bar{H}-Nf|\bar{A}{|}^2+⟨Y,\nabla (N\bar{H})⟩d\sigma .\end{array}$$

Full text

Combining the evolution formula for the quasi-local energy with a localized Penrose inequality proved in [10], we prove a rigidity theorem for compact $3$-manifolds with nonnegative scalar curvature, with boundary. This rigidity theorem in turn gives a characterization of the equality case of the localized Penrose inequality in $3$-dimension.

The first named author’s research was partially supported by CSC fellowship. The second named author’s research was partially supported by the Simons Foundation Collaboration Grant for Mathematicians #281105.

1. Introduction

The purpose in this paper is twofold. We derive a derivative formula for the integral

[TABLE]

along a family of hypersurfaces $\{\Sigma_{t}\}$ evolving in a Riemannian manifold $(M,g)$ with an assumption that $\Sigma_{t}$ can be isometrically embedded in a static space $(\mathbb{N},\bar{g})$ as a comparison hypersurface $\bar{\Sigma}_{t}$. Here $H$, $\bar{H}$ are the mean curvature of $\Sigma_{t}$, $\bar{\Sigma}_{t}$ in $(M,g)$, $(\mathbb{N},\bar{g})$, respectively, and $N$ is the static potential on $(\mathbb{N},\bar{g})$. When $\{\Sigma_{t}\}$ is a family of closed $2$-surfaces in a $3$-manifold $(M,g)$, integral (1.1) represents the Wang-Yau quasi-local energy in reference to the static space $(\mathbb{N},\bar{g})$, introduced by Chen-Wang-Wang-Yau [4]. In this case, if $(\mathbb{N},\bar{g})$ represents a mass minimizing, static extension of the initial surface $\Sigma_{0}$, we find that the derivative of the quasi-local energy agrees with the derivative of the Bartnik quasi-local mass at $\Sigma_{0}$ (see (2.8) in Section 2).

We also apply the derivative formula of (1.1) to prove a rigidity theorem for compact $3$-manifolds with nonnegative scalar curvature, with boundary. Precisely, we have

Theorem 1.1**.**

Let $(\Omega,\breve{g})$ be a compact, connected, orientable, $3$-dimensional Riemannian manifold with nonnegative scalar curvature, with boundary $\partial\Omega$. Suppose $\partial\Omega$ is the disjoint union of two pieces, $\Sigma_{{}_{O}}$ and $\Sigma_{{}_{H}}$, where

1. (i)

$\Sigma_{{}_{O}}$* has positive mean curvature ${H}$; and* 2. (ii)

$\Sigma_{{}_{H}}$* is a minimal hypersurface (with one or more components) and there are no other closed minimal hypersurfaces in $(\Omega,\breve{g})$.*

Let $\mathbb{M}^{3}_{m}$ be a $3$-dimensional spatial Schwarzschild manifold with mass $m>0$ outside the horizon. Suppose $\Sigma_{{}_{O}}$ is isometric to a convex surface $\Sigma\subset\mathbb{M}^{3}_{m}$ which encloses a domain $\Omega_{m}$ with the horizon $\partial\mathbb{M}^{3}_{m}$. Suppose $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$ on $\Sigma$, where ${\overline{\mathrm{Ric}}}$ is the Ricci curvature of the Schwarzschild metric $\bar{g}$ on $\mathbb{M}^{3}_{m}$ and $\nu$ is the outward unit normal to $\Sigma$. Let $H_{m}$ be the mean curvature of $\Sigma$ in $\mathbb{M}^{3}_{m}$ and $|\Sigma_{{}_{H}}|$ be the area of $\Sigma_{{}_{H}}$ in $(\Omega,\breve{g})$. If $H=H_{m}$ and $\sqrt{\frac{|\Sigma_{{}_{H}}|}{16\pi}}=m$, then $(\Omega,\breve{g})$ is isometric to $(\Omega_{m},\bar{g})$.

Theorem 1.1 gives a characterization of the equality case of a localized Penrose inequality proved in [10].

Theorem 1.2** ([10]).**

Let $(\Omega,\breve{g})$ be a compact, connected, orientable, $3$-dimensional Riemannian manifold with nonnegative scalar curvature, with boundary $\partial\Omega$. Suppose $\partial\Omega$ is the disjoint union of two pieces, $\Sigma_{{}_{O}}$ and $\Sigma_{{}_{H}}$, where

1. (i)

$\Sigma_{{}_{O}}$* has positive mean curvature ${H}$; and* 2. (ii)

$\Sigma_{{}_{H}}$, if nonempty, is a minimal hypersurface (with one or more components) and there are no other closed minimal hypersurfaces in $(\Omega,\breve{g})$.

Let $\mathbb{M}^{3}_{m}$ be a $3$-dimensional spatial Schwarzschild manifold with mass $m>0$ outside the horizon. Suppose $\Sigma_{{}_{O}}$ is isometric to a convex surface $\Sigma\subset\mathbb{M}^{3}_{m}$ which encloses a domain $\Omega_{m}$ with the horizon $\partial\mathbb{M}^{3}_{m}$. Suppose $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$ on $\Sigma$, where ${\overline{\mathrm{Ric}}}$ is the Ricci curvature of the Schwarzschild metric $\bar{g}$ on $\mathbb{M}^{3}_{m}$ and $\nu$ is the outward unit normal to $\Sigma$. Then

[TABLE]

Here $N$ is the static potential on $\mathbb{M}^{3}_{m}$, $H_{m}$ is the mean curvature of $\Sigma$ in $\mathbb{M}^{3}_{m}$, and $|\Sigma_{{}_{H}}|$ is the area of $\Sigma_{{}_{H}}$ in $(\Omega,\breve{g})$. Furthermore, equality in (1.2) holds if and only if

[TABLE]

By Theorems 1.1 and (1.3), we have the following rigidity statement concerning the equality case of (1.2).

Theorem 1.3**.**

Equality in (1.2) in Theorem 1.2 holds if and only if $(\Omega,\breve{g})$ is isometric to $(\Omega_{m},\bar{g})$.

Our motivation to consider the evolution of (1.1) and the proof of Theorem 1.1 are inspired by a recent paper of Chen and Zhang [5]. In [5], Chen-Zhang proved the global rigidity of a convex surface $\Sigma$ with $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$ among all isometric surfaces $\Sigma^{\prime}$ in $\mathbb{M}^{3}_{m}$ having the same mean curvature and enclosing the horizon. As a key step in their proof, they computed the first variation of the quasi-local energy of $\Sigma^{\prime}$ in reference to $\mathbb{M}^{3}_{m}$. Such a variational consideration is made possible by the openness result of solutions to the isometric embedding problem into warped product space, which is due to Li and Wang [8]. Combining the variation formula with inequality (1.2), Chen-Zhang established the rigidity of $\Sigma$ in $\mathbb{M}^{3}_{m}$.

This paper may be viewed as a further application of the method of Chen-Zhang. In Section 2, we compute the derivative of (1.1) (see Formula 2.1) and relate it to the derivative of the Bartnik quasi-local mass. In Section 3, we prove Theorem 1.1 by applying Formula 2.1 and Theorem 1.2. In Section 4, we discuss the implication of (2.8) on the relation between the Bartnik mass and the Wang-Yau quasi-local energy.

2. Evolution of quasi-local mass

In this section we derive a formula that is inspired by [5, Lemma 2]. First we fix some notations. Let $(M,g)$ be an $(n+1)$-dimensional Riemannian manifold and $\Sigma$ be an $n$-dimensional closed manifold. Consider a family of embedded hypersurfaces $\{\Sigma_{t}\}$ evolving in $(M,g)$ according to

[TABLE]

Here $F$ is a smooth map, $I$ is some open interval containing [math], $\Sigma_{t}=F_{t}(\Sigma)$ with $F_{t}(\cdot)=F(\cdot,t)$, $\nu$ is a chosen unit normal to $\Sigma_{t}=F_{t}(\Sigma)$, and $\eta$ denotes the speed of the evolution of $\{\Sigma_{t}\}$.

Let $(\mathbb{N},\bar{g})$ denote an $(n+1)$-dimensional static Riemannian manifold. Here $(\mathbb{N},\bar{g})$ is called static (cf. [6]) if there exists a nontrivial function $N$ such that

[TABLE]

where $\bar{Ric}$ is the Ricci curvature of $(\mathbb{N},\bar{g})$, $\bar{D}^{2}N$ is the Hessian of $N$ and $\bar{\Delta}$ is the Laplacian of $N$. The function $N$ is called a static potential on $(\mathbb{N},\bar{g})$.

In what follows, we consider another family of embedded hypersurfaces $\{\bar{\Sigma}_{t}\}$ evolving in $(\mathbb{N},\bar{g})$ according to

[TABLE]

with $\bar{\Sigma}_{t}=\bar{F}_{t}(\Sigma)$ and $\bar{F}_{t}(\cdot)=\bar{F}(\cdot,t)$. We will make an important assumption:

[TABLE]

In particular, this means that $\bar{\Sigma}_{t}$ is assumed to be isometric to $\Sigma_{t}$ for each $t$.

Remark 2.1*.*

We emphasize that, when $n=2$, given any $\{\Sigma_{t}\}$ in $(M,g)$, if $\Sigma_{0}$ admits an isometric embedding into $(\mathbb{N},\bar{g})$, there exists a family of $\{\bar{\Sigma}_{t}\}$ in $(\mathbb{N},\bar{g})$ satisfying condition (2.2). This is guaranteed by the openness result of solutions to the isometric embedding problem, which is due to Li and Wang [8, 9].

We will compute

[TABLE]

where $N_{t}=\bar{F}_{t}^{*}(N)$ is the pull back of the static potential $N$ on $(\mathbb{N},\bar{g})$; $H_{t}$, $\bar{H}_{t}$ are the mean curvature of $\Sigma_{t}$, $\bar{\Sigma}_{t}$ in $(M,g)$, $(\mathbb{N},\bar{g})$, respectively; and $d\sigma_{t}$ is the area element of the pull back metric $\gamma_{t}=\bar{F}_{t}^{*}(\bar{g})=F_{t}^{*}(g)$. For simplicity, the lower index $t$ is omitted below.

Formula 2.1**.**

Given $\{\Sigma_{t}\}$, $\{\bar{\Sigma}_{t}\}$ evolving in $(M,g)$, $(\mathbb{N},\bar{g})$ as specified above,

[TABLE]

Here $A$, $\bar{A}$ are the second fundamental forms of $\Sigma_{t}$, $\bar{\Sigma}_{t}$ in $(M,g)$, $(\mathbb{N},\bar{g})$, respectively; $R$, $\bar{R}$ are the scalar curvature of $(M,g)$, $(\mathbb{N},\bar{g})$, respectively; $f$ and $Y$ are the lapse and the shift associated to $\displaystyle\frac{\partial\bar{F}}{\partial t}$, i.e. $\displaystyle\frac{\partial\bar{F}}{\partial t}=f\bar{\nu}+Y$, where $f$ is a function and $Y$ is tangential to $\bar{\Sigma}_{t}$; and $\nabla$ denotes the gradient on $(\bar{\Sigma}_{t},\gamma)$.

Remark 2.2*.*

Suppose $(M,g)$ and $(\mathbb{N},\bar{g})$ both are $\mathbb{M}^{3}_{m}$ and suppose $H=\bar{H}$ at $t=0$, (2.3) becomes

[TABLE]

This is the formula in [5, Lemma 2].

Remark 2.3*.*

If $Y=0$ and $R=\bar{R}$, (2.3) reduces to

[TABLE]

This is the formula in [10, Proposition 2.2].

We now comment on the physical meaning of (2.3). Suppose $n=2$. In [4], Chen, Wang, Wang and Yau introduced a notion of quasi-local energy of a $2$-surface $\Sigma$ in reference to the static spacetime $\mathcal{S}=(\mathbb{R}^{1}\times\mathbb{N},-N^{2}dt^{2}+\bar{g})$. The notion is a generalization of the Wang-Yau quasi-local energy [15, 16] for which the reference $\mathcal{S}$ is the Minkowski spacetime $\mathbb{R}^{3,1}$. For this reason, we denote this quasi-local energy of $\Sigma$ by $E^{S}_{{}_{WY}}(\Sigma,\mathcal{S},X)$, where $X:\Sigma\rightarrow\mathcal{S}$ is an associated isometric embedding. When $\Sigma$ lies in a time-symmetric slice in the physical spacetime, one may focus on the case $X$ embeds $\Sigma$ into a constant $t$-slice of $\mathcal{S}$, i.e. $X:\Sigma\rightarrow(\mathbb{N},\bar{g})$. In this case, setting $\tau=0$ in equation (2.10) in [4], one has

[TABLE]

Therefore, up to a multiplicative constant, (2.3) is a formula of

[TABLE]

where $X_{t}=\bar{F}_{t}\circ F_{t}^{-1}$ is the isometric embedding of $\Sigma_{t}$ in $(\mathbb{N},\bar{g})$ as $\bar{\Sigma}_{t}$.

Next, we tie (2.3) with the evolution formula of the Bartnik quasi-local mass $\mathfrak{m}_{{}_{B}}(\cdot)$. We defer the detailed definition of the Bartnik mass $\mathfrak{m}_{{}_{B}}(\cdot)$ to Section 4. For the moment, we recall the following evolution formula of $\mathfrak{m}_{{}_{B}}(\cdot)$ derived in [12, Theorem 3.1] under a stringent condition.

Formula 2.2** ([12]).**

Suppose $\Sigma_{t}$ has a mass minimizing, static extension $(M_{t}^{s},g_{t}^{s})$ such that $\{(M_{t}^{s},g_{t}^{s})\}$ depends smoothly on $t$. One has

[TABLE]

To relate (2.3) to (2.7), we assume that $(\mathbb{N},\bar{g})$ represents a mass minimizing, static extension of the surface $\Sigma_{0}\subset(M,g)$. Then, by assumption, $H=\bar{H}$ at $t=0$. It follows from (2.3), (2.6) and (2.7) that

[TABLE]

We will reflect more on this relation in Section 4.

In the remainder of this section, we give a proof of Formula 2.1.

Proof of Formula 2.1.

By the evolution equations $\frac{\partial F}{\partial t}=\eta\nu$ and $\frac{\partial\bar{F}}{\partial t}=f\bar{\nu}+Y$, we have

[TABLE]

and

[TABLE]

where $\mbox{div}Y$ is the divergence of $Y$ on $(\Sigma,\gamma)$. Thus,

[TABLE]

We first compute

[TABLE]

Let $\bar{\nabla}$ denote the gradient on $(\mathbb{N},\bar{g})$. We have

[TABLE]

Hence,

[TABLE]

Recall that

[TABLE]

where $\nabla^{2}$ denotes the Hessian on $(\Sigma,\gamma)$. Hence,

[TABLE]

By (2.10),

[TABLE]

Thus,

[TABLE]

One checks that

[TABLE]

Hence,

[TABLE]

Thus,

[TABLE]

Here we have used

[TABLE]

which follows from the static equation (2.1).

By (2.14) and (2.20),

[TABLE]

On the other hand, by (2.10),

[TABLE]

Therefore, it follows from (2.21) and (2.22) that

[TABLE]

To proceed, we note that by (2.10),

[TABLE]

Thus,

[TABLE]

Integrating by parts, we have

[TABLE]

By the Codazzi equation and the static equation,

[TABLE]

Here

[TABLE]

Hence,

[TABLE]

Therefore, (2.23) can be rewritten as

[TABLE]

We now turn to the term $\int_{\Sigma}NH\,d\sigma$. We have

[TABLE]

Here

[TABLE]

Therefore,

[TABLE]

We group the zero order terms of $N$ in $\displaystyle\frac{d}{dt}\int_{\Sigma}N(\bar{H}-H)\,d\sigma$ first. Using $\gamma^{\prime}=2\eta A$, we have

[TABLE]

Thus, omitting the terms $\eta$ and $N$, using the Gauss equation, we have

[TABLE]

Integrating by part and using the fact $\eta H=f\bar{H}+\mbox{div}Y$, we conclude

[TABLE]

∎

3. Equality case of the localized Penrose inequality

In this section, we apply Formula 2.1, the openness result of the isometric embedding problem [8], and Theorem 1.2 to prove Theorem 1.1.

Proof of Theorem 1.1.

Let $A$, $\bar{A}$ be the second fundamental form of $\Sigma_{{}_{O}}$, $\Sigma$ in $(\Omega,\breve{g})$, $\mathbb{M}^{3}_{m}$, respectively. Viewing $\bar{A}$ as a tensor on $\Sigma_{{}_{O}}$ via the surface isometry, we want to show $A=\bar{A}$.

In $(\Omega,\breve{g})$, consider a smooth family of $2$-surfaces $\{\Sigma_{t}\}_{-\epsilon<t\leq 0}$ such that $\Sigma_{0}=\Sigma_{{}_{O}}$ and $\Sigma_{t}$ is $|t|$-distance away from $\Sigma_{{}_{O}}$. We can parametrize $\{\Sigma_{t}\}$ so that, as $t$ increases, $\Sigma_{t}$ evolves in a direction normal to $\Sigma_{t}$ and has constant unit speed. Applying the openness result of the isometric embedding problem in [8], we obtain a smooth family of $2$-surfaces $\{\bar{\Sigma}_{t}\}_{-\epsilon<t\leq 0}$ in $\mathbb{M}^{3}_{m}$ so that $\bar{\Sigma}_{0}=\Sigma$ and condition (2.2) is satisfied by $\{\Sigma_{t}\}$ and $\{\bar{\Sigma}_{t}\}$. By (2.3) and the assumption $H=H_{m}$, we have

[TABLE]

Here $N$ is the static potential on $\mathbb{M}^{3}_{m}$, which is positive away from the horizon, and $R$ is the scalar curvature of $(\Omega,\breve{g})$.

Suppose $A\neq\bar{A}$. Then, by (3.1) and the assumption $R\geq 0$,

[TABLE]

Thus, for small $t<0$,

[TABLE]

We claim (3.3) contradicts Theorem 1.2. To see this, we can first consider the case $\overline{\mathrm{Ric}}(\nu,\nu)<0$ on $\Sigma$. By choosing $\epsilon$ small, we may assume $\overline{\mathrm{Ric}}(\nu,\nu)<0$ on each $\bar{\Sigma}_{t}$. Hence, we can apply Theorem 1.2 to the region in $\Omega$ enclosed by $\Sigma_{t}$ and $\Sigma_{{}_{H}}$. It follows from (1.2) and the assumption $m=\sqrt{\frac{|\Sigma_{{}_{H}}|}{16\pi}}$ that

[TABLE]

This is a contradiction to (3.3).

To include the case $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$ on $\Sigma$, we point out that this assumption was imposed in [10] only to guarantee that the flow in $\mathbb{M}^{3}_{m}$, which starts from $\Sigma$ and satisfies equation (4.2) in [10], has the property that its leaves have positive scalar curvature (see Lemma 3.8 in [10]). Now, if $\Sigma$ is slightly perturbed to a nearby surface $\Sigma^{\prime}$ in $\mathbb{M}^{3}_{m}$, though $\Sigma^{\prime}$ may not satisfy $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$, the flow to (4.2) in [10] starting from $\Sigma^{\prime}$ remains to have such a property. (More precisely, this follows from estimates in Lemmas 3.6, 3.7 and 3.11 of [10].) Therefore, for small $t<0$, we can still apply Theorem 1.2 to conclude (3.4), which contradicts (3.3).

Thus we have $A=\bar{A}$. For the same reason, we also know $R=0$ along $\Sigma_{{}_{O}}$ in $(\Omega,\breve{g})$. Next, we consider the manifold $(\hat{M},\hat{g})$ obtained by gluing $(\Omega,\breve{g})$ and $(\mathbb{M}^{3}_{m}\setminus\Omega_{m},\bar{g})$ along $\Sigma_{{}_{O}}$ that is identified with $\Sigma$. Since $A=\bar{A}$, the metric $\hat{g}$ on $\hat{M}$ is $C^{1,1}$ across $\Sigma_{{}_{O}}$ and is smooth up to $\Sigma_{{}_{O}}$ from its both sides in $\hat{M}$. To finish the proof, we check that the rigidity statement of the Riemannian Penrose inequality holds on this $(\hat{M},\hat{g})$.

We apply the conformal flow used by Bray [2] in his proof of the Riemannian Penrose inequality. Since $\hat{g}$ is $C^{1,1}$, equations (13) - (16) in [2] which define the flow hold in the classical sense when $g_{0}$ is replaced by $\hat{g}$. Existence of this flow with initial condition $\hat{g}$ follows from Section 4 in [2]. The difference is that, along the flow which we denote by $\{\hat{g}(t)\}$, the outer minimizing horizon $\Sigma(t)$ is $C^{2,\alpha}$ and the green function in Theorems 8 and 9 in [2] is $C^{2,\alpha}$, for any $0<\alpha<1$. These regularities are sufficient to show Theorem 6 in [2] holds, i.e. the area of $\Sigma(t)$ stays the same; and the results on the mass and the capacity in Theorems 8 and 9 in [2] remain valid. Moreover, at $t=0$, by the proof of Theorem 10 in [2], i.e. equation (113), we have

[TABLE]

where $\mathcal{E}(\Sigma_{{}_{H}},\hat{g})$ is the capacity of $\Sigma_{{}_{H}}$ in $(\hat{M},\hat{g})$ and the inequality in (3.5) is given by Theorem 9 in [2].

Now, if $\frac{d}{dt^{+}}m(t)|_{t=0}<0$, then for $t$ small, we would have

[TABLE]

where $m(t)$ is the mass of $\hat{g}(t)$. But (3.6) violates the Riemannian Penrose inequality (for metrics possibly with corner along a hypersurface, cf. [11]). Thus, we must have

[TABLE]

Since Theorem 9 in [2] holds on $(\hat{M},\hat{g})$, by its rigidity statement we conclude that $(\hat{M},\hat{g})$ is isometric to $\mathbb{M}^{3}_{m}$. ∎

Remark 3.1*.*

As mentioned in [10, Remark 5.1], Theorem 1.1 would also follow if one could establish the rigidity statement for the Riemannian Penrose inequality on manifolds with corners along a hypersurface (cf. [11, Proposition 3.1]). Results along this direction can be found in [14].

4. Bartnik mass and Wang-Yau quasi-local energy

In (2.8) of Section 2, we have observed that, if $(\mathbb{N},\bar{g})$ represents a mass minimizing, static extension of $\Sigma_{0}\subset(M,g)$, then

[TABLE]

This observation was based on (2.7), which requires a stringent assumption that mass minimizing, static extensions of $\{\Sigma_{t}\}$ exist and depend smoothly on $t$. In this section, we will give a rigorous proof that (2.7) is true whenever the Bartnik data of $\Sigma_{0}$ corresponds to that of a surface in a spatial Schwarzschild manifold. We will also discuss the implication, suggested by (4.1), on the relation between the Bartnik mass and the Wang-Yau quasi-local energy.

First, we recall the definition of $\mathfrak{m}_{{}_{B}}(\cdot)$. Given a closed $2$-surface $\Sigma$, which bounds a bounded domain, in a $3$-manifold $(M,g)$ with nonnegative scalar curvature, $\mathfrak{m}_{{}_{B}}(\Sigma)$ is given by

[TABLE]

Here $\mathfrak{m}(\tilde{g})$ is the mass of $(\tilde{M},\tilde{g})$, which is an asymptotically flat $3$-manifold with nonnegative scalar curvature, with boundary $\partial\tilde{M}$. $(\tilde{M},\tilde{g})$ is called an admissible extension of $\Sigma$ if $\partial\tilde{M}$ is isometric to $\Sigma$ and the mean curvature of $\partial\tilde{M}$ equals the mean curvature $H$ of $\Sigma$. Moreover, it is assumed that $(\tilde{M},\tilde{g})$ satisfies certain non-degeneracy condition that prevents $\mathfrak{m}(\tilde{g})$ from becoming trivially small. For instance, one often assumes that $(\tilde{M},\tilde{g})$ contains no closed minimal surfaces or $\partial\tilde{M}$ is outer minimizing in $(\tilde{M},\tilde{g})$ (cf. [1, 2, 3, 7]).

Theorem 4.1**.**

Let $\Sigma$ be a $2$-surface with positive mean curvature in a $3$-manifold $(M,g)$ of nonnegative scalar curvature. Suppose $\Sigma$ is isometric to a convex surface $\bar{\Sigma}$ with $\overline{\mathrm{Ric}}(\nu,\nu)\leq 0$ in a spatial Schwarzschild manifold $(\mathbb{M}^{3}_{m},\bar{g})$ of mass $m>0$. Suppose $\bar{\Sigma}$ encloses a domain $\Omega_{m}$ with the horizon of $(\mathbb{M}^{3}_{m},\bar{g})$.

1. (i)

Let $X:\Sigma\rightarrow(\mathbb{M}^{3}_{m},\bar{g})$ be an isometric embedding such that $X(\Sigma)=\bar{\Sigma}$. Let $N$ be the static potential on $\mathbb{M}^{3}_{m}$ and let $\mathcal{S}_{m}$ denote the Schwarzschild spacetime, i.e. $\mathcal{S}_{m}=(\mathbb{R}^{1}\times\mathbb{M}^{3}_{m},-Ndt^{2}+\bar{g})$. Then

[TABLE]

Moreover, equality holds if and only if $H=\bar{H}$ and $\mathfrak{m}_{{}_{B}}(\Sigma)=m$. Here $H$, $\bar{H}$ are the mean curvature of $\Sigma$, $\bar{\Sigma}$ in $(M,g)$, $(\mathbb{M}^{3}_{m},\bar{g})$, respectively. 2. (ii)

Suppose $H=\bar{H}$. Let $\{\Sigma_{t}\}_{|t|<\epsilon}$ be a smooth family of $2$-surfaces evolving in $(M,g)$ according to $\frac{\partial F}{\partial t}=\eta\nu$ and satisfying $\Sigma_{0}=\Sigma$. If $\mathfrak{m}_{{}_{B}}(\Sigma_{t})$ is differentiable at $t=0$, then

[TABLE]

Here $A$, $\bar{A}$ are the second fundamental form of $\Sigma$, $\bar{\Sigma}$ in $(M,g)$, $(\mathbb{M}^{3}_{m},\bar{g})$, respectively, and $R$ is the scalar curvature of $(M,g)$.

Proof.

Part (i) was proved in [10, Theorem 5.1]. To show part (ii), we first note that the assumption $H=\bar{H}$ implies $\mathfrak{m}_{{}_{B}}(\Sigma)=m$. This is because, if $(\tilde{M},\tilde{g})$ is any admissible extension of $\Sigma$, by gluing $(\tilde{M},\tilde{g})$ with $\Omega_{m}$ along $\bar{\Sigma}_{0}$ and applying the Riemannian Penrose inequality, one has $\mathfrak{m}(\tilde{g})\geq m$. On the other hand, $\mathbb{M}^{3}_{m}\setminus\Omega_{m}$ is an admissible extension of $\Sigma$. Hence, $\mathfrak{m}_{{}_{B}}(\Sigma)=m$.

Next, we proceed as in the proof of Theorem 1.1. By the result of Li-Wang [8], for small $\epsilon$, there exists a smooth family of embeddings $\{X_{t}\}_{|t|<\epsilon}$ which isometrically embeds $\Sigma_{t}$ in $(\mathbb{M}^{3}_{m},\bar{g})$ such that $X_{0}=X$. By (i), for each small $t$, we have

[TABLE]

Note that $\mathfrak{m}_{{}_{B}}(\Sigma_{0})=m$ and $E^{S}_{{}_{WY}}(\Sigma_{0},\mathcal{S}_{m},X_{0})=0$. Hence, it follows from (4.3) that

[TABLE]

Here in the last step we have used (2.3). ∎

We propose a conjecture that is inspired by Theorem 4.1.

Conjecture 4.1**.**

Let $\Sigma$ be a $2$-surface, bounding some finite domain, in a $3$-manifold $(M,g)$ of nonnegative scalar curvature. Let $(\mathbb{N},\bar{g})$ be a complete, asymptotically flat $3$-manifold with nonnegative scalar curvature such that $(\mathbb{N},\bar{g})$ is static outside a compact set $K$. Suppose the static potential $N$ on $\mathbb{N}\setminus K$ is positive. Let $\mathcal{S}$ be the static spacetime generated by $(\mathbb{N}\setminus K,\bar{g})$, i.e. $\mathcal{S}=(\mathbb{R}^{1}\times(\mathbb{N}\setminus K),-N^{2}dt^{2}+\bar{g})$. Suppose there exists an isometric embedding $X:\Sigma\rightarrow(\mathbb{N},\bar{g})$ such that $\bar{\Sigma}=X(\Sigma)$ encloses $K$. Let $H$, $\bar{H}$ be the mean curvature of $\Sigma$, $\bar{\Sigma}$ in $(M,g)$, $(\mathbb{N},\bar{g})$, respectively. Then, under suitable conditions on $\Sigma$ and $\bar{\Sigma}$,

[TABLE]

Moreover, equality holds if and only if

[TABLE]

in which case $(\mathbb{N},\bar{g})$, outside $\bar{\Sigma}$, is a mass minimizing, static extension of $\Sigma$.

By results in [13], Conjecture 4.1 is true when $(\mathbb{N},\bar{g})$ is $\mathbb{R}^{3}$. By Theorem 4.1 (i), Conjecture 4.1 is also true when $(\mathbb{N}\setminus K,\bar{g})$ is an exterior region in $(\mathbb{M}^{3}_{m},\bar{g})$.

If Conjecture 4.1 is valid and if a mass minimizing, static extension of $\Sigma$ exists, then it would follow that

[TABLE]

Acknowledgement. The authors would like to thank Po-Ning Chen and Xiangwen Zhang for helpful comments on this work.