Least energy nodal solutions of Hamiltonian elliptic systems with Neumann boundary conditions

TL;DR

This paper investigates the existence, symmetry, and qualitative properties of least energy nodal solutions to Hamiltonian elliptic systems with Neumann boundary conditions, introducing a novel transformation to handle non-standard settings.

Contribution

It introduces a new $L^t$-norm-preserving transformation for analyzing solutions, revealing symmetry-breaking and monotonicity properties in Neumann problems.

Findings

Least energy solutions are foliated Schwarz symmetric.

Radial solutions are strictly monotone due to the transformation.

The approach applies to scalar models, providing new insights.

Abstract

We study existence, regularity, and qualitative properties of solutions to the system \[ -\Delta u = |v|^{q-1} v\quad \text{ in }\Omega,\qquad -\Delta v = |u|^{p-1} u\quad \text{ in }\Omega,\qquad \partial_\nu u=\partial_\nu v=0\quad \text{ on }\partial\Omega, \] with $\Omega\subset \mathbb R^N$ bounded; in this setting, all nontrivial solutions are sign changing. Our proofs use a variational formulation in dual spaces, considering sublinear $pq< 1$ and superlinear $pq>1$ problems in the subcritical regime. In balls and annuli we show that least energy solutions (l.e.s.) are foliated Schwarz symmetric and, due to a symmetry-breaking phenomenon, l.e.s. are not radial functions; a key element in the proof is a new $L^t$-norm-preserving transformation, which combines a suitable flipping with a decreasing rearrangement. This combination allows us to treat annular domains, sign-changing functions, and Neumann problems, which are non-standard settings to use rearrangements and symmetrizations. In particular, we show that our transformation diminishes the (dual) energy and, as a consequence, radial l.e.s. are strictly monotone. We also study unique continuation properties and simplicity of zeros. Our theorems also apply to the scalar associated model, where our approach provides new results as well as alternative proofs of known facts.

Figures

Equations

$$-\Delta u=|v{|}^{q-1}v\text{ in }\Omega ,-\Delta v=|u{|}^{p-1}u\text{ in }\Omega ,{\partial }_{\nu }u={\partial }_{\nu }v=0\text{ on }\partial \Omega ,$$

$$p,q>0,pq\ne 1,\text{ and }\frac{1}{p+1}+\frac{1}{q+1}>\frac{N-2}{N}.$$

$${\int }_{\Omega }|u{|}^{p-1}u={\int }_{\Omega }|v{|}^{q-1}v=0.$$

$$W^{2,\frac{p+1}{p}}(\Omega )\hookrightarrow L^{q+1}(\Omega )\text{ and }{W}^{2,\frac{q+1}{q}}(\Omega )\hookrightarrow L^{p+1}(\Omega )\text{ compactly}.$$

$$(u,v)\mapsto I(u,v)={\int }_{\Omega }\nabla u\cdot \nabla v-\frac{|u{|}^{p+1}}{p+1}-\frac{|v{|}^{q+1}}{q+1}dx.$$

$$c:=\inf \left\{I(u,v):(u,v)\not\equiv (0,0),(u,v)\text{ is a strong solution of eqrefNHS}\right\}.$$

$$\displaystyle X^{s}=\Big{\{}f\in L^{s}(\Omega):\ \int_{\Omega}f=0\Big{\}},\qquad\alpha:=\frac{p+1}{p},\qquad\beta:=\frac{q+1}{q},\qquad\text{and}\qquad X:=X^{\alpha}\times X^{\beta},$$

$$\varphi (f,g):={\int }_{\Omega }\frac{|f{|}^{\alpha }}{\alpha }+\frac{|g{|}^{\beta }}{\beta }-gKfdx,(f,g)\in X.$$

$$K_{t}h:=Kh+c_t(h)\text{ for some }{c}_t(h)\in \mathbb{R}\text{ such that }{\int }_{\Omega }|K_{t}h{|}^{t-1}{K}_{t}h=0.$$

$$\mathcal{N}$$

$$\varphi (f,g)=\underset{X}{\inf }\varphi \text{ if }pq<1\text{ or }\varphi (f,g)=\underset{\mathcal{N}}{\inf }\varphi \text{ if }pq>1.$$

$$\Omega$$

$$\varphi (f,g)=\underset{X_{rad}}{\inf }\varphi \text{ if }pq<1\text{ or }\varphi (f,g)=\underset{{\mathcal{N}}_{rad}}{\inf }\varphi \text{ if }pq>1.$$

$$\mathcal{I}:L_{rad}^{\infty }(\Omega )\to C_{rad}(\overline{\Omega }),\mathcal{I}h(x):={\int }_{\{\delta <|y|<|x|\}}h(y)dy=N{\omega }_N{\int }_{\delta }^{|x|}h(\rho ){\rho }^{N-1}d\rho ,$$

$$\mathfrak{F}:C_{rad}(\overline{\Omega })\to L_{rad}^{\infty }(\Omega ),\mathfrak{F}h:=({\chi }_{\{\mathcal{I}h>0\}}-{\chi }_{\{\mathcal{I}h\le 0\}})h.$$

$$h^{\divideontimes }\in L_{rad}^{\infty }(\Omega ),h^{\divideontimes }(x):=(\mathfrak{F}h{)}^{\#}({\omega }_N|x{|}^N-{\omega }_N{\delta }^N).$$

$$\varphi (f^{\divideontimes },g^{\divideontimes })\le \varphi (f,g).$$

$$\text{If (f,g) satisfies eqreflee, then f and g are not radial}.$$

$$-\Delta u_{x_1}$$

$${\int }_{\Omega (e_1)}p|u{|}^{p-1}{u}_{x_1}(u_{x_1}-K\bar{g})+q|v{|}^{q-1}{v}_{x_1}(v_{x_1}-K\bar{f})dx\ge 0,\Omega (e_1):=\Omega \cap \{x_1>0\}.$$

$$p>0,p\ne 1,(N-2)p<(N+2).$$

$$-\Delta u=|u{|}^{p-1}u\text{ in }\Omega ,{\partial }_{\nu }u=0\text{ on }\partial \Omega ,$$

$$E(u):=\frac{I(u,u)}{2}={\int }_{\Omega }\frac{1}{2}|\nabla u{|}^2-\frac{1}{p}|u{|}^{p}dx$$

$$N_p=\{u\in H^1(\Omega ):{\int }_{\Omega }|u{|}^{p-1}u=0\},$$

$$u,v\in W:=\{w\in C^1(\overline{\Omega }):\nabla w(x)\ne 0\text{ if }x\in \overline{\Omega }\text{ satisfies }|x|>\delta \text{ and }w(x)=0\},$$

$$\text{Sublinear: }pq$$

$$\text{Superlinear and subcritical: }pq$$

$$\alpha :=\frac{p+1}{p},{\alpha }^{\prime }=p+1,\beta :=\frac{q+1}{q},{\beta }^{\prime }=q+1.$$

$$-\Delta u=h\text{ in }\Omega ,{\partial }_{\nu }u=0\text{ on }\partial \Omega ,{\int }_{\Omega }u=0$$

$${\int }_{\Omega }\nabla u\nabla \phi ={\int }_{\Omega }h\phi \text{ for all }\phi \in W^{1,s^{\prime }}(\Omega ),s^{\prime }=\frac{s}{s-1},$$

Full text

[TABLE]

with $\Omega\subset\mathbb{R}^{N}$ bounded; in this setting, all nontrivial solutions are sign changing. Our proofs use a variational formulation in dual spaces, considering sublinear $pq<1$ and superlinear $pq>1$ problems in the subcritical regime. In balls and annuli we show that least energy solutions (l.e.s.) are foliated Schwarz symmetric and, due to a symmetry-breaking phenomenon, l.e.s. are not radial functions; a key element in the proof is a new $L^{t}$-norm-preserving transformation, which combines a suitable flipping with a decreasing rearrangement. This combination allows us to treat annular domains, sign-changing functions, and Neumann problems, which are non-standard settings to use rearrangements and symmetrizations. In particular, we show that our transformation diminishes the (dual) energy and, as a consequence, radial l.e.s. are strictly monotone. We also study unique continuation properties and simplicity of zeros. Our theorems also apply to the scalar associated model, where our approach provides new results as well as alternative proofs of known facts.

2010 MSC 35J50 (Primary); 35B05, 35B06, 35B07, 35J47, 35J15

Keywords Dual method, subcritical, Hamiltonian elliptic systems, flipping techniques, symmetry breaking, unique continuation.

1 Introduction

Let $N\geq 1$, $\Omega\subset\mathbb{R}^{N}$ be a smooth bounded domain, and consider the following Hamiltonian elliptic system with Neumann boundary conditions

[TABLE]

where $\nu$ is the outer normal vector on $\partial\Omega$ and we consider $p,q>0$ in the sublinear ($pq<1$) or the superlinear ($pq>1$) cases satisfying a subcritical condition, that is,

[TABLE]

In fact, in this setting the more general notion of linearity is $pq=1$ or, equivalently, $1/(p+1)+1/(q+1)=1$ [14]. On the other hand, the last inequality in (1.2) means that the exponents $(p,q)$ are below the critical hyperbola (i.e., $(p,q)$ is subcritical) [13, 26], and this condition is trivially satisfied if $N=1,2$ or if $pq<1$.

Systems with a Hamiltonian structure such as (1.1) have been extensively studied in the past 25 years and many results are known regarding existence, multiplicity, concentration phenomena, positivity, symmetry, Liouville theorems, etc. We refer to the surveys [15, 31, 6] for an overview of the topic and to [8, 12, 24, 11] for more recent results. Most of these papers use Dirichlet boundary conditions and, up to our knowledge, the few papers addressing Neumann problems are [37, 4, 29, 27], where existence of positive solutions and concentration phenomena are studied, and [9], which centers on existence of positive radial solutions. However, these papers focus on a different operator of the form $Lw=-\Delta w+V(x)w$, with $V$ positive. In comparison with (1.1), the shape of solutions changes drastically; for instance, the operator $L$ with Neumann b.c. induces a norm, and this allows the existence of positive solutions, while all nontrivial solutions of (1.1) are sign-changing. Indeed, if $(u,v)$ is a classical solution of (1.1), then by the Neumann b.c. and the divergence theorem,

[TABLE]

Since $u\equiv 0$ if and only if $v\equiv 0$, (1.3) is only satisfied if $(u,v)$ is trivial or if both components are sign-changing. Condition (1.3) is called a compatibility condition. As far as we know, our paper is the first to study problem (1.1).

We remark that if $p=q>0$ and $(u,v)$ is a classical solution of (1.1), then $u\equiv v$ in $\Omega$ and (1.1) reduces to the scalar equation $-\Delta u=|u|^{p-1}$ in $\Omega$ with $\partial_{\nu}u=0$ on $\partial\Omega$, see Lemma 2.6. Therefore, all our results cover the single equations case.

Condition (1.2) together with Sobolev embeddings and the Rellich–Kondrachov theorem implies that

[TABLE]

A strong solution of (1.1) is defined as a pair $(u,v)\in W^{2,\frac{q+1}{q}}(\Omega)\times W^{2,\frac{p+1}{p}}(\Omega)$ satisfying the equations a.e. in $\Omega$, and the boundary conditions in the trace sense. Problem (1.1) has a variational structure, and (1.1) are the Euler-Lagrange equations of the energy functional

[TABLE]

We define a least energy (nodal) solution as a nontrivial strong solution of (1.1) achieving the level

[TABLE]

In view of (1.2) and (1.4), the functional $I$ is well defined at strong solutions. Our main result is concerned with existence, regularity, and qualitative properties of least energy solutions.

Theorem 1.1**.**

Let $N\geq 1$, $\Omega\subset\mathbb{R}^{N}$ be a smooth bounded domain, and let $p$ and $q$ satisfy (1.2). The set of least energy solutions is nonempty. If $(u,v)$ is a least energy solution, then $(u,v)\in C^{2,\varepsilon}(\overline{\Omega})\times C^{2,\varepsilon}(\overline{\Omega})$ is a classical solution of (1.1) and the following holds.

• (i)

(Monotonicity in 1D) If $N=1$ and $\Omega=(-1,1)$, then $u^{\prime}v^{\prime}>0$ in $\Omega$; in particular, $u$ and $v$ are both strictly monotone increasing or both strictly monotone decreasing in $\Omega$.

• (ii)

(Symmetry & symmetry breaking) If $N\geq 2$ and $\Omega=B_{1}(0)$ or $\Omega=B_{1}(0)\backslash B_{\delta}(0)$ for some $\delta\in(0,1)$, then there is $e\in\partial B_{1}(0)$ such that $u$ and $v$ are foliated Schwarz symmetric with respect to $e$ and the functions $u$ and $v$ are not radially symmetric.

• (iii)

(Unique continuation property) If $pq<1$, then the zero sets of $u$ and $v$ have zero Lebesgue measure, i.e., $|\{x\in\Omega:\ u(x)=0\}|=|\{x\in\Omega:\ v(x)=0\}|=0.$

Results for the scalar equation follow immediately from Theorem 1.1, see Corollary 1.4 below. Our approach to show Theorem 1.1 is based on a variant of the dual method [14, 3]. Later in this introduction we motivate the use of this approach and also the relationship between our results and previously known results for the single-equation problem. To describe the dual framework, we introduce some notation used throughout the whole paper. Let $p$ and $q$ satisfy (1.2) and, for $s>1$, let

[TABLE]

endowed with the norm $\|(f,g)\|_{X}=\|f\|_{\alpha}+\|g\|_{\beta}$. Let $K$ denote the inverse (Neumann) Laplace operator with zero average, that is, if $h\in X^{s}(\Omega)$, then $u:=Kh\in W^{2,s}(\Omega)$ is the unique strong solution of $-\Delta u=h$ in $\Omega$ satisfying $\partial_{\nu}u=0$ on $\partial\Omega$ and $\int_{\Omega}u=0$, see Lemma 2.1 below. In this setting, the (dual) energy functional $\phi:X\to\mathbb{R}$ is given by

[TABLE]

Since (1.3) holds for any nontrivial solution, we require a suitable translation of $K$. For $t>0$, let $K_{t}:X^{\frac{t+1}{t}}\to W^{2,\frac{t+1}{t}}(\Omega)$ be given by

[TABLE]

Then, a critical point $(f,g)$ of $\phi$ solves the dual system $K_{q}f=|g|^{\frac{1}{q}-1}g$ and $K_{p}g=|f|^{\frac{1}{p}-1}f$ in $\Omega$, see Lemma 2.3.

In the sublinear case ($pq<1$), the operator $\phi$ achieves its global minimum in $X$ (see Lemma 2.8); whereas, in the superlinear case ($pq>1$), $\phi$ is unbounded from below, but it can be minimized (see Lemma 2.10) in the Nehari-type set

[TABLE]

Therefore, we often refer to a minimizer $(f,g)\in X$ satisfying

[TABLE]

In particular, if $(f,g)$ satisfies (1.10), then $(u,v):=(K_{p}g,K_{q}f)$ is a least energy solution, that is, $\phi(f,g)=I(u,v)=c$, with $c$ as in (1.6) and solves (1.1), see Lemmas 2.5, 2.8, and 2.10 below.

We now describe in more detail the different techniques involved in the proof of Theorem 1.1. The existence is obtained using the subcriticality assumption (1.2) and the compactness of the operator $K$, while the regularity follows from a bootstrap argument (Proposition 2.4). The unique continuation property is shown by extending the techniques from [25, Section 3] to the setting of Hamiltonian elliptic systems and to the dual-method framework (later in this introduction we compare in more detail the results and strategies from [25] with ours). Moreover, the fact that least energy solutions are foliated Schwarz symmetric is due to a characterization of sets of functions with this symmetry in terms of invariance under polarizations, see Lemma 4.7 below and [5, 25, 8, 10, 32] for similar results in other settings.

One of our main contributions, from the methodological point of view, is the proof of the strict monotonicity of solutions (for $N=1$) and of the symmetry-breaking phenomenon (i.e., that least energy solutions are not radial). These results require first a deep understanding of the radial setting: consider $\Omega\subset\mathbb{R}^{N}$ to be either

[TABLE]

If $Y$ is a set of functions, we use $Y_{rad}$ to denote the subset of radial functions in $Y$ and we call $(u,v):=(K_{p}g,K_{q}f)$ a least energy radial solution if $(f,g)$ satisfies

[TABLE]

Our next result shows that these radial solutions are remarkably rigid. We denote the radial derivative of $w$ by $w_{r}$ and use a slight abuse of notation setting $w(|x|)=w(x)$.

Theorem 1.2**.**

Let $p,q$ satisfy (1.2) and $\Omega,$ $\delta$ as in (1.11). The set of least energy radial solutions is nonempty and, if $(u,v)$ is a least energy radial solution, then $(u,v)\in C^{2,\varepsilon}(\overline{\Omega})\times C^{2,\varepsilon}(\overline{\Omega})$ is a classical radial solution of (1.1) and $u_{r}v_{r}>0$ in $(\delta,1)$; in particular, $u$ and $v$ are both strictly monotone increasing or both strictly monotone decreasing in the radial variable.

The existence and regularity follows similarly as in Theorem 1.1. The strict monotonicity relies on the following new transformation denoted with a superindex $\divideontimes$: let $\Omega$, $\delta$ as in (1.11) and

[TABLE]

Then, for $h\in C_{rad}(\overline{\Omega})$, the $\divideontimes$-transformation of $h$ is given by

[TABLE]

where $\#$ is the one-dimensional decreasing rearrangement (see Subsection 3.1) and $\omega_{N}=|B_{1}|$ is the volume of the unitary ball in $\mathbb{R}^{N}$. The function ${\mathfrak{F}}h$ can be seen as a suitable flipping of $h$ on the set $\{{\mathcal{I}}h\leq 0\}$, and this step is very important to construct monotone decreasing solutions. In Remark 3.5 we motivate further the definition of $h^{\divideontimes}$ and in Figure 1 in Section 3.2 we illustrate the construction of $h^{\divideontimes}$ in a ball and compare it with the Schwarz symmetrization $h^{*}(x)=h^{\#}(\omega_{N}|x|^{N})$.

Observe that annuli, sign-changing functions, and Neumann boundary data are non-standard conditions to work with rearrangements; in fact, many results relying on symmetrizations fail in these settings; for example, the standard Polya-Szegő inequality only holds for nonnegative functions with zero boundary data. Our approach is able to cover these cases mainly because of two reasons: the radiality assumption and the fact that we use Lebesgue spaces within a dual framework, which gives us more flexibility in the construction of our transformation; in this sense, we are transforming the dual variables $(f,g)$ to obtain, together with variational techniques, monotonicity information of solutions $(u,v):=(K_{p}g,K_{q}f)$.

In general, $h^{\divideontimes}$ is not a (level-set) rearrangement of $h$, since the maximum value of $h$ may vary due to the flipping ${\mathfrak{F}}h$; however, $L^{t}$-norms and (zero) averages are preserved (see Proposition 3.4) and the transformed functions have less energy, as stated in the following result.

Theorem 1.3**.**

Let $p,q>0$, $\Omega$ as in (1.11), and let $f,g:\overline{\Omega}\to\mathbb{R}$ be continuous and radially symmetric functions with $\int_{\Omega}f=\int_{\Omega}g=0$. Then $(f^{\divideontimes},g^{\divideontimes})\in X$ and

[TABLE]

Furthermore, if $f$ and $g$ are nontrivial and $\phi(f^{\divideontimes},g^{\divideontimes})=\phi(f,g)$, then $f$ and $g$ are monotone in the radial variable and if $(u,v):=(K_{p}g,K_{q}f)$, then $u$ and $v$ are radially symmetric and strictly monotone in the radial variable.

The proof exploits the one-dimensionality of the problem and uses elementary rearrangement techniques. Note that, if $\Omega$ is a ball, then $h^{\divideontimes}$ is actually the Schwarz symmetrization of $\mathfrak{F}h$. The flipping ${\mathfrak{F}}$, however, is necessary, since it can be shown that (1.13) does not hold in general using merely the Schwarz symmetrization (even in the scalar case $p=q$), see Remark 3.6 below.

Theorem 1.3 is the main tool to show the monotonicity claims in Theorem 1.1 and Theorem 1.2. These results are of independent interest and are new even in the single-equation case (1.17). Furthermore, Theorem 1.3 is also the starting point for the proof of the symmetry-breaking phenomenon, which, in the dual setting, reads as follows.

[TABLE]

The proof of this claim is done by contradiction: if $(f,g)$ is radially symmetric and $(u,v):=(K_{p}g,K_{q}f)$, then $(u_{x_{1}},v_{x_{1}})$ is a strong (Dirichlet) solution of

[TABLE]

Therefore, from the minimality of $(f,g)$ (approximating $\phi^{\prime\prime}$ in a suitable sense) we infer that

[TABLE]

Here, Theorem 1.2 is very important to control the (possibly singular) terms $|u|^{p-1}$ and $|v|^{p-1}$, since the strict monotonicity implies that the nodal sets of $u$ and $v$ are merely two inner spheres. In the end, a contradiction is obtained by showing—using the radiality assumption, maximum principles, and Hopf’s boundary point Lemma—that the Neumann solution $(K\bar{g},K\bar{f})$ dominates the Dirichlet solution $(u_{x_{1}},v_{x_{1}})$ in $\Omega(e_{1})$, and this would imply that the integral in (1.15) is strictly negative. Observe that the symmetry-breaking statement in Theorem 1.1 follows directly from (1.14). For other symmetry-breaking results for single equations we refer to [1] for Dirichlet boundary conditions, to [19, 25] for Neumann boundary conditions, and to [6] for a (perturbative) symmetry-breaking result for Dirichlet Hamiltonian systems. See also the survey [35] and the references therein.

We now focus on the particular case $p=q$, where (1.2) reduces to

[TABLE]

In this situation, we show in Lemma 2.6 below that any classical solution $(u,v)$ of (1.1) satisfies $u\equiv v$, and problem (1.1) is equivalent to

[TABLE]

whose solutions are critical points of

[TABLE]

Then, as a particular case of Theorems 1.1 and 1.2 we have the following.

Corollary 1.4**.**

Let $N\geq 1$, $\Omega\subset\mathbb{R}^{N}$ be a smooth bounded domain, and let $p$ satisfy (1.16). The set of least energy solutions of (1.17) is nonempty and it is contained in $C^{2,\varepsilon}(\overline{\Omega})$.

• (i)

(Unique continuation) If $p<1$, then the zero set of every least energy solution has zero Lebesgue measure.

• (ii)

(Monotonicity, symmetry, and symmetry breaking) If $N=1$ and $\Omega=(-1,1)$, then every least energy solution is strictly monotone in $\Omega$. If $N\geq 2$ and $\Omega$ is either a ball or an annulus as in (1.11), then every least energy solution is foliated Schwarz symmetric and it is not radially symmetric.

• (iii)

(Radial solutions) Let $\Omega$ be a ball or an annulus as in (1.11), then the set of least energy radial solutions is nonempty. If $u$ is a least energy radial solution, then $u\in C^{2,\varepsilon}(\overline{\Omega})$ is a classical radial solution of (1.16) and $u$ is strictly monotone in the radial variable.

Up to our knowledge, the monotonicity of least energy radial solutions (part $(iii)$ of Corollary 1.4) is new. Part (ii) is also new in the subcritical superlinear regime as well as the symmetry breaking result for annuli in the sublinear case. In this sense, this corollary complements the results in [25], where unique continuation, symmetry, and symmetry breaking for least energy solutions of (1.17) in the case $0<p<1$ are studied (the case $p=0$ is also considered in [25], interpreting (1.17) as $-\Delta u=\text{sign}(u)$). Note that the unique continuation property when $p>1$ is classical, see for instance [22, 16, 21]. Next, we comment on the similarities and differences between our approach and that of [25]; in particular, we explain the difficulties when passing from the single-equation (1.17) to system (1.2) and why the dual method is convenient in this last case.

To show existence of solutions (in the sublinear case $0<p<1$), the authors in [25] minimize the functional (1.18) on the set

[TABLE]

proving that the least energy level is achieved (note that $N_{p}$ is not a $C^{1}$-manifold because $p<1$). Several difficulties arise when trying to use such a direct method for a Hamiltonian system. For instance, in dimension $N\geq 3$, the (direct) functional $I$, given in (1.5), is not well defined in $H:=H^{1}(\Omega)\times H^{1}(\Omega)$ under (1.2). In fact, even working on a range of $(p,q)$ where $I$ is well defined in $H$ (or using (1.2) with $p,q>1$ and a truncation method as in [28]), the functional $I$ is strongly indefinite, in the sense that the principal part $\int_{\Omega}\nabla u\cdot\nabla v$ does not have a sign, and it is actually positive in an infinite dimensional subspace, and negative in another. To control this difficulty, a direct approach is based, for example, on abstract linking theorems or on special Nehari-type sets. Different alternatives are available in the literature to study Hamiltonian systems variationally (see, e.g., the survey [6]), each one with its own advantages and disadvantages. For example, a common strategy is to reduce the system to a single higher-order problem; however, it is not clear how to study Neumann b.c. in this setting and using higher-order Sobolev spaces brings additional complications (for the use of rearrangements, for example).

Dual methods, on the other hand, offer a flexible and elegant alternative. This approach entails the challenge of controlling the effects of the nonlocal operator $K$ in the functional $\phi$; but it compensates this difficulty with many advantages, for example, the compatibility conditions (1.3) translate to $\int_{\Omega}f=\int_{\Omega}g=0$ in the dual formulation; in particular, this allows in the sublinear case to minimize and differentiate the functional $\phi$ in the Banach space $X$ (recall that in [25] the functional (1.18) is minimized on (1.19), which is not a manifold); whereas, in the superlinear case, the dual formulation allows to minimize in a Nehari manifold in a Banach space.

The use of a direct or a dual approach has, of course, a strong influence on the methods to study qualitative properties of solutions; this is particularly clear in the proof of the symmetry breaking, described above, where our proof relies on a transformation in dual spaces to obtain monotonicity and then on comparison principles to control the effects of the nonlocal operator $K$. The proof of the symmetry breaking result in [25] (although is also done by contradiction finding a direction along which the energy would decrease) is very different from ours, and it relies on a unique continuation property for minimizers, which is obtained using known uniqueness results [36] and nonoscillation criteria [20] for sublinear ODEs of type $y^{\prime\prime}+a(t)|y|^{p-1}y=0$. These general theorems are not known for systems of ODEs, and in fact, they may fail in general. Using elementary manipulations in an ODE setting, one can find some extensions of these techniques to systems. Although we do not need these results for any of our proofs, we believe they can be of independent interest; in particular, we use them to show a result on the simplicity of zeros of any radial solution that satisfies a unique continuation property.

Theorem 1.5**.**

Let $p,q>0$, $N\geq 2$, $\Omega$, $\delta$ as in (1.11), and $(u,v)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ be a radial classical solution of (1.1) such that $u^{-1}(0)\cap v^{-1}(0)$ has empty interior. Then

[TABLE]

This theorem yields that, if $(u,v)$ is a radial solution of (1.1) satisfying a (weak) unique continuation property, then the nodal set of $u$ and of $v$ is at most a countable union of spheres and thus $|u^{-1}(0)|=|v^{-1}(0)|=0$.

To close this introduction, we mention some open questions about system (1.1): it is unknown whether or not least energy solutions are (up to rotations) unique; in fact, uniqueness is open even for least energy radial solutions. Moreover, when $pq<1$ it is unclear if the unique continuation property (i.e., that the nodal sets of solutions have zero Lebesgue measure) holds in general for all solutions, and not just for minimizers. When $pq>1$, with the exception of the single equation case $p=q$, the unique continuation property is an open question even for least energy solutions. In Remark 5.5 below we observe that, in the superlinear case, if a least energy solution $(u,v)$ vanishes on a set of positive measure, then it has a zero of infinite order. For single equations of type $-\Delta u+V(x)u=0$ such a result is available for a large class of potentials (see[16, Proposition 3] and [21, Proposition 1.1]) and having a zero of infinite order typically implies that a solution is identically zero (see [22]).

The paper is organized as follows. In Section 2 we provide some general auxiliary results in our dual framework; in particular, we show Lemmas 2.8 and 2.10, which prove the existence and regularity claims in Theorem 1.1, as well as Lemma 2.6 which shows (together with Theorems 1.1 and 1.2) Corollary 1.4. Section 3 is devoted to the study of monotonicity properties of radial minimizers, here the full details of the construction of the $\divideontimes$-transformation can be found as well as the proof of Theorems 1.2 and 1.3. These results are then used in Section 4 to prove the symmetry breaking result in Theorem 1.1 as well as the foliated Schwarz symmetry claim. Finally, in Section 5 we show Theorem 1.5 as well as the unique continuation statement in Theorem 1.1.

1.1 Notation

Throughout the paper $p$ and $q$ always denote the exponents in (1.1). We divide our proofs in two groups:

[TABLE]

Note that $pq<1$ readily implies the subcriticality condition, since $pq<1$ implies $1/(p+1)+1/(q+1)>1>(N-2)/N$. If $N=1,2$ the subcriticality condition is satisfied by any $p,q>0$. We fix

[TABLE]

Note that $\alpha^{\prime}$ and $\beta^{\prime}$ are the corresponding conjugate exponents of $\alpha$ and $\beta$, that is $\alpha^{-1}+(\alpha^{\prime})^{-1}=1$ and $\beta^{-1}+(\beta^{\prime})^{-1}=1$.

Moreover, we define $X$, $X_{s}$, $K$, and $K_{s}$ as in the Introduction. For $s\geq 1$, $\|\cdot\|_{s}$ and $\|\cdot\|_{2,s}$ are the standard norms in $L^{s}(\Omega)$ and $W^{2,s}(\Omega)$ respectively and, if $s>1$ is such that $N>2s$, then we denote by $s^{*}:=\frac{Ns}{N-2s}>s$ the critical Sobolev exponent, that is, $s^{*}$ is the biggest exponent which allows the (continuous) embedding $W^{2,s}(\Omega)\hookrightarrow L^{s^{*}}(\Omega)$. If $N=1,2$ we set $s^{*}=\infty$.

If $Y$ is a vector space, then $[Y]^{2}:=Y\times Y$. We denote by $B_{r}$ the ball in $\mathbb{R}^{N}$ of radius $r>0$ centered at 0. Finally, the nodal set of $u:\Omega\to\mathbb{R}$ is denoted by $u^{-1}(0):=\{x\in\Omega\>:\>u(x)=0\}$.

For a measurable set $A\subset\mathbb{R}^{N}$, $|A|$ denotes its Lebesgue measure, and the function $\chi_{A}$ is the characteristic function of $A$, that is, $\chi_{A}(x)=1$ if $x\in A$ and $\chi_{A}(x)=0$ if $x\not\in A$. Finally, we use $\omega_{N}$ to denote the measure of the unitary ball in $\mathbb{R}^{N}$.

2 Variational framework and existence results

In this section we describe our dual approach. In the following $\Omega$ is a smooth bounded domain in $\mathbb{R}^{N}$, $p$ and $q$ satisfy (1.2), and let $X^{s}$, $X$, $K$, $K_{t}$, $\alpha$, and $\beta$ be as above.

2.1 The dual method

The following lemma recalls some well-known regularity for Neumann problems, see for example [30, Theorem and Lemma in page 143] (see also [2, Theorem 15.2]).

Lemma 2.1**.**

If $s>1$, $\Omega$ be a smooth bounded domain in $\mathbb{R}^{N}$, and $h\in L^{s}(\Omega)$ with $\int_{\Omega}h=0$, then there is a unique strong solution $u\in W^{2,s}(\Omega)$ of

[TABLE]

in particular,

[TABLE]

and there is $C(\Omega,s)=C>0$ such that $\|u\|_{2,s}\leq C\|h\|_{s}.$

Recall that if $f\in L^{s}(\Omega)$ for some $s>1$, then $Kf$ is the solution of (2.1) with $h=f$. Observe that

[TABLE]

by integration by parts and because $\int_{\Omega}f=\int_{\Omega}g=0$. Our next result shows that $\phi$ is continuously differentiable.

Lemma 2.2**.**

The functional $\phi:X\to\mathbb{R}$ as defined in (1.8) is continuously differentiable in $X$ with

[TABLE]

Proof.

Let $\phi=\psi-T$, where

[TABLE]

Since $\alpha,\beta>1$ it is standard to show that $\Psi$ is continuously differentiable with

[TABLE]

Now we show that $T$ is bilinear and bounded. Indeed, this follows directly from the fact that $K(\lambda h)=\lambda Kh$ for $h\in L^{s}(\Omega)$, $s\in\{\alpha,\beta\}$, and the following integration by parts

[TABLE]

Moreover, by Hölder’s inequality, Lemma 2.1, and the first embedding in (1.4), there are $C_{1},C_{2}>0$ such that

[TABLE]

Thus $T$ is a (continuously differentiable) bilinear bounded mapping and, by (2.6),

[TABLE]

Therefore, (2.3) follows from (2.5) and (2.8). ∎

Before we argue existence of solutions, we establish a one-to-one relationship between critical points of $\phi$ and strong solutions of (1.1), see [6, Proposition 3.1] for the Dirichlet case.

Lemma 2.3**.**

An element $(f,g)\in X$ is a critical point of $\phi$, i.e.,

[TABLE]

if and only if $(u,v):=(K_{p}g,K_{q}f)\in W^{2,\beta}(\Omega)\times W^{2,\alpha}(\Omega)$ is a strong solution of (1.1), that is, $(u,v)$ solves (1.1) a.e. in $\Omega$.

Proof.

Let $(f,g)\in X$ satisfy (2.9) and let $(u,v):=(K_{p}g,K_{q}f)$. Then, using (2.2),

[TABLE]

Let $w:=|f|^{\alpha-2}f-u$ and, for a function $\zeta\in L^{\alpha}(\Omega)$, set $\overline{\zeta}:=|\Omega|^{-1}\int_{\Omega}\zeta\in\mathbb{R}$. Then $(\zeta-\overline{\zeta})\in X^{\alpha}$ and

[TABLE]

and therefore $w-\overline{w}=0$ a.e. in $\Omega$, which implies that $|f|^{\alpha-2}f=u+\overline{w}$ a.e. in $\Omega$ and thus $f=|u+\overline{w}|^{p-1}(u+\overline{w})$ a.e. in $\Omega$. Furthermore, since $(f,g)\in X$ and $u=K_{p}g$, we have that $\int_{\Omega}|u|^{p-1}u=0=\int_{\Omega}f=\int_{\Omega}|u+\overline{w}|^{p-1}(u+\overline{w})$, which implies that $\overline{w}=0$, and therefore $-\Delta v=-\Delta(K_{q}f)=f=|u|^{p-1}u$ a.e. in $\Omega$. Analogously, $-\Delta u=|v|^{q-1}v$ a.e. in $\Omega$, as claimed.

For the converse implication, let $(u,v):=(K_{p}g,K_{q}f)\in W^{2,\beta}(\Omega)\times W^{2,\alpha}(\Omega)$ be a strong solution of (1.1) for some $(f,g)\in X$. Then, necessarily $f=|u|^{p-1}u$ and $g=|v|^{q-1}v$ a.e. in $\Omega$, which implies that $u=|f|^{\alpha-2}f$ and $v=|g|^{\beta-2}g$ a.e. in $\Omega$, which implies (2.9), by (2.2).

∎

The next Proposition states that critical points of $\phi$ are in fact classical solutions. We refer to [33, Theorem 1] and [6, Lemma 5.16] for analogous results in the superlinear Dirichlet case.

Proposition 2.4**.**

Let $(f,g)\in X$ be a critical point of $\phi$, then $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ for some $\varepsilon>0$ and satisfies (1.1) pointwise.

Proof.

The proof follows closely [6, Lemma 5.16]. For $t,s\geq 1$ denote

[TABLE]

Let $(f,g)\in X\subset L(\alpha,\beta)$ be a critical point of $\phi$ then, by Lemma 2.3, $(u,v)\in W(\beta,\alpha)\hookrightarrow L(\beta^{*},\alpha^{*})$ is a strong solution of (1.1). For $n\in{\mathbb{N}}_{0},$ let

[TABLE]

Here we are using the notation given in Section 1.1 for the critical Sobolev exponent. Note that $(|v|^{q},|u|^{p})\in L(\alpha^{*}/q,\beta^{*}/p)=L(\beta_{1},\alpha_{1})$ and then Lemma 2.1 and (1.1) imply $(u,v)\in W(\beta_{1},\alpha_{1})\hookrightarrow L(\beta^{*}_{1},\alpha_{1}^{*})$. But then $(|v|^{q},|u|^{p})\in L(\beta_{2},\alpha_{2})$, which gives $(u,v)\in W(\beta_{2},\alpha_{2})\hookrightarrow L(\beta^{*}_{2},\alpha_{2}^{*})$. Iterating this procedure we obtain that $(u,v)\in W(\beta_{n},\alpha_{n})$ as long as $N>2\alpha_{n-1}$ and $N>2\beta_{n-1}$.

We claim that

[TABLE]

and then the proposition follows from Lemma 2.3, the embedding $W^{2,s}(\Omega)\hookrightarrow C^{\mu}(\Omega)$ for some $\mu>0$, and standard Schauder estimates (see [18, Theorem 6.31] and the Remark after the theorem), since $t\mapsto|t|^{r-1}t$ for $r>0$ is Hölder continuous.

Indeed, assume by contradiction that $N>2s$ for all $s\in\{\alpha_{n},\beta_{n}:n\in{\mathbb{N}}_{0}\}$ and consider the sequence $S_{n}:=(q\beta_{n},p\alpha_{n})$. By the subcriticality assumption we have that $W(\beta,\alpha)\hookrightarrow L(\alpha^{\prime},\beta^{\prime})=L(p\alpha,q\beta)$ and therefore $S_{1}=(q\beta_{1},p\alpha_{1})=(\alpha^{*},\beta^{*})>(q\beta,p\alpha)=S_{0}$ and $(S_{n})_{n\in{\mathbb{N}}}$ is, by induction, a (component-wise) increasing (bounded) sequence. Let $l_{1},l_{2}>0$ be such that $S_{n}\to(l_{1},l_{2})$ and note that, by the monotonicity,

[TABLE]

Moreover

[TABLE]

Since the function $t\mapsto h(t):=\frac{N(pt-1)}{2(t+1)}$ is increasing for $t>0$ and $q<t_{0}:=\frac{2p+N+2}{pN-2p-2}$, by the subcriticality condition (1.22), we have that $l_{2}=h(q)<h(t_{0})=p+1$ , which contradicts (2.12). Then (2.11) holds and this ends the proof. ∎

The next Lemma shows the relationship between the dual and the direct energy functionals when evaluated on solutions.

Lemma 2.5**.**

Let $(f,g)\in X$ be a critical point of $\phi$ and let $(u,v):=(K_{p}g,K_{q}f)$. Then

[TABLE]

Proof.

Let $f,g,u$ and $v$ as in the statement. By Proposition 2.4, we have that $(u,v)$ is a solution of (1.1) and, integrating by parts,

[TABLE]

Therefore $\phi(f,g)-I(u,v)=\int_{\Omega}|u|^{p+1}+|v|^{q+1}-2\nabla u\nabla v\,dx=0$, by (1.1). ∎

We finish this Section with a result in the case $p=q$, in which (1.1) reduces to a single equation. The proof is the same as in the superlinear Dirichlet case [8, Theorem 1.5] and we include it for the reader’s convenience.

Lemma 2.6**.**

Let $p=q>0$ and $(u,v)\in C^{2}(\overline{\Omega})\times C^{2}(\overline{\Omega})$ be a classical solution to (1.1), then $u=v$.

Proof.

By testing (1.1) with $u,v$ and integrating by parts we have that

[TABLE]

Then, by Hölder’s inequality,

[TABLE]

which implies $\nabla u=\nabla v$ in $\Omega$ and then $v-u\equiv c$ in $\Omega$ for some constant $c\in\mathbb{R}$. Therefore, by (1.1), $|u|^{p-1}u=-\Delta u=-\Delta(u+c)=|u+c|^{p-1}(u+c)$ in $B$. If $u\equiv 0$ then also $v\equiv 0$. If $u\not\equiv 0$ then $u$ changes sign and there is $x\in B$ with $u(x)=0$, but then $|c|^{p-1}c=0$, and $u=v$ in $\Omega$ as claimed. ∎

2.2 Existence of least energy solutions: sublinear case

Assume (1.21), that is, $p,q>0$ and $pq<1$ and recall the notation (1.23).

Lemma 2.7**.**

The functional $\phi:X\to\mathbb{R}$ as defined in (1.8) is coercive.

Proof.

Let $\varepsilon:=\frac{1}{2}\min\{\alpha^{-1},\beta^{-1}\}$. Then from (2.7), Young’s inequality, and the fact that $\beta^{\prime}<\alpha$ (since $pq<1$) there is $C_{2}(\alpha,\beta)=C_{2}>0$ such that $|T(f,g)|\leq\varepsilon(\|f\|^{\alpha}_{\alpha}+\|g\|^{\beta}_{\beta})+C_{2}.$ This yields that

[TABLE]

that is, $\phi$ is coercive. ∎

Lemma 2.8**.**

The functional $\phi$ achieves a negative minimum in $X$, that is,

[TABLE]

Moreover, $\phi^{\prime}(f,g)=0$ in $X$, $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ is a classical solution of (1.1), and $(u,v)$ is a least energy solution, that is, $I(u,v)=c$ with $c$ as in (1.6).

Proof.

For $n\in{\mathbb{N}}$ let $x_{n}:=(f_{n},g_{n})$ be a minimizing sequence in $X$, which exists by Lemma 2.7. Since $X$ is reflexive there is $x:=(f,g)\in X$ such that $x_{n}\rightharpoonup x$ weakly in $X$ and $(x_{n})$ is bounded in $X$ in virtue of (2.14). Since the operator $K$ is compact by Lemma 2.1 and the compact embedding (1.4), we have that $K(f_{n})$ converges strongly to $Kf$ in $L^{\beta^{\prime}}(\Omega)$. Using the lower semicontinuity of norms we have that $\min\limits_{X}\phi=\liminf\limits_{n\to\infty}\phi(f_{n},g_{n})\geq\phi(f,g)$, and therefore $\phi$ achieves its minimum.

To see that the minimum is strictly negative, let $T$ and $\Psi$ as in (2.4) and let $\varphi\in C^{\infty}_{c}(\Omega)\backslash\{0\}$ such that $\int_{\Omega}\varphi=0$. Then $(\varphi,\varphi)\in X\backslash\{(0,0)\}$ and

[TABLE]

Thus, since $pq<1$ is equivalent to $\alpha\beta>\alpha+\beta$,

[TABLE]

Then $\phi^{\prime}(f,g)=0$ in $X$ by Lemma 2.2, $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ is a classical solution of (1.1), by Proposition 2.4, and $(u,v)$ is a least energy solution, by Lemma 2.5. ∎

2.3 Existence of least energy solutions: superlinear case

Let $p,q\in(0,\infty)$ such that (1.22) holds.

[TABLE]

where $\gamma<1$ because $pq>1$. In particular, $\gamma_{1}\alpha=\gamma_{2}\beta=\gamma$ and $\gamma_{1}+\gamma_{2}=1$. We define the Nehari-type set

[TABLE]

Lemma 2.9**.**

Let $(f,g)\in X\backslash\{(0,0)\}$ such that $\int_{\Omega}fKg>0$, then

[TABLE]

is the unique maximum of the function $s\mapsto\phi(s^{\gamma_{1}}f,s^{\gamma_{2}}g)$ and $(t^{\gamma_{1}}f,t^{\gamma_{2}}g)\in{\mathcal{N}}$.

Proof.

Since $\phi(s^{\gamma_{1}}f,s^{\gamma_{2}})=s^{\gamma}\int_{\Omega}\frac{|f|^{\alpha}}{\alpha}+\frac{|g|^{\beta}}{\beta}\,dx-s\int_{\Omega}fKg,$ $\gamma<1$, and $\int_{\Omega}fKg>0$, the map $s\mapsto\phi(s^{\gamma_{1}}f,s^{\gamma_{2}}g)$ has a unique positive critical point, which is a global maximum. The claim now follows by direct computations. ∎

Lemma 2.10**.**

There is $(f,g)\in{\mathcal{N}}$ such that $\phi(f,g)=\inf_{\mathcal{N}}\phi$. Moreover, $\phi^{\prime}(f,g)=0$ in $X$, $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ is a classical solution of (1.1), and $(u,v)$ is a least energy solution, that is, $I(u,v)=c$ with $c$ as in (1.6).

Proof.

1. The functional $\phi$ is bounded from below on $\mathcal{N}$, since

[TABLE]

Thus $\inf_{\mathcal{N}}\phi\in\mathbb{R}$. Take a minimizing sequence $(f_{n},g_{n})\in\mathcal{N}$. From (2.19), we have that $\|f_{n}\|_{\alpha}$ and $\|g_{n}\|_{\beta}$ are bounded and thus, up to a subsequence, $f_{n}\rightharpoonup f$ in $L^{\alpha}$, $g_{n}\rightharpoonup g$ in $L^{\beta}$, for some $(f,g)\in X$.

2. Recalling (2.7) and using Young’s inequality, we have

[TABLE]

for some $C>0$ and, since $\frac{1}{\gamma_{1}}>\alpha$, $\frac{1}{\gamma_{2}}>\beta$, there exists $\delta>0$ such that $\int_{\Omega}f_{n}Kg_{n}\geq\delta$ for every $n$.

3. Combining this with the properties of weak convergence and the compactness of the operator $K$, we have $(f,g)\neq(0,0)$ and, by definition of $\mathcal{N}$,

[TABLE]

Therefore, by Lemma 2.9, there exists $0<t\leq 1$ such that $(t^{\gamma_{1}}f,t^{\gamma_{2}}g)\in\mathcal{N}$. Then,

[TABLE]

thus $t=1$, and $(f,g)\in\mathcal{N}$ and achieves $\inf_{\mathcal{N}}\phi$.

4. Defining $\tau(h,k)=\phi^{\prime}(h,k)(\gamma_{1}h,\gamma_{2}k)$ for $(h,k)\in\mathcal{N}$, we have

[TABLE]

Therefore $\mathcal{N}$ is a manifold and, since $(f,g)$ achieves $\inf_{\mathcal{N}}\phi$, then by Lagrange’s multiplier rule there exists $\lambda\in\mathbb{R}$ such that $\phi^{\prime}(f,g)=\lambda\tau^{\prime}(f,g)$. By testing this identity with $(\gamma_{1}f,\gamma_{2}g)$, we see that actually $\lambda=0$ and $(f,g)$ is a critical point of $\phi$ in $X$. Thus $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ is a classical solution of (1.1) by applying Proposition 2.4. Finally, $(u,v)$ is a least energy solution, by Lemma 2.5. ∎

3 Monotonicity of radial minimizers

In this subsection $\Omega\subset\mathbb{R}^{N}$ is either

[TABLE]

We emphasize that the case $B_{1}(0)\backslash\{0\}$ is not considered in any of our results. Our main goal is to show Theorem 1.2. This result is of independent interest, but it is also instrumental in the proof of the symmetry-breaking result in Theorem 1.1. The proof of Theorem 1.2 is based on a new transformation using rearrangement techniques and a suitable flipping compatible with our dual method approach. We introduce first some notation and show some preliminary results.

3.1 The decreasing rearrangement

Let $U\subset\mathbb{R}^{N}$ be an open set and $h:U\to\mathbb{R}$ a measurable function. The decreasing rearrangement of $h$ is given by

[TABLE]

In particular, $h^{\#}$ is a non-increasing and left-continuous function [23, Proposition 1.1.1], $h^{\#}$ is a (level-set) rearrangement of $h$, which yields in particular that $L^{p}$-norms and averages are preserved, that is,

[TABLE]

see [23, Corollary 1.1.3]. By [23, Proposition 1.2.2], we know that if $E\subset U$, then

[TABLE]

We now focus on the case where $U=I:=[0,l]\subset\mathbb{R}$, for some $l>0$. If $h:I\to\mathbb{R}$ is non-increasing in $I$, then $h=h^{\#}$ a.e. in $I$, see [23, Lemma 1.1.1]. We use the following decomposition for the integral of a product of two functions.

Lemma 3.1** (Particular case of Lemma 1.2.2 in [23]).**

Let $\varphi,\psi:I\to\mathbb{R}$ be measurable functions with $\varphi\in L^{1}(I)$, $\psi\in L^{\infty}(I)$, and $a,b\in\mathbb{R}$ with $a\leq\psi\leq b$. Then

[TABLE]

A well-known consequence of Lemma 3.1 and (3.3) is the Hardy-Littlewood inequality.

Theorem 3.2** (Particular case of Theorem 1.2.2 in [23]).**

Let $\varphi\in L^{1}(I)$ and $\psi\in L^{\infty}(I)$ be (possibly sign-changing) functions, then

[TABLE]

Under some additional assumptions, the equality case in $\eqref{HL}$ can be used to deduce monotonicity properties of the involved functions.

Lemma 3.3**.**

Let $\varphi,\psi:I\to\mathbb{R}$ with $\varphi\in L^{1}(I)$ and $\psi\in C(I)$ be a strictly decreasing function. If $\int_{I}\varphi\,\psi=\int_{I}\varphi^{\#}\,\psi$, then $\varphi=\varphi^{\#}$ a.e. in $I$.

Proof.

Let $b:=\psi(0)$ and $a:=\psi(l)$. Then, by Lemma 3.1,

[TABLE]

Since $\psi$ is strictly decreasing and continuous, $\{\psi>t\}=[0,\psi^{-1}(t))$ for every $t\in[a,b]$. Therefore, by (3.3), $\int_{\{\psi>t\}}\varphi(s)\ ds\leq\int_{\{\psi>t\}}\varphi^{\#}(s)\ ds$ for all $t\in(a,b)$ and thus, by (3.5),

[TABLE]

Furthermore, for every $t\in I$ we have $\{\psi>\psi(t)\}=[\,0\,,\,t\,)$, because $\psi$ is strictly monotone decreasing and continuous. Then, (3.6) yields that $\int_{0}^{t}\varphi=\int_{0}^{t}\varphi^{\#}$ for all $t\in I$, and $\varphi=\varphi^{\#}$ a.e. in $I$, by Lebesgue differentiation theorem. ∎

3.2 A decreasing transformation in dual spaces

In the following we do a slight abuse of notation and use $w(|x|)=w(x)$ for a radial function $w$. We use $L^{\infty}_{rad}(\Omega)$ and $C_{rad}(\overline{\Omega})$ to denote the subspace of radial functions in $L^{\infty}(\Omega)$ and $C(\overline{\Omega})$, respectively. Let $\Omega$, $\delta$ as in (3.1) and let

[TABLE]

Then, for $h\in C_{rad}(\overline{\Omega})$, the $\divideontimes$-transformation is given by

[TABLE]

where $\#$ is the decreasing rearrangement (see the previous section) and $\omega_{N}=|B_{1}|$ is the volume of the unitary ball in $\mathbb{R}^{N}$. The function ${\mathfrak{F}}h$ can be seen as a suitable flipping of $h$ on the set $\{{\mathcal{I}}h\leq 0\}$, and this step is very important to construct monotone decreasing solutions, while preserving the (zero) average and the $L^{p}$-norm of $h$, see Proposition 3.4 and Remark 3.5. See also Figure 1 below for some examples.

In general, the function $h^{\divideontimes}$ is not a (level-set) rearrangement of $h$, since the maximum value of $h$ may vary due to ${\mathfrak{F}}$, but it has the following important properties.

Proposition 3.4**.**

Let $h:\overline{\Omega}\to\mathbb{R}$ be a continuous radial function such that $\int_{\Omega}h=0$. Then, for any $t\geq 1$,

[TABLE]

Proof.

The second claim in (3.7) follows by (3.2), the definition of $f^{\divideontimes}$, the fact that $|{\mathfrak{F}}h|=|h|$ in $\Omega$, and polar coordinates, since

[TABLE]

For the last property in (3.7), we claim that,

[TABLE]

Indeed, by continuity and since ${\mathcal{I}}h(\delta)={\mathcal{I}}h(1)=0$, the set $\{{\mathcal{I}}h>0\}$ is open in $(\delta,1)$ and therefore $\{{\mathcal{I}}h>0\}=\cup_{i=1}^{\infty}I_{i}$, where $(I_{i})_{i\in{\mathbb{N}}}$ are disjoint open intervals in $(\delta,1)$. Let $I_{i}=(a,b)$ for some $\delta\leq a<b\leq 1$, then $N\omega_{N}\int_{I_{i}}h(s)s^{N-1}\ ds={\mathcal{I}}h(b)-{\mathcal{I}}h(a)=0$, since $a,b\in\partial\{{\mathcal{I}}h>0\}$.

Let $r>\delta$ arbitrary and let

[TABLE]

(recall that ${\mathcal{I}}h(\delta)=0$ and hence $\zeta$ is well defined). If $\zeta=r$ then $\chi_{\{{\mathcal{I}}h>0\}}(r)=0$ and $\{{\mathcal{I}}h>0\}\cap[\delta,r]$ is open in $(\delta,r)$, therefore

[TABLE]

If $\zeta<r$, then $\chi_{\{{\mathcal{I}}h>0\}}(r)=1$, ${\mathcal{I}}h(\zeta)=0$, and

[TABLE]

Thus the first equality in (3.8) follows, but then, by the linearity of the integral,

[TABLE]

which shows the second equality in (3.8). The last property in (3.7) follows from the definition of ${\mathfrak{F}}h$, because

[TABLE]

Finally, since $\int_{\Omega}h^{\divideontimes}=\int_{I}({\mathfrak{F}}h)^{\#}=\int_{\Omega}{\mathfrak{F}}h={\mathcal{I}}({\mathfrak{F}}h)(1)=|\,{\mathcal{I}}h(1)\,|=|\int_{\Omega}h|=0$, the first equality in (3.7) also holds, and the proof is finished. ∎

3.3 Monotonicity results

In this subsection we show the energy-decreasing property of the $\divideontimes$-transformation. We introduce first a useful notation. Recall that $\Omega,$ $\delta$ are as in (3.1) and $I:=[0,|\Omega|]$. For $h\in L_{rad}^{1}(\Omega)$ let

[TABLE]

(recall that we use $h(|x|)=h(x)$). The function $\widetilde{h}$ is the one-dimensional equimeasurable version of $h$, since, for $s\in I$ and $r=\tau^{-1}(s)\in[\delta,1]$,

[TABLE]

Similarly, observe that

[TABLE]

and therefore,

[TABLE]

Proof of Theorem 1.3.

Let $f$ and $g$ as in the statement. By Proposition 3.4, we have that $(f^{\divideontimes},g^{\divideontimes})\in X$ and $\int_{\Omega}\frac{|f|^{\alpha}}{\alpha}+\frac{|g|^{\beta}}{\beta}=\int_{\Omega}\frac{|f^{\divideontimes}|^{\alpha}}{\alpha}+\frac{|g^{\divideontimes}|^{\beta}}{\beta},$ so it suffices to show that $\int_{\Omega}gKf\leq\int_{\Omega}g^{\divideontimes}Kf^{\divideontimes}.$ Let $(u,v):=(Kg,Kf)$ and observe that

[TABLE]

Then,

[TABLE]

Thus, by Proposition 3.4,

[TABLE]

By (3.10),

[TABLE]

Let $\varphi({s}):=(N\omega_{N})^{-1}(\omega_{N}^{-1}{s}+\delta^{N})^{2(\frac{1}{N}-1)}\geq 0$. Since $\widetilde{{\mathfrak{F}}f}^{\#}$ is non-increasing in $I$ and $\int_{I}\widetilde{{\mathfrak{F}}f}^{\#}={\mathcal{I}}({\mathfrak{F}}f)(1)=0$ (by (3.10) and because averages are preserved by rearrangements), we have that $\int_{0}^{{s}}\widetilde{{\mathfrak{F}}f}^{\#}\geq 0$ for ${s}\in I$ . Moreover, by Proposition 3.4, $\int_{0}^{{s}}\widetilde{{\mathfrak{F}}g}={\mathcal{I}}({\mathfrak{F}}g)(\tau^{-1}(s))\geq 0$ for $s\in I$. Therefore,

[TABLE]

are non-increasing in $I$. Then, by (the Hardy-Littlewood inequality) Theorem 3.2 and Fubini’s theorem,

[TABLE]

However, using (3.10), (3.11) and reasoning as in (3.16), we have

[TABLE]

Therefore, arguing as in (3.15), we obtain that

[TABLE]

and (1.13) follows.

We now show that if $f$ and $g$ are nontrivial and $\phi(f^{\divideontimes},g^{\divideontimes})=\phi(f,g)$, then $f$ and $g$ are monotone in the radial variable. Observe that, since $f$ is nontrivial, $\widetilde{{\mathfrak{F}}f}^{\#}$ is non-increasing, and $\int_{I}\widetilde{{\mathfrak{F}}f}^{\#}={\mathcal{I}}({\mathfrak{F}}h)(1)=0$, then $\int_{0}^{{s}}\widetilde{{\mathfrak{F}}f}^{\#}>0$ for all ${s}\in(0,|\Omega|)$. But then the function ${\sigma}\mapsto\int_{\sigma}^{|\Omega|}\int_{0}^{{s}}\widetilde{{\mathfrak{F}}f}^{\#}({t})\ d{t}\,\varphi({s})\ d{s}$ is a strictly decreasing continuous function in $I$ and, in virtue of Lemma 3.3, equality in (3.18) may only hold if $\widetilde{{\mathfrak{F}}g}=\widetilde{{\mathfrak{F}}g}^{\#}$ a.e. in $I$, which, by (3.9), implies that ${\mathfrak{F}}g$ coincides a.e. with a radially monotone function. Then $g$ must also be radially monotone, because $|{\mathfrak{F}}g|=|g|$ in $\Omega$, $g$ is a nontrivial continuous function, and $\int_{\Omega}g=0$. Arguing similarly using (3.17) we conclude that $f$ is monotone in the radial variable as well. As a consequence, if $(u,v):=(K_{p}g,K_{q}f)$, then $u$ and $v$ are strictly monotone in the radial variable by (3.12) and the fact that either ${\mathcal{I}}f>0$, ${\mathcal{I}}g>0$ or ${\mathcal{I}}f<0$, ${\mathcal{I}}g<0$ in $(\delta,1)$. This ends the proof. ∎

Remark 3.5**.**

To motivate the definition of $f^{\divideontimes}$, observe that the flipping ${\mathfrak{F}}f$ is very natural in virtue of (3.15), but further insight on ${\mathfrak{F}}f$ can be gained by observing that, by (3.12), the (Neumann radial) solution of $-\Delta v={\mathfrak{F}}f$ in $\Omega$ satisfies that $v_{r}(r)=-(N\omega_{N})^{-1}r^{1-N}{\mathcal{I}}({\mathfrak{F}}f)(r)\leq 0$ in $(0,1)$, i.e. $v$ is decreasing in the radial variable. By (3.15), this is indeed a very simple way to guarantee that there is at least one minimizer which is monotone in the radial variable; but the flipping ${\mathfrak{F}}f$ is too simple to extract any further information. To show that all least energy solutions $(u,v):=(K_{p}g,K_{q}f)$ are strictly monotone decreasing, we need to combine the flipping ${\mathfrak{F}}$ with the decreasing rearrangement $\#$ in order to apply the powerful properties of rearrangements (Theorem 3.2 and Lemma 3.3).

Remark 3.6**.**

In general, it is not true that $\phi(f^{*},g^{*})\leq\phi(f,g)$, where $*$ denotes the Schwarz symmetrization and $(f,g)$ is as in Theorem 1.3. A first observation is that $f^{*}$ is defined in $\Omega^{*}$ (a ball with the same measure as $\Omega$), and this would be a first complication when considering annular domains. But even if $\Omega$ is a unitary ball centered at zero $B$ and we consider the scalar problem (i.e., $p=q$, see Lemma 2.6), we exhibit next a continuous radial function $f:B\to\mathbb{R}$ with $\int_{B}f=0$ such that $\phi(f^{*},f^{*})>\phi(f,f)$. Let $N=5$, $\varepsilon=\frac{1}{2}$, and $f(x):=-(\omega_{N}|x|^{N}+\varepsilon)^{-\frac{1}{2}}+k$, where $k:=2\omega_{N}^{-1}((\omega_{N}+\varepsilon)^{\frac{1}{2}}-\varepsilon^{\frac{1}{2}})$ is such that $\int_{B}f=0$. In particular, $\widetilde{f}(s)=-(s+\varepsilon)^{-\frac{1}{2}}+k$ for $s\in I=[0,\omega_{N}]=[0,\frac{8\pi^{2}}{15}]$. Observe that $\widetilde{f}$ is a strictly increasing function; therefore, by (3.11), $f^{*}(x)=f^{\#}(\omega_{N}\,|x|^{N})=\widetilde{f}^{\#}(\omega_{N}\,|x|^{N})=\widetilde{f}(\omega_{N}-\omega_{N}|x|^{N})$. We have that

[TABLE]

These integrals can be computed via hypergeometric functions; for simplicity, here we just give a numerical approximation. Then, since ${f^{\#}}=\widetilde{f}^{*}$ in $I$, we may use the techniques and notation from the proof of Theorem 1.3 and we find that

[TABLE]

where $\varphi({s}):=(N\omega_{N})^{-1}(\omega_{N}^{-1}{s})^{2(\frac{1}{N}-1)}$ for $s\in I$. Therefore $\phi(f^{*},f^{*})>\phi(f,f)$, because $\|f^{*}\|_{\alpha}=\|f\|_{\alpha}$ by (3.2).

Remark 3.7**.**

The Neumann b.c. are used at (3.12) (if $\Omega$ is an annulus) and in the integration by parts performed in (3.15). The Neumann b.c. is also the reason to consider zero-average functions.

Theorem 3.8**.**

(Sublinear case) Let $(p,q)$ satisfy (1.21) and $\Omega$ and $\delta$ be as in (3.1). The set of minimizers of $\phi$ in $X_{rad}$ is nonempty. If $(f,g)\in X_{rad}$ is such that $\phi(f,g)=\inf_{X_{rad}}\phi$, then $f$ and $g$ are monotone in the radial variable and, if $(u,v):=(K_{p}g,K_{q}f)$, then $u_{r}v_{r}>0$ in $(\delta,1)$; that is, $u$ and $v$ are both strictly monotone increasing or strictly monotone decreasing in the radial variable.

Proof.

Arguing as in Lemma 2.8 we have that the set of minimizers of $\phi$ in $X_{rad}$ is nonempty and contains only nontrivial functions. By Proposition 2.4 we have that all such minimizers $(f,g)$ are continuous up to the boundary $\partial\Omega$ and $(u,v):=(K_{p}g,K_{q}f)$ solves (1.1) pointwise, in particular $(f,g)=(|u|^{p-1}u,|v|^{q-1}v)$. By Theorem 1.3, $u$ and $v$ are strictly monotone. To show that $u_{r}v_{r}>0$ in $(\delta,1)$, assume without loss of generality that $u$ is increasing; then $u_{r}>0$ in $(\delta,1)$ and, by (1.1), (1.3) and (3.12), $v_{r}=-(N\omega_{N})^{-1}{\mathcal{I}}f(r)r^{1-N}=-(N\omega_{N})^{-1}{\mathcal{I}}(|u|^{p-1}u)(r)r^{1-N}>0$ in $(\delta,1)$, thus $v$ is also strictly increasing. This ends the proof. ∎

Theorem 3.9**.**

(Superlinear case) Let $(p,q)$ satisfy (1.22) and $\Omega$ and $\delta$ be as in (3.1). The set of minimizers of $\phi$ in ${\mathcal{N}}_{rad}$ is nonempty. If $(f,g)\in{\cal N}_{rad}$ is such that $\phi(f,g)=\inf_{{\cal N}_{rad}}\phi$, then $f$ and $g$ are monotone in the radial variable and, if $(u,v):=(K_{p}g,K_{q}f)$, then $u_{r}v_{r}>0$ in $(\delta,1)$.

Proof.

Arguing precisely as in Lemma 2.10, there exists at least one minimizer $(f,g)$ of $\phi$ in ${\mathcal{N}}_{rad}$. Since the $L^{s}$ norms are preserved by the $\divideontimes$-transformation, and (3.19) holds, then reasoning as in part 3) of the proof of Lemma 2.10 we have that $(f^{\divideontimes},g^{\divideontimes})$ is also a minimizer of $\phi$ in $\mathcal{N}_{rad}$. The rest of the proof follows as in the proof of Theorem 3.8. ∎

Since the proof of Theorem 3.8 is based on one-dimensional techniques, one can also apply them to a least energy solution in the case $N=1$ to obtain monotonicity properties.

Corollary 3.10**.**

Let $\Omega=(-1,1)$, $(f,g)$ as in (1.10), and $(u,v):=(K_{p}g,K_{q}f)$. Then $u_{r}v_{r}>0$ in $(-1,1)$, that is, $u$ and $v$ are both strictly monotone increasing or strictly monotone decreasing in the radial variable.

Proof.

For $N=1$ and $\Omega=(-1,1)$, let ${\cal I}_{1}:L^{\infty}(\Omega)\to C(\overline{\Omega})$ and ${\mathfrak{F}}_{1}:C(\overline{\Omega})\to L^{\infty}(\Omega)$ be given by

[TABLE]

Let $\tau_{1}:[-1,1]\to[0,2]$, $x\mapsto\tau(x)=x+1$, and for $h\in C(\overline{\Omega})$, the $\divideontimes_{1}$-transformation of $h$ is given by

[TABLE]

The proof now follows by replacing ${\cal I}_{1},$ ${\mathfrak{F}}_{1}$, $\tau_{1}$, and ${\divideontimes_{1}}$ instead of ${\cal I},$ ${\mathfrak{F}}$, $\tau$, and ${\divideontimes}$ in the proofs of Proposition 3.4 and Theorems 3.8 and 3.9 using $\delta:=-1$ and doing some obvious changes (for example, the use of polar coordinates is not needed). ∎

4 Symmetry and symmetry breaking of minimizers

4.1 Symmetry breaking in radial domains

This subsection is devoted to the following result.

Theorem 4.1**.**

Let $N\geq 1$, $\Omega$ as in (1.11), (1.2) hold, and $(f,g)$ satisfy (1.10), then $f$ and $g$ are not radial.

The proof of this result is long, and we split it in several lemmas. The argument goes by contradiction and is also based on the results from Section 3, i.e., that if $f$ and $g$ are radial, then both functions are either strictly increasing or strictly decreasing. This allows to construct a direction $(\varphi,\psi)$ along which the second variation of $\phi$ at $(f,g)$ is strictly negative, contradicting the minimality property of $(f,g)$. In the following, $\Omega\subset\mathbb{R}^{N}$ is either a ball or an annulus as in (3.1). We begin with an auxiliary lemma.

Lemma 4.2**.**

Let $U\subset\mathbb{R}^{N}$ be a domain of class $C^{2,1}$ and

[TABLE]

which is an open subset of $C^{1}(\overline{U})$. For $s>0$ the following holds.

1. (1)

There is $\delta>1$ depending only on $s$ such that the map $\gamma:{\cal W}(U)\to L^{\delta}(U)$, $\gamma(w):=|w|^{s-1}$ is well-defined and continuous. 2. (2)

If $w\in{\cal W}(U)$, then $(|w|^{s-1}w)_{x_{i}}=s|w|^{s-1}w_{x_{i}}$, $i=1,\ldots,N$, and

[TABLE]

The proof of Lemma 4.2 follows from (the proof of) [25, Proposition 2.3], where only the case $s\in(0,1)$ is considered, but the case $s\geq 1$ follows easily from standard arguments.

The next result state that the second variation of $\phi$ is nonnegative along some directions. We use $X_{rad}$ to denote the subspace of radially symmetric functions in $X$.

Lemma 4.3**.**

Let $pq<1$ and $(f,g)\in X_{rad}$ be a global minimizer of $\phi$ in $X$, $(u,v):=(K_{p}g,K_{q}f)$, and let $(\varphi,\psi)\in[C^{1}(\overline{\Omega})]^{2}\cap X$, with ${\rm supp}(\varphi)\subset\Omega\setminus f^{-1}(0)$ and ${\rm supp}(\psi)\subset\Omega\setminus g^{-1}(0)$. Then the map $s\mapsto\phi(f+s\varphi,g+s\psi)$ belongs to $C^{\infty}(\mathbb{R})$ and

[TABLE]

Proof.

For $s\in\mathbb{R}$, we write $\phi(f+s\varphi,g+s\psi)=\Psi(s)-T(s)$, with

[TABLE]

Then $T\in C^{\infty}(\mathbb{R})$ and, by integration by parts,

[TABLE]

As for $\Psi$, since $\alpha=\frac{p+1}{p}>1$ and $\beta=\frac{q+1}{q}>1$, it is standard to show that

[TABLE]

Since $u\in C^{2}(\overline{\Omega})$ and $U:={\rm supp}(\varphi)\subset\Omega\setminus f^{-1}(0)$, then $f=|u|^{p-1}u$ is bounded away from 0 on $U$, and $|f|^{\alpha-2}\in C^{1}(U)$. Analogously, $|g|^{\beta-2}\in C^{1}(V)$ with $V:={\rm supp}(\psi)$. Therefore $\psi\in C^{\infty}(\mathbb{R})$ and

[TABLE]

as required. ∎

Let ${\mathcal{N}}_{rad}$ be the subset of radial functions in ${\mathcal{N}}$. We say that a function $\varphi:\overline{\Omega}\to\mathbb{R}$ is antisymmetric with respect to $x_{1}$ if $\varphi(x_{1},x^{\prime})=-\varphi(-x_{1},x^{\prime})$ for all $(x_{1},x^{\prime})\in\overline{\Omega}$.

Lemma 4.4**.**

Let $pq>1$ and $(f,g)\in{\mathcal{N}}_{rad}$ be a minimizer of $\phi$ in ${\mathcal{N}}_{rad}$, $(u,v):=(K_{p}g,K_{q}f)$. Let $(\varphi,\psi)\in[C^{1}(\overline{\Omega})]^{2}\cap X$ be antisymmetric with respect to $x_{1}$ and such that ${\rm supp}(\varphi)\subset\Omega\backslash f^{-1}(0)$ and ${\rm supp}(\psi)\subset\Omega\backslash g^{-1}(0)$. Then the map $s\mapsto\phi(f+s\varphi,g+s\psi)$ belongs to $C^{\infty}(\mathbb{R})$ and (4.2) holds.

Proof.

By Lemma 4.3 the map $s\mapsto\phi(f+s\varphi,g+s\psi)$ is $C^{2}$ at $s=0$. We now argue as in [25, Theorem 4.2]. Assume, by contradiction, that there is $(\varphi,\psi)\in[C^{1}(\overline{\Omega})]^{2}\cap X$ antisymmetric with respect to $x_{1}$ satisfying ${\rm supp}(\varphi)\subset\Omega\backslash f^{-1}(0)$, ${\rm supp}(\psi)\subset\Omega\backslash g^{-1}(0)$, and such that $\frac{d^{2}}{ds^{2}}\phi(f+s\varphi,g+s\psi)<-2\kappa$ for $s\in(-\kappa_{1},\kappa_{1})$ and some $\kappa,\kappa_{1}>0$. Let $\gamma_{1}$ and $\gamma_{2}$ as in (2.16). Using the fact that $(f,g)$ is radially symmetric and $(\varphi,\psi)$ is antisymmetric with respect to $x_{1}$, we obtain that $\phi^{\prime}(c^{\gamma_{1}}f,c^{\gamma_{2}}g)(c^{\gamma_{1}}\varphi,c^{\gamma_{2}}\psi)=0$ for $c\in[\frac{1}{2},2]$. Then, by a Taylor expansion,

[TABLE]

where $o(s^{2})$ is a remainder term uniform in $c\in[\frac{1}{2},2]$. For $s\in(-\kappa_{1},\kappa_{1})$ sufficiently small, let $t=t({f+s\varphi,g+s\psi})>0$ be given by Lemma 2.9; using (2.18), we may assume that $t\in[\frac{1}{2},2]$, then, by Lemma 2.9 and using that $(f,g)\in{\mathcal{N}}$,

[TABLE]

which yields a contradiction to the minimality of $(f,g)$, because $(t^{\gamma_{1}}(f+s\varphi),t^{\gamma_{2}}(g+s\psi))\in{\mathcal{N}}$, and the claim follows. ∎

In the case of least energy radial solutions, Lemma 4.3 and Lemma 4.4 allow us to conclude the following. Let $\Omega(e_{1}):=\{x\in\Omega\>:\>x_{1}>0\}$.

Lemma 4.5**.**

Let $(f,g)\in X_{rad}$ satisfy (1.12) and $(u,v):=(K_{p}g,K_{q}f)$. If $u$ is increasing in the radial variable and $(\bar{f},\bar{g}):=(p|u|^{p-1}u_{x_{1}},q|v|^{q-1}v_{x_{1}}),$ then

[TABLE]

and

[TABLE]

Proof.

Note that $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}\ \backslash\ \{(0,0)\}$ is a radial classical solution of (1.1), by Proposition 2.4. By Theorem 1.2 we know that $u_{r}v_{r}>0$ in $(\delta,1)$, with $\delta=\inf_{x\in\Omega}|x|$. By assumption $u_{r}\geq 0$ in $(\delta,1)$, thus $u$ and $v$ are strictly monotone increasing in the radial variable. In particular,

[TABLE]

Moreover, since $u$ and $v$ are sign-changing, there exist $r_{1},r_{2}\in(\delta,1)$ such that

[TABLE]

i.e., the nodal sets are two spheres contained in $\Omega$ and $u,v\in{\cal W}(\Omega)$, with ${\cal W}(\Omega)$ as defined in (4.1). To control these nodal lines we use the following cutoff functions: for $\varepsilon,r\geq 0$, let $\rho_{\varepsilon}^{r}$ be a smooth radial function in $\mathbb{R}^{N}$ such that

[TABLE]

and let $(\bar{f}_{\varepsilon},\bar{g}_{\varepsilon}):=(\bar{f}\rho^{r_{1}}_{\varepsilon},\bar{g}\rho^{r_{2}}_{\varepsilon})$ for $\varepsilon\geq 0$; note that $(\bar{f}_{\varepsilon},\bar{g}_{\varepsilon})\in[C^{1}(\overline{\Omega})]^{2}$ for $\varepsilon>0$ and $(\bar{f}_{\varepsilon},\bar{g}_{\varepsilon})$ is antisymmetric in $\Omega$ with respect to $x_{1}$ for all $\varepsilon\geq 0$, because $(u,v)$ is radially symmetric in $\Omega$. Moreover,

[TABLE]

for some $t>0$ (see Lemma 4.2), and $(K\bar{g}_{\varepsilon},K\bar{f}_{\varepsilon})$ is also antisymmetric in $\Omega$ with respect to $x_{1}$ for all $\varepsilon\geq 0$, by uniqueness. In particular, $K\bar{f}_{\varepsilon}=K\bar{g}_{\varepsilon}=0$ on $\partial\Omega(e_{1})\cap\{x_{1}=0\}$ and

[TABLE]

by the maximum principle and Hopf’s boundary point lemma. Since $(\bar{f}_{\varepsilon},\bar{g}_{\varepsilon})\to(\bar{f},\bar{g})$ in $L^{t}(\Omega(e_{1}))$ as $\varepsilon\to 0$, then

[TABLE]

and therefore (4.7) implies (4.4). Furthermore, since the product of two antisymmetric functions is symmetric, we have that

[TABLE]

by applying Lemma 4.3 in the sublinear case (1.21) or Lemma 4.4 in the superlinear case (1.22) with $(\varphi,\psi):=(\bar{f}_{\varepsilon},\bar{g}_{\varepsilon})$ for small $\varepsilon>0$. The claim (4.5) follows from Lebesgue’s dominated convergence theorem once we show the existence of a suitable majorant. Indeed, let $(\xi_{\varepsilon},\zeta_{\varepsilon}):=(K\bar{g}_{\varepsilon},K\bar{f}_{\varepsilon})$ for $\varepsilon\geq 0$, then, by (4.8),

[TABLE]

Moreover, we have that $-\Delta(\xi_{\varepsilon}-\xi_{0})=\bar{g}_{\varepsilon}-\bar{g}=q|v|^{q-1}v_{x_{1}}(\rho_{\varepsilon}^{r_{2}}-1)\leq 0$ in $\Omega(e_{1})$, with $\partial_{\nu}(\xi_{\varepsilon}-\xi_{0})=0\text{ on }\partial\Omega(e_{1})\backslash\{x_{1}=0\}$ and $\xi_{\varepsilon}-\xi_{0}=0$ on $\partial\Omega(e_{1})\cap\{x_{1}=0\}$. Thus (testing the equation with $(\xi_{\varepsilon}-\xi_{0})_{+}$ and integrating by parts), $0<\xi_{\varepsilon}\leq\xi_{0}$ in $\Omega(e_{1})$ for $\varepsilon$ small, and

[TABLE]

Note that $|u|^{p-1}u^{2}_{x_{1}}\in L^{1}(\Omega)$, because $|u|^{p-1}\in L^{t}(\Omega)\subset L^{1}(\Omega)$ and $u\in C^{2}(\overline{\Omega})$. It remains to show that

[TABLE]

Since $-\Delta(u_{x_{1}}-\xi_{0})=0$ in $\Omega$ in the strong sense, interior elliptic regularity (see, e.g., [18, Theorem 9.19] ) implies that $u_{x_{1}}-\xi_{0}\in C^{\infty}(\Omega)$; therefore $\xi_{0}\in C^{1}(\Omega)\cap W^{2,t}(\Omega)$, because $u_{x_{1}}\in C^{1}(\overline{\Omega})$. This directly implies (4.11) for $p\geq 1$. If $p\in(0,1)$, then let $\gamma>0$ be such that $A:=\{x\in\Omega:r_{1}-\gamma<|x|<r_{1}+\gamma\}\subset\Omega$. Then

[TABLE]

and (4.11) also follows. A majorant for the term $\bar{g}_{\varepsilon}(v_{x_{1}}\rho^{r_{2}}_{\varepsilon}-K\bar{f}_{\varepsilon})$ in (4.9) can be obtained similarly, and this ends the proof. ∎

The following lemma shows that an antisymmetric Neumann solution dominates the corresponding Dirichlet solution in a half radial domain $\Omega(e_{1})$.

Lemma 4.6**.**

Let $t>1$ and $h\in L^{t}(\Omega)\backslash\{0\}$ be an antisymmetric function in $\Omega$ with respect to $x_{1}$ and let $w^{N}:=Kh\in W^{2,t}(\Omega)$, that is, $w^{N}$ is the unique strong solution of

[TABLE]

Moreover, let $w^{D}\in W^{2,t}(\Omega)\cap C^{1}(\overline{\Omega})$ be a strong solution of $-\Delta w^{D}=h$ in $\Omega$ with $w^{D}=0$ on $\partial\Omega$. If $h\geq 0$ and $w^{N}\geq 0$ in $\Omega(e_{1})$ then $w^{D}<w^{N}$ in $\Omega(e_{1})$.

Proof.

By uniqueness of solutions we have that $w^{D}\in W^{2,t}(\Omega)\cap C^{1}(\overline{\Omega})$ and $w^{N}:=Kh\in W^{2,t}(\Omega)$ are antisymmetric functions in $\Omega$ with respect to $x_{1}$ and therefore, since $w^{N}\geq 0$ in $\Omega(e_{1})$ by assumption,

[TABLE]

Observe that $w^{D}\not\equiv w^{N}$ in $\Omega(e_{1})$, because otherwise $w^{D}=w^{N}\geq 0$ in $\Omega(e_{1})$ with $\partial_{\nu}w^{D}=w^{D}=0$ on $\partial\Omega(e_{1})\cap\{x_{1}>0\}$, which contradicts Hopf’s boundary point lemma. Then, since $-\Delta(w^{D}-w^{N})=0$ in $\Omega$, $w^{D}\not\equiv w^{N}$ in $\Omega(e_{1})$, and (4.12) holds, the maximum principle yields that $w^{D}<w^{N}$ in $\Omega(e_{1})$. ∎

Proof of Theorem 4.1.

Let $\Omega\subset\mathbb{R}^{N}$ as in (3.1), $(f,g)$ as in the statement, $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{B})]^{2}$, and $(\bar{f},\bar{g}):=(p|u|^{p-1}u_{x_{1}},q|v|^{q-1}v_{x_{1}})$. We argue by contradiction. Assume without loss of generality that $f$ is radial. Using the relations $u=|f|^{\frac{1}{p}}f,$ $v=K_{q}f$, and $g=|v|^{q-1}v,$ we obtain that also $u$, $v$, and $g$ must be radially symmetric. By Lemma 4.2, there is some $t>1$ such that $|u|^{p-1},|v|^{q-1}\in L^{t}(\Omega)$ and $\partial_{x_{1}}(|v|^{q-1}v)=q|v|^{q-1}v_{x_{1}}\in L^{t}(\Omega)$. Therefore, we have that $-\Delta u=|v|^{q-1}v\in W^{1,t}(\Omega)$ and (interior) elliptic regularity (see for instance [18, Theorem 9.19]) yields that $u\in W_{\rm loc}^{3,t}(\Omega)$. Thus, we may interchange derivatives, and $(-\Delta u)_{x_{1}}=-\Delta(u_{x_{1}})$ in $\Omega$. Arguing analogously for $-\Delta v$ we conclude that $(u_{x_{1}},v_{x_{1}})\in[W^{2,t}(\Omega)\cap C^{1,\varepsilon}(\overline{\Omega})]^{2}$ is the unique strong solution of the Dirichlet problem

[TABLE]

where the boundary conditions follow from the fact that $u$ and $v$ are radially symmetric and $\partial_{\nu}u=\partial_{\nu}v=0$ on $\partial\Omega$. By Theorem 1.2, we may assume that $u$ and $v$ are strictly increasing in the radial variable (the other case follows similarly). Then $u_{x_{1}}$ and $v_{x_{1}}$ are nonnegative in $\Omega(e_{1})$ and, by Lemmas 4.5 and 4.6,

[TABLE]

a contradiction. ∎

4.2 Foliated Schwarz symmetry

In this section we show that least energy solutions are foliated Schwarz symmetric whenever the domain $\Omega$ is a ball or an annulus centered at zero in $\mathbb{R}^{N}$ in dimension $N\geq 2$. For $N=1$, see Corollary 3.10. We introduce first some notation. Let ${\mathbb{S}}^{N-1}=\{x\in\mathbb{R}^{N}:|x|=1\}$ be the unit sphere and fix $e\in{\mathbb{S}}^{N-1}$. We consider the halfspace $H(e):=\{x\in\mathbb{R}^{N}:x\cdot e>0\}$ and the half domain $\Omega(e):=\{x\in\Omega:x\cdot e>0\}.$

The composition of a function $w:\overline{\Omega}\to\mathbb{R}$ with a reflection with respect to $\partial H(e)$ is denoted by $w_{e}$, that is,

[TABLE]

The polarization $u^{H}$ of $u:\overline{\Omega}\to\mathbb{R}$ with respect to a hyperplane $H=H(e)$ is given by

[TABLE]

Following [34], we say that $u\in C(\overline{\Omega})$ is foliated Schwarz symmetric with respect to some unit vector $p\in{\mathbb{S}}^{N-1}$ if $u$ is axially symmetric with respect to the axis $\mathbb{R}p$ and nonincreasing in the polar angle $\theta:=\operatorname{arccos}(\frac{x}{|x|}\cdot p)\in[0,\pi].$ We use the following characterization of foliated Schwarz symmetry given in [32]. We remark that these kind of characterizations appeared for the first time in [10], see also [35] for a survey on symmetry via reflection methods.

Lemma 4.7** (Particular case of Proposition 3.2 in [32]).**

There is $p\in{\mathbb{S}}^{N-1}$ such that $u,v\in C(\overline{\Omega})$ are foliated Schwarz symmetric with respect to $p$ if and only if for every $e\in{\mathbb{S}}^{N-1}$ either

[TABLE]

The main result of this section is the following. For similar results under Dirichlet boundary conditions, we refer to [7, Theorem 1.3] and [8, Theorem 1.2].

Theorem 4.8** (Sublinear case).**

Let $\Omega$ be either a ball or an annulus centered at the origin of $\mathbb{R}^{N}$, $N\geq 2$. Let $(p,q)$ satisfy (1.21) and $(f,g)\in X$ be a global minimizer of $\phi$ in $X$ and $(u,v):=(K_{p}g,K_{q}f)$. There is $p\in{\mathbb{S}}^{N-1}$ such that $u$ and $v$ are foliated Schwarz symmetric with respect to $p$.

Proof.

Let $(f,g)$ and $(u,v)$ as in the statement and fix a hyperplane $H=H(e)$ for some $|e|=1$. By Proposition 2.4, $(u,v)\in[C^{2,\mu}(\Omega)]^{2}$ for some $\mu\in(0,1)$, $(u,v)$ solves (1.1) pointwise, and $(f,g)=(-\Delta v,-\Delta u)\in[C^{\mu}(\overline{\Omega})]^{2}$; thus $(f^{H},g^{H})\in[L^{\infty}(\Omega)]^{2}$ and $(\widetilde{u},\widetilde{v}):=(K_{p}(g^{H}),K_{q}(f^{H}))\in[W^{2,N}({\Omega})\cap C^{1}(\overline{\Omega})]^{2}$, by Sobolev embeddings.

Let $V:=v_{e}+v-\widetilde{v}-\widetilde{v}_{e}$, then using the definition of $f^{H}$ we have that

[TABLE]

testing this equation with $V$ and integrating by parts we obtain that $V=k$ for some $k\in\mathbb{R}$. Then

[TABLE]

Let

[TABLE]

$w_{1}:=\widetilde{v}-v+k/2$, and $w_{2}:=\widetilde{v}-v_{e}+k/2$. Since $v=v^{e}$ and $\widetilde{v}=\widetilde{v}_{e}$ on $\Gamma_{1}$ we have that $w_{1}=w_{2}=0$ on $\Gamma_{1}$, by (4.15), and $\partial_{\nu}w_{1}=\partial_{\nu}w_{2}=0$ on $\Gamma_{2}$. Furthermore,

[TABLE]

which implies by the maximum principle and Hopf’s Lemma that $w_{1}\geq 0$ and $w_{2}\geq 0$ in $\Omega(e)$. Therefore, using that $\widetilde{v}_{e}=v_{e}+v-\widetilde{v}-k$ (by (4.15)) and $g^{H}_{e}=g_{e}+g-g^{H}$ (by definition of $g^{H}$),

[TABLE]

because $\int_{\Omega}g=0$.

To show that $u$ and $v$ are foliated Schwarz symmetric with respect to the same vector $p\in{\mathbb{S}}^{N-1}$ we use Lemma 4.7 and argue by contradiction. Assume that (4.14) does not hold. Then, without loss of generality, there are $e\in{\mathbb{S}}^{N-1}$ and the corresponding halfspace $H=H(e)$ such that $v\neq v^{H}$ in $B(e)$ and either $v_{e}\neq v^{H}$ in $\Omega(e)$ or $u_{e}\neq u^{H}$ in $\Omega(e)$. Since $t\mapsto h(t):=|t|^{s}t$ is a strictly monotone increasing function in $\mathbb{R}$ for $s>-1$, this implies that

[TABLE]

As a consequence, $w_{1}>0$ and $0\neq g-g^{H}\leq 0$ in $B(e)$ and either $w_{2}>0$ in $B(e)$ or $0\neq g_{e}-g^{H}\leq 0$ in $B(e)$. In any case, (4.17) implies that

[TABLE]

But then, since $L^{s}$ norms are preserved under polarizations, we obtain that $\phi(f,g)>\phi(f^{H},g^{H})$, a contradiction to the minimality of $(f,g)$. Therefore (4.14) holds and the theorem follows from Lemma 4.7. ∎

Theorem 4.9** (Superlinear case).**

Let $\Omega$ be either a ball or an annulus centered at the origin of $\mathbb{R}^{N}$, $N\geq 2$. Let $(p,q)$ satisfy (1.22) and $(f,g)\in X$ be a minimizer of $\phi$ in $\mathcal{N}$ and $(u,v):=(K_{p}g,K_{q}f)$. There is $p\in{\mathbb{S}}^{N-1}$ such that $u$ and $v$ are foliated Schwarz symmetric with respect to $p$.

Proof.

Arguing by contradiction as in the proof of Theorem 4.8, we obtain (4.19). Since $(f,g)\in\mathcal{N}$ and $L^{s}$-norms are preserved under polarizations we have, by (2.17),

[TABLE]

and $\int_{\Omega}f^{H}Kg^{H}>0$, then by Lemma 2.9 there exists $0<t<1$ such that $(t^{\gamma_{1}}f^{H},t^{\gamma_{2}}g^{H})\in\mathcal{N}$. This gives a contradiction, because, by (2.17),

[TABLE]

5 Further results

5.1 Unique continuation principle for minimizers

In this section, $\Omega$ is again a general smooth bounded domain. We prove that if $(u,v)$ is a solution of (1.1) associated to a minimizer of $\phi$, then the nodal sets $u^{-1}(0):=\{x\in\Omega:\ u(0)=0\}$ and $v^{-1}(0):=\{x\in\Omega:\ v(x)=0\}$ have zero Lebesgue measure. We do it by extending the results and techniques from [25, Section 3] to the setting of Hamilitonian elliptic systems and to the dual method framework. Recall that if $(f,g)\in X$ is a critical point of $\phi$ then $(u,v):=(K_{p}g,K_{q}f)\in[C^{2,\varepsilon}(\overline{\Omega})]^{2}$ is a classical solution of (1.1). Our main result is the following.

Theorem 5.1**.**

Let $(f,g)\in X$ be a critical point of $\phi$ in $X$ and $(u,v):=(K_{p}g,K_{q}f)$, then $u^{-1}(0)=v^{-1}(0)$ a.e. Moreover, if $p$ and $q$ satisfy the sublinear condition (1.21) and $(f,g)\in X$ is a global minimizer of $\phi$ in $X$, then $|u^{-1}(0)|=|v^{-1}(0)|=0$.

To show Theorem 5.1 we rely on the following preliminary results.

Lemma 5.2** (Lemma 3.1 in [25]).**

Let $\gamma>0$ and $f:(0,\infty)\to[0,\infty)$ be such that $f$ is bounded in $[\varepsilon,\infty)$ for every $\varepsilon>0$ and $\lim_{r\to 0}r^{\gamma}f(r)=0$. Then for every $r>0$ there is $s>0$ such that $f(s)\geq f(r)$ and $f(t)\leq 2^{\gamma}f(s)$ for all $t\in[\frac{s}{2},2s]$.

We now characterize the decay of any solution $(u,v)$ of (1.1) close to a common zero.

Proposition 5.3**.**

Let $(u,v)\in[C^{2}(\overline{\Omega})]^{2}$ be a solution of (1.1) with $p,q>0$, $pq<1$. If $x_{0}\in\Omega$ is a point of density one for the set $u^{-1}(0)\cap v^{-1}(0)$ then $|u(x)|^{p+1}+|v(x)|^{q+1}=o(|x-x_{0}|^{\gamma})$, where

[TABLE]

In other words,

[TABLE]

Proof.

Without loss of generality we assume that $x_{0}=0$ and we set $u\equiv v\equiv 0$ in $\mathbb{R}^{N}\backslash\Omega$. Let

[TABLE]

Step 1. We show first that $f\in L^{\infty}(0,\infty)$. Indeed, assume by contradiction that there is a sequence $(r_{n})_{n\in{\mathbb{N}}}$ such that $f(r_{n})\to\infty$ as $n\to\infty$. Then, by Lemma 5.2, there is $(s_{n})_{n\in{\mathbb{N}}}$ such that $f(s_{n})\geq f(r_{n})$ and $f(t)\leq 2^{\gamma}f(s_{n})$ for all $t\in[\frac{s_{n}}{2},2s_{n}]$ and $n\in{\mathbb{N}}$. Then $f(s_{n})\to\infty$ as $n\to\infty$ and $s_{n}\to 0$, by the definition of $f$. Working if necessary with a subsequence, we may assume that $B_{2s_{n}}\subset\Omega$ for all $n\in{\mathbb{N}}$. Set $\Omega_{0}:=B_{2}\backslash B_{\frac{1}{2}}$ and let $u_{n},v_{n}:\Omega_{0}\to\mathbb{R}$ be given by

[TABLE]

By (1.1), and since $2-\frac{\gamma}{p+1}+q\frac{\gamma}{q+1}=2-\frac{\gamma}{q+1}+p\frac{\gamma}{p+1}=0$ we have that

[TABLE]

where

[TABLE]

Observe next that

[TABLE]

for all $x\in\Omega_{0}$ and $n\in{\mathbb{N}}.$ By (5.3), (5.4), (5.5), and interior elliptic regularity, there are subsequences $u_{n}\to u^{*}$, $v_{n}\to v^{*}$ in $C^{1}_{loc}(\Omega_{0})$. Furthermore, by definition of $f$ and the regularity of $u,v$, there is $(x_{n})_{n\in{\mathbb{N}}}\subset\mathbb{S}^{N-1}$ such that $|u_{n}(x_{n})|^{p+1}+|v_{n}(x_{n})|^{q+1}=1$ for all $n\in{\mathbb{N}}$. Thus, up to a subsequence, $x_{n}\to x^{*}\in\mathbb{S}^{N-1}$ with

[TABLE]

However, since [math] is a point of density one for $u^{-1}(0)\cap v^{-1}(0)$, we have that

[TABLE]

which implies that $u^{*}\equiv 0$ in $\Omega_{0}$. Analogously, we obtain that $v^{*}\equiv 0$ in $\Omega_{0}$. This contradicts (5.6) and therefore $f\in L^{\infty}(0,\infty)$.

Step 2. Now, it suffices to show that $\lim_{r\to 0}f(r)=0$. We argue again by contradiction: assume there is a sequence $r_{n}\to 0$ as $n\to\infty$ such that $f(r_{n})\geq\varepsilon$ for all $n\in{\mathbb{N}}$ and for some $\varepsilon>0$. Passing if necessary to a subsequence we have that $B_{2r_{n}}\subset\Omega$. Since $f$ is bounded (by Step 1), the rescaled functions $\widetilde{u}_{n},\widetilde{v}_{n}:\Omega_{0}\to\mathbb{R}$ given by

[TABLE]

are uniformly bounded. Moreover,

[TABLE]

by (1.1), and there is $(\widetilde{x}_{n})_{n\in{\mathbb{N}}}\subset\mathbb{S}^{N-1}$ such that $|\widetilde{u}_{n}(x_{n})|^{p+1}+|\widetilde{v}_{n}(x_{n})|^{q+1}=f(r_{n})\geq\varepsilon$ for all $n\in{\mathbb{N}}$. But arguing as before, we obtain subsequences $\widetilde{u}_{n}\to\widetilde{u}$, $\widetilde{v}_{n}\to\widetilde{v}$ in $C^{1}_{loc}(\Omega_{0})$, and $\widetilde{x}_{n}\to\widetilde{x}\in\mathbb{S}^{N-1}$ such that $|\widetilde{u}(\widetilde{x})|+|\widetilde{v}(\widetilde{x})|\geq\varepsilon$. But (5.4) with $\widetilde{u}_{n}$ instead of $u_{n}$ yields that $\widetilde{u}\equiv 0$, and we have analogously that $\widetilde{v}\equiv 0$, a contradiction. Therefore $\lim_{r\to 0}f(r)=0$, and the proof is finished. ∎

We now use Proposition 5.3 to construct directions along which the energy $\phi$ decreases.

Proposition 5.4**.**

Let $(f,g)\in X$ be a critical point of $\phi$, $(u,v):=(K_{p}g,K_{q}f)$, and assume that (1.21) holds. If $|u^{-1}(0)\cap v^{-1}(0)|>0$, then there is $(\varphi,\psi)\in X$ such that $\phi(f,g)>\phi(f+\varphi,g+\psi)$.

Proof.

If $|u^{-1}(0)\cap v^{-1}(0)|>0$ then, by Lebesgue’s density theorem (see, for example, [17, page 45]), there exists a point of density one $x_{0}$ of $u^{-1}(0)\cap v^{-1}(0)$. Without loss of generality we assume that $x_{0}=0$. Since $(u,v):=(K_{p}g,K_{q}f)$ is a classical solution of (1.1) by Proposition 2.4, we obtain by Proposition 5.3 that

[TABLE]

as $|x|\to 0$, where $\gamma$ is as in (5.1). Let $\zeta\in C_{c}^{\infty}(\mathbb{R}^{N})\backslash\{0\}$ such that $\operatorname{supp}\zeta\subset\Omega$, $\int_{\Omega}\zeta=0$, and fix $t>0$ such that

[TABLE]

(such number $t$ exists, since $\alpha\beta/(\alpha+\beta)>1$, which is equivalent to $pq<1$).

For $r>0$ small denote $\Omega_{r}:=r\Omega\subset\Omega$ and let $\varphi_{r},\psi_{r}:\Omega\to\mathbb{R}$ be given by

[TABLE]

Notice that $\varphi_{r}\equiv\psi_{r}\equiv 0$ in $\mathbb{R}^{N}\backslash\Omega_{r}$ and note that $w:=K(\zeta(\frac{x}{r}))\in C^{2}(\overline{\Omega})\cap C^{1}(\overline{\Omega})$ solves classically $-\Delta w=\zeta(\frac{x}{r})$ in $\Omega$ and $\partial_{\nu}w=0$ on $\partial\Omega$, therefore

[TABLE]

On the other hand, multiplying $-\Delta w=\zeta(\frac{x}{r})$ in $\Omega$ by $\zeta(\frac{x}{r})$ and using Hölder’s inequality,

[TABLE]

that is, $\|\nabla w\|_{L^{2}(\Omega)}\geq r^{1+\frac{N}{2}}\frac{\|\zeta\|_{L^{2}(\Omega)}^{2}}{\|\nabla\zeta\|_{L^{2}(\Omega)}},$ and therefore, by (5.12),

[TABLE]

Then, by (5.11), (5.13), and observing that $\gamma$ satisfies $\gamma(1/\alpha+1/\beta)+2=\gamma$ (cf. (5.1)) we obtain

[TABLE]

We claim that $E:=\phi(f+\varphi_{r},g+\psi_{r})-\phi(f,g)<0$. Indeed, by (5.10),

[TABLE]

and analogously, $\int_{{\Omega_{r}}}|g+\psi_{r}|^{\beta}-|g|^{\beta}-|\psi_{r}|^{\beta}\ dx=o(r^{\gamma+N})$ as $r\to 0$. Furthermore, by Proposition 5.3,

[TABLE]

as $r\to 0$, and analogously, $\int_{{\Omega_{r}}}|\varphi_{r}Kg|=o(r^{\gamma+N})$ as $r\to 0$. Therefore, by (2.6), (5.14), (5.15), (5.16), and the fact that $\varphi_{r}\equiv 0$ in $\mathbb{R}^{N}\backslash\Omega_{r}$,

[TABLE]

for $r>0$ sufficiently small, and the proof is finished. ∎

We are now ready to conclude the proof of the main result of this section.

Proof of Theorem 5.1.

Let $(f,g)\in X$ be a critical point of $\phi$ in $X$ and $(u,v):=(K_{p}g,K_{q}f)$. By [18, Lemma 7.7], Proposition 2.4, and (1.1), we have that $|v|^{q}=|\Delta u|=0$ a.e. in $u^{-1}(0)$ and $|u|^{p}=|\Delta v|=0$ a.e. in $v^{-1}(0)$, therefore $u^{-1}(0)=v^{-1}(0)$ a.e. Now, assume that (1.21) holds and $(f,g)\in X$ is a global minimizer of $\phi$ in $X$. Then, by Proposition 5.4, we have that $|u^{-1}(0)\cap v^{-1}(0)|=0$ and thus $|u^{-1}(0)|=|v^{-1}(0)|=0$, since $u^{-1}(0)=v^{-1}(0)$ a.e. ∎

Remark 5.5**.**

(Superlinear case) Let $(u,v)\in[C^{2}(\overline{\Omega})]^{2}$ be a solution of (1.1) with $p,q>0$, $pq>1$. If $|u^{-1}(0)\cap v^{-1}(0)|>0$, then $(u,v)$ has a zero of infinite order; indeed, if $x_{0}\in\Omega$ is a point of density one for the set $u^{-1}(0)\cap v^{-1}(0)$ and $\gamma>0$, then $|u(x)|^{p+1}+|v(x)|^{q+1}=o(|x-x_{0}|^{\gamma})$ as $x\to x_{0}$. A proof can be obtained by repeating almost word by word the proof of Proposition 5.3 with the following changes: in Step 1, the functions $u_{n},v_{n}$ defined in (5.2) satisfy system (5.3) with $\widetilde{A}_{n}:=s_{n}^{2}(\sup_{|x|=s_{n}}|u(x)|^{p+1}+|v(x)|^{q+1})^{\frac{pq-1}{(p+1)(q+1)}}$ instead of $A_{n}$ as in (5.4), observe that $\widetilde{A}_{n}\to 0$ as $n\to\infty$; moreover, in Step 2, the functions $\widetilde{u}_{n},\widetilde{v}_{n}$ defined in (5.8) solve (instead of (5.9)) the system $-\Delta\widetilde{u}_{n}=B_{n}|\widetilde{v}_{n}|^{q-1}\widetilde{v}_{n},-\Delta\widetilde{v}_{n}=B_{n}|\widetilde{u}_{n}|^{p-1}\widetilde{u}_{n}$, with $B_{n}=r_{n}^{2+\gamma\frac{pq-1}{(p+1)(q+1)}}\to 0$ as $n\to\infty$.

5.2 Simplicity of the zeros for radial solution

Proof of Theorem 1.5.

Let

[TABLE]

$b:=\lim_{s\to\delta}t(s)\in(1,\infty]$ ($b=\infty$ if $\Omega=B_{1}$), $J:=[1,b)$, and $\varphi,\psi:J\to\mathbb{R}$ satisfy $\varphi(t(|x|))=u(x)$ and $\psi(t(|x|))=v(x)$. Then $(\varphi,\psi)$ is a bounded solution of

[TABLE]

where $\zeta(t):=(N-2)^{-2}t^{-2\frac{N-1}{N-2}}\geq 0$ for $N\geq 3$ and $\zeta(t)=e^{-2(t-1)}\geq 0$ for $N=2$.

To prove (1.20) it suffices to show that

[TABLE]

We argue by contradiction: assume there is $x\in J$ such that

[TABLE]

Then, by (5.18),

[TABLE]

Indeed, if $|\varphi|>0$ in $(x,b)$ then $|\psi^{\prime\prime}|>0$ in $(x,b)$ and, in virtue of (5.20), $\psi$ must be unbounded if $b=\infty$ or $|\psi^{\prime}(b)|>0$ if $b<\infty$, which is impossible by assumption, and therefore (5.21) follows.

Now, it suffices to rule out the next three cases:

[TABLE]

Case a) Assume that $\varphi(x)=0$. Take the first integral $\Phi:J\to\mathbb{R}$ given by

[TABLE]

which satisfies

[TABLE]

by (5.18) and the fact that $\zeta$ is decreasing. Moreover, since $\varphi(x)=0$, (5.20) implies that $\Phi(x)=0$ and therefore

[TABLE]

by (5.22). By (5.21), $\varphi(x_{0})=0$ for some $x_{0}>x$. We claim that $\varphi\equiv 0$ in $[x,x_{0}]$. By (5.18), this implies that also $\psi\equiv 0$ in $[x,x_{0}]$, which contradicts the assumption that $\varphi^{-1}(0)\cap\psi^{-1}(0)$ has empty interior, ruling out case a).

To prove the claim, suppose that $\varphi\not\equiv 0$, and take $y^{*}\in(x,x_{0})$ such that $|\varphi(y^{*})|\neq 0$. Without loss of generality, assume that $\varphi(y^{*})>0$ (the other case is analogous). Then (5.23) implies $\varphi^{\prime}(y^{*})\psi^{\prime}(y^{*})<0$ and (by replacing the point $y^{*}$ if necessary) we may assume that $\varphi^{\prime}(y^{*})>0$. But then, using (5.23) and a simple continuity argument, we have $\varphi,\varphi^{\prime}>0$ in $[y^{*},b)$, in contradiction with (5.21).

Case b) Let $\varphi(x)>0$ and $\varphi^{\prime}(x)\geq 0$ (the case $\varphi(x)<0$ and $\varphi^{\prime}(x)\leq 0$ is similar). By adjusting the point $x_{0}$ from (5.21) if necessary, we may assume that $\varphi>0$ in $(x,x_{0})$, and by system (5.18) also $-\psi^{\prime\prime}>0$ in $(x,x_{0})$. By (5.20), this implies that $\psi<0$ in $(x,x_{0})$ and therefore $-\varphi^{\prime\prime}<0$ in $(x,x_{0})$, i.e., $\varphi$ is convex in $(x,x_{0})$, which would contradict $\varphi(x_{0})=0$.

Case c) Finally, we rule out $\varphi(x)\varphi^{\prime}(x)<0$. Suppose that $\varphi(x)>0$ and $\varphi^{\prime}(x)<0$ (the case $\varphi(x)<0$ and $\varphi^{\prime}(x)>0$ is similar). Observe that, by (5.18) we deduce $\psi^{\prime\prime}(x)<0$, which, combined with (5.20), yields $\psi<0$ in some interval $(w_{0},x)\subseteq(1,x)$. We claim that actually $\psi<0$ in $(1,x)$, which yields a contradiction since this would imply $\varphi^{\prime\prime}>0$ in $(1,x)$ and thus $\varphi^{\prime}(1)<0$, contradicting the boundary conditions in (5.18) .

To prove the claim, assume $w_{0}>1$ and $\psi(w_{0})=0$. Then, by the mean value theorem, there is $w_{1}\in(w_{0},x)$ such that $\psi^{\prime}(w_{1})=0$. Since $\psi^{\prime}(x)=0$ we may use again the mean value theorem to obtain $w_{2}\in(w_{1},x)$ such that $\psi^{\prime\prime}(w_{2})=0$. But then $\varphi(w_{2})=0$ by (5.18), which is impossible since $\varphi(x)<0$, $\varphi^{\prime}(x)>0$, and $-\varphi^{\prime\prime}=\zeta|\psi|^{q-1}\psi>0$ in $(w_{0},x)$.

Since we have reached a contradiction in all cases, we conclude that (5.20) cannot happen. The case $\varphi(x)=\varphi^{\prime}(x)=0$ is completely analogous to (5.20) and therefore the claim (5.19) follows.

∎

Acknowledgments. A. Saldaña is supported by the Alexander von Humboldt Foundation, Germany. H. Tavares is partially supported by ERC Advanced Grant 2013 n. 339958 “Complex Patterns for Strongly Interacting Dynamical Systems - COMPAT”. Both authors were partially supported by FCT/Portugal through UID/MAT/04459/2013. The graphs in Figure 1 and the numerical approximations in Remark 3.6 were done using Mathematica 11.1, Wolfram Research Inc. The authors would like to thank Tobias Weth for fruitful discussions and the anonymous referee for the careful reading of the paper and helpful comments and suggestions.