A simple variational approach to weakly coupled competitive elliptic systems

TL;DR

This paper introduces a straightforward variational framework for identifying fully nontrivial solutions to weakly coupled elliptic systems, linking solutions to critical points of a smooth functional on a product of spheres.

Contribution

It presents a novel, simple variational approach that simplifies finding solutions to weakly coupled elliptic systems and extends existing results in the field.

Findings

Established a variational setting for the system

Linked solutions to critical points of a smooth functional

Extended known results for weakly coupled systems

Abstract

The main purpose of this paper is to exhibit a simple variational setting for finding fully nontrivial solutions to the weakly coupled elliptic system (1.1). We show that such solutions correspond to critical points of a $\mathcal{C}^1$-functional $\Psi:\mathcal{U}\to\mathbb{R}$ defined in an open subset $\mathcal{U}$ of the product $\mathcal{T}:=S_1\times\cdots\times S_M$ of unit spheres $S_i$ in an appropriate Sobolev space. We use our abstract setting to extend and complement some known results for the system (1.1).

Equations

$$\{\begin{array}{l}-\Delta u_i+{\kappa }_i{u}_i={\mu }_i|u_i{|}^{p-2}{u}_i+\sum _{j\ne i}{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_i,\\ u_i\in H,i,j=1,\dots ,M,\end{array}$$

$$-\Delta u+{\kappa }_{i}u={\mu }_i|u{|}^{p-2}u,u\in H,$$

$$\{\begin{array}{l}-\Delta u_i+{\kappa }_i{u}_i={\mu }_i|u_i{|}^{p-2}{u}_i+\sum _{j\ne i}{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_i,\\ u_i\in H_0^1(\Omega ),i,j=1,\dots ,M,\end{array}$$

$$\{\begin{array}{l}-\Delta u_i={\mu }_i|u_i{|}^{2^{\ast }-2}{u}_i+\sum _{j\ne i}{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_i,\\ u_i\in D^{1,2}({\mathbb{R}}^N),i,j=1,\dots ,M,\end{array}$$

$$\{\begin{array}{l}-\Delta u_i+{\kappa }_i{u}_i={\mu }_i|u_i{|}^{2^{\ast }-2}{u}_i+\sum _{j\ne i}{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_i,\\ u_i\in D_0^{1,2}(\Omega ),i,j=1,\dots ,M,\end{array}$$

$$J(s):=\sum _{i=1}^M{a}_i{s}_i^2-\sum _{i=1}^M{b}_i{s}_i^p+\sum _{i\ne j}{d}_{ij}{s}_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}},$$

$${\partial }_{i}J(s)$$

$$=2a_i{s}_i-p{b}_i{s}_i^{p-1}+2\sum _{j\ne i}{d}_{ij}{\beta }_{ij}{s}_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}-1}.$$

$$\underset{s\in (0,\infty {)}^M}{\max }J(s)=\underset{s\in [r,R{]}^M}{\max }J(s).$$

$$2a_{i}t-\left(p{b}_i-2\sum _{j\ne i}{d}_{ij}{\beta }_{ij}\right)t^{p-1}<0\text{if }t\in [R,\infty )$$

$$2a_{i}t-p{b}_i{t}^{p-1}>0\text{if }t\in (0,r].$$

$${\partial }_{i}J(s)\le 2a_i{s}_i-\left(p{b}_i-2\sum _{j\ne i}{d}_{ij}{\beta }_{ij}\right)s_i^{p-1}<0,$$

$${\partial }_{i}J(s)\ge 2a_i{s}_i-p{b}_i{s}_i^{p-1}>0.$$

$$0<2a_i=p{b}_i-2\sum _{j\ne i}{d}_{ij}{\beta }_{ij}\text{for all }i=1,\dots ,M.$$

$$2a_i(s_i-s_i^{p-1})=2\sum _{j\ne i}{d}_{ij}{\beta }_{ij}(s_i^{p-1}-s_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}-1}).$$

$$\bar{J}(s):=\sum _{i=1}^M{\bar{a}}_i{s}_i^2-\sum _{i=1}^M{\bar{b}}_i{s}_i^p+\sum _{i\ne j}{\bar{d}}_{ij}{s}_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}},$$

$$\sum _{i=1}^M(|{\tilde{a}}_i-a_i|+|{\tilde{b}}_i-b_i|)+\sum _{i\ne j}|{\tilde{d}}_{ij}-d_{ij}|<\delta ,$$

$$\tilde{J}(s):=\sum _{i=1}^M{\tilde{a}}_i{s}_i^2-\sum _{i=1}^M{\tilde{b}}_i{s}_i^p+\sum _{i\ne j}{\tilde{d}}_{ij}{s}_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}},$$

$$⟨v,w{⟩}_i:={\int }_{\Omega }(\nabla v\cdot \nabla w+{\kappa }_{i}vw)\text{and}\Vert v{\Vert }_i:={\left({\int }_{\Omega }(|\nabla v{|}^2+{\kappa }_i{v}^2)\right)}^{1/2}.$$

$$\Vert (u_1,\dots ,u_M)\Vert :={\left(\sum _{i=1}^M\Vert u_i{\Vert }_i^2\right)}^{1/2},$$

$$\mathcal{J}(u_1,\dots ,u_M):=\frac{1}{2}\sum _{i=1}^M\Vert u_i{\Vert }_i^2-\frac{1}{p}\sum _{i=1}^M{\int }_{\Omega }{\mu }_i|u_i{|}^p-\frac{1}{2}\sum _{j\ne i}{\int }_{\Omega }{\lambda }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}}.$$

$${\partial }_i\mathcal{J}(u_1,\dots ,u_M)v=⟨u_i,v{⟩}_i-{\int }_{\Omega }{\mu }_i|u_i{|}^{p-2}{u}_{i}v$$

$$-\frac{1}{2}\sum _{j\ne i}{\int }_{\Omega }{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_{i}v-\frac{1}{2}\sum _{j\ne i}{\int }_{\Omega }{\lambda }_{ji}{\alpha }_{ji}|u_i{|}^{{\alpha }_{ji}-2}{u}_{i}v|u_j{|}^{{\beta }_{ji}}$$

$$=⟨u_i,v{⟩}_i-{\int }_{\Omega }{\mu }_i|u_i{|}^{p-2}{u}_{i}v-\sum _{j\ne i}{\int }_{\Omega }{\lambda }_{ij}{\beta }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}-2}{u}_{i}v,$$

$$\mathcal{N}:=\{(u_1,\dots ,u_M)\in \mathcal{H}:u_i\ne 0,{\partial }_i\mathcal{J}(u_1,\dots ,u_M)u_i=0,\forall i=1,\dots ,M\}.$$

$$\mathcal{J}(u)=\frac{p-2}{2p}\sum _{i=1}^M\Vert u_i{\Vert }_i^2\text{if }u=(u_1,\dots ,u_M)\in \mathcal{N}.$$

$$su:=(s_1{u}_1,\dots ,s_M{u}_M),$$

$$J_u(s):=\mathcal{J}(su)=\sum _{i=1}^M{a}_{u,i}{s}_i^2-\sum _{i=1}^M{b}_{u,i}{s}_i^p+\sum _{i\ne j}{d}_{u,ij}{s}_j^{{\alpha }_{ij}}{s}_i^{{\beta }_{ij}},$$

$$a_{u,i}:=\frac{1}{2}\Vert u_i{\Vert }_i^2,b_{u,i}:=\frac{1}{p}{\int }_{\Omega }{\mu }_i|u_i{|}^p,d_{u,ij}:=-\frac{1}{2}{\int }_{\Omega }{\lambda }_{ij}|u_j{|}^{{\alpha }_{ij}}|u_i{|}^{{\beta }_{ij}}.$$

$$s_i{\partial }_i{J}_u(s)={\partial }_i\mathcal{J}(su)[s_i{u}_i],i=1,\dots ,M,$$

$$\tilde{\mathcal{U}}:=$$

$$=$$

Full text

Keywords: Weakly coupled elliptic system, simple variational setting, subcritical system in exterior domain, entire solutions to critical system, Brezis-Nirenberg problem.

Mathematics Subject Classification: 35J50 (35J47, 35B08, 35B33, 58E30).

1 Introduction

We study the weakly coupled elliptic system

[TABLE]

where $\Omega$ is a domain in $\mathbb{R}^{N}$, $N\geq 3$, $\mu_{i}>0$, $\lambda_{ij}=\lambda_{ji}<0$, $\alpha_{ij},\beta_{ij}>1$, $\alpha_{ij}=\beta_{ji}$, and $\alpha_{ij}+\beta_{ij}=p\in(2,2^{*}]$. As usual, $2^{*}:=\frac{2N}{N-2}$ is the critical Sobolev exponent. The space $H$ is, either $H^{1}_{0}(\Omega)$, or $D^{1,2}_{0}(\Omega)$, and the operators $-\Delta+\kappa_{i}$ are assumed to be well defined and coercive in $H$.

The cubic system (1.1) in $\mathbb{R}^{3}$ with $\alpha_{ij}=\beta_{ij}=2$ arises as a model in many physical phenomena, for example, in the study of standing waves for a mixture of Bose-Einstein condensates of $M$-hyperfine states which overlap in space. The sign of $\mu_{i}$ reflects the interaction of the particles within each single state, whereas that of $\lambda_{ij}$ reflects the interaction between particles in two different states. The interaction is attractive if the sign is positive, and it is repulsive if the sign is negative. The system is called competitive if, as we are assuming here, all of the $\lambda_{ij}$’s are negative.

A solution $u_{i}$ to the equation

[TABLE]

gives rise to a solution of the system (1.1) whose $i$-th component is $u_{i}$ and all other components are trivial, i.e., $u_{j}=0$ if $j\neq i$. A solution with at least one trivial and one nontrivial component is called semitrivial. We are interested in finding solutions all of whose components are nontrivial. These are called fully nontrivial solutions. A fully nontrivial solution is said to be positive if every component $u_{i}$ is nonnegative.

The main purpose of this paper is to exhibit a simple variational setting for finding fully nontrivial solutions to the system (1.1). Our approach is inspired by the ideas introduced by Szulkin and Weth in [19, 20].

We will show that the fully nontrivial solutions to (1.1) correspond to the critical points of a $\mathcal{C}^{1}$-functional $\Psi:\mathcal{U}\to\mathbb{R}$ defined in an open subset $\mathcal{U}$ of the product $\mathcal{T}:=S_{1}\times\cdots\times S_{M}$ of unit spheres $S_{i}$ in $H$. The functional $\Psi$ tends to infinity at the boundary of $\mathcal{U}$ in $\mathcal{T}$, thus allowing the application of the usual descending gradient flow techniques to obtain existence and multiplicity of critical points.

This variational setting can be easily extended to systems whose coefficients $\kappa_{i},\mu_{i},\lambda_{ij}$ are functions defined in $\Omega$ and satisfying suitable assumptions. It may also be extended, with some care, to systems having more general nonlinearities. We chose to treat only the constant coefficient system (1.1) in order to make the ideas more transparent.

Our abstract results (Theorems 3.3 and 3.4) apply to many interesting types of systems. Here we consider the following three.

Firstly, we consider the subcritical system

[TABLE]

with $\kappa_{i},\mu_{i}>0$, $\lambda_{ij}=\lambda_{ji}<0$, $\alpha_{ij},\beta_{ij}>1$, $\alpha_{ij}=\beta_{ji}$, and $\alpha_{ij}+\beta_{ij}=p\in(2,2^{*})$, in an exterior domain $\Omega$ of $\mathbb{R}^{N}$ (i.e., $\mathbb{R}^{N}\smallsetminus\Omega$ is bounded, possibly empty), $N\geq 3$.

We assume that $\Omega$ is invariant under the action of a closed subgroup $G$ of the group $O(N)$ of linear isometries of $\mathbb{R}^{N}$, and look for $G$-invariant solutions, i.e., solutions whose components are $G$-invariant.

Let $Gx:=\{gx:g\in G\}$ denote the $G$-orbit of $x\in\mathbb{R}^{N}$. We prove the following result.

Theorem 1.1**.**

If $\dim(Gx)>0$ for every $x\in\mathbb{R}^{N}\smallsetminus\{0\}$ and $\Omega$ is a $G$-invariant exterior domain in $\mathbb{R}^{N}$, then the system (1.2) has an unbounded sequence of $G$-invariant fully nontrivial solutions. One of them is positive and has least energy among all $G$-invariant fully nontrivial solutions.

There is an extensive literature on subcritical systems in bounded domains and in the whole of $\mathbb{R}^{3}$. We refer to [17] for a detailed account. Theorem 1.1 seems to be the first existence result for the system (1.2) in an exterior domain. A cubic system of two equations with variable coefficients in an expanding exterior domain was recently considered in [10].

Our second application concerns the critical system

[TABLE]

where $N\geq 3$, $\mu_{i}>0$, $\lambda_{ij}=\lambda_{ji}<0$, $\alpha_{ij},\beta_{ij}>1$, $\alpha_{ij}=\beta_{ji}$, and $\alpha_{ij}+\beta_{ij}=2^{*}$.

We look for solutions which are invariant under the conformal action of the group $\Gamma:=O(m)\times O(n)$ on $\mathbb{R}^{N}$, with $m+n=N+1$ and $n,m\geq 2$, which is induced by the isometric action of $\Gamma$ on the standard $N$-dimensional sphere, by means of the stereographic projection. We prove the following result.

Theorem 1.2**.**

The system (1.3) has an unbounded sequence of $\Gamma$-invariant fully nontrivial solutions. One of them is positive and has least energy among all $\Gamma$-invariant fully nontrivial solutions.

Theorem 1.2 extends some earlier results obtained in [5, 6] for a system of two equations; see also [9]. Existence and multiplicity results for the purely critical system in a bounded domain may be found in [5, 13, 14]. Supercritical systems were recently considered in [4].

Finally, we consider the critical system

[TABLE]

where $\Omega$ is a bounded domain with $\mathcal{C}^{2}$-boundary in $\mathbb{R}^{N}$, $N\geq 4$, $\kappa_{i}\in(-\lambda_{1}(\Omega),0)$, $\mu_{i}>0$, $\lambda_{ij}=\lambda_{ji}<0$, $\alpha_{ij},\beta_{ij}>1$, $\alpha_{ij}=\beta_{ji}$, and $\alpha_{ij}+\beta_{ij}=2^{*}$. As usual, $\lambda_{1}(\Omega)$ denotes the first Dirichlet eigenvalue of $-\Delta$ in $\Omega$.

We prove the following result.

Theorem 1.3**.**

Let $N\geq 4$. Assume that $\min\{\alpha_{ij},\beta_{ij}\}\geq\frac{4}{3}$ if $N=5$ and that $\alpha_{ij}=\beta_{ij}=2$ if $N=4$, for all $i,j=1,\ldots,M$. Then, the system (1.4) has a positive least energy fully nontrivial solution.

Note that there is no condition on $\alpha_{ij},\beta_{ij}$, other than $\alpha_{ij},\beta_{ij}>1$ and $\alpha_{ij}+\beta_{ij}=2^{*}$, if $N\geq 6$.

Theorem 1.3 extends some earlier results obtained in [2, 3] for a system of two equations. Multiple positive solutions were constructed in [15] when $N=4$, and the existence of infinitely many sign-changing solutions was established in [11] when $N\geq 7$ and $\alpha_{ij}=\beta_{ij}=\frac{2^{*}}{2}$; see also [12].

Our variational approach is based on some elementary properties of a certain function in $M$ variables, which are established in Section 2. In Section 3 we introduce our variational setting and we derive some abstract results concerning the existence and multiplicity of fully nontrivial solutions to the system (1.1). Section 4 is devoted to the proof of Theorems 1.1, 1.2 and 1.3.

2 On a function in M variables

Let $J:(0,\infty)^{M}\to\mathbb{R}$ be the function given by

[TABLE]

where $s=(s_{1},\ldots,s_{M})$, $a_{i},b_{i}>0$, $d_{ij}\geq 0$, $d_{ij}=d_{ji}$, $\alpha_{ij},\beta_{ij}>1$, $\alpha_{ij}+\beta_{ij}=p>2$, and $\alpha_{ji}=\beta_{ij}$. Then, for $i=1,\ldots,M$,

[TABLE]

Lemma 2.1**.**

If $pb_{i}>2\sum_{j\neq i}d_{ij}\beta_{ij}$ for all $i=1,\ldots,M$, then there exist $0<r<R<\infty$ such that

[TABLE]

In particular, $J$ attains its maximum on $(0,\infty)^{M}$.

Proof.

Fix $R>r>0$ such that, for all $i=1,\ldots,M$,

[TABLE]

and

[TABLE]

Let $s=(s_{1},\ldots,s_{M})\in(0,\infty)^{M}$. If $s_{i}\geq R\text{ and }s_{i}=\max\{s_{1},\ldots,s_{M}\}$, we have that

[TABLE]

whereas, if $s_{i}\leq r$, then

[TABLE]

Therefore (2.2) holds true. ∎

Lemma 2.2**.**

If $J$ has a critical point in $(0,\infty)^{M}$, then it is unique and it is a global maximum of $J$ in $(0,\infty)^{M}$.

Proof.

Assume first that $(1,\ldots,1)$ is a critical point of $J$. Then, from (2.1) we get that

[TABLE]

If $s=(s_{1},\ldots,s_{M})$ is a critical point of $J$ in $(0,\infty)^{M}$, then, for each $i=1,\ldots,M$, (2.1) and (2.5) yield

[TABLE]

Arguing by contradiction, assume that $s\neq(1,\ldots,1)$. We consider two cases. Suppose first that $s_{i}>1$ for some $i$. We may assume without loss of generality that $s_{i}\geq s_{j}$ for all $j$. Then, the left-hand side in (2.6) is negative whereas the right-hand side is $\geq 0$. This is a contradiction. Now suppose that $s_{i}<1$ for some $i$. Again, we may assume that $s_{i}\leq s_{j}$ for all $j$. Now the left-hand side in (2.6) is positive while the right-hand side is not, a contradiction again. Hence $(1,\ldots,1)$ is the only critical point of $J$ in $(0,\infty)^{M}$. The inequalities (2.5) allow us to apply Lemma 2.1 to conclude that $(1,\ldots,1)$ is a global maximum.

Now, if $s^{0}=(s_{1}^{0},\ldots,s_{M}^{0})$ is a critical point of $J$ in $(0,\infty)^{M}$, then $(1,\ldots,1)$ is a critical point of

[TABLE]

where $\bar{a}_{i}:=a_{i}s_{i}^{0}$, $\bar{b}_{i}:=b_{i}(s_{i}^{0})^{p-1}$ and $\bar{d}_{ij}:=d_{ij}(s_{j}^{0})^{\alpha_{ij}}(s_{i}^{0})^{\beta_{ij}-1}$, and the conclusion follows from the special case considered above. ∎

Lemma 2.3**.**

Assume that $J$ has a critical point $s^{0}$ in $(0,\infty)^{M}$. Then, for each $\varepsilon>0$, there exists $\delta>0$ such that, if $\widetilde{d}_{ij}\geq 0$ for all $i\neq j$ and

[TABLE]

then the function

[TABLE]

has a unique critical point $\widetilde{s}^{0}$ in $(0,\infty)^{M}$ which is a global maximum and satisfies $|\widetilde{s}^{0}-s^{0}|<\varepsilon$.

Proof.

As in the proof of Lemma 2.2, we may assume without loss of generality that $s^{0}=(1,\ldots,1)$. Then, (2.5) holds true. So, choosing $\delta>0$ small enough, we have that $\widetilde{a}_{i},\,\widetilde{b}_{i}>0$ and $p\widetilde{b}_{i}-2\sum_{j\neq i}\widetilde{d}_{ij}\beta_{ij}>0$ if (2.7) is satisfied. Thus, by Lemma 2.1, $\widetilde{J}$ has a global maximum $\widetilde{s}^{0}$ in $(0,\infty)^{M}$ and, by Lemma 2.2, it is the only critical point of $\widetilde{J}$ in $(0,\infty)^{M}$.

Taking smaller $\delta,r>0$ and a larger $R>r$ if necessary, we have that $\widetilde{J}$ satisfies the same inequalities and, therefore, $\widetilde{s}^{0}\in(r,R)^{M}$. Since $(1,\ldots,1)$ is a strict maximum, it is easy to see that $|\widetilde{s}^{0}-(1,\ldots,1)|<\varepsilon$, possibly after choosing a still smaller $\delta$. ∎

3 The variational setting

The results of this section also apply to the case $N=1$ or 2 and $p\in(2,\infty)$.

Let $H$ be either $H^{1}_{0}(\Omega)$ or $D^{1,2}_{0}(\Omega)$ and, for $v,w\in H$, set

[TABLE]

Since, by assumption, the operators $-\Delta+\kappa_{i}$ are well defined and coercive in $H$, we have that $\|\cdot\|_{i}$ is a norm in $H$, equivalent to the standard one.

Let $\mathcal{H}:=H^{M}$ with the norm

[TABLE]

and let $\mathcal{J}:\mathcal{H}\to\mathbb{R}$ be given by

[TABLE]

This function is of class $\mathcal{C}^{1}$ and, since $\lambda_{ij}=\lambda_{ji}$ and $\beta_{ij}=\alpha_{ji}$,

[TABLE]

for each $v\in H$, $i=1,\ldots,M$. So the critical points of $\mathcal{J}$ are the solutions to the system (1.1). The fully nontrivial ones belong to the set

[TABLE]

This Nehari-type set was introduced in [7], and has been used in many works. Note that

[TABLE]

Given $u=(u_{1},\ldots,u_{M})\in\mathcal{H}$ and $s=(s_{1},\ldots,s_{M})\in(0,\infty)^{M}$, we write

[TABLE]

and we define $J_{u}:(0,\infty)^{M}\to\mathbb{R}$ by

[TABLE]

where

[TABLE]

If $u_{i}\neq 0$ for all $i=1,\ldots,M$, then, as

[TABLE]

we have that $s$ is a critical point of $J_{u}$ iff $su\in\mathcal{N}$. Define

[TABLE]

By Lemma 2.2, if $u\in(H\smallsetminus\{0\})^{M}$ and $J_{u}$ has a critical point in $(0,\infty)^{M}$, then this critical point is unique and it is a global maximum of $J_{u}$. We denote it by $s_{u}=(s_{u,1},\ldots,s_{u,M})$, and we define $\widetilde{\mathfrak{m}}:\widetilde{\mathcal{U}}\to\mathcal{N}$ by

[TABLE]

Then,

[TABLE]

Let $S_{i}:=\{v\in H:\|v\|_{i}=1\}$, $\mathcal{T}:=S_{1}\times\cdots\times S_{M}$, $\mathcal{U}:=\widetilde{\mathcal{U}}\cap\mathcal{T}$, and let $\mathfrak{m}:\mathcal{U}\to\mathcal{N}$ be the restriction of $\widetilde{\mathfrak{m}}$ to $\mathcal{U}$. We write $\partial\mathcal{U}$ for the boundary of $\mathcal{U}$ in $\mathcal{T}$.

Proposition 3.1**.**

• $(a)$

If $u=(u_{1},\ldots,u_{M})\in\mathcal{T}$ is such that $u_{i}$ and $u_{j}$ have disjoint supports for every $i\neq j$, then $u\in\mathcal{U}$. Hence $\mathcal{U}\neq\emptyset$. Moreover, $\mathcal{U}$ is an open subset of $\mathcal{T}$.

• $(b)$

$\mathcal{U}\neq\mathcal{T}$* if $-\lambda_{ij}\geq\max\{\frac{\mu_{i}}{\beta_{ij}},\,\frac{\mu_{j}}{\beta_{ji}}\}$ for some $i\neq j$.*

• $(c)$

$\widetilde{\mathfrak{m}}:\widetilde{\mathcal{U}}\to\mathcal{N}$* is continuous, and $\mathfrak{m}:\mathcal{U}\to\mathcal{N}$ is a homeomorphism.*

• $(d)$

There exists $d_{0}>0$ such that $\min_{i=1,\ldots,M}\|u_{i}\|_{i}\geq d_{0}$ if $(u_{1},\ldots,u_{M})\in\mathcal{N}$. Thus, $\mathcal{N}$ is a closed subset of $\mathcal{H}$.

• $(e)$

If $(u_{n})$ is a sequence in $\mathcal{U}$ such that $u_{n}\to u\in\partial\mathcal{U}$, then $\|\mathfrak{m}(u_{n})\|\to\infty$.

Proof.

$(a):$ Let $u=(u_{1},\ldots,u_{M})\in\mathcal{T}$ be such that $u_{i}$ and $u_{j}$ have disjoint supports if $i\neq j$. Then, $d_{u,ij}=0$ for every $i\neq j$, and, setting $s_{i}:=(\mu_{i}\int_{\Omega}|u_{i}|^{p})^{-1/(p-2)}$, we have that $(s_{1}u_{1},\ldots,s_{M}u_{M})\in\mathcal{N}$. This proves that $u\in\mathcal{U}$. Moreover, as $a_{u,i},b_{u,i},d_{u,ij}$ are continuous functions of $u$, Lemma 2.3 implies that $\mathcal{U}$ is open.

$(b):$ We assume without loss of generality that $i=1$ and $j=2$. Let $v,v_{3},\ldots,v_{M}\in H$ be nontrivial functions. Assume there exist $t_{1},t_{2}>0$ such that $(t_{1}v,t_{2}v,v_{3},\ldots,v_{M})\in\mathcal{N}$. Then, as $\alpha_{ij}+\beta_{ij}=p$ and $\lambda_{ij}<0$ for all $i,j$, we have that

[TABLE]

Since $\lambda_{12}=\lambda_{21}$ and the right-hand sides above must be positive, we get that

[TABLE]

which is impossible if $-\lambda_{12}\geq\max\{\frac{\mu_{1}}{\beta_{12}},\,\frac{\mu_{2}}{\beta_{21}}\}$. So, if this last inequality holds true, then

[TABLE]

$(c):$ If $(u_{n})$ is a sequence in $\widetilde{\mathcal{U}}$ and $u_{n}\to u\in\widetilde{\mathcal{U}}$, then, for each $i,j=1,\ldots,M$ with $i\neq j$, we have that $a_{u_{n},i}\to a_{u,i}$,  $b_{u_{n},i}\to b_{u}$ and $d_{u_{n},ij}\to d_{u,ij}$. So, from Lemma 2.3 we get that $s_{u_{n},i}\to s_{u,i}$. Hence, $\widetilde{\mathfrak{m}}:\widetilde{\mathcal{U}}\to\mathcal{N}$ is continuous.

The inverse of $\mathfrak{m}:\mathcal{U}\to\mathcal{N}$ is given by

[TABLE]

which is, obviously, continuous.

$(d):$ If $(u_{1},\ldots,u_{M})\in\mathcal{N}$ then, as $\lambda_{ij}<0$ for every $i\neq j$, we have that $\|u_{i}\|_{i}^{2}\leq\mu_{i}\int_{\Omega}|u_{i}|^{p}$ for all $i=1,\ldots,M$. The statement now follows from Sobolev’s inequality.

$(e):$ Let $(u_{n})$ be a sequence in $\mathcal{U}$ such that $u_{n}\to u\in\partial\mathcal{U}$. If the sequence $(s_{u_{n},i})$ were bounded for every $i=1,\ldots,M$, then, after passing to a subsequence, $s_{u_{n},i}\to s_{i}$. Since $\mathcal{N}$ is closed, we would have that $(s_{1}u_{1},\ldots,s_{M}u_{M})\in\mathcal{N}$ and, therefore, $u\in\mathcal{U}$. This is impossible because $u\in\partial\mathcal{U}$ and $\mathcal{U}$ is open in $\mathcal{T}$. ∎

A fully nontrivial solution $u$ to (1.1) will be called synchronized if $u_{i}=t_{i}v$ and $u_{j}=t_{j}v$ for some $i\neq j$ and $t_{i},t_{j}\in\mathbb{R}$.

Proposition 3.2**.**

There exists $\Lambda_{0}<0$ such that if $\lambda_{ij}<\Lambda_{0}$ for all $i,j$, then the system (1.1) has no fully nontrivial synchronized solutions.

Proof.

Choose $\Lambda_{0}$ such that $-\Lambda_{0}\geq\max\{\frac{\mu_{i}}{\beta_{ij}},\,\frac{\mu_{j}}{\beta_{ji}}\}$ for all $i\neq j$. Then (3.3) holds true and so $u$ cannot be a solution to (1.1). ∎

$\mathcal{T}$ is a smooth Hilbert submanifold of $\mathcal{H}$. The tangent space to $\mathcal{T}$ at a point $u=(u_{1},\ldots,u_{M})\in\mathcal{T}$ is the space

[TABLE]

Let $\widetilde{\Psi}:\widetilde{\mathcal{U}}\to\mathbb{R}$ be given by $\widetilde{\Psi}(u):=\mathcal{J}(\widetilde{\mathfrak{m}}(u))$, and let $\Psi$ be the restriction of $\widetilde{\Psi}$ to $\mathcal{U}$. Then,

[TABLE]

If $u\in\mathcal{U}$ and the derivative $\Psi^{\prime}(u)$ of $\Psi$ at $u$ exists, then

[TABLE]

i.e., $\|\cdot\|_{*}$ is the norm in the cotangent space $\mathrm{T}_{u}^{*}(\mathcal{T})$ to $\mathcal{T}$ at $u$. A sequence $(u_{n})$ in $\mathcal{U}$ is called a $(PS)_{c}$-sequence for $\Psi$ if $\Psi(u_{n})\to c$ and $\|\Psi^{\prime}(u_{n})\|_{*}\to 0$, and $\Psi$ is said to satisfy the $(PS)_{c}$-condition if every such sequence has a convergent subsequence.

As usual, a $(PS)_{c}$-sequence for $\mathcal{J}$ is a sequence $(u_{n})$ in $\mathcal{H}$ such that $\mathcal{J}(u_{n})\to c$ and $\|\mathcal{J}^{\prime}(u_{n})\|_{\mathcal{H}^{-1}}\to 0$, and $\mathcal{J}$ satisfies the $(PS)_{c}$-condition if any such sequence has a convergent subsequence.

Theorem 3.3**.**

• $(i)$

$\Psi\in\mathcal{C}^{1}(\mathcal{U},\mathbb{R})$* and*

[TABLE]

• $(ii)$

If $(u_{n})$ is a $(PS)_{c}$-sequence for $\Psi$, then $(\mathfrak{m}(u_{n}))$ is a $(PS)_{c}$-sequence for $\mathcal{J}$. Conversely, if $(u_{n})$ is a $(PS)_{c}$-sequence for $\mathcal{J}$ and $u_{n}\in\mathcal{N}$ for all $n\in\mathbb{N}$, then $(\mathfrak{m}^{-1}(u_{n}))$ is a $(PS)_{c}$-sequence for $\Psi$.

• $(iii)$

$u$* is a critical point of $\Psi$ if and only if $\mathfrak{m}(u)$ is a fully nontrivial critical point of $\mathcal{J}$.*

• $(iv)$

If $(u_{n})$ is a sequence in $\mathcal{U}$ such that $u_{n}\to u\in\partial\mathcal{U}$, then $\Psi(u_{n})\to\infty$.

• $(v)$

$\Psi$* is even, i.e., $\Psi(u)=\Psi(-u)$ for every $u\in\mathcal{U}$.*

Proof.

We adapt the arguments of Proposition 9 and Corollary 10 in [20].

$(i):$ Let $u\in\widetilde{\mathcal{U}}$ and $v\in\mathcal{H}$. As $s_{u}$ is the maximum of $J_{u}$, using the mean value theorem we obtain

[TABLE]

for $|t|$ small enough and some $\tau_{1}\in(0,1)$. Similarly,

[TABLE]

for some $\tau_{2}\in(0,1)$. From the continuity of $s_{u}$ and these two inequalities we obtain

[TABLE]

The right-hand side is linear in $v$ and continuous in $v$ and $u$. Therefore $\widetilde{\Psi}$ is of class $\mathcal{C}^{1}$. If $u\in\mathcal{U}$ and $v\in\mathrm{T}_{u}(\mathcal{T})$, then $\mathfrak{\widetilde{m}}(u)=\mathfrak{m}(u)$, and the statement is proved.

$(ii):$ Note that $\mathcal{H}=\mathrm{T}_{u}(\mathcal{T})\oplus(\mathbb{R}u_{1},\ldots,\mathbb{R}u_{M})$ for each $u\in\mathcal{U}$. Since $\mathfrak{m}(u)\in\mathcal{N}$, we have that $\mathcal{J}^{\prime}(\mathfrak{m}(u))w=0$ if $w\in(\mathbb{R}u_{1},\ldots,\mathbb{R}u_{M})$. So, from $(i)$ we get

[TABLE]

If $(\Psi(u_{n}))$ converges, then $(s_{u_{n}})$ is bounded in $\mathbb{R}^{M}$ by (3.4). Moreover, by Proposition 3.1$(d)$, this sequence is bounded away from [math]. Therefore, $(\mathfrak{m}(u_{n}))$ is a $(PS)_{c}$-sequence for $\mathcal{J}$ iff $(u_{n})$ is a $(PS)_{c}$-sequence for $\Psi$, as claimed.

$(iii):$ As $\mathcal{J}^{\prime}(\mathfrak{m}(u))w=0$ if $w\in(\mathbb{R}u_{1},\ldots,\mathbb{R}u_{M})$, it follows from $(i)$ that $\Psi^{\prime}(u)=0$ if and only if $\mathcal{J}^{\prime}(\mathfrak{m}(u))=0$.

$(iv):$ This statement follows from Proposition 3.1$(e)$ and (3.1).

$(v):$ Since $-u\in\mathcal{N}$ iff $u\in\mathcal{N}$, we have that $s_{u}=s_{-u}$. So, as $\mathcal{J}$ is even, $\Psi(-u)=\mathcal{J}(s_{-u}(-u))=\mathcal{J}(s_{u}u)=\Psi(u)$. ∎

Let $Z$ be a subset of $\mathcal{T}$ such that $-u\in Z$ iff $u\in Z$. If $Z\neq\emptyset$, the genus of $Z$ is the smallest integer $k\geq 1$ such that there exists an odd continuous function $Z\to\mathbb{S}^{k-1}$ into the unit sphere $\mathbb{S}^{k-1}$ in $\mathbb{R}^{k}$. We denote it by $\mathrm{genus}(Z)$. If no such $k$ exists, we define $\mathrm{genus}(Z):=\infty$. We set $\mathrm{genus}(\emptyset):=0$.

As usual, we write

[TABLE]

The previous theorem yields the following one.

Theorem 3.4**.**

• $(a)$

If $\inf_{\mathcal{N}}\mathcal{J}$ is attained by $\mathcal{J}$ at some $u=(u_{1},\ldots,u_{M})\in\mathcal{N}$, then $u$ and $|u|:=(|u_{1}|,\ldots,|u_{M}|)$ are fully nontrivial solutions of (1.1).

• $(b)$

If $\Psi:\mathcal{U}\to\mathbb{R}$ satisfies the $(PS)_{c}$-condition for every $c\leq a$, then the system (1.1) has, either an infinite (in fact, uncountable) set of fully nontrivial solutions with the same norm, or it has at least $\mathrm{genus}(\Psi^{\leq a})$ fully nontrivial solutions with pairwise different norms.

• $(c)$

If $\Psi:\mathcal{U}\to\mathbb{R}$ satisfies the $(PS)_{c}$-condition for every $c\in\mathbb{R}$ and $\mathrm{genus}(\mathcal{U})=\infty$, then the system (1.1) has an unbounded sequence of fully nontrivial solutions.

Proof.

Theorem 3.3$(iii)$ states that $u$ is a critical point of $\Psi$ iff $\mathfrak{m}(u)$ is a fully nontrivial critical point of $\mathcal{J}$. Note that $\Psi(u)=\frac{p-2}{2p}\|\mathfrak{m}(u)\|^{2}$, by (3.1).

If $\inf_{\mathcal{N}}\mathcal{J}=\mathcal{J}(u)$ and $u\in\mathcal{N}$, then $\mathfrak{m}^{-1}(u)\in\mathcal{U}$ and $\Psi(\mathfrak{m}^{-1}(u))=\inf_{\mathcal{U}}\Psi$. So $u$ is a fully nontrivial critical point of $\mathcal{J}$. As $|u|\in\mathcal{N}$ and $\mathcal{J}(|u|)=\mathcal{J}(u)$ the same is true for $|u|$. This proves $(a)$.

Theorem 3.3$(iv)$ implies that $\mathcal{U}$ is positively invariant under the negative pseudogradient flow of $\Psi$, so the usual deformation lemma holds true for $\Psi$; see, e.g., [18, Section II.3] or [21, Section 5.3]. Set

[TABLE]

Standard arguments show that, under the assumptions of $(b)$, $c_{j}$ is a critical value of $\Psi$ for every $j=1,\ldots,\mathrm{genus}(\Psi^{\leq a})$. Moreover, if some of these values coincide, say $c:=c_{j}=\cdots=c_{j+k}$, then $\mathrm{genus}(K_{c})\geq k+1\geq 2$. Hence, $K_{c}$ is an infinite set; see, e.g., [18, Lemma II.5.6]. On the other hand, under the assumptions of $(c)$, $c_{j}$ is a critical value for every $j\in\mathbb{N}$, and a well known argument (see, e.g., [16, Proposition 9.33]) shows that $c_{j}\to\infty$ as $j\to\infty$. This completes the proof. ∎

4 Some applications

4.1 Subcritical systems in exterior domains

Consider the subcritical system (1.2) in an exterior domain $\Omega$. First, we show that this system cannot be solved by minimization. Set

[TABLE]

where   $\|w\|_{i}^{2}:=\int_{\mathbb{R}^{N}}(|\nabla w|^{2}+\kappa_{i}w^{2})$   and   $|w|_{p,i}^{p}:=\int_{\mathbb{R}^{N}}\mu_{i}|w|^{p}.$

Proposition 4.1**.**

We have that

[TABLE]

and this infimum is not attained by $\mathcal{J}$ on $\mathcal{N}$.

Proof.

We consider $H^{1}_{0}(\Omega)$ to be a subspace of $H^{1}(\mathbb{R}^{N})$, via trivial extension.

If $(u_{1},\ldots,u_{M})\in\mathcal{N}$ then, as $\lambda_{ij}<0$ for every $i\neq j$, we have that $\|u_{i}\|_{i}^{2}\leq|u_{i}|_{p,i}^{p}$ for all $i=1,\ldots,M$. Hence,

[TABLE]

It follows from (3.1) that $\mathcal{J}(u)\geq\frac{p-2}{2p}\sum_{i=1}^{M}S_{p,i}^{\frac{p}{p-2}}$.

To prove the opposite inequality, set $B_{r}(x):=\{y\in\mathbb{R}^{N}:|y-x|<r\}$, and let $w_{i,R}$ be a least energy solution to the problem

[TABLE]

It is easy to verify that $\lim_{R\to\infty}\|w_{i,R}\|_{i}^{2}=S_{p,i}^{\frac{p}{p-2}}$. Fix $\xi_{i,R}\in\Omega$, $i=1\ldots,m$, such that $B_{R}(\xi_{i,R})\subset\Omega$ and $B_{R}(\xi_{i,R})\cap B_{R}(\xi_{j,R})=\emptyset$ if $i\neq j$, and set $u_{R}:=(u_{1,R},\ldots,u_{M,R})$ with $u_{i,R}(x):=w_{i,R}(x-\xi_{i,R})$. Then, $u_{R}\in\mathcal{N}$ and

[TABLE]

This completes the proof of (4.1).

To show that the infimum is not attained, we argue by contradiction. Assume that $(u_{1},\ldots,u_{M})\in\mathcal{N}$ and $\mathcal{J}(u)=\frac{p-2}{2p}\sum_{i=1}^{M}S_{p,i}^{\frac{p}{p-2}}$. We may assume that $u_{i}\geq 0$ for all $i=1,\ldots,M$. We fix $i$ and consider two cases. If $\int_{\Omega}u_{j}^{\alpha_{ij}}u_{i}^{\beta_{ij}}\neq 0$ for some $j\neq i$, then $\|u_{i}\|_{i}^{2}<|u_{i}|_{p,i}^{p}$ and, hence, $S_{p,i}^{p/(p-2)}<\|u_{i}\|_{i}^{2}$. This implies that $\mathcal{J}(u)>\frac{p-2}{2p}\sum_{i=1}^{M}S_{p,i}^{p/(p-2)}$, contradicting our assumption. On the other hand, if $\int_{\Omega}u_{j}^{\alpha_{ij}}u_{i}^{\beta_{ij}}=0$ for all $j\neq i$, then $\|u_{i}\|_{i}^{2}=|u_{i}|_{p,i}^{p}=S_{p,i}^{p/(p-2)}$. Hence, $u_{i}$ is a nontrivial solution to the problem

[TABLE]

Moreover, $\int_{\Omega}u_{j}^{\alpha_{ij}}u_{i}^{\beta_{ij}}=0$ also implies that $u_{j}^{\alpha_{ij}}u_{i}^{\beta_{ij}}=0$ a.e. in $\Omega$. As $u_{j}\not\equiv 0$ for all $j$, we have that $u_{i}=0$ in some subset of positive measure of $\mathbb{R}^{N}$. This contradicts the maximum principle. ∎

To obtain multiple solutions to the system $\eqref{eq:exterior}$ we introduce some symmetries.

Let $G$ be a closed subgroup of $O(N)$ and $Gx:=\{gx:g\in G\}$. Set $\mathbb{S}^{N-1}:=\{x\in\mathbb{R}^{N}:|x|=1\}$. We start with the following lemma.

Lemma 4.2**.**

If $\dim(Gx)>0$ for every $x\in\mathbb{R}^{N}\smallsetminus\{0\}$, then, for each $k\in\mathbb{N}$, there exists $d_{k}>0$ such that, for every $x\in\mathbb{S}^{N-1}$, there exist $g_{1},\ldots,g_{k}\in G$ with

[TABLE]

Proof.

Arguing by contradiction, assume that for some $k\in\mathbb{N}$ and every $n\in\mathbb{N}$ there exists $x_{n}\in\mathbb{S}^{N-1}$ such that

[TABLE]

After passing to a subsequence, we have that $x_{n}\to x$ in $\mathbb{S}^{N-1}$. Since $\dim(Gx)>0$, there exist $\bar{g}_{1},\ldots,\bar{g}_{k}\in G$ such that $\bar{g}_{i}x\neq\bar{g}_{j}x$ if $i\neq j$. Fix $i\neq j$ such that, after passing to a subsequence, $|\bar{g}_{i}x_{n}-\bar{g}_{j}x_{n}|=\min_{i\neq j}|\bar{g}_{i}x_{n}-\bar{g}_{j}x_{n}|$ for every $n\in\mathbb{N}$. Then,

[TABLE]

This is a contradiction. ∎

We assume that $\Omega$ is $G$-invariant and define

[TABLE]

Recall that $\Omega$ is called $G$-invariant if $Gx\subset\Omega$ for all $x\in\Omega$, and a function $v:\Omega\to\mathbb{R}$ is $G$-invariant if it is constant on $Gx$ for every $x\in\Omega$. An $M$-tuple $(v_{1},\ldots,v_{M})$ will be called $G$-invariant if each component $v_{i}$ is $G$-invariant.

Lemma 4.3**.**

Assume that $\dim(Gx)>0$ for every $x\in\mathbb{R}^{N}\smallsetminus\{0\}$ and let $\Omega$ be a $G$-invariant exterior domain. Then, the embedding $H^{1}_{0}(\Omega)^{G}\hookrightarrow L^{p}(\Omega)$ is compact for every $p\in(2,2^{*})$.

Proof.

Let $(w_{n})$ be a bounded sequence in $H_{0}^{1}(\Omega)^{G}$. Then, after passing to a subsequence, $w_{n}\rightharpoonup w$ weakly in $H_{0}^{1}(\Omega)^{G}$. Set $v_{n}:=w_{n}-w$. A subsequence of $(v_{n})$ satisfies $v_{n}\rightharpoonup 0$ weakly in $H_{0}^{1}(\Omega)^{G}$, $v_{n}\to 0$ in $L_{loc}^{2}(\Omega)$ and $v_{n}(x)\to 0$ a.e. in $\Omega$. We claim that

[TABLE]

To prove this claim, let $\varepsilon>0$, and let $C>0$ be such that $\|v_{n}\|^{2}\leq C$ for all $n\in\mathbb{N}$, where $\|\cdot\|$ is the standard norm in $H_{0}^{1}(\Omega)$. We choose $k\in\mathbb{N}$ such that $C<\varepsilon k$ and $d_{k}>0$ as in Lemma 4.2, and we fix $R_{k}>2/d_{k}$. We consider two cases.

Assume first that $|x|\geq R_{k}$. By Lemma 4.2, there exist $g_{1},\ldots,g_{k}\in G$ such that

[TABLE]

Since $|x|\geq R_{k}$, we have that $|g_{i}x-g_{j}x|>2$. Hence, $B_{1}(g_{i}x)\cap B_{1}(g_{j}x)=\emptyset$ if $i\neq j$ and, as $v_{n}$ is $G$-invariant, we obtain

[TABLE]

Therefore,

[TABLE]

Now assume that $|x|\leq R_{k}$. Then, since $v_{n}\to 0$ strongly in $L^{2}(B_{R_{k}+1}(0))$, there exists $n_{0}\in\mathbb{N}$ such that

[TABLE]

Inequalities (4.3) and (4.4) yield (4.2). Applying Lions’ lemma [21, Lemma 1.21] we conclude that $v_{n}\to 0$ strongly in $L^{p}(\Omega)$ for any $p\in(2,2^{*})$. ∎

Lemma 4.4**.**

Assume that $\dim(Gx)>0$ for every $x\in\mathbb{R}^{N}\smallsetminus\{0\}$ and let $\Omega$ be a $G$-invariant exterior domain. Then, the functional $\mathcal{J}$ satisfies the Palais-Smale condition in $\mathcal{H}^{G}$, i.e., every sequence $(u_{n})$ in $\mathcal{H}^{G}$ such that $\mathcal{J}(u_{n})\to c$ and $\mathcal{J}^{\prime}(u_{n})\to 0$ in $(\mathcal{H}^{G})^{\prime}$, contains a convergent subsequence.

Proof.

Since

[TABLE]

$(u_{n})$ is bounded. The rest of the proof follows from Lemma 4.3 by standard arguments. ∎

Lemma 4.5**.**

Let $\mathcal{U}^{G}:=\mathcal{U}\cap\mathcal{H}^{G}$. Then, $\mathrm{genus}(\mathcal{U}^{G})=\infty$.

Proof.

Given $k\geq 1$, for each $j=1,\ldots,k$, $i=1,\ldots,M$, we choose $u_{j,i}\in H_{0}^{1}(\Omega)^{G}$ such that $\|u_{j,i}\|_{i}=1$ and $\mathrm{supp}(u_{j,i})\cap\mathrm{supp}(u_{j^{\prime},i^{\prime}})=\emptyset$ if $(j,i)\neq(j^{\prime},i^{\prime})$.

Let $\{e_{j}:1\leq j\leq k\}$ be the canonical basis of $\mathbb{R}^{k}$, and $Q$ be the set

[TABLE]

Note that $Q$ is homeomorphic to the unit sphere $\mathbb{S}^{k-1}$ in $\mathbb{R}^{k}$ by an odd homeomorphism.

For each $i=1,\ldots,M$, define $\sigma_{i}:Q\to H_{0}^{1}(\Omega)^{G}$ by setting $\sigma_{i}(e_{j}):=u_{j,i}$, $\sigma_{i}(-e_{j}):=-u_{j,i}$, and

[TABLE]

Note that, since $u_{j,i}$ and $u_{j^{\prime},i^{\prime}}$ have disjoint supports if $(j,i)\neq(j^{\prime},i^{\prime})$, these maps are well defined and $\mathrm{supp}(\sigma_{i}(z))\cap\mathrm{supp}(\sigma_{i^{\prime}}(z))=\emptyset$ if $i\neq i^{\prime}$ for every $z\in Q$. So, by Proposition 3.1$(a)$, the map $\sigma:Q\to\mathcal{U}^{G}$ given by $\sigma(z):=(\sigma_{1}(z),\ldots,\sigma_{M}(z))$ is well defined. As each $\sigma_{i}$ is continuous and odd, so is $\sigma$. Hence, $\mathrm{genus}(\mathcal{U}^{G})\geq\mathrm{genus}(Q)=k$. ∎

Proof of Theorem 1.1.

The functional $\mathcal{J}$ is $G$-invariant, so, by the principle of symmetric criticality, the critical points of the restriction of $\mathcal{J}$ to $\mathcal{H}^{G}$ are the $G$-invariant critical points of $\mathcal{J}$; see, e.g., [21, Theorem 1.28].

It is readily seen that the results of Section 3 are also true for $H:=H_{0}^{1}(\Omega)^{G}$. Theorem 3.3$(ii)$ and Lemma 4.4 imply that $\Psi$ satisfies the $(PS)_{c}$-condition for every $c\in\mathbb{R}$. This, together with Lemma 4.5 and Theorem 3.4, yields Theorem 1.1. ∎

4.2 Entire solutions to critical systems

Next, we consider the Yamabe system (1.3).

As usual, we denote

[TABLE]

where   $\|w\|^{2}:=\int_{\mathbb{R}^{N}}|\nabla w|^{2}$   and   $|w|_{2^{*}}^{2^{*}}:=\int_{\mathbb{R}^{N}}|w|^{2^{*}}$. The next result says that the system (1.3) cannot be solved by minimization.

Proposition 4.6**.**

We have that

[TABLE]

and this infimum is not attained by $\mathcal{J}$ on $\mathcal{N}$.

Proof.

Following the argument given in [6, Proposition 2.2] for $M=2$ one can easily prove this statement. ∎

To obtain multiple solutions to the system (1.3) we consider a conformal action on $\mathbb{R}^{N}$, as in [8, 6].

Let $\Gamma=O(m)\times O(n)$ with $m+n=N+1$ and $m,n\geq 2$ act on $\mathbb{R}^{N+1}\equiv\mathbb{R}^{m}\times\mathbb{R}^{n}$ in the obvious way. Then, $\Gamma$ acts isometrically on the unit sphere $\mathbb{S}^{N}:=\{x\in\mathbb{R}^{N+1}:|x|=1\}$. The stereographic projection $\sigma:\mathbb{S}^{N}\to\mathbb{R}^{N}\cup\{\infty\}$, which maps the north pole $(0,\ldots,0,1)$ to $\infty$, induces a conformal action of $\Gamma$ on $\mathbb{R}^{N}$, given by

[TABLE]

Note that the map $\widetilde{\gamma}$ is well defined except at a single point.

The group $\Gamma$ acts on the Sobolev space $D^{1,2}(\mathbb{R}^{N})$ by linear isometries as follows:

[TABLE]

see [6, Section 3]. We shall say that $w$ is $\Gamma$-invariant if $\gamma w=w$ for all $\gamma\in\Gamma$, and that $(u_{1},\ldots,u_{M})$ is $\Gamma$-invariant if each $u_{i}$ is $\Gamma$-invariant. We set

[TABLE]

One has the following results.

Lemma 4.7**.**

The embedding $D^{1,2}(\mathbb{R}^{N})^{\Gamma}\hookrightarrow L^{2^{*}}(\mathbb{R}^{N})$ is compact.

Proof.

This follows from Proposition 3.3 and Example 3.4(1) in [6]. ∎

Lemma 4.8**.**

The functional $\mathcal{J}$ satisfies the Palais-Smale condition in $\mathcal{H}^{\Gamma}$.

Proof.

The proof follows from Lemma 4.7 by standard arguments (boundedness of Palais-Smale sequences is proved as in Lemma 4.4). ∎

Lemma 4.9**.**

Let $\mathcal{U}^{\Gamma}:=\mathcal{U}\cap\mathcal{H}^{\Gamma}$. Then, $\mathrm{genus}(\mathcal{U}^{\Gamma})=\infty$.

Proof.

The proof is the same as that of Lemma 4.5. ∎

Proof of Theorem 1.2.

The functional $\mathcal{J}$ is $\Gamma$-invariant; see [6, Section 3]. Thus, the critical points of the restriction of $\mathcal{J}$ to $\mathcal{H}^{\Gamma}$ are the $\Gamma$-invariant critical points of $\mathcal{J}$.

The results of Section 3 hold true for $H=D^{1,2}(\mathbb{R}^{N})^{\Gamma}$. Theorem 3.3$(ii)$ and Lemma 4.8 imply that $\Psi$ satisfies the $(PS)_{c}$-condition for every $c\in\mathbb{R}$. This, together with Lemma 4.9 and Theorem 3.4, yields Theorem 1.2. ∎

4.3 Brezis-Nirenberg systems

Finally, we consider the Brezis-Nirenberg type system (1.4).

For each $I\subset\{1,\ldots,M\}$, let $(\mathscr{S}_{I})$ be the system of $M-|I|$ equations obtained by replacing $\kappa_{i},\mu_{i},\lambda_{ij},\lambda_{ji}$ with [math] if $i\in I$, where $|I|$ is the cardinality of $I$, i.e.,

[TABLE]

The fully nontrivial solutions of $(\mathscr{S}_{I})$ correspond to the solutions $(u_{1},\ldots,u_{M})$ of (1.4) which satisfy $u_{i}=0$ iff $i\in I$. We set

[TABLE]

Lemma 4.10**.**

If

[TABLE]

then this infimum is attained by $\mathcal{J}$ on $\mathcal{N}$.

Proof.

Note that $\inf_{v\in\mathcal{N}}\mathcal{J}(v)=\inf_{v\in\mathcal{U}}\Psi(v)$. So, by Ekeland’s variational principle [21, Theorem 8.5] and Theorem 3.3, there exists a sequence $(u_{n})$ in $\mathcal{N}$ such that $\mathcal{J}(u_{n})\to c_{0}$ and $\mathcal{J}^{\prime}(u_{n})\to 0$. It follows from (3.1) that $(u_{n})$ is bounded in $\mathcal{H}:=(D^{1,2}_{0}(\Omega))^{M}$. So, after passing to a subsequence, $u_{n}\rightharpoonup u$ weakly in $\mathcal{H}$, $u_{n}\to u$ strongly in $L^{2}(\Omega)$ and $u_{n}\to u$ a.e. in $\Omega$. A standard argument shows that $u$ is a solution to the system (1.4). We claim that $u$ is fully nontrivial.

Arguing by contradiction, assume that some components of $u$ are trivial. Let $I:=\{i\in\{1,\ldots,M\}:u_{i}=0\}$. Then, for each $i\in I$, we have that $u_{n,i}\to 0$ strongly in $L^{2}(\Omega)$. As $u_{n}\in\mathcal{N}$ and $\lambda_{ij}<0$, we get that

[TABLE]

where

[TABLE]

Hence,

[TABLE]

i.e.,  $\mu_{i}^{-\frac{N-2}{2}}S^{\frac{N}{2}}\leq\|u_{n,i}\|_{i}^{2}+o(1)$  for every $i\in I$. As $u$ solves (1.4), we obtain

[TABLE]

This contradicts our assumption.

Therefore, $u$ is fully nontrivial. This implies that $u\in\mathcal{N}$, and (3.1) yields

[TABLE]

Hence, $\mathcal{J}(u)=c_{0}$, as claimed. ∎

Lemma 4.11**.**

Let $N\geq 4$. Assume that $\min\{\alpha_{ij},\beta_{ij}\}\geq\frac{4}{3}$ if $N=5$ and $\alpha_{ij}=\beta_{ij}=2$ if $N=4$, for all $i,j=1,\ldots,M$. Then

[TABLE]

Proof.

We prove this statement by induction on $M$.

If $M=1$ the system reduces to the single equation

[TABLE]

and the statement was proved by Brezis and Nirenberg in [1].

Assume that the statement is true for every system $(\mathscr{S}_{I})$ with $|I|\geq 1$ (i.e., for every system of $M-1$ equations). Then, the right-hand side of (4.6) reduces to

[TABLE]

Without loss of generality, we may assume that $I=\{M\}$. By Lemma 4.10 and our induction hypothesis, there exists a positive, least energy, fully nontrivial solution $(u_{1},\ldots,u_{M-1})$ to the system $(\mathscr{S}_{I})$. Fix $\xi\in\Omega$ and $\varrho\in(0,1)$ such that $B_{\varrho}(\xi)\subset\Omega$, and a cut-off function $\chi\in\mathcal{C}^{\infty}_{c}(B_{\varrho}(\xi))$ such that $0\leq\chi\leq 1$ and $\chi\equiv 1$ in $B_{\varrho/2}(\xi)$. Set

[TABLE]

where

[TABLE]

It is shown in [1] that

[TABLE]

for some $d_{0}>0$; see also [21, Lemma 1.46]. Inspecting the proofs in [1, 21] one sees that $d_{0}$ may be chosen independently of $\varrho$. Moreover, if $\alpha,\beta>1$ and $\alpha+\beta=2^{*}$, we have that

[TABLE]

for some $c_{1},d_{1}>0$ and $\varepsilon$ small enough. By a regularity result in [18, Appendix B], $u_{i}\in\mathcal{C}^{0}(\overline{\Omega})$ and we get that

[TABLE]

Hence, there exists $\varepsilon_{0}>0$ such that, for every $\varepsilon\in(0,\varepsilon_{0})$,

[TABLE]

Therefore we may use Lemma 2.1 in order to obtain $0<r<R<\infty$ and $s_{\varepsilon,1},\ldots,s_{\varepsilon,M}\in[r,R]$ such that

[TABLE]

As $(u_{1},\ldots,u_{M-1})$ is a least energy solution to the system $(\mathscr{S}_{I})$, from (3.2) and estimates (4.7) we obtain

[TABLE]

So, if either $N\geq 6$, or $N=5$ and $\min\{\alpha_{ij},\beta_{ij}\}>\frac{4}{3}$ for all $i,j=1,\ldots,M$, we derive from (4.8) and (4.9) that, for $\varepsilon$ small enough,

[TABLE]

and (4.6) follows. In the remaining cases we need to be careful when selecting $\xi$ and $\varrho$. If $N=4$ and $\alpha_{ij}=\beta_{ij}=2$ for all $i,j=1,\ldots,M$, we choose them in such a way that

[TABLE]

This can be done because $u_{i}=0$ on $\partial\Omega$. Then, from (4.8) we get

[TABLE]

If $N=5$ and $\min\{\alpha_{ij},\beta_{ij}\}=\frac{4}{3}$ for some pairs $i,j=1,\ldots,M$ we argue in a similar way. Hence, in all cases,

[TABLE]

for $\varepsilon$ small enough, as claimed. ∎

Proof of Theorem 1.3.

The result follows from Lemmas 4.10, 4.11 and Theorem 3.4$(a)$. ∎

Remark 4.12**.**

If $N=5$ and $\min\{\alpha_{ij},\beta_{ij}\}>\frac{4}{3}$ or $N\geq 6$, then the condition $\partial\Omega\in\mathcal{C}^{2}$ is not necessary because $u_{i}\in\mathcal{C}^{0}(\Omega)$ according to the results in [18, Appendix B] and we may choose any $\xi,\varrho$ such that $\overline{B}_{\varrho}(\xi)\subset\Omega$.**