Multiplicity and concentration of solutions for fractional Schr\"odinger systems via penalization method

TL;DR

This paper establishes the existence, multiplicity, and concentration of positive solutions for a system of fractional Schrödinger equations using penalization, variational methods, and topological tools, highlighting the influence of potential minima.

Contribution

It introduces a novel approach combining penalization and topological methods to analyze fractional Schrödinger systems with variable potentials.

Findings

Multiple positive solutions exist depending on the topology of potential minima.

Solutions concentrate near the minima of the potentials as the parameter tends to zero.

The number of solutions relates to the topological complexity of the potential's minimum set.

Abstract

The aim of this paper is to investigate the existence, multiplicity and concentration of positive solutions for the following nonlocal system of fractional Schr\"odinger equations \begin{equation*} \left\{ \begin{array}{ll} \varepsilon^{2s} (-\Delta)^{s}u+V(x)u=Q_{u}(u, v) &\mbox{ in } \mathbb{R}^{N}, \varepsilon^{2s} (-\Delta)^{s}v+W(x)v=Q_{v}(u, v) &\mbox{ in } \mathbb{R}^{N}, u, v>0 &\mbox{ in } \mathbb{R}^{N}, \end{array} \right. \end{equation*} where $\varepsilon>0$ is a parameter, $s\in (0, 1)$, $N>2s$, $(-\Delta)^{s}$ is the fractional Laplacian, $V:\mathbb{R}^{N}\rightarrow \mathbb{R}$ and $W:\mathbb{R}^{N}\rightarrow \mathbb{R}$ are positive continuous potentials, $Q$ is a homogeneous $C^{2}$-function with subcritical growth. In order to relate the number of solutions with the topology of the set where the potentials $V$ and $W$ attain their minimum values, we apply penalization techniques, Nehari manifold arguments and Ljusternik-Schnirelmann theory.

Equations

$$\left\{\begin{array}[]{ll}\operatorname{\varepsilon}^{2s}(-\Delta)^{s}u+V(x)u=Q_{u}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ \operatorname{\varepsilon}^{2s}(-\Delta)^{s}v+W(x)v=Q_{v}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ u,v>0&\mbox{ in }\mathbb{R}^{N},\end{array}\right.$$

$$pQ(u,v)=u{Q}_u(u,v)+v{Q}_v(u,v)\text{mbox}forany(u,v)\in {\mathbb{R}}^2,$$

$$p(p-1)Q(u,v)=u^2{Q}_{uu}(u,v)+2uv{Q}_{uv}(u,v)+v^2{Q}_{vv}(u,v)\text{mbox}forany(u,v)\in {\mathbb{R}}^2.$$

$${\mathcal{P}}_q(u,v)=\sum _{{\alpha }_i+{\beta }_i=q}{a}_i{u}^{{\alpha }_i}{v}^{{\beta }_i},$$

$$Q_1(u,v)={\mathcal{P}}_p(u,v),Q_2(u,v)=\sqrt[r]{{\mathcal{P}}_{\ell }(u,v)}\text{mbox}and{Q}_3(u,v)=\frac{{\mathcal{P}}_{{\ell }_1}(u,v)}{{\mathcal{P}}_{{\ell }_2}(u,v)},$$

$$(-\Delta {)}^{s}u(x)=C_{N,s}P.V.{\int }_{{\mathbb{R}}^N}\frac{(u(x)-u(y))}{|x-y{|}^{N+2s}}dy(x\in {\mathbb{R}}^N)$$

$${\epsilon }^{2s}(-\Delta {)}^{s}u+V(x)u=f(x,u)\text{mbox}in{\mathbb{R}}^N.$$

$$i\frac{\partial \Phi }{\partial t}=(-\Delta {)}^s\Phi +V(x)\Phi -f(x,|\Phi |)\text{mbox}for(t,x)\in \mathbb{R}\times {\mathbb{R}}^N,$$

$$\left\{\begin{array}[]{ll}-\operatorname{\varepsilon}^{2}\Delta u+V(x)u=G_{u}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ -\operatorname{\varepsilon}^{2}\Delta v+W(x)v=G_{v}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ u,v>0&\mbox{ in }\mathbb{R}^{N}.\end{array}\right.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(\xi)u=Q_{u}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ (-\Delta)^{s}v+W(\xi)v=Q_{v}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ u,v>0&\mbox{ in }\mathbb{R}^{N}.\end{array}\right.$$

$$J_{\xi }(u,v)=\frac{1}{2}\Vert (u,v){\Vert }_{\xi }^2-{\int }_{{\mathbb{R}}^N}Q(u,v)dx,$$

$$\Vert (u,v){\Vert }_{\xi }^2:={\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^2+|(-\Delta {)}^{\frac{s}{2}}v{|}^{2}dx+{\int }_{{\mathbb{R}}^N}(V(\xi )u^2+W(\xi )v^2)dx.$$

$$C(\xi ):=\underset{\gamma \in \Gamma }{\inf }\underset{t\in [0,1]}{\max }{J}_{\xi }(\gamma (t))$$

$$\Gamma :=\{\gamma \in C([0,1],H_0):\gamma (0)=0,J_{\xi }(\gamma (1))\le 0\}.$$

$$C(\xi )=\underset{(u,v)\in {\mathcal{N}}_{\xi }}{\inf }{J}_{\xi }(u,v),$$

$$M:=\left\{x\in {\mathbb{R}}^N:C(x)=\underset{\xi \in {\mathbb{R}}^N}{\inf }C(\xi )\right\}\ne \varnothing .$$

$$M_{\delta }=\{x\in {\mathbb{R}}^N:dist(x,M)\le \delta \}\subset \Lambda ,$$

$$u_{\epsilon }(x)\le \frac{C{\epsilon }^{N+2s}}{{\epsilon }^{N+2s}+|x-P_{\epsilon }{|}^{N+2s}}\text{mbox}and{v}_{\epsilon }(x)\le \frac{C{\epsilon }^{N+2s}}{{\epsilon }^{N+2s}+|x-Q_{\epsilon }{|}^{N+2s}}\forall x\in {\mathbb{R}}^N.$$

$${\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^{2}dx=\frac{C_{N,s}}{2}{\iint }_{{\mathbb{R}}^{2N}}\frac{|u(x)-u(y){|}^2}{|x-y{|}^{N+2s}}dxdy,$$

$${\mathcal{D}}^{s,2}({\mathbb{R}}^N)=\left\{u\in L^{2_s^{\ast }}({\mathbb{R}}^N):{\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^{2}dx<\infty \right\}.$$

$$H^s({\mathbb{R}}^N):=\left\{u\in L^2({\mathbb{R}}^N):{\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^{2}dx<\infty \right\}$$

$$\Vert u{\Vert }_{H^s({\mathbb{R}}^N)}:=\sqrt{{\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^{2}dx+{\int }_{{\mathbb{R}}^N}|u{|}^{2}dx}.$$

$${\left({\int }_{{\mathbb{R}}^N}|u{|}^{2_s^{\ast }}dx\right)}^{\frac{2}{2_s^{\ast }}}\le S_{\ast }^{-1}{\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^{2}dx.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(\xi)u=Q_{u}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ (-\Delta)^{s}v+W(\xi)v=Q_{v}(u,v)&\mbox{ in }\mathbb{R}^{N},\\ u,v>0&\mbox{ in }\mathbb{R}^{N}.\end{array}\right.$$

$$\Vert (u,v){\Vert }_{\xi }^2:={\int }_{{\mathbb{R}}^N}|(-\Delta {)}^{\frac{s}{2}}u{|}^2+|(-\Delta {)}^{\frac{s}{2}}v{|}^{2}dx+{\int }_{{\mathbb{R}}^N}(V(\xi )u^2+W(\xi )v^2)dx.$$

$$J_{\xi }(u,v)=\frac{1}{2}\Vert (u,v){\Vert }_{\xi }^2-{\int }_{{\mathbb{R}}^N}Q(u,v)dx.$$

$$C(\xi ):=\underset{\gamma \in \Gamma }{\inf }\underset{t\in [0,1]}{\max }{J}_{\xi }(\gamma (t))$$

$$\Gamma :=\{\gamma \in C([0,1],H_0):\gamma (0)=0,J_{\xi }(\gamma (1))\le 0\}.$$

$$J_{\xi }(w)=C(\xi )\text{mbox}and{J}_{\xi }^{\prime }(w)=0.$$

Full text

[TABLE]

where $\operatorname{\varepsilon}>0$ is a parameter, $s\in(0,1)$, $N>2s$, $(-\Delta)^{s}$ is the fractional Laplacian, $V:\mathbb{R}^{N}\rightarrow\mathbb{R}$ and $W:\mathbb{R}^{N}\rightarrow\mathbb{R}$ are positive continuous potentials, $Q$ is a homogeneous $C^{2}$-function with subcritical growth. In order to relate the number of solutions with the topology of the set where the potentials $V$ and $W$ attain their minimum values, we apply penalization techniques, Nehari manifold arguments and Ljusternik-Schnirelmann theory.

1. Introduction

In this paper we deal with the existence, multiplicity and concentration phenomena of positive solutions for the following nonlinear fractional Schrödinger system

[TABLE]

where $\operatorname{\varepsilon}>0$ is a parameter, $s\in(0,1)$, $N>2s$, $V:\mathbb{R}^{N}\rightarrow\mathbb{R}$ and $W:\mathbb{R}^{N}\rightarrow\mathbb{R}$ are Hölder continuous potentials, $Q$ is a homogeneous $C^{2}$-function with subcritical growth.

We assume that there exist a bounded open set $\Lambda\subset\mathbb{R}^{N}$, $x_{0}\in\mathbb{R}^{N}$ and $\rho_{0}>0$ such that:

1. $(H1)$

$V(x),W(x)\geq\rho_{0}$ for any $x\in\partial\Lambda$; 2. $(H2)$

$V(x_{0}),W(x_{0})<\rho_{0}$; 3. $(H3)$

$V(x)\geq V(x_{0})>0$, $W(x)\geq W(x_{0})>0$ for any $x\in\mathbb{R}^{N}$.

Concerning the function $Q:\mathbb{R}^{2}_{+}\rightarrow\mathbb{R}$, where $\mathbb{R}^{2}_{+}=[0,\infty)\times[0,\infty)$, we suppose that $Q\in C^{2}(\mathbb{R}^{2}_{+},\mathbb{R})$ satisfies the following conditions:

1. $(Q1)$

there exists $p\in(2,2^{*}_{s})$, with $2^{*}_{s}=\frac{2N}{N-2s}$, such that $Q(tu,tv)=t^{p}Q(u,v)$ for any $t>0$, $(u,v)\in\mathbb{R}^{2}_{+}$; 2. $(Q2)$

there exists $C>0$ such that $|Q_{u}(u,v)|+|Q_{v}(u,v)|\leq C(u^{p-1}+v^{p-1})$ for any $(u,v)\in\mathbb{R}^{2}_{+}$; 3. $(Q3)$

$Q_{u}(0,1)=0=Q_{v}(1,0)$; 4. $(Q4)$

$Q_{u}(1,0)=0=Q_{v}(0,1)$; 5. $(Q5)$

$Q(u,v)>0$ for any $u,v>0$; 6. $(Q6)$

$Q_{u}(u,v),Q_{v}(u,v)\geq 0$ for any $(u,v)\in\mathbb{R}^{2}_{+}$.

Since we are interested in positive solutions of (1.1), we extend the function $Q$ to the whole of $\mathbb{R}^{2}$ by setting $Q(u,v)=0$ if $u\leq 0$ or $v\leq 0$. We note that the $p$-homogeneity of $Q$ implies that the following identity holds:

[TABLE]

and

[TABLE]

As a model for $Q$, we can provide the following example given in [22]. Let $q\geq 1$ and

[TABLE]

where $i\in\{1,\dots,k\}$, $\alpha_{i},\beta_{i}\geq 1$ and $a_{i}\in\mathbb{R}$. The following functions and their possible combinations, with appropriate choice of the coefficients $a_{i}$, satisfy assumptions $(Q1)$-$(Q5)$ on $Q$

[TABLE]

with $r=\ell p$ and $\ell_{1}-\ell_{2}=p$.

The nonlocal operator $(-\Delta)^{s}$ appearing in (1.1), it is the fractional Laplacian operator which can be defined for any $u:\mathbb{R}^{N}\rightarrow\mathbb{R}$ smooth enough by setting

[TABLE]

where $P.V.$ stands for the Cauchy principal value, and $C_{N,s}$ is a positive constant depending only on $N$ and $s$; see for instance [DPV, 35] for more details.

In the scalar case, problem (1.1) reduces to the following fractional Schrödinger equation

[TABLE]

We recall that a basic motivation to consider (1.4) arises in the study of standing wave solutions $\Phi(t,x)=u(x)e^{-\imath ct}$ for the following time-dependent fractional Schrödinger equation

[TABLE]

which plays a fundamental role in fractional quantum mechanics. Equation (1.5) was introduced by Laskin [32, 33] as an extension of the classical nonlinear Schrödinger equation [15, 23, 30, 36, 38] in which the Brownian motion of the quantum paths is replaced by a Lévy flight.

In the last decade a great attention has been paid to the existence and multiplicity of solutions to (1.4) under several assumptions on the potential $V(x)$, and involving nonlinearities $f(x,u)$ with subcritical or critical growth. Felmer et al. [27] investigated existence, regularity and qualitative properties of positive solution to (1.4) when $V=1$ and $f$ is a superlinear function with subcritical growth and satisfying the Ambrosetti-Rabinowitz condition. Dávila et al. [21] used Lyapunov-Schmidt reduction method to prove that (1.4) has a multi-peak solution when the potential $V\in C^{1,\alpha}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, $\inf_{x\in\mathbb{R}^{N}}V(x)>0$ and $f(x,u)=|u|^{p-1}u$. Fall et al. [26] showed that the concentration points of the solutions of (1.4) must be the critical points for $V$, as $\operatorname{\varepsilon}$ tends to zero. Dipierro et al. [24] proved some existence results to (1.4) with $V=0$, $f(x,u)=\operatorname{\varepsilon}hu^{q}+u^{2^{*}_{s}-1}$, where $q\in(0,1)$ and $h\in L^{1}(\mathbb{R}^{N})\cap L^{\infty}(\mathbb{R}^{N})$, via Concentration-Compactness Principle and mountain pass arguments. Alves and Miyagaki in [5] (see also [10]) used the extension method [19] and the penalization technique in [23] to investigate the existence and concentration of positive solutions to (1.4) when $f$ is a continuous function having a subcritical growth, and the potential $V$ is a continuous function having a local minimum. Further results related to (1.4) can be found in [2, 8, 13, 29, 31, 37] in which the authors established several existence and multiplicity results by using appropriate and different variational and topological methods.

In this paper we focus our attention on the multiplicity and concentration of positive solutions for fractional Schrödinger systems.

We recall that in the classical literature, many interesting papers [1, 3, 4, 6, 14, 17, 28] considered the existence, multiplicity and symmetry of solutions for elliptic systems of the type

[TABLE]

In particular way, in [1, 3, 4], the authors investigated positive solutions to (1.6), via a suitable variant of the penalization method introduced by del Pino and Felmer in [23] to study a class of nonlinear Schrödinger equations.

Differently from the local case, in the fractional context there are only few papers [9, 20, 34, 39] dealing with fractional systems in $\mathbb{R}^{N}$, and, as far as we know, no results on the multiplicity and concentration of solutions for fractional nonlinear Schrödinger systems are available.

The goal of this work is to give a first result in this direction, generalizing the multiplicity and concentration results in [3] for nonlocal system (1.1).

Before stating our results, we need to introduce some notations. Fix $\xi\in\mathbb{R}^{N}$, and we consider the following autonomous system

[TABLE]

Let $\operatorname{\mathcal{J}}_{\xi}:H^{s}(\mathbb{R}^{N})\times H^{s}(\mathbb{R}^{N})\rightarrow\mathbb{R}$ be the Euler-Lagrange functional associated with the above problem, i.e.

[TABLE]

where

[TABLE]

As in [9], we can see that assumptions $(H3)$, $(Q1)$ and $(Q2)$, show that $\operatorname{\mathcal{J}}_{\xi}$ possesses a mountain pass geometry, so we can consider the mountain pass value

[TABLE]

where

[TABLE]

Moreover, we can prove (see Section $2$) that $\xi\mapsto C(\xi)$ is a continuous function and that $C(\xi)$ can be also characterized as

[TABLE]

where $\mathcal{N}_{\xi}$ is the Nehari manifold associated with $\operatorname{\mathcal{J}}_{\xi}$. From the results in [9], we know that, for any fixed $\xi\in\mathbb{R}^{N}$, $C(\xi)$ is achieved and in view of condition $(H3)$ we can deduce that $C(x_{0})\leq C(\xi)$ for any $\xi\in\mathbb{R}^{N}$, which yields

[TABLE]

We recall that if $Y$ is a given closed set of a topological space $X$, we denote by $cat_{X}(Y)$ the Ljusternik-Schnirelmann category of $Y$ in $X$, that is the least number of closed and contractible sets in $X$ which cover $Y$.

With the above notations, the statement of our main result is the following one.

Theorem 1.1**.**

Assume that $(H1)$-$(H3)$ and $(Q1)$-$(Q6)$ hold. Then, for any $\delta>0$ satisfying

[TABLE]

there exists $\operatorname{\varepsilon}_{\delta}>0$ such that, for any $\operatorname{\varepsilon}\in(0,\operatorname{\varepsilon}_{\delta})$, system (1.1) admits at least $cat_{M_{\delta}}(M)$ solutions. Moreover, if $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$ is a solution to (1.1) and $P_{\operatorname{\varepsilon}}$ and $Q_{\operatorname{\varepsilon}}$ are global maximum points of $u_{\operatorname{\varepsilon}}$ and $v_{\operatorname{\varepsilon}}$ respectively, then $C(P_{\operatorname{\varepsilon}}),C(Q_{\operatorname{\varepsilon}})\rightarrow C(x_{0})$ as $\operatorname{\varepsilon}\rightarrow 0$, and we have the following estimates:

[TABLE]

The proof of Theorem 1.1 is obtained by combining in a suitable way some variational arguments inspired by [1, 3] with some ideas used in [5, 10] to deal with fractional Schrödinger equations. Firstly, we use the penalization technique introduced by Alves [1] modifying appropriately the function $Q(u,v)$ outside the set $\Lambda$. In this way, the energy functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ associated with the modified problem satisfies the assumptions of the mountain pass theorem [7], and we can find a nontrivial solution of the modified problem. Since we are interested in obtaining a multiplicity result for the modified problem, we study the energy functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ restricted to its Nehari Manifold $\mathcal{N}_{\operatorname{\varepsilon}}$, and we employ a technique introduced by Benci and Cerami in [16]. The main ingredient is to make precisely comparisons between the category of some sublevel sets of the functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ and the category of the set $M$. Therefore, using Ljusternik-Schnirelmann theory, we obtain the existence of multiple solutions $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$ for the modified problem. Now, in order to prove that these solutions are also solutions to (1.1) provided that $\operatorname{\varepsilon}>0$ is sufficiently small, we use a different approach from [1, 3], because the techniques developed for the local case can not be adapted in our context due to the presence of the nonlocal operator $(-\Delta)^{s}$. More precisely, motivated by [2, 5, 8, 10], we use a Moser iteration argument to estimate the $L^{\infty}$-norm of $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$, and by constructing suitable comparison functions based on the Bessel kernel [27], we are able to show that $|(u_{\operatorname{\varepsilon}}(x),v_{\operatorname{\varepsilon}}(x))|\rightarrow 0$ as $|x|\rightarrow\infty$ uniformly in $\operatorname{\varepsilon}$. This fact will be fundamental to achieve our aim. Finally, we also study the behavior of the maximum points of solutions to (1.1).

We would like to point out that Theorem 1.1 is in clear accordance with the local case, and it can be seen as the nonlocal counterpart of Theorem $1.1$ in [3].

We also emphasize that, to our knowledge, this is the first result in which the penalization technique combined with Ljusternik-Schnirelmann theory allows us to obtain multiple solutions for subcritical fractional system (1.1).

The paper is organized as follows. In Section $2$ we collect some preliminary facts about the fractional Sobolev spaces and fractional autonomous systems. In Section $3$ we introduce the modified problem. In Section $4$ we prove some compactness results for the modified functional. In Section $5$ we present the proof of Theorem 1.1.

2. preliminaries and technical results

In this preliminary section we recall some results concerning the fractional Sobolev spaces and we introduce the functional setting.

For any $s\in(0,1)$ we define $\mathcal{D}^{s,2}(\mathbb{R}^{N})$ as the completion of $C^{\infty}_{0}(\mathbb{R}^{N})$ with respect to

[TABLE]

or equivalently

[TABLE]

Let us introduce the fractional Sobolev space

[TABLE]

endowed with the natural norm

[TABLE]

We recall the following fundamental embeddings:

Theorem 2.1**.**

[DPV]* Let $s\in(0,1)$ and $N>2s$. Then there exists a sharp constant $S_{*}=S(N,s)>0$ such that for any $u\in H^{s}(\mathbb{R}^{N})$*

[TABLE]

Moreover, $H^{s}(\mathbb{R}^{N})$ is continuously embedded in $L^{q}(\mathbb{R}^{N})$ for any $q\in[2,2^{*}_{s}]$ and compactly in $L^{q}_{loc}(\mathbb{R}^{N})$ for any $q\in[1,2^{*}_{s})$.

Now we collect some technical results which will be useful later. Fixed $\xi\in\mathbb{R}^{N}$, let us consider the following subcritical autonomous system

[TABLE]

We set $\operatorname{\mathbb{H}}_{0}=H^{s}(\mathbb{R}^{N})\times H^{s}(\mathbb{R}^{N})$ endowed with the following norm

[TABLE]

Clearly, $\operatorname{\mathbb{H}}_{0}$ is a Hilbert space. Let us introduce the functional $\operatorname{\mathcal{J}}_{\xi}:\operatorname{\mathbb{H}}_{0}\rightarrow\mathbb{R}$ defined as

[TABLE]

Since $\operatorname{\mathcal{J}}_{\xi}$ has a mountain pass geometry (see [9]), we can define the minimax level

[TABLE]

where

[TABLE]

Using Theorem $3.1$ in [9], we know that problem (2.2) admits a weak solution.

Next we give the proof of the following result which plays an important role to study (1.1).

Lemma 2.1**.**

The map $\xi\mapsto C(\xi)$ is continuous.

Proof.

For $\xi\in\mathbb{R}^{N}$, let $\{\zeta_{n}\},\{\lambda_{n}\}\subset\mathbb{R}^{N}$ be two sequences such that

1. $(a)$

$\zeta_{n}\rightarrow\xi$ and $C(\zeta_{n})\geq C(\xi)$ for all $n\in\mathbb{N}$, 2. $(b)$

$\lambda_{n}\rightarrow\xi$ and $C(\lambda_{n})\leq C(\xi)$ for all $n\in\mathbb{N}$.

We aim to prove that $C(\zeta_{n}),C(\lambda_{n})\rightarrow C(\xi)$ as $n\rightarrow\infty$. Using Theorem $3.1$ in [9], we know that there exists $w=(u,v)\in\operatorname{\mathbb{H}}_{0}$ such that

[TABLE]

For any $n\in\mathbb{N}$, let $t_{n}>0$ be such that

[TABLE]

We can show that $t_{n}\rightarrow 1$. Indeed $\operatorname{\mathcal{J}}_{\xi}^{\prime}(w)=0$ and (1.2) imply that

[TABLE]

By the definition of $t_{n}>0$ we know that $\frac{d}{dt}\operatorname{\mathcal{J}}_{\zeta_{n}}(tu,tv)\mid_{t=t_{n}}=0$, so, using $(Q1)$ and (1.2), we get

[TABLE]

Thus, using the continuity of $V$ and $W$, and the fact that $\zeta_{n}\rightarrow\xi$, we deduce that $t_{n}\rightarrow 1$ as $n\rightarrow\infty$. Moreover, we can see that $\operatorname{\mathcal{J}}_{\zeta_{n}}(t_{n}w)\rightarrow\operatorname{\mathcal{J}}_{\xi}(w)$ as $n\rightarrow\infty$. Therefore

[TABLE]

From $(a)$ we can deduce that $\liminf_{n\rightarrow\infty}C(\zeta_{n})\geq C(\xi)$, which implies that $C(\zeta_{n})\rightarrow C(\xi)$ as $n\rightarrow\infty$. Now we show that $C(\lambda_{n})\rightarrow C(\xi)$ as $n\rightarrow\infty$. Using Theorem $3.1$ in [9], there exists $w_{n}=(u_{n},v_{n})$ such that

[TABLE]

Let $p_{n},q_{n}\in\mathbb{R}^{N}$ be such that

[TABLE]

and we set $z_{n}=u_{n}+v_{n}$. By $(Q2)$, there exists $K>0$ such that $z_{n}$ satisfies

[TABLE]

where $\alpha=\min\{V(x_{0}),W(x_{0})\}$. If we denote by $z_{n}(r_{n})=\max_{x\in\mathbb{R}^{N}}z_{n}(x)$, we can use the integral representation formula for the fractional Laplacian (see [DPV]) to see that

[TABLE]

Therefore,

[TABLE]

Hence, there exists $\delta=(\frac{\alpha}{K})^{\frac{1}{p-2}}>0$ such that for any $n\in\mathbb{N}$

[TABLE]

Consequently, there exists an infinite subset $\mathbb{M}\subset\mathbb{N}$ such that at least one of the following cases occurs:

1. $(i)$

$u_{n}(p_{n})\geq\frac{\delta}{4}$ for any $n\in\mathbb{M}$, 2. $(ii)$

$v_{n}(q_{n})\geq\frac{\delta}{4}$ for any $n\in\mathbb{M}$.

Let us assume that $(i)$ occurs, and define

[TABLE]

From (2.3), we may assume, up to a subsequence, that $\hat{u}_{n}\rightharpoonup\hat{u}$ and $\hat{v}_{n}\rightharpoonup\hat{v}$ in $\operatorname{\mathbb{H}}_{0}$. Since $\lambda_{n}\rightarrow\xi$, we can note that the function $\hat{w}=(\hat{u},\hat{v})$ verifies $\operatorname{\mathcal{J}}_{\xi}(\hat{w})=C(\xi)$ and $\operatorname{\mathcal{J}}^{\prime}_{\xi}(\hat{w})=0$. Using (2.3), we can see that $\langle\operatorname{\mathcal{J}}_{\lambda_{n}}^{\prime}(w_{n}),\hat{w}(\cdot-p_{n})\rangle=0$, that is

[TABLE]

By the weak convergence in $\operatorname{\mathbb{H}}_{0}$ and $\lambda_{n}\rightarrow\xi$ as $n\rightarrow\infty$, we can pass to the limit in the above relation and we find $\langle\operatorname{\mathcal{J}}_{\xi}^{\prime}(\hat{w}),\hat{w}\rangle=0$. Now, fix $\theta\in(2,p)$. Using the weak convergence, Fatou’s Lemma, (1.3) and $\langle\operatorname{\mathcal{J}}_{\xi}^{\prime}(\hat{w}),\hat{w}\rangle=0$, we can see that

[TABLE]

This and condition $(b)$ yields $C(\lambda_{n})\rightarrow C(\xi)$ as $n\rightarrow\infty$.

∎

Let us note that

[TABLE]

where

[TABLE]

Since the minimax level $C(\xi)$ is achieved and using $(H1)$-$(H3)$, we can see that

[TABLE]

Now we prove the following fundamental result.

Lemma 2.2**.**

$C^{*}:=C(x_{0})=\inf_{\xi\in\Lambda}C(\xi)<\min_{\xi\in\partial\Lambda}C(\xi)$.

Proof.

Let us denote by $b_{\rho_{0}}$ the minimax level of mountain pass theorem associated with the functional $F_{\rho_{0}}:\operatorname{\mathbb{H}}_{0}\rightarrow\mathbb{R}$ given by

[TABLE]

Using the definition of $F_{\rho_{0}}$ and $(H1)$, we have for $\xi\in\partial\Lambda$

[TABLE]

so we get

[TABLE]

Thus

[TABLE]

On the other hand, from $(H2)$, we can see that

[TABLE]

from where we can conclude that

[TABLE]

which implies

[TABLE]

Putting together (2.4) and (2.5) we obtain

[TABLE]

This ends the proof of lemma. ∎

Finally, we recall the following compactness property related to minimizing sequences of the autonomous system, whose proof follows the lines of Theorem $3.1$ in [9].

Theorem 2.2**.**

[9]** Let $\{(u_{n},v_{n})\}\subset\mathcal{N}_{\xi}$ be a sequence such that $\operatorname{\mathcal{J}}_{\xi}(u_{n},v_{n})\rightarrow C^{*}$ and $(u_{n},v_{n})\rightharpoonup(u,v)$ in $\operatorname{\mathbb{H}}_{0}$. Then there exists $\{\tilde{y}_{n}\}\subset\mathbb{R}^{N}$ such that the translated sequence $\{(u_{n}(\cdot+\tilde{y}_{n}),v_{n}(\cdot+\tilde{y}_{n}))\}$ strongly converges to $(\tilde{u},\tilde{v})\in\mathcal{N}_{\xi}$ with $\operatorname{\mathcal{J}}_{\xi}(\tilde{u},\tilde{v})=C^{*}$. Moreover, if $(u,v)\neq 0$, then $\{\tilde{y}_{n}\}$ can be taken identically zero and therefore $(u_{n},v_{n})\rightarrow(u,v)$ in $\operatorname{\mathbb{H}}_{0}$.

3. the modified problem

In this section we introduce a penalty function in order to study solutions of problem (1.1). Firstly, we observe that, by using the change of variable $x\mapsto\operatorname{\varepsilon}x$, the analysis of (1.1) is equivalent to consider the following problem

[TABLE]

At this point we choose $a>0$ and $\eta\in C^{2}(\mathbb{R},\mathbb{R})$ a non-increasing function such that

[TABLE]

Using $\eta$, we introduce the following function $\hat{Q}:\mathbb{R}^{2}\rightarrow\mathbb{R}$ by setting

[TABLE]

where

[TABLE]

Let us observe that $A\rightarrow 0$ as $a\rightarrow 0^{+}$, so we may assume that $A<\min\{V(x_{0}),W(x_{0})\}$. Now we define $H:\mathbb{R}^{N}\times\mathbb{R}^{2}\rightarrow\mathbb{R}$ by setting

[TABLE]

As in [1], we can prove the following useful properties of the penalized function $H$.

Lemma 3.1**.**

The function $H$ satisfies the following estimates

[TABLE]

and

[TABLE]

Moreover, for any $k>0$ fixed, we can choose the constant $a>0$ sufficiently small such that

[TABLE]

and

[TABLE]

where $\alpha:=\min\{V(x_{0}),W(x_{0})\}$.

Now we consider the following modified problem

[TABLE]

Then, by the definition of $H$ and $\hat{Q}$, to study solutions of (3.1), we will look for solutions $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$ to (3.7) such that

[TABLE]

where $\Lambda_{\operatorname{\varepsilon}}:=\{x\in\mathbb{R}^{N}:\operatorname{\varepsilon}x\in\Lambda\}$ and $|(u,v)|:=\sqrt{u^{2}+v^{2}}$ for any $u,v\in\mathbb{R}$.

For any $\operatorname{\varepsilon}>0$, we introduce the fractional space

[TABLE]

endowed with the norm

[TABLE]

Let us introduce the Euler-Lagrange functional associated with (3.7), that is

[TABLE]

for any $(u,v)\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$.

We define

[TABLE]

It is standard to check that for any $(u,v)\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}\setminus\{(0,0)\}$, the function $t\rightarrow\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(tu,tv)$ achieves its maximum at a unique $t_{u}>0$ such that $t_{u}(u,v)\in\mathcal{N}_{\operatorname{\varepsilon}}$.

Let us observe that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}\in C^{1}(\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}},\mathbb{R})$ has a mountain pass geometry, that is

1. $(MP1)$

$\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(0,0)=0$; 2. $(MP2)$

there exist $r,\rho>0$ such that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(u,v)\geq r$ for $\|(u,v)\|_{\xi}=\rho$; 3. $(MP3)$

there exists $e\in\operatorname{\mathbb{H}}_{0}$ with $\|e\|_{\operatorname{\varepsilon}}>\rho$ such that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(e)<0$.

Indeed, using (3.3)-(3.5), we can see that

[TABLE]

which together with $(Q2)$ and Theorem 2.1 yields

[TABLE]

where $k>2$ is fixed. Hence, $(MP2)$ holds. On the other hand, for any $(\phi_{1},\phi_{2})\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$ such that $Q(\phi_{1},\phi_{2})\geq 0$ and $Q(\phi_{1},\phi_{2})\not\equiv 0$, we have, in view of $(Q1)$, that

[TABLE]

Moreover, $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ satisfies the Palais-Smale compactness condition:

Lemma 3.2**.**

Any sequence $\{(u_{n},v_{n})\}$ in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$ such that $\{\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(u_{n},v_{n})\}$ is bounded and $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}^{\prime}(u_{n},v_{n})\rightarrow 0$, admits a convergent subsequence in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$.

Proof.

First of all, we show that $\{(u_{n},v_{n})\}$ is bounded in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$. Indeed, using conditions (3.3)-(3.4), it follows that

[TABLE]

On the other hand,

[TABLE]

so, in view of (3.5), we can see that

[TABLE]

Taking into account (3.3), (3.8) and (3.9) we have

[TABLE]

Choosing $k$ such that $k>\frac{1}{2}(\frac{1}{2}-\frac{1}{p})^{-1}$ it follows that $\{(u_{n},v_{n})\}$ is bounded. Since $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$ is reflexive, there exists $(u,v)\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$ and a subsequence, still denoted by $\{(u_{n},v_{n})\}$, such that $\{(u_{n},v_{n})\}$ is weakly convergent to $(u,v)$ and $u_{n}\rightarrow u$, $v_{n}\rightarrow v$ in $L^{q}_{loc}(\mathbb{R}^{N})$ for any $q\in[1,2^{*}_{s})$. Then, it is easy to check that $(u,v)$ is a critical point of $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ and

[TABLE]

Now we show that $\{(u_{n},v_{n})\}$ strongly converges to $(u,v)$. To do this, we will prove the following claim.

Claim 1. For each $\delta>0$, there exists $R>0$ such that

[TABLE]

where $B_{R}$ denotes the ball with center at [math] and radius $R$.

First of all, we may assume that $R$ is chosen so that $\Lambda_{\operatorname{\varepsilon}}\subset B_{R}$. Let $\eta_{R}$ be a cut-off function such that $\eta_{R}=0$ on $B_{R}$, $\eta_{R}=1$ on $\mathbb{R}^{N}\setminus B_{2R}$, $0\leq\eta\leq 1$ and $|\nabla\eta_{R}|\leq\frac{c}{R}$. Since $\{(u_{n},v_{n})\}$ is a bounded (PS) sequence, we have

[TABLE]

Thus, using (3.5) with $k>1$, we have

[TABLE]

from which we deduce that

[TABLE]

Using the Hölder inequality and the boundedness of $\{(u_{n},v_{n})\}$ in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$, we get

[TABLE]

Therefore, if we prove that

[TABLE]

we can use (3) to conclude that Claim $1$ holds true.

Then, in what follows, we show that (3.13) is satisfied. Firstly, we note that $\mathbb{R}^{2N}$ can be written as

[TABLE]

Hence,

[TABLE]

Now, we estimate each integral in (3). Since $\eta_{R}=1$ in $\mathbb{R}^{N}\setminus B_{2R}$, we get

[TABLE]

Let $K>4$. Clearly, we have

[TABLE]

Let us observe that, if $(x,y)\in(\mathbb{R}^{N}\setminus B_{kR})\times B_{2R}$, then

[TABLE]

Then, taking into account $0\leq\eta_{R}\leq 1$, $|\nabla\eta_{R}|\leq\frac{C}{R}$ and applying the Hölder inequality, we have

[TABLE]

Now, take $\operatorname{\varepsilon}\in(0,1)$, and we observe that

[TABLE]

Let us estimate the integrals on the right hand side in (3). Then we can see that

[TABLE]

and

[TABLE]

which yield

[TABLE]

Now, using the definition of $\eta_{R}$, $\operatorname{\varepsilon}\in(0,1)$, and $0\leq\eta_{R}\leq 1$, we obtain

[TABLE]

where we used the fact that if $(x,y)\in B_{\varepsilon R}\times(\mathbb{R}^{N}\setminus B_{R})$, then $|x-y|>(1-\operatorname{\varepsilon})R$.

Taking into account (3), (3.18) and (3) we deduce

[TABLE]

Putting together (3), (3.15), (3) and (3), we can infer

[TABLE]

Since $\{u_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{N})$, by Theorem 2.1 we may assume that $u_{n}\rightarrow u$ in $L^{2}_{loc}(\mathbb{R}^{N})$ for some $u\in H^{s}(\mathbb{R}^{N})$. Then, taking the limit as $n\rightarrow\infty$ in (3), we have

[TABLE]

where in the last passage we used the Hölder inequality.

Since $u\in L^{2^{*}_{s}}(\mathbb{R}^{N})$, $K>4$ and $\operatorname{\varepsilon}\in(0,1)$, we obtain

[TABLE]

Choosing $\operatorname{\varepsilon}=\frac{1}{K}$, we get

[TABLE]

that is (3.13) holds true.

Then, using (3.11) and (3.5) of Lemma 3.1, we obtain that

[TABLE]

for any $n$ big enough. On the other hand, taking $R$ larger if necessary, we can suppose that

[TABLE]

Therefore, we see that (3.22) and (3.23) yield

[TABLE]

for $n$ large enough. Now, observing that $B_{R}$ is bounded, we can use the dominated convergence theorem and the strong convergence in $L^{q}_{loc}(\mathbb{R}^{N})$ to deduce that

[TABLE]

as $n\rightarrow\infty$. Putting together $\langle\operatorname{\mathcal{J}}^{\prime}_{\operatorname{\varepsilon}}(u_{n},v_{n}),(u_{n},v_{n})\rangle=o_{n}(1)$, (3.10), (3.24) and (3.25), we can infer that

[TABLE]

which implies that $\{(u_{n},v_{n})\}$ strongly converges to $(u,v)$ in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$.

∎

In light of mountain pass theorem [7], there exists $(u,v)\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}\setminus\{0\}$ such that

[TABLE]

where

[TABLE]

and

[TABLE]

Finally, we prove the following result.

Lemma 3.3**.**

If $(u,v)$ is a critical point of $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$, we have that $u,v\geq 0$ in $\mathbb{R}^{N}$.

Proof.

Since $(u,v)$ is a critical point of $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$, we know that for any $(\phi,\psi)\in\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}\times\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$ it holds

[TABLE]

Taking $\phi=u^{-}$ and $\psi=v^{-}$ in (3), where $x^{-}=\min\{x,0\}$, and recalling that $(x-y)(x^{-}-y^{-})\geq|x^{-}-y^{-}|^{2}$ for any $x,y\in\mathbb{R}$, we can see that

[TABLE]

Now, we can note that for any $x\in\mathbb{R}^{N}\setminus\Lambda$,

[TABLE]

and

[TABLE]

Then, we deduce that

[TABLE]

Taking into account (3) and the definitions of $A$ and $\chi$, we can find a constant $C>0$ such that

[TABLE]

Recalling that $A\rightarrow 0$ as $a\rightarrow 0$, we can see that (3) and (3.29) imply

[TABLE]

for any $a$ sufficiently small. By the definition of $Q$, we know that $Q_{u}(u^{-},v)=0=Q_{v}(u,v^{-})$, so we get

[TABLE]

Using (3), we can infer that $\|(u^{-},v^{-})\|_{\operatorname{\varepsilon}}^{2}=0$, that is $u^{-}=v^{-}=0$ in $\mathbb{R}^{N}$.

∎

4. compactness properties

This section is devoted to prove compactness properties related to the functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$. Since we are interested in obtaining multiple critical points, we work with the functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ restricted to the Nehari manifold $\mathcal{N}_{\operatorname{\varepsilon}}$. We begin by proving some useful properties of $\mathcal{N}_{\operatorname{\varepsilon}}$.

Lemma 4.1**.**

There exist positive constants $a_{1}$, $\delta$ such that, for each $a\in(0,a_{1})$, $(u,v)\in\mathcal{N}_{\operatorname{\varepsilon}}$, there hold

[TABLE]

and

[TABLE]

Proof.

Using (3.5), $(Q2)$ and Theorem 2.1, we can see that for any $(u,v)\in\mathcal{N}_{\operatorname{\varepsilon}}$ it holds

[TABLE]

which implies that there is $\hat{\delta}>0$ such that

[TABLE]

Thus, using (1.2) and (3.5) (with $k=2$), we obtain

[TABLE]

which gives

[TABLE]

Therefore, (4.1) holds with $\delta=\frac{\hat{\delta}^{2}}{2p}$.

Now, taking into account $(u,v)\in\mathcal{N}_{\operatorname{\varepsilon}}$, (1.2) and (3.5), we get

[TABLE]

which implies that (4.2) is satisfied. ∎

Now we aim to show that the functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ restricted to $\mathcal{N}_{\operatorname{\varepsilon}}$, satisfies the Palais-Smale condition. To achieve our goal we prove the following technical lemma.

Lemma 4.2**.**

Let $\phi_{\operatorname{\varepsilon}}:\mathbb{H}_{\operatorname{\varepsilon}}\rightarrow\mathbb{R}$ be given by

[TABLE]

Then, there exist $a_{2},b>0$ such that, for each $a\in(0,a_{2})$,

[TABLE]

Proof.

Given $(u,v)\in\mathcal{N}_{\operatorname{\varepsilon}}$, we can use the definition of $H$, (1.2) and (1.3) to get

[TABLE]

where

[TABLE]

We set $|z|=\sqrt{u^{2}+v^{2}}$. By the definitions of $\hat{Q}$ and $\eta$, and using (1.2) again, we can see that

[TABLE]

Since $A\rightarrow 0$ as $a\rightarrow 0^{+}$, the last inequality together with $(H_{3})$ implies that

[TABLE]

where $o(1)\rightarrow 0$ as $a\rightarrow 0^{+}$.

Now we aim to estimate the last integral in (4.4). Firstly we observe that

[TABLE]

with

[TABLE]

and

[TABLE]

Thanks to (3.2), we obtain that

[TABLE]

On the other hand, by the definition of $A$, we have

[TABLE]

It follows from (1.2) that

[TABLE]

and (1.3) implies that

[TABLE]

Taking into account the above estimates, we deduce that

[TABLE]

Thus, by (4.4) and (4.5), we get

[TABLE]

Applying Lemma 4.1 we have, for $a$ small enough,

[TABLE]

∎

At this point, we are able to deduce the following compactness result.

Proposition 4.1**.**

The functional $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ restricted to $\mathcal{N}_{\operatorname{\varepsilon}}$ satisfies $(PS)_{c}$ for each $c\in\mathbb{R}$.

Proof.

Let $\{(u_{n},v_{n})\}\subset\mathcal{N}_{\operatorname{\varepsilon}}$ be such that

[TABLE]

where $o_{n}(1)$ goes to zero when $n\rightarrow\infty$. Then, there exists $\{\lambda_{n}\}\subset\mathbb{R}$ satisfying

[TABLE]

with $\phi_{\operatorname{\varepsilon}}$ as in Lemma 4.2. Due to the fact that $(u_{n},v_{n})\in\mathcal{N}_{\operatorname{\varepsilon}}$, we get

[TABLE]

Proceeding as in the proof of Lemma 3.2 we can see that there exists $C>0$ such that

[TABLE]

On the other hand, from Lemma 4.2, we may assume that $\langle\phi_{\operatorname{\varepsilon}}^{\prime}(u_{n},v_{n}),(u_{n},v_{n})\rangle\rightarrow\ell<0$.

Then, in view of (4.7), we can deduce that $\lambda_{n}\rightarrow 0$ and that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}^{\prime}(u_{n},v_{n})\rightarrow 0$ in the dual space of $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$. Invoking Lemma 3.2 we can infer that $\{(u_{n},v_{n})\}$ admits a convergent subsequence in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$. ∎

5. barycenter map and multiplicity of solutions to (3.7)

In this section our main purpose is to apply the Ljusternik-Schnirelmann category theory to prove a multiplicity result for system (3.7). In order to accomplish our goal, we first give some useful lemmas. We start by proving the following result.

Lemma 5.1**.**

Let $\operatorname{\varepsilon}_{n}\rightarrow 0^{+}$ and $\{(u_{n},v_{n})\}\subset\mathcal{N}_{\operatorname{\varepsilon}_{n}}$ be such that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}_{n}}(u_{n},v_{n})\rightarrow C^{*}$. Then there exists $\{\tilde{y}_{n}\}\subset\mathbb{R}^{N}$ such that the translated sequence

[TABLE]

has a subsequence which converges in $\operatorname{\mathbb{H}}_{0}$. Moreover, up to a subsequence, $\{y_{n}\}:=\{\operatorname{\varepsilon}_{n}\tilde{y}_{n}\}$ is such that $y_{n}\rightarrow y\in M$.

Proof.

Since $\langle\operatorname{\mathcal{J}}^{\prime}_{\operatorname{\varepsilon}_{n}}(u_{n},v_{n}),(u_{n},v_{n})\rangle=0$ and $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}_{n}}(u_{n},v_{n})\rightarrow C^{*}$, it is easy to see that $\{(u_{n},v_{n})\}$ is bounded in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}}$. Let us observe that $\|(u_{n},v_{n})\|_{\operatorname{\varepsilon}_{n}}\nrightarrow 0$ since $C^{*}>0$. Therefore, arguing as in [9], we can find a sequence $\{\tilde{y}_{n}\}\subset\mathbb{R}^{N}$ and constants $R,\gamma>0$ such that

[TABLE]

and we may assume that

[TABLE]

where $(\tilde{u}_{n}(x),\tilde{v}_{n}(x)):=(u_{n}(x+\tilde{y}_{n}),v_{n}(x+\tilde{y}_{n}))$ and $(\tilde{u},\tilde{v})\neq(0,0)$.

Let $\{t_{n}\}\subset(0,+\infty)$ be such that $(\hat{u}_{n},\hat{v}_{n}):=(t_{n}\tilde{u}_{n},t_{n}\tilde{v}_{n})\in\mathcal{N}_{x_{0}}$, and set $y_{n}:=\operatorname{\varepsilon}_{n}\tilde{y}_{n}$.

Using the definition of $H$ and $(H3)$ we can see that

[TABLE]

which gives $\operatorname{\mathcal{J}}_{x_{0}}(\hat{u}_{n},\hat{v}_{n})\rightarrow C^{*}$.

Now, the sequence $\{t_{n}\}$ is bounded since $\{(\tilde{u}_{n},\tilde{v}_{n})\}$ and $\{(\hat{u}_{n},\hat{v}_{n})\}$ are bounded in $\operatorname{\mathbb{H}}_{0}$, and $(\tilde{u}_{n},\tilde{v}_{n})\nrightarrow 0$. Therefore, up to a subsequence, $t_{n}\rightarrow t_{0}\geq 0$. Indeed $t_{0}>0$. Otherwise, if $t_{0}=0$, from the boundedness of $\{(\tilde{u}_{n},\tilde{v}_{n})\}$, we get $(\hat{u}_{n},\hat{v}_{n})=t_{n}(\tilde{u}_{n},\tilde{v}_{n})\rightarrow(0,0)$, that is $\operatorname{\mathcal{J}}_{x_{0}}(\hat{u}_{n},\hat{v}_{n})\rightarrow 0$ in contrast with the fact that $C^{*}>0$. Thus, $t_{0}>0$ and, up to a subsequence, we have $(\hat{u}_{n},\hat{v}_{n})\rightharpoonup t_{0}(\tilde{u},\tilde{v})=(\hat{u},\hat{v})\neq 0$ weakly in $\operatorname{\mathbb{H}}_{0}$. Hence it holds

[TABLE]

From Theorem 2.2 we deduce that $(\hat{u}_{n},\hat{v}_{n})\rightarrow(\hat{u},\hat{v})$ in $\operatorname{\mathbb{H}}_{0}$, that is $(\tilde{u}_{n},\tilde{v}_{n})\rightarrow(\tilde{u},\tilde{v})$ in $\operatorname{\mathbb{H}}_{0}$.

Now we show that $\{y_{n}\}$ has a subsequence, still denoted by itself, such that $y_{n}\rightarrow y\in M$. Assume by contradiction that $\{y_{n}\}$ is not bounded, that is there exists a subsequence, still denoted by $\{y_{n}\}$, such that $|y_{n}|\rightarrow+\infty$. Since $(u_{n},v_{n})\in\mathcal{N}_{\operatorname{\varepsilon}_{n}}$, we can see that

[TABLE]

Take $R>0$ such that $\Lambda\subset B_{R}$. Since we may assume that $|y_{n}|>2R$, for any $x\in B_{R/\operatorname{\varepsilon}_{n}}$ we get $|\operatorname{\varepsilon}_{n}x+y_{n}|\geq|y_{n}|-|\operatorname{\varepsilon}_{n}x|>R$. Then, we deduce that

[TABLE]

where we used the strong convergence of $(\tilde{u}_{n},\tilde{v}_{n})$ and that $|\mathbb{R}^{N}\setminus B_{R/\operatorname{\varepsilon}_{n}}|\rightarrow 0$ as $n\rightarrow\infty$. In virtue of $(H3)$ we get

[TABLE]

which is impossible due to $(\tilde{u}_{n},\tilde{v}_{n})\rightarrow(\tilde{u},\tilde{v})\neq 0$. Thus $\{y_{n}\}$ is bounded and, up to a subsequence, we may suppose that $y_{n}\rightarrow y$. If $y\notin\overline{\Lambda}$, then there exists $r>0$ such that $y_{n}\in B_{r/2}(y)\subset\mathbb{R}^{N}\setminus\overline{\Lambda}$ for any $n$ large enough. Reasoning as before, we get a contradiction. Hence $y\in\overline{\Lambda}$.

Now, we prove that $y\in M$. Taking into account Lemma 2.2, it is enough to prove that $C(y)=C^{*}$. Assume by contradiction that $C^{*}<C(y)$. Since $(\hat{u}_{n},\hat{v}_{n})\rightarrow(\hat{u},\hat{v})$ strongly in $\operatorname{\mathbb{H}}_{0}$, by Fatou’s Lemma we have

[TABLE]

which gives a contradiction. ∎

Now, we aim to relate the number of positive solutions of (3.7) with the topology of the set $M$. For this reason, we take $\delta>0$ such that

[TABLE]

and we choose $\psi\in C^{\infty}_{0}(\mathbb{R}_{+},[0,1])$ a non-increasing function satisfying $\psi(t)=1$ if $0\leq t\leq\frac{\delta}{2}$ and $\psi(t)=0$ if $t\geq\delta$. For any $y\in M$, we define

[TABLE]

and denote by $t_{\operatorname{\varepsilon}}>0$ the unique number such that

[TABLE]

where $(w_{1},w_{2})\in\operatorname{\mathbb{H}}_{0}$ is a solution to autonomous system (2.2) with $\xi=x_{0}$, such that $w_{1},w_{2}>0$ in $\mathbb{R}^{N}$ and $\operatorname{\mathcal{J}}_{x_{0}}(w_{1},w_{2})=C(x_{0})=C^{*}$ (such solution there exists in view of Theorem $3.1$ in [9]).

Finally, we consider $\Phi_{\operatorname{\varepsilon}}:M\rightarrow\mathcal{N}_{\operatorname{\varepsilon}}$ defined by setting

[TABLE]

Let us prove the following important relationship between $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ and $M$.

Lemma 5.2**.**

The functional $\Phi_{\operatorname{\varepsilon}}$ satisfies the following limit

[TABLE]

Proof.

Assume by contradiction that there there exist $\delta_{0}>0$, $\{y_{n}\}\subset M$ and $\operatorname{\varepsilon}_{n}\rightarrow 0$ such that

[TABLE]

We first show that $\lim_{n\rightarrow\infty}t_{\operatorname{\varepsilon}_{n}}<\infty$. Let us observe that by using the change of variable $z=\frac{\operatorname{\varepsilon}_{n}x-y_{n}}{\operatorname{\varepsilon}_{n}}$, if $z\in B_{\frac{\delta}{\operatorname{\varepsilon}_{n}}}$, it follows that $\operatorname{\varepsilon}_{n}z\in B_{\delta}$ and then $\operatorname{\varepsilon}_{n}z+y_{n}\in B_{\delta}(y_{n})\subset M_{\delta}\subset\Lambda$.

Then, recalling that $H=Q$ on $\Lambda$ and $\psi(t)=0$ for $t\geq\delta$, we have

[TABLE]

Now, let assume that $t_{\operatorname{\varepsilon}_{n}}\rightarrow\infty$. By the definition of $t_{\operatorname{\varepsilon}_{n}}$, $(Q1)$ and (1.2), we get

[TABLE]

Since $\psi=1$ in $B_{\frac{\delta}{2}}$ and $B_{\frac{\delta}{2}}\subset B_{\frac{\delta}{2\operatorname{\varepsilon}_{n}}}$ for $n$ big enough, and $w_{1}$, $w_{2}$ are continuous and positive in $\mathbb{R}^{N}$ we obtain

[TABLE]

for some $C_{\delta,p}>0$. Taking the limit as $n\rightarrow\infty$ in (5.5) we can infer that

[TABLE]

which is a contradiction because of

[TABLE]

in view of the dominated convergence theorem.

Thus, $\{t_{\operatorname{\varepsilon}_{n}}\}$ is bounded and we can assume that $t_{\operatorname{\varepsilon}_{n}}\rightarrow t_{0}\geq 0$. Clearly, if $t_{0}=0$, by limitation of $\|(\Psi_{1,\operatorname{\varepsilon}_{n},y_{n}},\Psi_{2,\operatorname{\varepsilon}_{n},y_{n}})\|^{2}_{\operatorname{\varepsilon}_{n}}$, the growth assumptions on $Q$, and (5.4), we can deduce that $\|(\Psi_{1,\operatorname{\varepsilon}_{n},y_{n}},\Psi_{2,\operatorname{\varepsilon}_{n},y_{n}})\|^{2}_{\operatorname{\varepsilon}_{n}}\rightarrow 0$, which is impossible. Hence, $t_{0}>0$.

Now, using $(Q2)$ and the dominated convergence theorem we can see that as $n\rightarrow\infty$

[TABLE]

Then, taking the limit as $n\rightarrow\infty$ in (5.4) we obtain

[TABLE]

In light of $(w_{1},w_{2})\in\mathcal{N}_{x_{0}}$, we deduce that $t_{0}=1$. Moreover, from (5), we have

[TABLE]

which contradicts (5.2).

∎

At this point, we are in the position to define the barycenter map. We take $\rho=\rho_{\delta}>0$ such that $M_{\delta}\subset B_{\rho}$, and we consider $\varUpsilon:\mathbb{R}^{N}\rightarrow\mathbb{R}^{N}$ given by

[TABLE]

We define the barycenter map $\beta_{\operatorname{\varepsilon}}:\mathcal{N}_{\operatorname{\varepsilon}}\rightarrow\mathbb{R}^{N}$ as follows

[TABLE]

Lemma 5.3**.**

The functional $\Phi_{\operatorname{\varepsilon}}$ satisfies the following limit

[TABLE]

Proof.

Suppose by contradiction that there exist $\delta_{0}>0$, $\{y_{n}\}\subset M$ and $\operatorname{\varepsilon}_{n}\rightarrow 0$ such that

[TABLE]

Using the definitions of $\Phi_{\operatorname{\varepsilon}_{n}}(y_{n})$, $\beta_{\operatorname{\varepsilon}_{n}}$, $\eta$ and the change of variable $z=\frac{\operatorname{\varepsilon}_{n}x-y_{n}}{\operatorname{\varepsilon}_{n}}$, we can see that

[TABLE]

Taking into account that $\{y_{n}\}\subset M\subset B_{\rho}$ and applying the dominated convergence theorem we can infer that

[TABLE]

which contradicts (5.7). ∎

We now introduce a subset $\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ of $\mathcal{N}_{\operatorname{\varepsilon}}$ by taking a function $h:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$ such that $h(\operatorname{\varepsilon})\rightarrow 0$ as $\operatorname{\varepsilon}\rightarrow 0$, and setting

[TABLE]

Fixed $y\in M$, we conclude from Lemma 5.2 that $h(\operatorname{\varepsilon})=|\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}(\Phi_{\operatorname{\varepsilon}}(y))-C^{*}|\rightarrow 0$ as $\operatorname{\varepsilon}\rightarrow 0$. Hence, $\Phi_{\operatorname{\varepsilon}}(y)\in\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ and $\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}\neq\emptyset$ for any $\operatorname{\varepsilon}>0$ small. Moreover, we have the following interesting relation between $\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ and $\beta_{\operatorname{\varepsilon}}$.

Lemma 5.4**.**

For any $\delta>0$, there holds that

[TABLE]

Proof.

Let $\operatorname{\varepsilon}_{n}\rightarrow 0$ as $n\rightarrow\infty$. For any $n\in\mathbb{N}$, there exists $(u_{n},v_{n})\in\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}_{n}}$ such that

[TABLE]

Therefore, it is suffices to find a sequence $\{y_{n}\}\subset M_{\delta}$ such that

[TABLE]

We note that $\{(u_{n},v_{n})\}\subset\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}_{n}}\subset\mathcal{N}_{\operatorname{\varepsilon}_{n}}$, from which we obtain that

[TABLE]

This yields that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}_{n}}(u_{n},v_{n})\rightarrow C^{*}$. Using Lemma 5.1, there exists $\{\tilde{y}_{n}\}\subset\mathbb{R}^{N}$ such that $y_{n}=\operatorname{\varepsilon}_{n}\tilde{y}_{n}\in M_{\delta}$ for $n$ sufficiently large. Setting $(\tilde{u}_{n}(x),\tilde{v}_{n}(x))=(u_{n}(\cdot+\tilde{y}_{n}),v_{n}(\cdot+\tilde{y}_{n}))$, we can see that

[TABLE]

Since $(\tilde{u}_{n},\tilde{v}_{n})\rightarrow(u,v)$ in $\operatorname{\mathbb{H}}_{0}$ and $\operatorname{\varepsilon}_{n}x+y_{n}\rightarrow y\in M_{\delta}$, we deduce that $\beta_{\operatorname{\varepsilon}_{n}}(u_{n},v_{n})=y_{n}+o_{n}(1)$, that is (5.8) holds.

∎

Now, we are ready to present the proof of the multiplicity result for (3.7).

Theorem 5.1**.**

For any $\delta>0$ satisfying $M_{\delta}\subset M$, there exists $\operatorname{\varepsilon}_{\delta}>0$ such that for any $\operatorname{\varepsilon}\in(0,\operatorname{\varepsilon}_{\delta})$, problem (3.7) has at least $cat_{M_{\delta}}(M)$ positive solutions.

Proof.

Given $\delta>0$ such that $M_{\delta}\subset\Lambda$, we can apply Lemma 5.2, Lemma 5.3 and Lemma 5.4 to find $\operatorname{\varepsilon}_{\delta}>0$ such that for any $\operatorname{\varepsilon}\in(0,\operatorname{\varepsilon}_{\delta})$, the following diagram

[TABLE]

is well-defined and $\beta_{\operatorname{\varepsilon}}\circ\Phi_{\operatorname{\varepsilon}}$ is homotopically equivalent to the embedding $\iota:M\rightarrow M_{\delta}$. Using the definition of $\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ and taking $\operatorname{\varepsilon}_{\delta}$ sufficiently small, we may assume that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ fulfills the Palais-Smale condition in $\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ (see Proposition 4.1). Therefore, standard Ljusternik-Schnirelmann theory [40] provides at least $cat_{\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}}(\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}})$ critical points $(u_{i},v_{i}):=(u^{i}_{\operatorname{\varepsilon}},v^{i}_{\operatorname{\varepsilon}})$ of $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}}$ restricted to $\mathcal{N}_{\operatorname{\varepsilon}}$. Using the arguments in [16], we know that $cat_{\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}}(\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}})\geq cat_{M_{\delta}}(M)$. Then, arguing as in the proof of Proposition 4.1, we can see that $(u_{i},v_{i})$ is also a critical point of the unconstrained functional and therefore a solution of problem (3.7).

∎

6. proof of theorem 1.1

In this last section we provide the proof of our main result.

Proof.

Take $\delta>0$ sufficiently small such that $M_{\delta}\subset\Lambda$. We begin by proving that there exists $\tilde{\operatorname{\varepsilon}}_{\delta}>0$ such that for any $\operatorname{\varepsilon}\in(0,\tilde{\operatorname{\varepsilon}}_{\delta})$ and any solution $u_{\operatorname{\varepsilon}}\in\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ of (3.7) it holds

[TABLE]

Assume by contradiction that there exist $\operatorname{\varepsilon}_{n}\rightarrow 0$, $(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\in\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}_{n}}$ such that $\operatorname{\mathcal{J}}^{\prime}_{\operatorname{\varepsilon}_{n}}(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})=0$ and $\|(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\|_{L^{\infty}(\mathbb{R}^{N}\setminus\Lambda_{\operatorname{\varepsilon}_{n}})}\geq a$. Since $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}_{n}}(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\leq C^{*}+h(\operatorname{\varepsilon}_{n})$ and $h(\operatorname{\varepsilon}_{n})\rightarrow 0$, we can argue as in the first part of the proof of Lemma 5.1, to deduce that $\operatorname{\mathcal{J}}_{\operatorname{\varepsilon}_{n}}(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\rightarrow C^{*}$. Then, invoking Lemma 5.1, we can find $\{\tilde{y}_{n}\}\subset\mathbb{R}^{N}$ such that $\operatorname{\varepsilon}_{n}\tilde{y}_{n}\rightarrow y\in M$.

Now, if we choose $r>0$ such that $B_{r}(y)\subset B_{2r}(y)\subset\Lambda$, we have $B_{\frac{r}{\operatorname{\varepsilon}_{n}}}(\frac{y}{\operatorname{\varepsilon}_{n}})\subset\Lambda_{\operatorname{\varepsilon}_{n}}$. In particular, for any $z\in B_{\frac{r}{\operatorname{\varepsilon}_{n}}}(\tilde{y}_{n})$ there holds

[TABLE]

Therefore $\mathbb{R}^{N}\setminus\Lambda_{\operatorname{\varepsilon}_{n}}\subset\mathbb{R}^{N}\setminus B_{\frac{r}{\operatorname{\varepsilon}_{n}}}(\tilde{y}_{n})$ for any $n$ big enough.

Now, let us denote by $(\tilde{u}_{n}(x),\tilde{v}_{n}(x))=(u_{\operatorname{\varepsilon}_{n}}(x+\tilde{y}_{n}),v_{\operatorname{\varepsilon}_{n}}(x+\tilde{y}_{n}))$ and $\tilde{z}_{n}=\tilde{u}_{n}+\tilde{v}_{n}\geq 0$. Using $(H3)$, the definition of $H$ and the growth conditions on $Q$, we can see that $\tilde{z}_{n}$ satisfies

[TABLE]

where $\alpha=\min\{V(x_{0}),W(x_{0})\}$ and $g_{n}$ is such that $|g_{n}|\leq\xi\tilde{z}_{n}+C_{\xi}\tilde{z}_{n}^{p-1}$, with $\xi>0$ fixed.

Then, for $\beta>0$ and $L>1$, we take $\tilde{z}_{n}\tilde{z}_{L,n}^{2(\beta-1)}$, where $\tilde{z}_{L,n}=\min\{\tilde{z}_{n},L\}$, as test function in (6.2), and arguing as in the proof of Lemma $6.1$ in [12] (see also Lemma $5.1$ in [11]) and observing that $\{\tilde{z}_{n}\}$ is bounded in $L^{2^{*}_{s}}(\mathbb{R}^{N})$ (since $\{(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\}$ is bounded in $\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}_{n}}$), we can use a Moser iteration scheme to deduce that $\tilde{z}_{n}\in L^{\infty}(\mathbb{R}^{N})$ and there exists a constant $K>0$ such that

[TABLE]

Consequently, $\{\tilde{u}_{n}\}$ and $\{\tilde{v}_{n}\}$ are bounded in $L^{\infty}(\mathbb{R}^{N})$, and by interpolation, $\tilde{u}_{n}\rightarrow u$ and $\tilde{v}_{n}\rightarrow v$ in $L^{q}(\mathbb{R}^{N})$ for any $q\in(2,\infty)$, for some $u,v\in L^{q}(\mathbb{R}^{N})$ for any $q\in(2,\infty)$.

Then, from the growth conditions on $Q$, we also have the following relations of limit in $L^{q}(\mathbb{R}^{N})$ for any $q\in(2,\infty)$:

[TABLE]

and

[TABLE]

Since $\tilde{z}_{n}$ satisfies

[TABLE]

where

[TABLE]

we have that

[TABLE]

for any $q\in[2,\infty)$, and we can find $K_{1}>0$ such that

[TABLE]

Hence $\tilde{z}_{n}(x)=(\mathcal{K}*\xi_{n})(x)=\int_{\mathbb{R}^{N}}\mathcal{K}(x-t)\xi_{n}(t)\,dt$, where $\mathcal{K}$ is the Bessel kernel which satisfies the following properties (see [27]):

1. $(i)$

$\mathcal{K}$ is positive, radially symmetric and smooth in $\mathbb{R}^{N}\setminus\{0\}$, 2. $(ii)$

there is $C>0$ such that $\mathcal{K}(x)\leq\frac{C}{|x|^{N+2s}}$ for any $x\in\mathbb{R}^{N}\setminus\{0\}$, 3. $(iii)$

$\mathcal{K}\in L^{q}(\mathbb{R}^{N})$ for any $q\in[1,\frac{N}{N-2s})$.

Then, arguing as in Lemma $2.6$ in [5], we can see that

[TABLE]

uniformly in $n\in\mathbb{N}$.

Therefore, there exists $R>0$ such that

[TABLE]

from which

[TABLE]

On the other hand, there exists $\nu\in\mathbb{N}$ such that for any $n\geq\nu$ and $\frac{r}{\operatorname{\varepsilon}_{n}}>R$, it holds

[TABLE]

which gives $|(u_{\operatorname{\varepsilon}_{n}}(x),v_{\operatorname{\varepsilon}_{n}}(x))|<a$ for any $x\in\mathbb{R}^{N}\setminus\Lambda_{\operatorname{\varepsilon}_{n}}$, that is a contradiction.

Now, let $\bar{\operatorname{\varepsilon}}_{\delta}$ be given by Theorem 5.1 and take $\operatorname{\varepsilon}_{\delta}=\min\{\tilde{\operatorname{\varepsilon}}_{\delta},\bar{\operatorname{\varepsilon}}_{\delta}\}$. Fix $\operatorname{\varepsilon}\in(0,\operatorname{\varepsilon}_{\delta})$. By Theorem 5.1 we know that problem (3.7) admits $cat_{M_{\delta}}(M)$ nontrivial solutions $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$. Since $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})\in\widetilde{\mathcal{N}}_{\operatorname{\varepsilon}}$ satisfies (6.1), by the definitions of $H$ and $\hat{Q}$ it follows that $(u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}})$ is a solution of (3.1). In light of $(Q6)$ and the maximum principle for the fractional Laplacian [18], we can infer that $u_{\operatorname{\varepsilon}},v_{\operatorname{\varepsilon}}>0$ in $\mathbb{R}^{N}$.

Now, we study the behavior of maximum points of solutions to (1.1).

Let $\operatorname{\varepsilon}_{n}\rightarrow 0$ and take $\{(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\}\subset\operatorname{\mathbb{H}}_{\operatorname{\varepsilon}_{n}}$ be a sequence of solutions to (3.7) as above. Using the definition of $H$ and $(Q2)$ we can see that there exists $\bar{a}\in(0,a)$ sufficiently small such that

[TABLE]

Arguing as before, we can find $R>0$ such that

[TABLE]

Up to a subsequence, we may assume that

[TABLE]

Indeed, if this case does not occur, we deduce that $\|(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\|_{L^{\infty}(\mathbb{R}^{N})}<\bar{a}$ and using the facts that $\langle\operatorname{\mathcal{J}}^{\prime}_{\operatorname{\varepsilon}_{n}}(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}}),(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\rangle=0$ and (6.4) we get

[TABLE]

which implies that $\|(u_{\operatorname{\varepsilon}_{n}},v_{\operatorname{\varepsilon}_{n}})\|_{\operatorname{\varepsilon}_{n}}\rightarrow 0$ as $n\rightarrow\infty$, that is a contradiction. Then (6.6) holds. Therefore, if we denote by $x_{n}$ and $\bar{x}_{n}$ the maximum points of $u_{\operatorname{\varepsilon}_{n}}$ and $v_{\operatorname{\varepsilon}_{n}}$ respectively, it follows from (6.5) and (6.6) that $x_{n}=\tilde{y}_{n}+p_{n}$ and $\bar{x}_{n}=\tilde{y}_{n}+q_{n}$ for some $p_{n},q_{n}\in B_{R}$.

Set $\hat{u}_{n}(x)=u_{\operatorname{\varepsilon}_{n}}(x/\operatorname{\varepsilon}_{n})$ and $\hat{v}_{n}(x)=v_{\operatorname{\varepsilon}_{n}}(x/\operatorname{\varepsilon}_{n})$. Then $\hat{u}_{n}$ and $\hat{v}_{n}$ are solutions to (1.1) with maximum points $P_{n}:=\operatorname{\varepsilon}_{n}\tilde{y}_{n}+\operatorname{\varepsilon}_{n}p_{n}$ and $Q_{n}:=\operatorname{\varepsilon}_{n}\tilde{y}_{n}+\operatorname{\varepsilon}_{n}q_{n}$ respectively. Since $|p_{n}|,|q_{n}|<R$ for all $n\in\mathbb{N}$ and $\operatorname{\varepsilon}_{n}\tilde{y}_{n}\rightarrow y\in M$ we can infer that $P_{n},Q_{n}\rightarrow y$. By using Lemma 2.1 we obtain

[TABLE]

Finally, we study the decay properties of $(\hat{u}_{n},\hat{v}_{n})$ and we prove that (1.7) holds.

Let us define $\tilde{z}_{n}(x)=\tilde{u}_{n}(x)+\tilde{v}_{n}(x)$. By (6.3) it follows that $\tilde{z}_{n}\rightarrow 0$ as $|x|\rightarrow\infty$ uniformly in $n$. We recall that $(Q2)$ gives

[TABLE]

Then, setting $V_{n}:=V(\operatorname{\varepsilon}_{n}x+\operatorname{\varepsilon}_{n}\tilde{y}_{n})$, $W_{n}:=W(\operatorname{\varepsilon}_{n}x+\operatorname{\varepsilon}_{n}\tilde{y}_{n})$, and using $(H3)$, $\alpha=\min\{V(x_{0}),W(x_{0})\}$, $\sqrt{x^{2}+y^{2}}\leq x+y$ for any $x,y\geq 0$, we can find $R_{1}>0$ sufficiently large such that

[TABLE]

On the other hand, by Lemma $4.3$ in [27], we know that there exists a function $w$ such that

[TABLE]

and

[TABLE]

for some suitable $R_{2}>0$. Take $R_{3}=\max\{R_{1},R_{2}\}$ and we set

[TABLE]

where $b=\sup_{n\in\mathbb{N}}\|\tilde{z}_{n}\|_{L^{\infty}(\mathbb{R}^{N})}<\infty$. In what follows we prove that

[TABLE]

First of all, we can note that

[TABLE]

Now we argue by contradiction and assume that there exists a sequence $\{\bar{x}_{j,n}\}\subset\mathbb{R}^{N}$ such that

[TABLE]

Using (6.3), $\eqref{HZ1}$ and the definition of $\tilde{w}_{n}$, we know that $\tilde{w}_{n}(x)\rightarrow 0$ as $|x|\rightarrow\infty$, uniformly in $n\in\mathbb{N}$. Therefore $\{\bar{x}_{j,n}\}$ is bounded, and, up to subsequence, we may assume that there exists $\bar{x}_{n}\in\mathbb{R}^{N}$ such that $\bar{x}_{j,n}\rightarrow\bar{x}_{n}$ as $j\rightarrow\infty$. Thanks to (6.14) we can see that

[TABLE]

From the minimality of $\bar{x}_{n}$ and the integral representation of the fractional Laplacian [DPV], we can see that

[TABLE]

In view of (6.12) and (6.14), we can infer that $\bar{x}_{n}\in\mathbb{R}^{N}\setminus B_{R_{3}}$. This fact combined with (6.15) and (6.16) yields

[TABLE]

which gives a contradiction due to (6.13). Accordingly, (6.11) holds true, and using (6.8) we have

[TABLE]

for some constant $\tilde{C}>0$. Recalling the definition of $\tilde{z}_{n}$ we can deduce that

[TABLE]

In a similar manner we can obtain the estimate for $\hat{v}_{n}$. This ends the proof of Theorem 1.1. ∎