Semiclassical states of a linearly coupled critical fractional Schr\"{o}dinger system

TL;DR

This paper studies the existence and properties of semiclassical states in a coupled fractional Schr"odinger system, revealing how solutions concentrate and decay as parameters vary, with new results on multiplicity under certain conditions.

Contribution

It establishes existence, decay, and concentration of positive solutions for a coupled fractional Schr"odinger system, including multiplicity results under specific potential assumptions.

Findings

Existence of positive ground states for small epsilon.

Decay estimates and concentration behavior of solutions.

Multiplicity of solutions under additional potential conditions.

Abstract

This paper focuses on the linearly coupled critical fractional Schr\"{o}dinger system \begin{equation*} \begin{cases} \epsilon^{2s}(-\triangle)^s u +a(x)u=u^p+\lambda v\quad &\text{in}\ \mathbb{R}^N,\\ \epsilon^{2s}(-\triangle)^s v +b(x)v=v^{2_s^*-1}+\lambda u\quad &\text{in}\ \mathbb{R}^N, \end{cases} \end{equation*} where $N>2s,$ $s\in(0,1),$ $p\in(1,2_s^*),$ $\epsilon$ and $\lambda$ are positive parameters, $a,b\in C{(\mathbb{R}^N)}$ are positive potentials, and $(-\triangle)^s$ is the fractional Laplacian operator. Under certain assumptions on $a$ and $\lambda,$ we obtain the existence, decay estimates and concentration property of positive vector ground states for small $\epsilon.$ Furthermore, under an additional assumption on potentials $a$ and $b$, we consider the multiplicity of positive vector solutions for small $\epsilon$, which turn out to have similar decay estimate and concentration property to those of the ground state for small $\epsilon$.

Equations

$$\{\begin{array}{ll}{ϵ}^{2s}(-△{)}^{s}u+a(x)u=u^p+\lambda v& \text{in}{\mathbb{R}}^N,\\ {ϵ}^{2s}(-△{)}^{s}v+b(x)v=v^{2_s^{\ast }-1}+\lambda u& \text{in}{\mathbb{R}}^N,\end{array}$$

$$\{\begin{array}{ll}iϵ\frac{\partial \psi }{\partial t}={ϵ}^{2s}(-△{)}^s\psi +P(x)\psi -|\psi {|}^{p-1}\psi -\lambda \varphi & \text{in}\mathbb{R}\times {\mathbb{R}}^N,\\ iϵ\frac{\partial \varphi }{\partial t}={ϵ}^{2s}(-△{)}^s\varphi +Q(x)\varphi -|\varphi {|}^{2_s^{\ast }-2}\varphi -\lambda \psi & \text{in}\mathbb{R}\times {\mathbb{R}}^N.\end{array}$$

$$\{\begin{array}{ll}{ϵ}^{2s}(-△{)}^{s}u+(a(x)-P)u=|u{|}^{p-1}u+\lambda v& \text{in}{\mathbb{R}}^N,\\ {ϵ}^{2s}(-△{)}^{s}v+(b(x)-Q)v=|v{|}^{2_s^{\ast }-2}v+\lambda u& \text{in}{\mathbb{R}}^N.\end{array}$$

$$\mathcal{F}\left((-△{)}^{s}u\right)(\xi )=|\xi {|}^{2s}\mathcal{F}(u)(\xi ),\xi \in {\mathbb{R}}^N,$$

$$(-△{)}^{s}u(x):=C_{N,s}PV{\int }_{{\mathbb{R}}^N}\frac{u(x)-u(y)}{|x-y{|}^{N+2s}}dy=C_{N,s}\underset{ϵ\to 0}{\lim }{\int }_{{\mathbb{R}}^N\setminus B_{ϵ}(x)}\frac{u(x)-u(y)}{|x-y{|}^{N+2s}}dy$$

$$L_s:=\left\{u\in L_{loc}^1({\mathbb{R}}^N):{\int }_{{\mathbb{R}}^N}\frac{|u(x)|}{1+|x{|}^{N+2s}}dx<\infty \right\}.$$

$$(-\Delta {)}^{s}u(x):=\frac{C_{N,s}}{2}{\int }_{{\mathbb{R}}^N}\frac{2u(x)-u(x+y)-u(x-y)}{|y{|}^{N+2s}}dy.$$

$${ϵ}^2(-△{)}^{s}u+V(x)u=f(u)\text{in }{\mathbb{R}}^N,$$

$$\{\begin{array}{ll}{ϵ}^{2s}(-△{)}^{2s}u+P_1(x)u=(|u{|}^{2p}+b|u{|}^{p-1}|v{|}^{p+1})u& \text{in}{\mathbb{R}}^N,\\ {ϵ}^{2s}(-△{)}^{2s}v+P_2(x)v=(|v{|}^{2p}+b|v{|}^{p-1}|u{|}^{p+1})v& \text{in}{\mathbb{R}}^N,\end{array}$$

$$\underset{\Lambda }{\inf }{P}_1(x)<\underset{\partial \Lambda }{\min }{P}_1(x)\text{and }\underset{\Lambda }{\inf }{P}_2(x)<\underset{\partial \Lambda }{\min }{P}_2(x).$$

$$\{\begin{array}{ll}{ϵ}^{2s}(-△{)}^{s}u+V(x)u+uv=K(x)|u{|}^{p-2}u& \text{in}{\mathbb{R}}^3,\\ {ϵ}^{2s}(-△{)}^{s}v=u^2& \text{in}{\mathbb{R}}^3,\end{array}$$

$$\{\begin{array}{ll}-{ϵ}^2△u+a(x)u=u^p+\lambda v& \text{in}{\mathbb{R}}^N,\\ -{ϵ}^2△v+b(x)v=v^{2_s^{\ast }-1}+\lambda u& \text{in}{\mathbb{R}}^N,\end{array}$$

$$\underset{x\in {\mathbb{R}}^N}{\inf }b(x)=b_0,\underset{x\in {\mathbb{R}}^N}{\inf }a(x)=a_0\le {\mu }_0:={\left(\frac{2s(p+1)}{N(p-1)}{S}^{\frac{N}{2s}}{C}_{p+1}^{-\frac{p+1}{p-1}}\right)}^{{\left(\frac{p+1}{p-1}-\frac{N}{2s}\right)}^{-1}},$$

$$\underset{x\in \Lambda }{\inf }a(x)\le {\mu }_0<a_1:=\underset{x\in \partial \Lambda }{\inf }a(x).$$

$$\{\begin{array}{ll}(-△{)}^{s}u+a(x_0)u=u^p+\lambda v& \text{in}{\mathbb{R}}^N,\\ (-△{)}^{s}v+b(x_0)v=v^{2_s^{\ast }-1}+\lambda u& \text{in}{\mathbb{R}}^N.\end{array}$$

$$\mathcal{O}=\left\{x\in \Lambda :c_x=c_0:=\underset{y\in \Lambda }{\inf }{c}_y\right\}.$$

$$u_{ϵ}(x_{ϵ})+v_{ϵ}(x_{ϵ})=\underset{x\in {\mathbb{R}}^N}{\max }(u_{ϵ}(x)+v_{ϵ}(x)).$$

$$u_{ϵ}(x)+v_{ϵ}(x)\le \frac{C{ϵ}^{N+2s}}{{ϵ}^{N+2s}+|x-ϵx_{ϵ}{|}^{N+2s}}\text{for any }x\in {\mathbb{R}}^N.$$

$$\left\{\begin{array}[]{ll}{\rm div}(y^{1-2s}\nabla U)=0&\text{in~{}}\mathbb{R}^{N}\times[0,\infty),\\ U(x,0)=u(x).\end{array}\right.$$

$$(-\Delta {)}^{s}u(x)=-k_s\underset{y\to 0^{+}}{\lim }{y}^{1-2s}\frac{\partial U(x,y)}{\partial y},x\in {\mathbb{R}}^N,$$

$$U(x,y)=P_y^s\ast u(x)=C_{N,s}{\int }_{{\mathbb{R}}^N}\frac{y^{2s}}{(|x-\xi {|}^2+y^2{)}^{\frac{N+2s}{2}}}u(\xi )d\xi ,(x,y)\in {\mathbb{R}}_{+}^{N+1},$$

$$||u|{|}_{\dot{H}}^2:=C_{N,s}{\int }_{{\mathbb{R}}^{2N}}\frac{(u(x)-u(y){)}^2}{|x-y{|}^{N+2s}}dxdy.$$

$$||U|{|}_{{\chi }_s}^2:=k_s{\int }_{{\mathbb{R}}_{+}^{N+1}}{y}^{1-2s}|\nabla U{|}^{2}dxdy.$$

$$\mathcal{E}:=\left\{U\in {\chi }_s:{\int }_{{\mathbb{R}}^N}|u(x){|}^{2}dx<\infty \right\}$$

$$||U|{|}_{\mathcal{E}}^2:=||U|{|}_{{\chi }_s}^2+||u|{|}_2^2.$$

$$S||u|{|}_{2_s^{\ast }}^2\le ||U|{|}_{{\chi }_s}^2\text{for any }U\in {\chi }_s.$$

$$w_{ϵ}=\frac{{ϵ}^{\frac{N-2s}{2}}}{({ϵ}^2+|x{|}^2{)}^{\frac{N-2s}{2}}},ϵ>0.$$

$${\left({\int }_{{\mathbb{R}}_{+}^{N+1}}{y}^{1-2s}|U{|}^{2\gamma }\right)}^{1/2\gamma }\le \hat{C}{\left({\int }_{{\mathbb{R}}_{+}^{N+1}}{y}^{1-2s}|\nabla U{|}^{2}dxdy\right)}^{1/2}.$$

$$||U|{|}_{L^2\left(y^{1-2s},{\mathbb{R}}_{+}^{N+1}\right)}^2:={\int }_{{\mathbb{R}}_{+}^{N+1}}{y}^{1-2s}{U}^{2}dxdy.$$

Full text

[TABLE]

where $N>2s,$ $s\in(0,1),$ $p\in(1,2_{s}^{*}),$ $\epsilon$ and $\lambda$ are positive parameters, $a,b\in C{(\mathbb{R}^{N})}$ are positive potentials, and $(-\triangle)^{s}$ is the fractional Laplacian operator. Under certain assumptions on $a$ and $\lambda,$ we obtain the existence, decay estimates and concentration property of positive vector ground states for small $\epsilon.$ Furthermore, under an additional assumption on potentials $a$ and $b$, we consider the multiplicity of positive vector solutions for small $\epsilon$, which turn out to have similar decay estimate and concentration property to those of the ground state for small $\epsilon$.

Keywords: Fractional Schrödinger system; critical nonlinearities; positive vector solutions; concentration property; decay estimates.

1 Introduction and main results

In this paper, we consider the existence, multiplicity and concentration property of positive vector solutions for the following linearly coupled fractional Schrödinger system

[TABLE]

where $N>2s,$ $s\in(0,1),$ $p\in(1,2_{s}^{*}),$ $2_{s}^{*}=2N/(N-2s),$ $\epsilon,\ \lambda>0$ are parameters, $(-\triangle)^{s}$ is the fractional Laplacian operator, and $a,b\in C{(\mathbb{R}^{N})}$ are positive potentials. System (1.1) arises in the study of time-dependent nonlinear Schrödinger system

[TABLE]

In fact, if $(\psi,\phi)$ is a standing wave of system (1.2), that is, a solution with the form $(\psi(t,x),\phi(t,x))$ $=(e^{-iEt/\epsilon}u(x),e^{-iEt/\epsilon}u(x))$ satisfying $u(x)\to 0$ and $v(x)\to 0$ as $|x|\to\infty$. Then $(u,v)$ is a solution of system

[TABLE]

For sufficiently small $\epsilon>0$, the standing wave is referred to as the semiclassical state. A solution $(u,v)$ of (1.1) is called a positive vector solution if $u>0$ and $v>0$ in $\mathbb{R}^{N}.$

The fractional Laplacian operator $(-\triangle)^{s}$ arises in many fields such as phase transitions, flame propagation, stratified materials and others, see [4, 9, 37] and references therein. In particular, it can be understood as the infinitesimal generator of a stable Levy process (see [40]). There are various equivalent definitions of the fractional Laplacian operator [32]. In particular, if $u$ belongs to the Schwartz class, then $(-\triangle)^{s}u$ can be defined as

[TABLE]

where $\mathcal{F}$ denotes the Fourier transform. The fractional Laplacian can also be defined by the singular integral

[TABLE]

for any real number $0<s<1$, where PV stands for the Cauchy principal value and $C_{N,s}$ is the normalized constant. If $u\in C_{\text{loc}}^{1.1}(\mathbb{R}^{N})\cap L_{s},$ then the integral in (1.3) converges (see, e.g., [19, 20]), and hence $(-\Delta)^{s}$ is well-defined for such $u$. Here

[TABLE]

Alternatively, it can be expressed without using the Cauchy principal value as

[TABLE]

The nonlinear Schrödinger equations and systems have attracted a great deal of attentions. In particular, there has been a great interest in the study of standing waves. For the local cases (s=1), there are many significant references, we refer to [10, 11, 12, 13, 16, 22, 23, 33, 34, 35] and references therein.

In recent years, an ever-growing interest has been devoted to consider Schrödinger equations and systems involving in the nonlocal operator. However, the study of these problems becomes much more complicated since the nonlocal character of operators causes some essential difficulties. For the fractional Schrödinger equation

[TABLE]

Fall, Mahmoudi and Valdinoci [27] showed that the concentration points must be critical points of $V$ under some suitable assumptions. Moreover, if the potential $V$ is coercive and has a unique global minimum, then ground states have the concentration property as $\epsilon$ tends to zero. In addition, if the potential $V$ is radial, then the minimizer is unique for small $\epsilon$. Dávila, del Pino and Wei [24] proved the existence of positive solutions which have multiple spikes near given topologically nontrivial critical points of $V$ or cluster near a given local maximum point of $V$ by the Lyapunov-Schmidt reduction method. Alves and Miyagaki [3] studied the existence and concentration property of positive solutions via penalization method developed in [22] for $f$ with subcritical growth. He and Zou [30] considered the existence, multiplicity and concentration property of positive solutions of (1.4) with critical nonlinearities. For more results concerning the existence and concentration property for fractional Schrödinger equations, we refer to [7, 6, 8, 29, 38] and references therein.

However, only few results are known in the literature on the study of concentration property of standing waves even for the subcritical case when the fractional Schödinger systems are incorporated into consideration. Guo and He [31] considered the existence and concentration property of ground states for the following weakly coupled fractional Schrödinger system with subcritical nonlinearities

[TABLE]

where $2p+2<2_{s}^{*},$ potentials $P_{i}\ (i=1,2)$ are continuous and have the positive global infimum, and there exists a smooth bounded open set $\Lambda\subset\mathbb{R}^{N}$ such that

[TABLE]

Yu, Zhao and Zhao [41] investigated the subcritical fractional Schrödinger-Poisson system

[TABLE]

where $3/4<s<1,$ $4<p<2_{s}^{*},$ potential $V\in C(\mathbb{R}^{3})\cap L^{\infty}(\mathbb{R}^{3})$ has a positive global minimum, and $K\in C(\mathbb{R}^{3})\cap L^{\infty}(\mathbb{R}^{3})$ is positive and has global maximum. The authors proved the existence of positive vector ground state by using variational methods for each $\epsilon>0$ sufficiently small, and determined a concrete set related to the potentials $V$ and $K$ as the concentration position of these ground state solutions as $\epsilon\to 0.$ For more results on the existence of solutions for fractional Schrödinger systems, we can refer to [5, 39, 42] and references therein.

Inspired by the works mentioned above, the current paper is devoted to the study of semiclassical states of the nonlocal critical Schrödinger system (1.1) by using the variational methods, Ljusternik-Schnirelmann theory and penalization approach. Clearly, if $s=1,$ then system (1.1) becomes the local Schrödinger system

[TABLE]

which has been investigated in [1, 2, 14, 15]. In particular, the authors in [14] considered the existence and nonexistence of positive vector solutions of (1.7) in autonomous case with $\epsilon=1$ by the Nehari manifold approach and blow up analysis. Concretely, the authors showed that the radial positive vector ground state $(u_{n},v_{n})$ of the subcritical system obtained by replacing $2_{s}^{*}-1$ by $2_{s}^{*}-1-\tau_{n}$ with $\tau_{n}\to 0$ as $n\to\infty$ approximates to a radial positive vector ground state of (1.7) if the ground state energy is less than $\frac{1}{N}\widetilde{S}^{\frac{N}{2}},$ where $\widetilde{S}$ denotes the sharp embedding constant from $H^{1}(\mathbb{R}^{N})$ to $L^{2^{*}}(\mathbb{R}^{N}).$ Based on the results established in [14], the authors in [15] further considered the concentration property of the ground state for (1.7) using the penalization method. Naturally, we except to investigate the existence, multiplicity and concentration property of positive vector solutions for the nonlocal case (1.1). Before stating our main results, we first give some assumptions on the potential $a$ and $b$ as follows.

• (P1)

There exist positive constants $a_{0}$ and $b_{0}$ such that

[TABLE]

where $S$ and $C_{p+1}$ are the sharp embedding constants from $\chi_{s}$ to $L^{2_{s}^{*}}(\mathbb{R}^{N})$ and $\mathcal{E}$ to $L^{p+1}(\mathbb{R}^{N})$ defined in (2.3) and (3.5) below, respectively .

• (P2)

There is a smooth bounded open domain $\Lambda\subset\mathbb{R}^{N}$ such that

[TABLE]

Noting that we only assume some local conditions (P1)-(P2) on $a$ and $b$ rather than assumptions in [41], where the authors posed the global boundedness to the potential. Moreover, compared with assumption (1.6) concerning both potentials $P_{1}$ and $P_{2}$ for system (1.5), in the present paper, we only need assumption (P2) on the potential $a$, which is not involved in the potential $b$.

As usual, if the ground state of (1.1) exists and has concentration property, we except that it converges to a ground state of a autonomous system. Consequently, for any fixed $x_{0}\in\mathbb{R}^{N},$ we first consider the system

[TABLE]

Theorem 1.1**.**

Let $N>2s,$ $s\in(0,1)$ and $p\in(1,2_{s}^{*}).$ Assume $\lambda<\sqrt{a(x_{0})b_{0}(x_{0})}$ and $a(x_{0})\leq\mu_{0},$ where $\mu_{0}$ is defined in $(P1).$ Then system (1.8) admits a positive vector ground state.

Remark 1.2**.**

If $p=2_{s}^{*}-1$ and $\lambda<\sqrt{a_{0}b_{0}},$ then there is no positive vector solution to (1.8) for any $x_{0}\in\mathbb{R}^{N}$ by the Pohozaev identity. Therefore, we always assume $p<2_{s}^{*}-1$ in this work.

In the present paper, we investigate the existence of positive vector ground state for (1.8) using the extension methods, mountain pass theorem and Nehari manifold approach, which is very different to that of [14] for local case. Indeed, in [14], the authors showed that the radial positive vector ground state $(u_{n},v_{n})$ of the subcritical system obtained by replacing $2_{s}^{*}-1$ by $2_{s}^{*}-1-\tau_{n}$ with $\tau_{n}\to 0$ as $n\to\infty$ is bounded uniformly in $H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N}).$ As a result, there exists a subsequence which converges weakly to some $(u,v)\in H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N}).$ Then the authors established the uniform $L^{\infty}(\mathbb{R}^{N})$ estimates of $(u_{n},v_{n})$ by blow up analysis when the ground state energy is less than $\frac{1}{N}\widetilde{S}^{\frac{N}{2}}$, which along with the radial character of $(u_{n},v_{n})$ concludes that $(u_{n},v_{n})$ converges strongly to $(u,v)\in H^{1}(\mathbb{R}^{N})\times H^{1}(\mathbb{R}^{N}).$ Consequently, $(u,v)$ is a positive vector solution of autonomous case of (1.7) with $\epsilon=1.$

For the scalar equation (1.4), as mentioned above, Fall, Mahmoudi and Valdinoci [27] showed that the concentration positions must be critical points of $V$ under suitable assumptions. While for systems, the positions of concentration points become more complicated. For any $x\in\mathbb{R}^{N},$ let $c_{x}$ be the ground state energy of (1.8) with $x_{0}$ replaced by $x.$ Furthermore, under assumption (P2), we define

[TABLE]

Theorem 1.3**.**

Assume that $(P1)$ and $(P2)$ hold. Let $N>2s,$ $s\in(0,1)$ and $p\in(1,2_{s}^{*})$. If $\lambda<\min\{\sqrt{a_{0}b_{0}},$ $\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Then $\mathcal{O}\neq\emptyset,$ $\mathcal{O}\subset\subset\Lambda$ and there exists $\epsilon_{0}>0$ such that for any $\epsilon\in(0,\epsilon_{0}),$ system (1.1) has a positive vector ground state $(u_{\epsilon},v_{\epsilon})$ with the following properties:

• (I)

There exist $x_{\epsilon}\in\Lambda$ and $x_{0}\in\mathcal{O}$ such that

[TABLE]

Moreover, $x_{\epsilon}\to x_{0}$ as $\epsilon\to 0.$

• (II)

Define $(\widetilde{u}_{\epsilon}(x),\widetilde{v}_{\epsilon}(x))=(u_{\epsilon}(\epsilon x+x_{\epsilon}),v_{\epsilon}(\epsilon x+x_{\epsilon}))$ for any $\epsilon\in(0,\epsilon_{0}).$ Then, up to a subsequence, $(\widetilde{u}_{\epsilon},\widetilde{v}_{\epsilon})$ converges as $\epsilon\to 0$ to a ground state of (1.8) with $x_{0}$ given in (I).

• (III)

There exists a positive constant $C$ independent of $\epsilon$ such that

[TABLE]

Noting that Theorem 1.3 focuses on the existence, concentration property and decay estimate of the positive vector ground state. Naturally, we want to ask weather other positive vector solutions exist or not for (1.1), and if exist, weather they have the same properties to those of the ground state obtained in Theorem 1.3.

Theorem 1.4**.**

Under all assumptions of Theorem 1.3, if we suppose in addition that

• (P3)

there exists $x_{0}\in\Lambda$ such that $a(x_{0})=a_{0}$ and $b(x_{0})=b_{0}.$

Then for any $\delta>0$ such that $\mathcal{O}_{\delta}:=\{x\in\mathbb{R}^{N}:dist(x,\mathcal{O})\leq\delta\}\subset\subset\Lambda,$ there exists $\epsilon_{\delta}>0$ such that system (1.1) admits at least $cat_{\mathcal{O}_{\delta}}(\mathcal{O})$ positive vector solutions for any $\epsilon\in(0,\epsilon_{\delta})$. Moreover, the properties (I)-(III) in Theorem 1.3 also hold for these solutions.

We obtain the polynomial decay results for the positive vector solutions of system (1.1) instead of the exponential decay in the local case. Moreover, we conclude not only the existence, concentration property and decay estimate of the positive vector ground state, but also the multiplicity of positive vector solutions and similar properties to those of the ground state in Theorem 1.4.

We would like to mention here that, there are some essential difficulties in dealing with our system (1.1). The first one of the main difficulties arises in the nonlocal character of the operator $(-\triangle)^{s}.$ One useful method to study the fractional Laplacian is the integral equations method, which turns a given fractional Laplacian equation into its equivalent integral equation, and then various properties of the original equation can be obtained by investigating the integral equation, see [18, 19] and references therein. Recently, Chen and Li et al. have developed a direct method to investigate the nonlocal problems, see [17, 19, 20] and references therein. However, these methods do not turn the nonlocal operator into a local one, which makes many traditional methods in studying the local differential operators no longer work. To overcome this difficulty, we use the extension method introduced by Cafarelli and Silvestre [21], which turns nonlocal problems involving the fractional Laplacian into local ones in higher dimensions, and therefore some additional difficulties followed with, for example, the extension functional is not homogeneous and the truncation argument becomes more delicate since we need to take care the trace of the involved functions which defined in an upper half space $\mathbb{R}^{N+1}_{+}.$ The second main difficulty comes from the critical nonlinearities in using variational methods due to the lack of compactness. To overcome this, we will use a version of the concentration compactness principle established in [25]. Another difficulty arises in the linearly coupled terms, which makes our analysis more complicated in establishing various estimates.

The rest of the paper is organized as follows. In section 2, we translate our system into its extension form, and then consider the variational character of a modified extension system. Section 3 is devoted to the study of the ground state for the autonomous system (1.8). In section 4, the multiplicity of the positive vector solutions are concluded for the modified extension system for small $\epsilon$. In section 5, we prove that the positive vector solutions obtained in section 4 for the modified extension system also solve the extension system of the original problem by making a rescaling, and then we complete the proofs of our main theorems.

Notation

• •

We use $S$ and $C_{p+1}$ to stand for the positive constants defined in (2.3) and (3.5) below, respectively.

• •

For any $x\in\mathbb{R}^{N},$ $y\geq 0$ and $r>0,$ the symbol $B_{r}(x)$ denotes the ball centered at $x$ with radius $r$ in $\mathbb{R}^{N},$ and $\mathcal{B}_{r}^{+}((x,y))$ denotes the ball centered at $(x,y)$ with radius $r$ in $\mathbb{R}^{N+1}_{+}=\mathbb{R}^{N}\times\mathbb{(}0,+\infty).$ Particularly, we denote respectively $B_{r}(0)$ and $\mathcal{B}^{+}_{r}((0,0))$ by $B_{r}$ and $\mathcal{B}_{r}^{+}$

• •

For any $U$ belongs to $\chi_{s}$ defined in section 2, we denote its trace by $u.$

• •

We denote the norm in $L^{q}(\mathbb{R}^{N})$ by $||\cdot||_{q}$.

2 The modified extension problem

Since the fractional Laplacian operator $(-\triangle)^{s}$ is nonlocal, many powerful methods for local elliptic equation are not available any more. To overcome this difficulty, we use the extension method developed by Cafarelli and Silvestre in [21]. Concretely, for a function $u:\mathbb{R}^{N}\to\mathbb{R}$, consider its extension $U:\mathbb{R}^{N}\times[0,\infty)\to\mathbb{R}$ that satisfies

[TABLE]

Then there hold

[TABLE]

and

[TABLE]

where $k_{s}$ and $C_{N,s}$ are normalized positive constants (see, e.g., [8, 21, 29]). The extension operator is well defined for any $u\in\dot{H}(\mathbb{R}^{N})$, which is defined as the completion of $C_{c}^{\infty}({\mathbb{R}^{N}})$ under the norm

[TABLE]

We define the weighted Sobolev space $\chi_{s}$ as the completion of $C_{c}^{\infty}(\overline{\mathbb{R}^{N+1}_{+}})$ under the norm

[TABLE]

For any $U\in\chi_{s}$, its trace on $\mathbb{R}^{N}\times\{y=0\}$ is well defined and denoted by $U(\cdot,0)$ or $u$ for simplicity in this paper. Furthermore, we define a Hilbert space $\mathcal{E}$ as

[TABLE]

equipped with the norm

[TABLE]

Next, we state some useful embedding results in [8, 29, 25].

Lemma 2.1**.**

There hold

• (I)

The embedding $\mathcal{E}\hookrightarrow L^{q}(\mathbb{R}^{N})$ for any $q\in[2,2_{s}^{*}]$ is continuous. Moreover, the embedding $\mathcal{E}\hookrightarrow L^{q}_{\text{loc}}(\mathbb{R}^{N})$ for any $q\in[2,2_{s}^{*})$ is compact.

• (II)

There exists a sharp constant $S=S(s,N)$ such that

[TABLE]

Moreover, the equality holds on the family of functions $W_{\epsilon}$ which is the extension of

[TABLE]

• (III)

There exist positive constants $\widehat{C}$ and $\gamma=1+\frac{2}{N-2s}$ such that, for any $U\in\chi_{s},$

[TABLE]

• (IV)

If $\{U_{n}\}$ is a bounded sequence in $\chi_{s},$ then it is pre-compact in $L_{\text{loc}}^{2}(y^{1-2s},\mathbb{R}^{N+1}_{+}),$ where $L^{2}(y^{1-2s},\mathbb{R}^{N+1}_{+})$ denotes the weighted Lebesgue space equipped with the norm

[TABLE]

For more information on the extension method and extension spaces, we refer to [8, 21, 29, 25] and references therein.

Now, let us turn to our system (1.1). Note that if $(\tilde{u},\tilde{v})$ is a solution of (1.1), then by a direct calculation, $(u(x),v(x)):=(\tilde{u}(\epsilon x),\tilde{v}(\epsilon x))$ solves the system

[TABLE]

By the extension method, we can translate system (2.4) into

[TABLE]

Now we define a function space

[TABLE]

where

[TABLE]

[TABLE]

[TABLE]

In what follows, we omit the normalized constant $k_{s}$ for the convenience.

We call $(U,V)\in X_{s,\epsilon}$ a weak solution of (2.5) if and only if for any $(\Phi,\Psi)\in X_{s,\epsilon}$, there holds

[TABLE]

Observe that if $(U,V)\in X_{s,\epsilon}$ is a weak solution of (2.5), then its trace $(u,v)$ is a weak solution of (2.4), and $(U,V)$ is a critical point of the functional $J_{\epsilon}:X_{s,\epsilon}\to\mathbb{R}$ defined as

[TABLE]

Define

[TABLE]

where

[TABLE]

$\delta_{0}$ is defined in (2.19) below, and $s^{+}:=\max\{s,0\}.$ Furthermore, we denote

[TABLE]

where $\Lambda$ is given in (P2). Set

[TABLE]

[TABLE]

and

[TABLE]

Note that if $x\in\Lambda_{\epsilon}$ then

[TABLE]

and

[TABLE]

If $x\notin\Lambda_{\epsilon},$ then for any $s\leq 0,$ there hold $f_{\epsilon}(x,s)=0=F_{\epsilon}(x,s)$ and $g_{\epsilon}(x,s)=0=G_{\epsilon}(x,s).$ For any $0<s\leq\alpha,$ the equalities (2.13) and (2.14) hold. Moreover, for any $s>\alpha$,

[TABLE]

Similarly, by a straightforward calculation we have $G_{\epsilon}(x,s)\leq\frac{1}{2}g_{\epsilon}(x,s)s$ for any $s>0,$ which along with (2.13), (2.14) and (2.15) implies that

[TABLE]

Now, we consider the following modified system

[TABLE]

Noting that if $(U,V)\in X_{s,\epsilon}$ is a weak solution of (2.17) defined by a similar way to (2.6), then $(U,V)$ is a critical point of the functional $I_{\epsilon}:X_{s,\epsilon}\to\mathbb{R}$ defined as

[TABLE]

Recall that $\lambda<\sqrt{a_{0}b_{0}},$ we can fix a positive constant $\delta_{0}=\delta_{0}(\lambda)$ such that

[TABLE]

This along with assumption (P1) implies that for any $\epsilon>0$ and $x\in\mathbb{R}^{N}$, there hold

[TABLE]

Lemma 2.2**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then $I_{\epsilon}$ satisfies the mountain geometry, that is,

• (I)

there exist $r,\tau>0$ independent of $\epsilon$ such that

[TABLE]

• (II)

there is $(U,V)\in X_{s,\epsilon}$ with $||(U,V)||_{X_{s,\epsilon}}>r$ such that $I_{\epsilon}(U,V)<0.$

Proof.

By the definitions of $F_{\epsilon}$ and $G_{\epsilon}$, we have that

[TABLE]

we then by (2.18) and (2.20) deduce that

[TABLE]

where the last inequality follows from Lemma 2.1. This implies that (I) holds for small $r>0.$ Choose a nonnegative function $\Phi\in C_{c}^{\infty}(\overline{\mathbb{R}^{N+1}_{+}})$ with $\emptyset\neq supp~{}\phi\subset\subset\Lambda_{\epsilon},$ then

[TABLE]

Then (II) holds. The proof is complete. ∎

Now, we define

[TABLE]

then $\Gamma\neq\emptyset$ thanks to Lemma 2.2. Furthermore, set

[TABLE]

then it follows from (2.22) that $c_{\epsilon}>\tau>0$ for any $\epsilon>0.$ We define the Nehari manifold [36] associated to $I_{\epsilon}$ by

[TABLE]

Lemma 2.3**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then

• (I)

there exists a positive constant $C_{0}>0$ independent of $\epsilon$ such that

[TABLE]

• (II)

For any $(U,V)\in N_{\epsilon},$ we have $|supp\ u^{+}\cap\Lambda_{\epsilon}|+|supp\ v^{+}\cap\Lambda_{\epsilon}|>0.$

• (III)

$I_{\epsilon}$* is bounded from below on $N_{\epsilon}$ by a positive constant.*

Proof.

For any $(U,V)\in N_{\epsilon},$ in view of $I^{\prime}_{\epsilon}(U,V)(U,V)=0$, we have

[TABLE]

It then follows from (2.20),(2.21) and Lemma 2.1 that

[TABLE]

Consequently, there exists a positive constant $C_{0}>0$ independent of $\epsilon$ such that

[TABLE]

Assume to the contrary that (II) is not true, then there exists $(U,V)\in N_{\epsilon}$ such that

[TABLE]

This along with $I^{\prime}_{\epsilon}(U,V)(U,V)=0,$ (2.10), (2.11), (2.20) and the choice of $\alpha$ shows that

[TABLE]

Then we conclude that $||(U,V)||_{X_{s,\epsilon}}^{2}=0,$ which contradicts (I). Therefore (II) holds.

For any $(U,V)\in N_{\epsilon},$ it follows from $I_{\epsilon}^{\prime}(U,V)(U,V)=0$ and (2.16) that

[TABLE]

We then from (II) obtain that $I_{\epsilon}(U,V)>0$ for any $(U,V)\in N_{\epsilon}.$ Now we suppose to the contrary that there exists a sequence $\{(U_{n},V_{n})\}\subset N_{\epsilon}$ such that $I_{\epsilon}(U_{n},V_{n})\to 0$ as $n\to\infty,$ then it follows from (2.24) that

[TABLE]

Combining with (2.18) (2.10) and (2.11), we have

[TABLE]

Hence $||(U_{n},V_{n})||_{X_{s,\epsilon}}\to 0$ as $n\to\infty,$ which contradicts (I). The proof is complete. ∎

Since the trace $(u,v)$ of any solution $(U,V)$ of (2.17) satisfies (II) in Lemma 2.3, we define

[TABLE]

and $S_{s,\epsilon}^{+}$ as the intersect of $X_{s,\epsilon}^{+}$ with the unit sphere $S_{s,\epsilon},$ namely, $S_{s,\epsilon}^{+}=S_{s,\epsilon}\cap X_{s,\epsilon}^{+}.$

Lemma 2.4**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then

• (I)

for any $(U,V)\in X_{s,\epsilon}^{+},$ if we define $h_{UV}(t)=I_{\epsilon}(tU,tV),$ then $h_{UV}\in C^{1}(0,\infty)$ and there exists a unique $t_{UV}>0$ such that

[TABLE]

Moreover, $h_{UV}^{\prime}(t)>0$ for any $t\in(0,t_{UV}),$ and $h_{UV}^{\prime}(t)<0$ for any $t\in(t_{UV},\infty).$

• (II)

$(t_{UV}U,t_{UV}V)\in N_{\epsilon}.$* Moreover if $\mathcal{W}\subset X_{s,\epsilon}^{+}$ is bounded and there is $\theta>0$ such that $||(U,V)||_{X_{s,\epsilon}}>\theta$ for any $(U,V)\in\mathcal{W}$ , then there exists a positive constant $c_{\mathcal{W}}$ independent of $\epsilon$ such that $t_{UV}\geq c_{\mathcal{W}}>0.$ In addition, if $\mathcal{W}$ is compact, then there is a positive constant $C_{\mathcal{W}}$ such that $t_{UV}\leq C_{\mathcal{W}}.$*

• (III)

*Set $\tau_{\epsilon}:(U,V)\mapsto(t_{UV}U,t_{UV}V),$ then $\tau_{\epsilon}$ is continuous from $X_{s,\epsilon}^{+}$ to $N_{\epsilon}$, and *$\overline{\tau_{\epsilon}}:=\tau_{\epsilon}|_{S_{s,\epsilon}^{+}}$is a homomorphism between $S_{s,\epsilon}^{+}$ and $N_{\epsilon}.$

Proof.

For any $(U,V)\in X_{s,\epsilon}^{+},$ by the definition of $h_{UV}(t)$ and straight computations, we have

[TABLE]

which implies that $h_{UV}\in C^{1}(0,\infty)$ and $h^{\prime}_{UV}(t)>0$ for small $t>0.$ On the other hand,

[TABLE]

we further from (2.25) and $h_{UV}(0)=0$ conclude that there exists $t_{UV}>0$ such that

[TABLE]

To prove (I), we only need to prove that the equation $h^{\prime}_{UV}(t)=0$ has a unique solution in $(0,\infty).$ In terms of $h^{\prime}_{UV}(t)=0,$ we have

[TABLE]

It follows from the definitions of $f_{\epsilon}$ and $g_{\epsilon}$ that $f_{\epsilon}(x,s)/s$ and $g_{\epsilon}(x,s)/s$ are nondecreasing with respect to $s$ in $(0,\infty)$ for any fixed $x\in\mathbb{R}^{N}.$ Moreover, if $x\in\Lambda_{\epsilon}$ then $f_{\epsilon}(x,s)/s$ and $g_{\epsilon}(x,s)/s$ are strictly increasing with respect to $s$ in $(0,\infty).$ As a result, there exists a unique $t_{UV}>0$ such that (2.26) holds. Namely, we have shown (I).

In view of the equality in (2.25) and a direct calculation, it follows that

[TABLE]

that is, $(t_{UV}U,t_{UV}V)\in N_{\epsilon}.$ Let $(U,V)\in\mathcal{W},$ then by a similar estimate to (2.25), we have

[TABLE]

As a result, there exists a positive constant $c_{\mathcal{W}}$ only depending on the uniformly upper and below bounds of the norm of elements in $\mathcal{W}$ such that $t_{UV}\geq c_{\mathcal{W}}>0.$ Assume that there exists a sequence $\{(U_{n},V_{n})\}\subset\mathcal{W}$ such that $t_{U_{n}V_{n}}\to\infty$ as $n\to\infty.$ We denote $t_{U_{n}V_{n}}$ by $t_{n}$ for convenience. Since $\mathcal{W}$ is compact, there exist a subsequence denoted still by $\{(U_{n},V_{n})\}$ and positive constants $c_{1},c_{2}$ and $N_{0}$ such that for any $n>N_{0}$, there hold

[TABLE]

We assume without loss of generality that the former one holds. Observing that

[TABLE]

This contradicts (III) in Lemma 2.3. Consequently, (II) holds.

Suppose that there exist $\{(U_{n},V_{n})\}\subset X_{s,\epsilon}^{+}$ and $(U,V)\in X_{s,\epsilon}^{+}$ such that $(U_{n},V_{n})\to(U,V)$ in $X_{s,\epsilon}^{+}.$ To verify the continuity of $\tau_{\epsilon},$ we only need to prove $\mathop{\lim}\limits_{n\to\infty}t_{U_{n}V_{n}}=t_{UV}.$ Thanks to (II), there exists $t_{0}>0$ such that, up to a subsequence, $\mathop{\lim}\limits_{n\to\infty}t_{U_{n}V_{n}}=t_{0}.$ Let $n$ tends to infinity in the equality $I^{\prime}_{\epsilon}(t_{U_{n}V_{n}}U_{n},t_{U_{n}V_{n}}V_{n})(t_{U_{n}V_{n}}U_{n},t_{U_{n}V_{n}}V_{n})=0$, we have

[TABLE]

Hence $h_{UV}^{\prime}(t_{0})=0.$ By virtue of (I), there holds $t_{UV}=t_{0}.$ Therefore, $\tau_{\epsilon}$ is continuous.

Define $\varsigma_{\epsilon}:N_{\epsilon}\to S_{s,\epsilon}^{+}$ as

[TABLE]

Then $\varsigma$ is well defined thanks to (II) in Lemma 2.3, and for any $(U,V)\in S_{s,\epsilon}^{+}$, we have

[TABLE]

Therefore $\overline{\tau_{\epsilon}}^{-1}=\varsigma_{\epsilon}.$ Namely, $\overline{\tau_{\epsilon}}$ is a homomorphism between $S_{s,\epsilon}^{+}$ and $N_{\epsilon}.$ The proof is complete. ∎

Lemma 2.5**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then

[TABLE]

Proof.

The first equality is a direct result of (I) and (II) of Lemma 2.4. We only prove the second one. In fact, for any $(U,V)\in X_{s,\epsilon}^{+}$ there exists $T>t_{UV}$ such that $I_{\epsilon}(TU,TV)<0.$ We define $\gamma(t)=(tTU,tTV)$ for any $t\in[0,1].$ Then $\gamma\in\Gamma$ and

[TABLE]

It then follows that

[TABLE]

On the other hand, note that for any $\gamma\in\Gamma,$ $I^{\prime}_{\epsilon}(\gamma(s))\gamma(s)>0$ if $s$ sufficiently small. Moreover, it follows from $I_{\epsilon}(\gamma(1))<0$ that $\gamma(1)\in X_{s,\epsilon}^{+}$ and $t_{\gamma(1)}<1,$ hence $I_{\epsilon}^{\prime}(\gamma(1))(\gamma(1))<0.$ We then drive that there exists $s_{0}\in(0,1)$ such that $I_{\epsilon}^{\prime}(\gamma(s_{0}))(\gamma(s_{0}))=0.$ Namely, $\gamma(s_{0})\in N_{\epsilon}.$ As a result, we have

[TABLE]

which together with (2.28) completes the proof. ∎

Now we define a function $\Phi_{\epsilon}:X_{s,\epsilon}^{+}\to\mathbb{R}$ as

[TABLE]

where $\tau_{\epsilon}$ is defined in Lemma 2.4. Moreover, set $\overline{\Phi}_{\epsilon}={\Phi_{\epsilon}}|_{S_{s,\epsilon}^{+}}.$ Then by a similar discussion to [36, Corollary 2.3], we have the following results.

Lemma 2.6**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then

• (I)

$\Phi_{\epsilon}\in C^{1}(X_{s,\epsilon}^{+})$* and*

[TABLE]

• (II)

$\overline{\Phi}_{\epsilon}\in C^{1}(S_{s,\epsilon}^{+})$* and*

[TABLE]

where

[TABLE]

• (III)

If $\{(U_{n},V_{n})\}\subset S_{s,\epsilon}^{+}$ is a $(PS)_{d}$ sequence of $\overline{\Phi}_{\epsilon}$, then $\{\overline{\tau}_{\epsilon}(U_{n},V_{n})\}$ is a $(PS)_{d}$ sequence of $I_{\epsilon}.$ Moreover, if $\{(U_{n},V_{n})\}\subset N_{\epsilon}$ is a bounded $(PS)_{d}$ sequence of $I_{\epsilon},$ then $\overline{\tau}_{\epsilon}^{-1}(U_{n},V_{n})$ is a $(PS)_{d}$ sequence of $\overline{\Phi}_{\epsilon}$.

• (IV)

$(U,V)\in S_{s,\epsilon}^{+}$* is a critical point of $\overline{\Phi}_{\epsilon}$ if and only if $\overline{\tau}_{\epsilon}(U,V)$ is a critical point of $I_{\epsilon}.$ Moreover, the critical values coincide and*

[TABLE]

Lemma 2.7**.**

Assume $(P1)$ and $\lambda<\sqrt{a_{0}b_{0}}$. If $\{(U_{n},V_{n})\}\subset X_{s,\epsilon}$ is a $(PS)_{d}$ sequence for $I_{\epsilon}$ with $d>0,$ then

• (I)

there exists $C_{0}>0$ such that $||(U_{n},V_{n})||_{X_{s,\epsilon}}\leq C_{0}.$

• (II)

For any $\theta>0$, there exists $r=r(\theta)$ such that

[TABLE]

Proof.

Let $\{(U_{n},V_{n})\}\subset X_{s,\epsilon}$ be a $(PS)_{d}$ sequence for $I_{\epsilon}$ with $d>0,$ then it follows from (2.10), (2.11), (2.15) and (2.20) that

[TABLE]

Hence there exists $C_{0}>0$ such that $||(U_{n},U_{n})||_{X_{s,\epsilon}}\leq C_{0}.$ Namely, (I) holds.

For any fixed $\epsilon,$ we first choose a constant $r>1$ such that $\Lambda_{\epsilon}\subset\subset B_{\frac{r}{2}}$ and a cut-off function $\eta\in C^{\infty}(\overline{\mathbb{R}^{N+1}_{+}})$ such that $\eta(x,y)=1$ if $(x,y)\in\overline{\mathbb{R}^{N+1}_{+}}\setminus\mathcal{B}_{r}^{+},$ $\eta(x,y)=0$ if $(x,y)\in\mathcal{B}_{\frac{r}{2}}^{+},$ $0\leq\eta(x,y)\leq 1$ and $|\nabla\eta(x,y)|\leq\frac{C}{r}$ with some positive constant $C$ for any $(x,y)\in\overline{\mathbb{R}^{N+1}_{+}}.$ It then follows that $\{(\eta U_{n},\eta V_{n})\}\subset X_{s,\epsilon}$ is bounded due to (I). Hence,

[TABLE]

By a direct calculation, this is equivalent to

[TABLE]

Since $\Lambda_{\epsilon}\subset\subset B_{\frac{r}{2}}$ and $\eta(x,y)=0$ if $(x,y)\in\mathcal{B}^{+}_{\frac{r}{2}},$ we then find

[TABLE]

It then follows from Lemma 2.1, (I) and the definition of $\eta$ that

[TABLE]

Let $r>\frac{2C}{\theta},$ then (II) holds. The proof is complete. ∎

Next, we conclude a compactness for the (PS) sequence of $I_{\epsilon}$. Firstly we state a version of the concentration compactness principle in [25].

Lemma 2.8**.**

Assume that $\{U_{n}\}\subset\chi_{s}$ converges weakly to $U$ in $\chi_{s}.$ Let $\mu,\nu$ be two nonnegative measures on $\mathbb{R}^{N+1}_{+}$ and $\mathbb{R}^{N}$ respectively and such that

[TABLE]

in the sense of measure. If for any $\delta>0$ there exists $\rho>0$ such that

[TABLE]

Then there exist an at most countable set $J$ and three families $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{N},$ $\{\mu_{j}\}_{j\in J}$ and $\{\nu_{j}\}_{j\in J},$ $\mu_{j},\nu_{j}\geq 0$ such that

[TABLE]

Lemma 2.9**.**

Assume $(P1)$ and $\lambda<\sqrt{a_{0}b_{0}}.$ If $\{(U_{n},V_{n})\}\subset X_{s,\epsilon}$ is a $(PS)_{d}$ sequence for $I_{\epsilon}$ with $0<d<\frac{s}{N}S^{\frac{N}{2s}},$ then there exists a convergent subsequence.

Proof.

Let $\{(U_{n},V_{n})\}\subset X_{s,\epsilon}$ be a $(PS)_{d}$ sequence for $I_{\epsilon}$ with $0<d<\frac{s}{N}S^{\frac{N}{2s}},$ then there exists $C_{0}>0$ such that $||(U_{n},V_{n})||_{X_{s,\epsilon}}\leq C_{0}$ by (I) in Lemma 2.7. Consequently, we have a subsequence (denoted still by $\{(U_{n},V_{n})\}$ for convenience) and $(U_{\epsilon},V_{\epsilon})$ such that

[TABLE]

[TABLE]

[TABLE]

It then follows from $I^{\prime}_{\epsilon}(U_{n},V_{n})\to 0$ that $(U_{\epsilon},V_{\epsilon})$ is a critical point of $I_{\epsilon}.$ Thanks to (2.10),(2.11) and (2.31), we have

[TABLE]

By virtue of (II) in Lemma 2.7, Lemma 2.8 and (2.10), we can conclude that for any $\theta>0$ there exist $r=r(\theta)$ and $N_{0}>0$ such that $\Lambda_{\epsilon}\subset\subset B_{r},$ and

[TABLE]

As a consequence, for any $n>N_{0}$, we find that

[TABLE]

that is

[TABLE]

In a similar manner, we obtain that $u_{n}v_{n}\to u_{\epsilon}v_{\epsilon}$ in $L^{1}(\mathbb{R}^{N})$ as $n\to\infty$. It follows from $I^{\prime}(U_{n},V_{n})(U_{n},0)=o_{n}(1)$ that

[TABLE]

By using (II) in Lemma (2.7) and the concentration compactness principle, we readily conclude that there exist an at most countable set $J$ and three families $\{x_{j}\}_{j\in J},$ $\{\mu_{j}\}_{j\in J}$ and $\{\nu_{j}\}_{j\in J}$ with $\mu_{j}\geq 0,$ $\nu_{j}\geq 0$ for any $j\in J$ such that

[TABLE]

If $\nu_{j}=0$ for any $j\in J$, then

[TABLE]

If there is some $j\in J$ such that $\nu_{j}\neq 0,$ then we claim that $x_{j}\notin\Lambda_{\epsilon}.$ Otherwise, there exists $\rho>0$ such that $B_{\rho}(x_{j})\subset\subset\Lambda.$ Let $\eta\in C_{c}^{\infty}(\overline{\mathbb{R}^{N+1}_{+}})$ be a cut-off function such that $\eta(x,y)=1$ in $\mathcal{B}_{\frac{1}{2}}^{+}$, $\eta(x,y)=0$ in $\overline{\mathbb{R}^{N+1}_{+}}\setminus\mathcal{B}_{1}^{+}$ and $0\leq\eta(x,y)\leq 1$ for any $(x,y)\in\overline{\mathbb{R}^{N+1}_{+}}.$ Define

[TABLE]

Owing to $I^{\prime}(U_{n},V_{n})(0,V_{n}\eta_{\rho})=o_{n}(1),$ we obtain

[TABLE]

Noting from (P1) and (2.31) that

[TABLE]

Similarly,

[TABLE]

By virtue of $supp~{}\eta_{\rho}(\cdot,0)\subset B_{\rho}(x_{j})\subset\subset\Lambda_{\epsilon}$ and (2.11),

[TABLE]

Similarly,

[TABLE]

Note that

[TABLE]

Using the Hölder inequality, we deduce

[TABLE]

It follows from Lemma 2.1 that $\mathop{\lim}\limits_{n\to\infty}A_{1}=0.$ From the Hölder inequality again, we have

[TABLE]

In view of $\gamma=1+\frac{2}{N-2s},$ there holds $\frac{(N+2-2s)(\gamma-1)}{2\gamma}-1=0,$ which together with Lemma 2.1 implies that

[TABLE]

We further from (2.35) conclude that

[TABLE]

By passing to the limit as $n\to\infty$ and then $\rho\to 0$ in (2.34), we have $\mu_{j}\leq\nu_{j},$ which combined with $\mu_{j}\geq S\nu_{j}^{\frac{2}{2_{s}^{*}}}$ and $\nu_{j}>0$ further shows that $\nu_{j}>S^{\frac{N}{2s}}.$ Since $0<d<\frac{s}{N}S^{\frac{N}{2s}},$ there exists $N_{0}>0$ such that for any $n>N_{0}$,

[TABLE]

we reach a contradiction. As a consequence, $x_{j}\notin\Lambda_{\epsilon}$ and then

[TABLE]

By a similar discussion to (2.32), we infer

[TABLE]

To this end, it follows from $I^{\prime}_{\epsilon}(U_{n},V_{n})(0,V_{n})\to 0$ that

[TABLE]

which together with (2.33) implies $(U_{n},V_{n})\to(U_{\epsilon},V_{\epsilon})$ in $X_{s,\epsilon}.$ The proof is complete. ∎

Corollary 2.10**.**

Assume $(P1)$ and $\lambda<\sqrt{a_{0}b_{0}}.$ Then the function $\overline{\Phi}_{\epsilon}$ defined by (2.29) satisfies $(PS)_{d}$ condition with $0<d<\frac{s}{N}S^{\frac{N}{2s}}$ on $S_{s,\epsilon}^{+}.$

Proof.

Assume that $\{(U_{n},V_{n})\}\subset S_{s,\epsilon}^{+}$ is a $(PS)_{d}$ sequence with $0<d<\frac{s}{N}S^{\frac{N}{2s}}$ for $\overline{\Phi}_{\epsilon}$, then $\{\overline{\tau}_{\epsilon}(U_{n},V_{n})\}$ is a $(PS)_{d}$ sequence for $I_{\epsilon}$ due to Lemma 2.6. It then follows from Lemma 2.9 that there exist $(U,V)\in X_{s,\epsilon}$ and a subsequence (still denoted by $\{\overline{\tau}_{\epsilon}(U_{n},V_{n})\}$) such that

[TABLE]

We then by(I) and (II) in Lemma 2.4 infer that $(U,V)\in N_{\epsilon}$ and $\overline{\tau}_{\epsilon}^{-1}(U,V)\in S_{s,\epsilon}^{+}.$ Thanks to (2.36) and (III) in Lemma 2.4, we have

[TABLE]

The proof is complete. ∎

Lemma 2.11**.**

Assume $(P1)$ and $\lambda<\sqrt{a_{0}b_{0}}.$ If there exists $x_{0}\in\Lambda$ such that system (1.8) has a positive vector ground state with energy $c_{x_{0}}$ less than $\frac{s}{N}S^{\frac{N}{2s}},$ then

[TABLE]

Moreover, system (2.17) admits a positive vector ground state $(U_{\epsilon},V_{\epsilon})\in X_{s,\epsilon}$.

Proof.

Let $(U,V)$ be a positive vector ground state of (1.8) with $x_{0}\in\Lambda$, then there exists $r>0$ such that $B_{r}(x_{o})\subset\subset\Lambda.$ Take a cut-off function $\eta\in C_{c}^{\infty}(\overline{\mathbb{R}^{N+1}_{+}})$ such that $\eta(x,y)=1$ if $(x,y)\in\mathcal{B}_{\frac{r}{2}}^{+},$ $\eta(x,y)=0$ if $(x,y)\in\overline{\mathbb{R}^{N+1}_{+}}\setminus\mathcal{B}_{r}^{+},$ $1\leq\eta(x,y)\leq 1$ and $|\nabla\eta(x,y)|\leq\frac{C}{r}$ with constant $C>0$ for any $(x,y)\in\overline{\mathbb{R}^{N+1}_{+}}.$ Set

[TABLE]

We omit the subscript $x_{0}$ for convenience. Then there exists $t_{\epsilon}>0$ such that $(t_{\epsilon}\widetilde{U}_{\epsilon},t_{\epsilon}\widetilde{V}_{\epsilon})\in N_{\epsilon}.$ Moreover, by a straight calculation, there exist constants $\theta,\widetilde{C}>0$ independent of $\epsilon$ such that

[TABLE]

Noting that $supp~{}\tilde{u}_{\epsilon}\subset B_{\frac{r}{\epsilon}}(\frac{x_{0}}{\epsilon})\subset\subset\Lambda_{\epsilon}$ and $supp~{}\tilde{v}_{\epsilon}\subset B_{\frac{r}{\epsilon}}(\frac{x_{0}}{\epsilon})\subset\subset\Lambda_{\epsilon},$ we obtain by a change of variables $\bar{x}=x-\frac{x_{0}}{\epsilon}$ that

[TABLE]

Similar arguments to (2.27) further implies that $\{t_{\epsilon}\}$ is bounded. Moreover, if there exists a subsequence denoted still by $\{t_{\epsilon}\}$ such that $t_{\epsilon}\to t_{0}$ as $\epsilon\to 0,$ then $t_{0}>0$ by a similar argument to (2.25). Furthermore, from a straight calculation and changes of variables, we conclude that

[TABLE]

By Lemma 2.1, Hölder inequality and $\gamma=1+\frac{2}{N-2s},$ we have

[TABLE]

and

[TABLE]

Similarly, we obtain that

[TABLE]

and

[TABLE]

It further follows from the dominated convergence theorem that

[TABLE]

As a consequence,

[TABLE]

Now we proceed to show that system (2.17) admits a positive vector ground state. Owing to Lemma 2.2, Lemma 2.9 and the mountain pass theorem, we find that $I_{\epsilon}$ admits a critical point $(U_{\epsilon},V_{\epsilon})\in X_{s,\epsilon}$ such that $I_{\epsilon}(U_{\epsilon},V_{\epsilon})=c_{\epsilon}.$ By virtue of Lemma 2.5, $(U_{\epsilon},V_{\epsilon})$ is a ground state for $I_{\epsilon}.$ Next, we prove that $u_{\epsilon}>0$ and $v_{\epsilon}>0$ in $\mathbb{R}^{N}.$ Observe that $I_{\epsilon}^{\prime}(U_{\epsilon},V_{\epsilon})(-U_{\epsilon}^{-},-V_{\epsilon}^{-})=0$, then

[TABLE]

which along with $\lambda\leq(1-\delta_{0})\sqrt{a_{0}b_{0}}$ and (P1) shows that $||(U_{\epsilon}^{-},V_{\epsilon}^{-})||_{X_{s,\epsilon}}=0.$ As a result, $u_{\epsilon}\geq 0,v_{\epsilon}\geq 0$ in $\mathbb{R}^{N}.$ Assume there exists $\bar{x}\in\mathbb{R}^{N}$ such that $u_{\epsilon}(\bar{x})=0,$ then

[TABLE]

On the other hand,

[TABLE]

Therefore, $(-\triangle)^{s}u_{\epsilon}(\bar{x})=0,$ and then $u_{\epsilon}=0$ in $\mathbb{R}^{N}$. By (2.17), we further see $v_{\epsilon}=0$ in $\mathbb{R}^{N}.$ Thanks to the extension formula (2.1), there holds $(U_{\epsilon},V_{\epsilon})=(0,0).$ This contradicts (I) in Lemma 2.2. The proof is complete. ∎

3 The autonomous system

In this section, we discuss the ground state of the linearly coupled autonomous system (1.8). By the extension method, we translate (1.8) into

[TABLE]

The Euler-Lagrange functional associated to system (3.1) is given by

[TABLE]

To consider system (3.1), we define a Hilbert space

[TABLE]

where

[TABLE]

[TABLE]

[TABLE]

Since we are interested in the positive vector solution, we first discuss the functional $I_{x_{0}}:X_{s,x_{0}}\to\mathbb{R}$ defined as

[TABLE]

It is easy to see that if $I_{x_{0}}$ has a critical point $(U,V)$ satisfying $u\geq 0$ and $v\geq 0$ in $\mathbb{R}^{N},$ then it is a critical point of $J_{x_{0}}$ as well. The Nehari manifold associated to $I_{x_{0}}$ is given by

[TABLE]

Remark 3.1**.**

By a minor modification, we can prove that the results of Lemma 2.2, (I) and (III) in Lemma 2.3 also hold for $I_{x_{0}}$ and $N_{x_{0}}.$ Moreover, For any $(U,V)\in N_{x_{0}},$ we have

[TABLE]

Now we define

[TABLE]

and let $S_{s,x_{0}}^{+}$ be the intersect of $X_{s,x_{0}}^{+}$ with the unit sphere $S_{s,x_{0}},$ namely, $S_{s,x_{0}}^{+}=S_{s,x_{0}}\cap X_{s,x_{0}}^{+}.$

Remark 3.2**.**

The results in Lemmas 2.4, 2.5 and 2.6 hold as well if we replace the subscript $\epsilon$ by $x_{0}$.

Lemma 3.3**.**

If there exist a bounded sequence $\{U_{n}\}\subset\chi_{s}$ and $U\in\chi_{s}$ such that

[TABLE]

Then

[TABLE]

Proof.

In fact, we only need to prove

[TABLE]

Note that

[TABLE]

Since for any $\epsilon>0$, there is $C(\epsilon)>0$ such that

[TABLE]

It then follows that

[TABLE]

As a consequence, we have

[TABLE]

The dominated convergence theorem then implies that

[TABLE]

Since $\{U_{n}\}\subset\chi_{s}$ is bounded, we conclude that

[TABLE]

which along with (3.4) and the definition of $h^{\epsilon}_{n}$ implies that

[TABLE]

Letting $\epsilon\to 0$, we get the result. The proof is complete. ∎

We now state a useful lemma which is a direct result of [30, Lemma 3.3] as follows.

Lemma 3.4**.**

Let $\{(U_{n},V_{n})\}\subset\chi_{s}\times\chi_{s}$ be bounded and satisfy

[TABLE]

Then

[TABLE]

We define a constant

[TABLE]

where the Hilbert space $\mathcal{E}$ is defined in (2.1).

Lemma 3.5**.**

If $p\in[1,2_{s}^{*}-1),$ then there exists a nonnegative function $U\in\mathcal{E}$ such that the constant $C_{p+1}$ defined by (3.5) is achieved at $U$.

Proof.

Noting that

[TABLE]

First, it follows from the interpolation theorem that, for any $U\in\mathcal{E}$ with $||u||_{p+1}=1$,

[TABLE]

where $C$ is independent of $U.$ Hence $C_{p+1}\geq\frac{1}{C}>0.$ Let $\{U_{n}\}\subset\mathcal{E}$ with $||u_{n}||_{p+1}=1$ be such that

[TABLE]

In view of Lemma 3.4, there exist $\{z_{n}\}\subset\mathbb{R}^{N}$ and $\theta>0$ such that

[TABLE]

Now, we define

[TABLE]

Then $||\tilde{u}_{n}||_{p+1}=1$ and

[TABLE]

Therefore, there exists a subsequence (denoted still by $\widetilde{U}_{n}$) and $U\in\mathcal{E}$ such that

[TABLE]

[TABLE]

[TABLE]

As a consequence, we further have

[TABLE]

[TABLE]

We now claim that $||u||_{p+1}=1.$ Define $W_{n}=\widetilde{U}_{n}-U$, then by Lemma 3.3 and (3.5),

[TABLE]

Then $||u||_{p+1}^{p+1}=1$ in terms of (3.6), and furthermore $||U||_{\mathcal{E}}=C_{p+1}$ due to (3.7). Since \big{|}\big{|}|U|\big{|}\big{|}_{\mathcal{E}}=\big{|}\big{|}U\big{|}\big{|}_{\mathcal{E}} and \big{|}\big{|}|u|\big{|}\big{|}_{p+1}=\big{|}\big{|}u\big{|}\big{|}_{p+1}=1, we can assume that $U$ is nonnegative. Define

[TABLE]

Then $||\widehat{U}||_{\mathcal{E}}^{2}=C_{p+1}||\hat{u}||_{p+1}^{2}.$ Moreover, $\widehat{U}$ is a ground state of

[TABLE]

Hence, $||\widehat{U}||_{\mathcal{E}}^{2}=||\hat{u}||_{p+1}^{p+1}=C_{p+1}^{\frac{p+1}{p-1}}.$ As a result, there hold

[TABLE]

∎

Theorem 3.6**.**

Under assumptions $(P1)$, $(P2)$ and $0<\lambda<\min\{\sqrt{a_{0}b_{0}}$, $\sqrt{(a_{1}-\mu_{0})b_{0}}\},$

• (I)

if $x_{0}\in\partial\Lambda$ then $c_{x_{0}}=\frac{s}{N}S^{\frac{N}{2s}}.$

• (II)

if $a(x_{0})\leq\mu_{0}$ for some $x_{0}\in\mathbb{R}^{N},$ then $c_{x_{0}}<\frac{s}{N}S^{\frac{N}{2s}}.$

Proof.

Inspired by some ideas in [14], we fist show $c_{x_{0}}\geq\frac{s}{N}S^{\frac{N}{2s}}$ for any $x_{0}\in\partial\Lambda.$ In fact, for any $(U,V)\in X^{+}_{s,x_{0}},$ it follows from Young inequality and (P1) that

[TABLE]

We consider the following three cases.

Case 1. if $u^{+}=0$ in $\mathbb{R}^{N},$ then

[TABLE]

where $W_{\epsilon}$ is defined in (III) of Lemma 2.1.

Case 2. if $v^{+}=0$ in $\mathbb{R}^{N},$ then

[TABLE]

Let $\beta^{2s}=a(x_{0})-\frac{\lambda^{2}}{b(x_{0})}$, $\gamma=\left(a(x_{0})-\frac{\lambda^{2}}{b(x_{0})}\right)^{\frac{1}{p-1}}$ and $\widetilde{U}(x,y)=\gamma\widehat{U}(\beta x,\beta y),$ where $\widehat{U}$ is defined by (3.8). Then by a direct calculation, we find $\widetilde{U}$ is a ground state of

[TABLE]

As a result, there hold

[TABLE]

Therefore,

[TABLE]

Case 3. If $u^{+}\neq 0,v^{+}\neq 0$ in $\mathbb{R}^{N},$ then there exist $t_{1}>0$ and $t_{2}>0$ such that

[TABLE]

Moreover, if $t_{1}\geq t_{2}$ then $g(t_{2},U)>0.$ As a consequence,

[TABLE]

Similarly, if $t_{1}<t_{2},$ then

[TABLE]

Summarizing the above three cases, we conclude that $c_{x_{0}}\geq\frac{s}{N}S^{\frac{N}{2s}}$ for any $x_{0}\in\partial\Lambda.$

Next, we prove that $c_{x_{0}}\leq\frac{s}{N}S^{\frac{N}{2s}}$ for any $x_{0}\in\mathbb{R}^{N}.$ Let $\phi_{0}\in C^{\infty}([0,\infty))$ be a cut-off function such that $\phi_{0}(t)=1$ if $t\in[0,1/2]$ and $\phi_{0}(t)=1$ if $t>1.$ Define

[TABLE]

and $\psi_{\epsilon}$ as the trace of $\Psi_{\epsilon},$ where $W_{\epsilon}$ is given by (III) in Lemma 2.1.Then we know from the argument of [26] that

[TABLE]

and

[TABLE]

As a consequence,

[TABLE]

that is, $t_{\epsilon}$ is bounded. Therefore, there exist a subsequence (denoted still by $t_{\epsilon}$) and $t_{0}>0$ such that $t_{\epsilon}\to t_{0}$ as $\epsilon\to 0.$ Direct calculations yields that

[TABLE]

As a result,

[TABLE]

Consequently, (I) holds.

We next prove (II). If $a(x_{0})\leq\mu_{0}$ for some $x_{0}\in\mathbb{R}^{N},$ define $\alpha=(a(x_{0}))^{\frac{1}{p-1}},$ $\beta=(a(x_{0}))^{\frac{1}{2s}}$,

[TABLE]

where $\widehat{U}$ is given by (3.8). Then $\widehat{W}$ is a ground state of

[TABLE]

Hence, there hold

[TABLE]

If $a(x_{0})<\mu_{0}$, then it follows from $p<2_{s}^{*},$ (P1)-(P2) and the definition of $\mu_{0}$ that

[TABLE]

If $a(x_{0})=\mu_{0},$ we claim that $c_{x_{0}}<\frac{s}{N}S^{\frac{N}{2s}}.$ Otherwise, we have

[TABLE]

due to (3.10) and (3.12), where $\widehat{W}$ is defined in (3.11). Hence $(\widehat{W},0)$ is a ground state of (3.1), which is impossible. The proof is complete. ∎

Lemma 3.7**.**

Assume $(P1)$ and $\lambda<\sqrt{a_{0}b_{0}}.$ If $\{(U_{n},V_{n})\}\subset X_{s,x_{0}}$ is a $(PS)_{d}$ sequence of $I_{x_{0}}$ with $0<d<\frac{s}{N}S^{\frac{N}{2s}},$ then there exists $\theta>0,$ $r>0$ and $\{z_{n}\}\subset\mathbb{R}^{N}$ such that

[TABLE]

Proof.

Assume to the contrary that

[TABLE]

then Lemma 3.4 implies that

[TABLE]

Moreover, we can cover $\mathbb{R}^{N}$ by a countable number of balls $\{B_{r}(x_{j})\}_{j\in J}$ in a way that each point of $\mathbb{R}^{N}$ is contained in at most $N+1$ balls, hence

[TABLE]

as $n\to\infty.$ It follows from boundedness of $\{(U_{n},V_{n})\}$ and $I_{x_{0}}^{\prime}(U_{n},V_{n})\to 0$ as $n\to\infty$ that

[TABLE]

Assume that

[TABLE]

Then we obtain by $I_{x_{0}}^{\prime}(U_{n},V_{n})(0,V_{n})\to 0$ as $n\to\infty$ that

[TABLE]

Thanks to Lemma 2.1, there is $l\geq Sl^{\frac{2}{2_{s}^{*}}}.$ Namely, either $l=0$ or $l\geq S^{\frac{N}{2s}}.$ If the later one holds, then

[TABLE]

This contradiction confirms that $l=0.$ Namely,

[TABLE]

which along with (3.14) further implies that $(U_{n},V_{n})\to(0,0)$ as $n\to\infty.$ This contradicts $I_{x_{0}}(U_{n},V_{n})\to d>0.$ The proof is complete. ∎

Theorem 3.8**.**

Assume $(P1)$. If $\lambda<\sqrt{a_{0}b_{0}},$ then for any $x_{0}\in\mathbb{R}^{N}$ with $0<c_{x_{0}}<\frac{s}{N}S^{{\frac{N}{2s}}},$ $I_{x_{0}}$ admits a positive vector critical point $(U,V)\in X_{s,x_{0}}$ such that $I_{x_{0}}(U,V)=c_{x_{0}}.$

Proof.

By a minor modification to the proof of Lemma 2.2, we can prove that $I_{x_{0}}$ satisfies the mountain pass geometry. As a consequence, there exists $\{(U_{n},V_{n})\}\subset X_{s,x_{0}}$ such that

[TABLE]

By virtue of Lemma 3.7, there exist $\theta>0,$ $r>0$ and $\{z_{n}\}\subset\mathbb{R}^{N}$ such that

[TABLE]

Now define

[TABLE]

Then $\{(\widetilde{U}_{n},\widetilde{V}_{n})\}$ is a $(PS)_{c_{x_{0}}}$ sequence for $I_{x_{0}}$ as well, and

[TABLE]

It then follows from the boundedness of $(PS)_{c_{x_{0}}}$ sequence that there exist a subsequence denoted still by $\{(\widetilde{U}_{n},\widetilde{V}_{n})\}$ and $(U,V)\in X_{s,x_{0}}$ such that

[TABLE]

[TABLE]

[TABLE]

By a direct calculation, we infer that $(U,V)$ is a critical point of $I_{x_{0}}.$ As a consequence,

[TABLE]

where the second inequality is obtained by $\lambda<\sqrt{a_{0}b_{0}}$ and Fatou’s lemma. Hence $(U,V)$ is a ground state of $I_{x_{0}}.$ By a similar argument to the proof of Theorem 2.11 and (3.15) , we conclude $u>0$ and $v>0$ in $\mathbb{R}^{N}$. The proof is complete. ∎

Lemma 3.9**.**

Under assumptions $(P1)$ and $(P2),$ if $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\},$ then

[TABLE]

Moreover, $c_{x}<\frac{s}{N}S^{\frac{N}{2s}}$ for any $x\in\mathcal{O}.$

Proof.

It follows from Theorem 3.6 and (P2) that $\mathcal{O}\neq\emptyset$ and $c_{x}<\frac{s}{N}S^{\frac{N}{2s}}$ for any $x\in\mathcal{O}.$ To complete the proof, we only need to verify that the function $l:\overline{\Lambda}\to\mathbb{R}$ defined as $l(x)=c_{x}$ is continuous. If $x_{n}\to x_{0}$ with $c_{x_{0}}<\frac{s}{N}S^{\frac{N}{2s}}$ as $n\to\infty,$ then it follows from Theorems 3.8 that there exist $(U_{0},V_{0})\in N_{x_{0}}$ and $t_{n}>0$ such that $I_{x_{0}}(U_{0},V_{0})=c_{x_{0}}$ and $(t_{n}U_{0},t_{n}V_{0})\in N_{x_{n}}.$ By a similar argument to the proof of (II) in Lemma 2.4, we see that $t_{n}$ is bounded. So we can assume that $t_{n}\to t_{0}>0$ as $n\to\infty.$ As a consequence,

[TABLE]

Hence, we can assume $c_{x_{n}}<\frac{s}{N}S^{\frac{N}{2s}}$ for large $n.$ It follows from Theorem 3.8 that $I_{x_{n}}$ has a positive vector ground state $(U_{n},V_{n})$ for large $n.$ By a similar argument to the proof of (I) in lemma 2.7, $\{(U_{n},V_{n})\}$ is bounded in $X_{s,x_{0}}.$ As a result, there exist a subsequence denoted still by $\{(U_{n},V_{n})\}$ and $(U,V)\in X_{s,x_{0}}$ such that

[TABLE]

and $(U,V)$ is a critical of $I_{x_{0}}.$ Moreover, Similar to (3.16), we see

[TABLE]

This together with (3.17) implies that $\mathop{\lim}\limits_{n\to\infty}l(x_{n})=l(x_{0})$ for $x_{0}$ with $c_{x_{0}}<\frac{s}{N}S^{\frac{N}{2s}}.$ Assume $x_{n}\to x_{0}$ with $c_{x_{0}}=\frac{s}{N}S^{\frac{N}{2s}},$ we only need to prove that the limit holds for $\{x_{n}\}$ with $c_{x_{n}}<\frac{s}{N}S^{\frac{N}{2s}}.$ In fact, we conclude by a similar argument to (3.18) that

[TABLE]

The proof is complete. ∎

It follows from a similar argument to (3.16) that the (PS)${}_{c_{x_{0}}}$ sequence for $I_{x_{0}}$ with $x_{0}\in\mathcal{O}$ has a convergent subsequence. Furthermore, we can investigate the compactness for a sequence in $N_{x_{0}},$ which plays a crucial role in the discussion of multiplicity.

Lemma 3.10**.**

Under assumptions $(P1)$, $(P2)$ and $0<\lambda<\min\{\sqrt{a_{0}b_{0}}$, $\sqrt{(a_{1}-\mu_{0})b_{0}}\},$ if $\{(U_{n},V_{n})\}\subset N_{x_{0}}$ be such that $I_{x_{0}}(U_{n},V_{n})\to c_{x_{0}}$ for some $x_{0}\in\mathcal{O},$ then there exists a convergent subsequence.

Proof.

We divide the proof into two steps.

Step 1. We claim that if $\{(U_{n},V_{n})\}\subset S_{s,x_{0}}^{+}$ satisfies dist$((U_{n},V_{n}),\partial S_{s,x_{0}}^{+})\to 0$ as $n\to\infty,$ then $||\overline{\tau}_{x_{0}}(U_{n},V_{n})||\to\infty$ and $I_{x_{0}}(\overline{\tau}_{x_{0}}(U_{n},V_{n}))\to\infty.$ In fact, for any $(U,V)\in\partial S_{s,x_{0}}^{+}$, we have $u(x)\leq 0$, $v(x)\leq 0$ in $\mathbb{R}^{N}.$ As a consequence, $u_{n}^{+}(x)\leq|u_{n}(x)-u(x)|$ for any $n.$ Hence, for any $(U,V)\in\partial S_{s,x_{0}}^{+},$

[TABLE]

Noting that dist$((U_{n},V_{n}),\partial S_{s,x_{0}}^{+})\to 0$, then

[TABLE]

Similarly, we have

[TABLE]

As a result, for any $t>0,$

[TABLE]

Hence $I_{x_{0}}(\overline{\tau}_{x_{0}}(U_{n},V_{n}))\to\infty.$ Thanks to Lemma 2.1, we conclude that if $\{\overline{\tau}_{x_{0}}(U_{n},V_{n})\}$ is bounded, then so is $\{I_{x_{0}}(\overline{\tau}_{x_{0}}(U_{n},V_{n}))\}$. The claim then holds.

Step 2. In view of $I_{x_{0}}(U_{n},V_{n})\to c_{x_{0}},$ we have

[TABLE]

Define $\Upsilon:\overline{S_{s,x_{0}}^{+}}\to\mathbb{R}\cup\{\infty\}$ by

[TABLE]

Then $\Upsilon\in C(\overline{S_{s,x_{0}}^{+}},\mathbb{R}\cup\{\infty\})$ is bounded from below by $c_{x_{0}}$. Therefore, there is a (PS)${}_{c_{x_{0}}}$ sequence $\{(\overline{W}_{n},\overline{Z}_{n})\}\subset S_{s,x_{0}}^{+}$ for $\overline{\Phi}_{x_{0}}$ such that

[TABLE]

Set $(W_{n},Z_{n})=\overline{\tau}_{x_{0}}(\overline{W}_{n},\overline{Z}_{n}),$ then $\{(W_{n},Z_{n})\}$ is a (PS)${}_{c_{x_{0}}}$ sequence of $I_{x_{0}}$ and

[TABLE]

Similar to the proof of Theorem 3.8, the sequence $\{(W_{n},Z_{n})\}$ has a convergent subsequence in $X_{s,x_{0}}$, and then so does $\{(U_{n},V_{n})\}$ in terms of (3.19). The proof is complete. ∎

4 Multiplicity of solutions for the modified system

In this section, we relate the number of positive vector solutions of (2.17) to the topology of the set $\mathcal{O}$. For this aim, we choose $\delta>0$ such that $\mathcal{O}_{\delta}\subset\subset\Lambda.$

Lemma 4.1**.**

Under assumptions $(P1)$ and $(P2),$ if $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\},$ then there holds

[TABLE]

where $\left(\widetilde{U}_{\epsilon,x_{0}},\widetilde{V}_{\epsilon,x_{0}}\right)$ is defined in (2.37), and $\tau_{\epsilon}$ is specified by (III) in Lemma 2.4.

Proof.

It follows from (II) in Lemma 2.4 that there exists $t_{\epsilon}>0$ such that

[TABLE]

By a similar argument to the proof of Lemma 2.11, we see that $t_{\epsilon}$ is bounded. Moreover, if there exists a subsequence denoted still by $\{t_{\epsilon}\}$ such that $t_{\epsilon}\to t_{0}$ as $\epsilon\to 0,$ then $t_{0}>0$. Furthermore, thanks to $I_{\epsilon}^{\prime}\left(t_{\epsilon}\widetilde{U}_{\epsilon,x_{0}},t_{\epsilon}\widetilde{V}_{\epsilon,x_{0}}\right)\left(t_{\epsilon}\widetilde{U}_{\epsilon,x_{0}},t_{\epsilon}\widetilde{V}_{\epsilon,x_{0}}\right)=0$, we conclude by a similar discussion to the proof of (2.38) that

[TABLE]

Namely, $(t_{0}U,t_{0}V)\in N_{x_{0}}.$ Recall that $(U,V)$ is a ground state of $I_{x_{0}}$, then $t_{0}=1$, this along with (2.38) implies that

[TABLE]

By virtue of $I_{\epsilon}^{\prime}\left(t_{\epsilon}\widetilde{U}_{\epsilon,x_{0}},t_{\epsilon}\widetilde{V}_{\epsilon,x_{0}}\right)\left(t_{\epsilon}\widetilde{U}_{\epsilon,x_{0}},t_{\epsilon}\widetilde{V}_{\epsilon,x_{0}}\right)=0$ and a direct calculation similar to that in the proof of Lemma 2.11, we can see the limit holds uniformly for $x_{0}\in\mathcal{O}.$ The proof is complete. ∎

We define $H_{\epsilon}:\mathcal{O}\to N_{\epsilon}$ by

[TABLE]

Then by Lemma 4.1, there holds

[TABLE]

Moreover, $t_{\epsilon}\to 1$ as $\epsilon\to 0.$ For any fixed $z\in\mathcal{O},$ let $h(\epsilon)=\left|I_{\epsilon}\left(\tau_{\epsilon}\left(\widetilde{U}_{\epsilon,z},\widetilde{V}_{\epsilon,z}\right)\right)-c_{0}\right|$, and

[TABLE]

then $h(\epsilon)\to 0$ as $\epsilon\to 0,$ and $\widetilde{N}_{\epsilon}\neq\emptyset$ because of $\tau_{\epsilon}\left(\widetilde{U}_{\epsilon,z},\widetilde{V}_{\epsilon,z}\right)\in\widetilde{N}_{\epsilon}$ for any $z\in\mathcal{O}.$

Next, we show a concentration property for the functions in $\widetilde{N}_{\epsilon}.$ More general, we have

Lemma 4.2**.**

Assume (P1)-(P2) and $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Let sequence $\{(U_{\epsilon},V_{\epsilon})\}$ satisfy $(U_{\epsilon},V_{\epsilon})\in N_{\epsilon}$ for any small $\epsilon$ and $I_{\epsilon}(U_{\epsilon},V_{\epsilon})\to c_{0}$ as $\epsilon\to 0.$ If either $(U_{\epsilon},V_{\epsilon})$ is a critical point of $I_{\epsilon}$ for each small $\epsilon$ or assumption (P3) holds, then

• (I)

there exist a subsequence $\{(U_{\epsilon_{n}},V_{\epsilon_{n}})\}$, a sequence $\{z_{\epsilon_{n}}\}\subset\mathbb{R}^{N}$ and $(\widetilde{U},\widetilde{V})\in\mathcal{E}\times\mathcal{E}$ such that

[TABLE]

where

[TABLE]

• (II)

there exists $z_{0}\in\mathcal{O}$ such that $\epsilon_{n}z_{\epsilon_{n}}\to z_{o}$ as $n\to\infty.$

• (III)

$(\widetilde{U},\widetilde{V})$* is a positive vector ground state of system (1.8) with $x_{0}$ replaced by $z_{0}.$*

Proof.

Note that $\{(U_{\epsilon},V_{\epsilon})\}$ is bounded by assumptions. Then we claim that there exist $\{z_{\epsilon}\}\subset\mathbb{R}^{N},$ $r>0$ and $\theta>0$ such that

[TABLE]

Otherwise, let $l=\mathop{\lim}\limits_{\epsilon\to 0}||(U_{\epsilon}.V_{\epsilon})||^{2}_{X_{s,\epsilon}},$ then it follows from Lemma 3.4 and $(U_{\epsilon}.V_{\epsilon})\in N_{\epsilon}$ that

[TABLE]

which along with Lemmas 2.3 and 2.1 that $l>S^{\frac{N}{2s}}.$ Furthermore,

[TABLE]

we reach a contradiction. Now define

[TABLE]

Then $\{(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})\}$ is bounded in $\mathcal{E}\times\mathcal{E},$ and then there exist a subsequence denoted still by $\{(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})\}$ for convenience and $(\widetilde{U},\widetilde{V})$ such that

[TABLE]

As a consequence,

[TABLE]

We consider two cases in the following.

Case 1. $(U_{\epsilon},V_{\epsilon})$ is a critical point of $I_{\epsilon}$ for each small $\epsilon.$ Then $u_{\epsilon}\geq 0,$ $v_{\epsilon}\geq 0$ in $\mathbb{R}^{N}$ for each small $\epsilon$. We claim that $\mathop{\lim}\limits_{\epsilon\to 0}{\rm dist}(\epsilon z_{\epsilon},\Lambda)=0.$ Otherwise, there exists $\rho>0$ such that ${\rm dist}(\epsilon z_{\epsilon},\Lambda)>\rho$ for small $\epsilon.$ As a result,

[TABLE]

It then follows from (2.17) and (4.5) that for any $(W,Z)\in C_{c}^{\infty}{(\overline{\mathbb{R}^{N+1}_{+}})}\times C_{c}^{\infty}{(\overline{\mathbb{R}^{N+1}_{+}})}$ with $W\geq 0,Z\geq 0$ in $\overline{\mathbb{R}^{N+1}_{+}},$

[TABLE]

Letting $\epsilon\to 0$, then

[TABLE]

By the density of $C_{c}^{\infty}{(\overline{\mathbb{R}^{N+1}_{+}})}$ in $\mathcal{E}$, we further get

[TABLE]

which contradicts (4.7). Hence the claim holds. As a consequence, $\{\epsilon z_{\epsilon}\}$ is bounded. Then there exist a subsequence denoted still by $\{\epsilon z_{\epsilon}\}$ and $z_{0}\in\overline{\Lambda}$ such that

[TABLE]

In view of $(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})\rightharpoonup(\widetilde{U},\widetilde{V})$ in $\mathcal{E}\times\mathcal{E}$, we conclude that $(\widetilde{U},\widetilde{V})$ is a solution of system

[TABLE]

where $\chi(x)=\mathop{\lim}\limits_{\epsilon\to 0}\chi_{\Lambda}(\epsilon x+\epsilon z_{\epsilon})$ and hence, if $z_{0}\in\Lambda$ then $\chi$ is the characteristic function of $\mathbb{R}^{N},$ and if $z_{0}\in\partial\Lambda$ then $\chi$ is the characteristic function of a half-space in $\mathbb{R}^{N}.$ The Euler-Lagrange functional associated to system (4.9) is given by

[TABLE]

It then follows from (2.8) and a similar result to Lemma 2.5 that

[TABLE]

On the other hand, by Fatou’s Lemma,

[TABLE]

It then follows from (4.11) that $c_{z_{0}}=c_{0}=\mathop{\inf}\limits_{y\in\Lambda}c_{y},$ that is, $z_{0}\in\mathcal{O}\subset\subset\Lambda.$ Hence $\chi=1$ in $\mathbb{R}^{N},$ and then $(\widetilde{U},\widetilde{V})$ is a ground state of system (1.8) with $x_{0}$ replaced by $z_{0}.$ Moreover,

[TABLE]

Furthermore, by straight calculations, we see

[TABLE]

On the other hand, we infer from (2.20) that

[TABLE]

The Fatou’s Lemma then implies that

[TABLE]

[TABLE]

Hence $(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})\to(\widetilde{U},\widetilde{V})$ as $\epsilon\to 0$ in $\mathcal{E}\times\mathcal{E}.$

Case 2. There is $x_{0}\in\Lambda$ such that $a(x_{0})=a_{0}$ and $b(x_{0})=b_{0}.$ Then there exists $t_{\epsilon}>0$ such that $(t_{\epsilon}\widetilde{U}_{\epsilon},t_{\epsilon}\widetilde{V}_{\epsilon})\in N_{x_{0}}.$ By (2.10), (2.11) and a change of variables, we have

[TABLE]

and then $\{(t_{\epsilon}\widetilde{U}_{\epsilon},t_{\epsilon}\widetilde{V}_{\epsilon})\}$ is bounded, which along with the boundedness of $\{(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})\}$ implies that there exists a subsequence of $t_{\epsilon}$ (still denoted by $t_{\epsilon}$) such that it converges to $t_{0}>0$. In view of Lemma 3.10 and (4.6), we then derive, up to a subsequence,

[TABLE]

We now claim that $\mathop{\lim}\limits_{\epsilon\to 0}{\rm dist}(\epsilon z_{\epsilon},\Lambda)=0.$ Otherwise, there exists $\rho>0$ such that ${\rm dist}(\epsilon z_{\epsilon},\Lambda)>\rho$ for small $\epsilon.$ As a result,

[TABLE]

Since $I^{\prime}_{\epsilon}({U}_{\epsilon},{V}_{\epsilon})({U}_{\epsilon},{V}_{\epsilon})=0$, we have

[TABLE]

which contradicts (4.7). The claim then holds. As a consequence, there is $z_{0}\in\overline{\Lambda}$ such that, up to a subsequence, $\epsilon z_{\epsilon}\to z_{0}$ as $\epsilon\to 0.$ Now we prove $z_{0}\in\mathcal{O}.$ In fact, it follows from Fatou’s lemma, (4.14), (2.10), (2.11) and $(t_{\epsilon}\widetilde{U}_{\epsilon},t_{\epsilon}\widetilde{V}_{\epsilon})\to(t_{0}\widetilde{U},t_{0}\widetilde{V})$ that

[TABLE]

On the other hand, $c_{0}=I_{x_{0}}(t_{0}\widetilde{U},t_{0}\widetilde{V})$ due to (4.13) and (4.14). Hence $a(z_{0})=a_{0}$ and $b(z_{0})=b_{0}$ by (4.16), that is $z_{0}\in\mathcal{O}.$ It then follows from (4.14) that $(\widetilde{U},\widetilde{V})$ is a ground state of (1.8) with $x_{0}$ replaced by $z_{0}.$ The proof is complete. ∎

Corollary 4.3**.**

Assume (P1)-(P2) and $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Let $(U_{\epsilon},V_{\epsilon})$ be a ground state of $I_{\epsilon},$ then $I_{\epsilon}(U_{\epsilon},V_{\epsilon})\to c_{0}$ as $\epsilon\to 0.$

Proof.

The proof follows directly from (4.11), (4.12), Lemma 2.11 and Theorem 3.6. ∎

Set $\rho=\rho(\delta)$ such that $\mathcal{O}_{\delta}\subset B_{\rho}.$ Define $\chi:\mathbb{R}^{N}\to\mathbb{R}^{N}$ as $\chi(x)=x$ if $x\in B_{\rho}$ and $\chi(x)=\frac{\rho x}{|x|}$ if $|x|>\rho.$ Consider the barycenter map $\beta_{\epsilon}:N_{\epsilon}\to\mathbb{R}^{N},$

[TABLE]

Then by a change of variables $\bar{x}=x-\frac{z}{\epsilon}$ and the dominated convergence theorem, we have

[TABLE]

where $\left(\widetilde{U}_{\epsilon,z},\widetilde{V}_{\epsilon,z}\right)$ is defined in (2.37), and $\tau_{\epsilon}$ is specified by (III) in Lemma 2.4. We further discuss the concentration property of barycenters for the functions in $\widetilde{N}_{\epsilon}$.

Lemma 4.4**.**

Assume (P1)-(P3) and $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Then for any $\delta>0$ such that $\mathcal{O}_{\delta}\subset\subset\Lambda,$ there holds

[TABLE]

Proof.

For any $\epsilon>0$, there exists $(U_{\epsilon},V_{\epsilon})\in\widetilde{N}_{\epsilon}$ such that

[TABLE]

In view of (4.11) and (4.12), we see

[TABLE]

Lemma 4.2 then implies that there exists $\{\tilde{z}_{\epsilon}\}\subset\mathbb{R}^{N}$ such that

[TABLE]

As a consequence, for small $\epsilon,$ we have $z_{\epsilon}\in\mathcal{O}_{\delta}$ and

[TABLE]

The dominated convergence theorem further yields that

[TABLE]

As a consequence,

[TABLE]

as $\epsilon\to 0.$ The proof is complete. ∎

Theorem 4.5**.**

Assume (P1)-(P3) and $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Then for any $\delta>0$ such that $\mathcal{O}_{\delta}\subset\subset\Lambda$ and small $\epsilon$, system (2.17) has at least $cat_{\mathcal{O}_{\delta}}(\mathcal{O})$ positive vector solutions. Moreover, if we denote by $\{(U_{\epsilon},V_{\epsilon})\}$ a sequence of these solutions, then

[TABLE]

Proof.

For any $\epsilon>0,$ define $\pi_{\epsilon}:\mathcal{O}\to S_{\epsilon}^{+}$ by

[TABLE]

where $\overline{\tau}_{\epsilon}$ is defined in Lemma 2.4. Moreover, by (2.29) and (4.1),

[TABLE]

As a consequence, there exists $\hat{\epsilon}>0$ such that for any $\epsilon\in(0,\hat{\epsilon})$,

[TABLE]

Consequently, we have

[TABLE]

On the other hand, for any $z\in\mathcal{O}$ and small $\epsilon$, there exists $\theta(\epsilon,z)\in\mathbb{R}^{N}$ with $|\theta(\epsilon,z)|<\frac{\delta}{2}$ such that $\beta_{\epsilon}(H_{\epsilon}(z))=z+\theta(\epsilon,z)$ thanks to (4.17). Define $Q:[0,1]\times\mathcal{O}\to\mathcal{O}_{\delta}$ by

[TABLE]

then $Q$ is continuous. Obviously, $Q(0,z)=\beta_{\epsilon}(H_{\epsilon}(z)),$ $Q(1,z)=z$ for all $z\in\mathcal{O}.$ Therefore, $Q$ is a homotopy between $\beta_{\epsilon}\circ H_{\epsilon}=\beta_{\epsilon}\circ\tau_{\epsilon}\circ\pi_{\epsilon}$ and the inclusion map $Id:\mathcal{O}\to\mathcal{O}_{\delta}.$ As a result,

[TABLE]

It follows from Corollary 2.10 and category theory (see [36, Corollary 28]) that $\overline{\Phi}_{\epsilon}$ has at least $cat_{\pi_{\epsilon}(\mathcal{O})}\pi_{\epsilon}(\mathcal{O})$ critical points on $\widetilde{S}_{\epsilon}^{+}:=\{(U,V)\in S_{\epsilon}^{+}:\overline{\Phi}_{\epsilon}(U,V)\leq c_{0}+h(\epsilon)\}.$ In view of Lemma 2.6, $I_{\epsilon}$ admits at least $cat_{\mathcal{O}_{\delta}}(\mathcal{O})$ critical points in $\widetilde{N}_{\epsilon}.$ Namely, system (2.17) has at least $cat_{\mathcal{O}_{\delta}}(\mathcal{O})$ solutions. By a similar argument to the proof of Theorem 2.11 and (3.15), we conclude that the components of these solutions are positive. The proof is complete. ∎

5 Proofs of main theorems

This section is devoted to the proofs of main theorems. To prove the existence and multiplicity of positive vector solutions of system (1.1), we only need to show that the solutions of system (2.17) obtained in Theorem 4.5 and Lemma 2.11 also solve system (2.5) for any small $\epsilon$. Then we further investigate the decay estimate and concentration property of positive vector solutions of system (1.1).

Lemma 5.1**.**

Assume (P1)-(P3) and $\lambda<\min\{\sqrt{a_{0}b_{0}},\sqrt{(a_{1}-\mu_{0})b_{0}}\}.$ Let $({U}_{\epsilon},{V}_{\epsilon})$ be the positive vector solution of system (2.17) obtained in Theorem 4.5 or Lemma 2.11 for any small $\epsilon$. Then there exists a positive constant $C$ independent of $\epsilon$ such that for any $(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})$ defined in (4.3), there holds

[TABLE]

Proof.

Let $\widetilde{W}_{\epsilon}=\widetilde{U}_{\epsilon}+\widetilde{V}_{\epsilon}$ in $\mathbb{R}^{N+1}_{+},$ then it follows from (2.10) and (2.11) that

[TABLE]

where $C$ is a positive constant independent of $\epsilon.$ Furthermore, we can verify from (2.17) and (4.3) that $\widetilde{W}_{\epsilon}$ is a subsolution of

[TABLE]

Then by the Moser’s iteration for scalar equations (see, e.g., [30, Lemma 4.1]), we can complete the proof. ∎

Proofs of Theorem 1.3 and 1.4.

To verify the existence and multiplicity of positive vector solutions of system (1.1), we only need to show that there exists $\epsilon_{0}$ such that any positive vector solution $({U}_{\epsilon},{V}_{\epsilon})$ of system (2.17) obtained in Theorem 4.5 and Lemma 2.11 also solves system (2.5) for any $0<\epsilon<\epsilon_{0}.$ Set $\widetilde{W}_{\epsilon}=\widetilde{U}_{\epsilon}+\widetilde{V}_{\epsilon}$, where $(\widetilde{U}_{\epsilon},\widetilde{V}_{\epsilon})$ is defined in (4.3). Then there exists $\widetilde{W}\in\mathcal{E}$ such that $\widetilde{W}_{\epsilon}\to\widetilde{W}$ as $\epsilon\to 0$ due to Lemma 4.2, and $||\tilde{w}_{\epsilon}||_{\infty}<C$ for some constant $C>0$ independent of $\epsilon.$ Moreover, $\widetilde{W}_{\epsilon}\in C(\overline{\mathbb{R}^{N+1}_{+}})$ by [29, Corollary 2.1], and $\widetilde{W}_{\epsilon}$ is a subsolution of

[TABLE]

It follows from [29, Proposition 2.4] that for any $x\in\mathbb{R}^{N},$

[TABLE]

Since $\widetilde{W}_{\epsilon}\to\widetilde{W}$ in $\mathcal{E},$ for any $\theta>0$ there exist $\epsilon_{0}>0$ and $R>0$ such that for any $0<\epsilon<\epsilon_{0}$,

[TABLE]

[TABLE]

which along with (5.2) implies that

[TABLE]

In other words, there exists $R_{\alpha}>0$ such that $\tilde{w}_{\epsilon}<\alpha$ for any $\epsilon\in(0,\epsilon_{0})$ and $|x|>R_{\alpha}.$ Consequently,

[TABLE]

On the other hand, since $\epsilon z_{\epsilon}\to z_{0}\in\mathcal{O}$ as $\epsilon\to 0,$ there exists $r>0$ such that $B_{r}(\epsilon z_{\epsilon})\subset\subset\Lambda$ for small $\epsilon.$ Namely, $\Lambda_{\epsilon}^{c}\subset\subset\mathbb{R}^{N}\setminus B_{\frac{r}{\epsilon}}(z_{\epsilon})$ for small $\epsilon.$ Let $\epsilon$ be sufficiently small such that $\epsilon R_{\alpha}<r$, then $\Lambda_{\epsilon}^{c}\subset\subset\mathbb{R}^{N}\setminus B_{R_{\alpha}}(z_{\epsilon}).$ It then follows from (5.4) that

[TABLE]

In terms of (2.10) and (2.11), we have

[TABLE]

That is to say, $({U}_{\epsilon},{V}_{\epsilon})$ is a solution of system (2.5).

We claim that $||w_{\epsilon}||_{L^{\infty}(\mathbb{R}^{N})}\geq\alpha$ for small $\epsilon$. Otherwise, by (2.10) and (2.11), we have

[TABLE]

Furthermore, it follows from $I_{\epsilon}^{\prime}(U_{\epsilon},V_{\epsilon})(U_{\epsilon},V_{\epsilon})=0$ and (2.20) that $||(U_{\epsilon},V_{\epsilon})||_{X_{s,\epsilon}}=0,$ which contradicts (I) in Lemma 2.3. The claim holds. Therefore, there exists $x_{\epsilon}\in\Lambda_{\epsilon}$ such that

[TABLE]

By a direct calculation we see that $(\hat{u}_{\epsilon}(x),\hat{v}_{\epsilon}(x)):=({u}_{\epsilon}(\frac{x}{\epsilon}),{v}_{\epsilon}(\frac{x}{\epsilon}))$ in $\mathbb{R}^{N}$ is a positive vector solution of system (1.1). Moreover, $\hat{u}_{\epsilon}+\hat{v}_{\epsilon}$ achieves its maximum at $\epsilon x_{\epsilon}.$ Next, we prove $\epsilon x_{\epsilon}\to z_{0}$ as $\epsilon\to 0.$ In fact, $\epsilon x_{\epsilon}\in B_{\epsilon R_{\alpha}}(\epsilon z_{\epsilon})$ for small $\epsilon$ due to (5.4), which along with Lemma 4.2 implies $\epsilon x_{\epsilon}\to z_{0}$ as $\epsilon\to 0,$ and then the property (II) in Theorems is a direct result of Lemma 4.2.

Thanks to $\lambda<(1-\delta_{0})\sqrt{a_{0}b_{0}},$ we have

[TABLE]

Let $\theta=\frac{1}{2}\left(\frac{\lambda}{(1-\delta_{0})a_{0}}+\frac{(1-\delta_{0})b_{0}}{\lambda}\right)$ and $\overline{W}_{\epsilon}=\theta\widetilde{U}_{\epsilon}+\widetilde{V}_{\epsilon}$. Then

[TABLE]

where $\bar{\theta}=\min\{\delta_{0}a_{0}\theta,\delta_{0}b_{0}\}.$ By virtue of (5.3), there exists $R>0$ such that for any $|x|>R$ and small $\epsilon$ there holds

[TABLE]

which together with (5.6) yields

[TABLE]

Noting from [28, Lemma 4.3] that there is a continuous function $w$ such that

[TABLE]

and $0<w<\frac{C_{2}}{|x|^{N+2s}}$ for some $C_{2}>0.$ Now we set $\phi(x)=w\left(\bar{\theta}^{\frac{1}{2s}}x\right)$ in $\mathbb{R}^{N},$ then

[TABLE]

and $0<\phi<\frac{C^{\prime}}{|x|^{N+2s}}$ with $C^{\prime}>0.$ Define

[TABLE]

and $\psi=\alpha\phi-\beta\overline{w}_{\epsilon}$ in $\mathbb{R}^{N},$ then

[TABLE]

We claim that $\psi\geq 0$ in $\mathbb{R}^{N}.$ Otherwise, there exists $\{x_{n}\}\subset\mathbb{R}^{N}$ such that

[TABLE]

Since $\psi(x)\to 0$ as $|x|\to\infty$, $\{x_{n}\}$ is bounded. Hence there exists $x_{0}\in\mathbb{R}^{N}$ such that, up to a subsequence, $x_{n}\to x_{0}.$ It then follows that

[TABLE]

which along with (5.7) implies that $x_{0}\in\mathbb{R}^{N}\setminus B_{R^{\prime}}.$ On the other hand, we see that

[TABLE]

As a result, we have

[TABLE]

which contradicts (5.7). Therefore $\psi\geq 0$ in $\mathbb{R}^{N}$. In other words, there holds

[TABLE]

for some positive constant $C$ independent of $\epsilon.$ As a consequence,

[TABLE]

where the last inequality is obtained by the fact that $\epsilon z_{\epsilon}\to z_{0}$ and $\epsilon x_{\epsilon}\to z_{0}$ as $\epsilon\to 0.$ The proof is complete. ∎

Acknowledgements

The first author would like to thank the China Scholarship Council of China (201706180064) for financial support during the period of his overseas study and to express his gratitude to the Department of Mathematical Sciences in Yeshiva University for its kind hospitality. The first author is also partially supported by the Fundamental Research Funds for the Central Universities (lzujbky-2017-it53) and the second author is partially supported by the National Natural Science Foundations of China (No.11471147).