Existence and concentration of positive ground state solutions for nonlinear fractional Schr\"odinger-Poisson system with critical growth

TL;DR

This paper proves the existence and concentration behavior of positive ground state solutions for a fractional Schrödinger-Poisson system with critical growth, using variational methods and concentration compactness techniques.

Contribution

It establishes the existence of a family of positive ground state solutions concentrating on minimal and maximal points of potential functions for small parameters.

Findings

Solutions concentrate on minimal points of V(x).

Solutions concentrate on maximal points of K(x) and Q(x).

Existence of solutions for small epsilon with polynomial growth.

Abstract

In this paper, we study the following fractional Schr\"{o}dinger-Poisson system involving competing potential functions \begin{equation*} \left\{ \begin{array}{ll} \varepsilon^{2s}(-\Delta)^su+V(x)u+\phi u=K(x)f(u)+Q(x)|u|^{2_s^{\ast}-2}u, & \hbox{in $\mathbb{R}^3$,} \varepsilon^{2t}(-\Delta)^t\phi=u^2,& \hbox{in $\mathbb{R}^3$,} \end{array} \right. \end{equation*} where $\varepsilon>0$ is a small parameter, $f$ is a function of $C^1$ class, superlinear and subcritical nonlinearity, $2_s^{\ast}=\frac{6}{3-2s}$, $s>\frac{3}{4}$, $t\in(0,1)$, $V(x)$ $K(x)$ and $Q(x)$ are positive continuous function. Under some suitable assumptions on $V$, $K$ and $Q$, we prove that there is a family of positive ground state solutions with polynomial growth for sufficiently small $\varepsilon>0$, of which it is concentrating on the set of minimal points of $V(x)$ and the sets of maximal points of $K(x)$ and $Q(x)$. The methods are based on the Nehari manifold, arguments of Brezis-Nirenberg and concentration compactness of P. L. Lions.

Equations

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=K(x)f(u)+Q(x)|u|^{2_{s}^{\ast}-2}u,&\hbox{in $\mathbb{R}^{3}$,}\\ \varepsilon^{2t}(-\Delta)^{t}\phi=u^{2},&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$${\Theta }_K=\{x\in {\mathbb{R}}^3:K(x)=K_0=\underset{x\in {\mathbb{R}}^3}{\max }K(x)\}{\Theta }_Q=\{x\in {\mathbb{R}}^3:Q(x)=Q_0=\underset{x\in {\mathbb{R}}^3}{\max }Q(x)\}.$$

$$(-\Delta {)}^{s}v(x)=C_{s}P.V.{\int }_{{\mathbb{R}}^3}\frac{v(x)-v(y)}{|x-y{|}^{3+2s}}\mathrm{d}y=C_s\underset{\epsilon \to 0}{\lim }{\int }_{{\mathbb{R}}^3\backslash B_{\epsilon }(x)}\frac{v(x)-v(y)}{|x-y{|}^{3+2s}}\mathrm{d}y$$

$$(-\Delta)^{s}u=\mathcal{F}^{-1}\Big{(}(2\pi|\xi|)^{2s}\mathcal{F}u(\xi)\Big{)},$$

$$\left\{\begin{array}[]{ll}-\varepsilon^{2}\Delta u+V(x)u+\phi u=g(x,u),&\hbox{in $\mathbb{R}^{3}$,}\\ -\varepsilon^{2}\Delta\phi=u^{2},&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(x)u+\phi u=|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u,&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2},&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=f(u),&\hbox{in $\mathbb{R}^{N}$,}\\ \varepsilon^{\theta}(-\Delta)^{\frac{\alpha}{2}}\phi=\gamma_{\alpha}u^{2},&\hbox{in $\mathbb{R}^{N}$,}\end{array}\right.$$

$$-h^2\Delta u+V(x)u=K(x)|u{|}^{p-1}u+Q(x)|u{|}^{q-1}ux\in {\mathbb{R}}^N$$

$$(-\Delta {)}^{\frac{s}{2}}u+V(x)u=K(x)|u{|}^{p-2}u+Q(x)|u{|}^{2_s^{\ast }-2}ux\in {\mathbb{R}}^N$$

$$\underset{\epsilon \to 0}{\lim }V(z_{\epsilon })=V_0,\underset{\epsilon \to 0}{\lim }K(z_{\epsilon })=K_0,\underset{\epsilon \to 0}{\lim }Q(z_{\epsilon })=Q_0,$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}v+V(x_{0})v+\phi v=K(x_{0})f(v)+Q(x_{0})|v|^{2_{s}^{\ast}-2}v,&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=v^{2},&\hbox{in $\mathbb{R}^{3}$;}\end{array}\right.$$

$$u_{\epsilon }\le \frac{C{\epsilon }^{3+2s}}{C_0{\epsilon }^{3+2s}+|x-z_{\epsilon }{|}^{3+2s}}\text{for all}x\in {\mathbb{R}}^3.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(\varepsilon z)u+\phi u=K(\varepsilon z)f(u)+Q(\varepsilon z)|u|^{2_{s}^{\ast}-2}u,&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2},&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$\mathcal{D}^{\alpha,2}(\mathbb{R}^{3})=\Big{\{}u\in L^{2_{\alpha}^{\ast}}(\mathbb{R}^{3})\,\,\Big{|}\,\,|\xi|^{\alpha}\widehat{u}(\xi)\in L^{2}(\mathbb{R}^{3})\Big{\}}$$

$$\|u\|_{\mathcal{D}^{\alpha,2}}=\Big{(}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{\alpha}{2}}u|^{2}\,{\rm d}x\Big{)}^{\frac{1}{2}}=\Big{(}\int_{\mathbb{R}^{3}}|\xi|^{2\alpha}|\widehat{u}(\xi)|^{2}\,{\rm d}\xi\Big{)}^{\frac{1}{2}}$$

$$\mathcal{S}_{\alpha}=\inf_{u\in\mathcal{D}^{\alpha,2}}\frac{\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{\alpha}{2}}u|^{2}\,{\rm d}x}{\Big{(}\int_{\mathbb{R}^{3}}|u(x)|^{2_{\alpha}^{\ast}}\,{\rm d}x\Big{)}^{\frac{2}{2_{\alpha}^{\ast}}}}.$$

$$H^{\alpha}(\mathbb{R}^{3})=\Big{\{}u\in L^{2}(\mathbb{R}^{3})\,\,\Big{|}\,\,\int_{\mathbb{R}^{3}}(|\xi|^{2\alpha}|\widehat{u}(\xi)|^{2}+|\widehat{u}(\xi)|^{2})\,{\rm d}\xi<+\infty\Big{\}}.$$

$$(u,v)={\int }_{{\mathbb{R}}^3}(|\xi {|}^{2\alpha }\hat{u}(\xi )\overline{\hat{v}(\xi )}+\hat{u}(\xi )\overline{\hat{v}(\xi )})\mathrm{d}\xi$$

$$\|u\|_{H^{\alpha}}=\bigg{(}\int_{\mathbb{R}^{3}}(|\xi|^{2\alpha}|\widehat{u}(\xi)|^{2}+|\widehat{u}(\xi)|^{2})\,{\rm d}\xi\bigg{)}^{\frac{1}{2}}.$$

$$\|u\|_{H^{\alpha}}=\bigg{(}\int_{\mathbb{R}^{3}}(|(-\Delta)^{\frac{\alpha}{2}}u(x)|^{2}+|u(x)|^{2})\,{\rm d}x\bigg{)}^{\frac{1}{2}},\quad\forall u\in H^{\alpha}(\mathbb{R}^{3}).$$

$$H^{\alpha}(\mathbb{R}^{3})=\Big{\{}u\in L^{2}(\mathbb{R}^{3})\,\,\Big{|}\,\,\frac{|u(x)-u(y)|}{|x-y|^{\frac{3}{2}+\alpha}}\in L^{2}(\mathbb{R}^{3}\times\mathbb{R}^{3})\Big{\}}$$

$$\|u\|=\bigg{(}\int_{\mathbb{R}^{3}}|u|^{2}\,{\rm d}x+\int_{\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{3+2\alpha}}\,{\rm d}x\,{\rm d}y\bigg{)}^{\frac{1}{2}}.$$

$$\Vert (-\Delta {)}^{\frac{\alpha }{2}}u{\Vert }_2^2={\int }_{{\mathbb{R}}^3}|\xi {|}^{2\alpha }|\hat{u}(\xi ){|}^2\mathrm{d}\xi =\frac{1}{C(\alpha )}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}\frac{|u(x)-u(y){|}^2}{|x-y{|}^{3+2\alpha }}\mathrm{d}x\mathrm{d}y.$$

$$\|u\|_{\varepsilon}=\Big{(}\int_{\mathbb{R}^{3}}(|(-\Delta)^{\frac{s}{2}}u|^{2}+V(\varepsilon x)|u|^{2})\,{\rm d}x\Big{)}^{\frac{1}{2}}.$$

$$f(u)\le \epsilon |u{|}^2+C_{\epsilon }|u{|}^{p-1},F(u)\le \epsilon |u{|}^4+C_{\epsilon }|u{|}^p\text{for all}u\in \mathbb{R}.$$

$$(\frac{1}{4}f(t)t-F(t){)}^{\prime }>0,f^{\prime }(u)u^2-3f(u)u>0,0<4F(u)<f(u)u,\text{for any}u\ne 0.$$

$${\int }_{{\mathbb{R}}^3}(-\Delta {)}^{\frac{t}{2}}{\varphi }_u^t(-\Delta {)}^{\frac{t}{2}}v\mathrm{d}x={\int }_{{\mathbb{R}}^3}{u}^{2}v\mathrm{d}x,\forall v\in {\mathcal{D}}^{t,2}({\mathbb{R}}^3),$$

$$(-\Delta {)}^t{\varphi }_u^t=u^2,x\in {\mathbb{R}}^3$$

$${\varphi }_u^t(x)=c_t{\int }_{{\mathbb{R}}^3}\frac{u^2(y)}{|x-y{|}^{3-2t}}\mathrm{d}y,x\in {\mathbb{R}}^3,c_t={\pi }^{-\frac{3}{2}}{2}^{-2t}\frac{\Gamma (\frac{3-2t}{2})}{\Gamma (t)},$$

Full text

[TABLE]

where $\varepsilon>0$ is a small parameter, $f$ is a function of $C^{1}$ class, superlinear and subcritical nonlinearity, $2_{s}^{\ast}=\frac{6}{3-2s}$, $s>\frac{3}{4}$, $t\in(0,1)$, $V(x)$ $K(x)$ and $Q(x)$ are positive continuous function. Under some suitable assumptions on $V$, $K$ and $Q$, we prove that there is a family of positive ground state solutions with polynomial growth for sufficiently small $\varepsilon>0$, of which it is concentrating on the set of minimal points of $V(x)$ and the sets of maximal points of $K(x)$ and $Q(x)$. The methods are based on the Nehari manifold, arguments of Brezis-Nirenberg and concentration compactness of P. L. Lions.

1. Introduction

In this paper, we are concerned with the existence of ground state solution and its concentration phenomenon for the following fractional Schrödinger-Poisson system

[TABLE]

where $s>\frac{3}{4},t\in(0,1)$, $2_{s}^{\ast}=\frac{6}{3-2s}$, $\varepsilon>0$ is a small parameter. We assume that the potentials $V(x)$, $K(x)$ and $Q(x)$ satisfy the following hypotheses:

$(V)$ $V(x)\in C(\mathbb{R}^{3},\mathbb{R})$ and $0<V_{0}=\inf\limits_{\mathbb{R}^{3}}V(x)<\liminf\limits_{|x|\rightarrow+\infty}V(x)=V_{\infty}<+\infty$.

$(K)$ $K(x)\in C(\mathbb{R}^{3},\mathbb{R})$ and $\lim\limits_{|x|\rightarrow+\infty}K(x)=K_{\infty}\in(0,+\infty)$ and $K(x)\geq K_{\infty}$ for $x\in\mathbb{R}^{3}$.

$(Q_{0})$ $Q(x)\in C(\mathbb{R}^{3},\mathbb{R})$ and $\lim\limits_{|x|\rightarrow+\infty}Q(x)=Q_{\infty}\in(0,+\infty)$ and $Q(x)\geq Q_{\infty}$.

$(Q_{1})$ There exists $\delta>0$ and $\rho_{0}>0$ such that $|Q(x)-Q(x_{0})|\leq\delta|x-x_{0}|^{\alpha}$ for $|x-x_{0}|\leq\rho_{0}$, where $\frac{3-2s}{2}\leq\alpha<\frac{3}{2}$ and $Q(x_{0})=\max_{x\in\mathbb{R}^{3}}Q(x)$.

$(\mathcal{H})$ $\Theta:=\Theta_{V}\cap\Theta_{K}\cap\Theta_{Q}\neq\emptyset$, where $\Theta_{V}=\{x\in\mathbb{R}^{3}:\,\,V(x)=V_{0}=\inf_{x\in\mathbb{R}^{3}}V(x)\}$ and

[TABLE]

Without loss of generality, we may assume that $0\in\Theta$. Here the non-local operator $(-\Delta)^{s}$ ($s\in(0,1)$), which is called fractional Laplace operator, can be defined by

[TABLE]

for $v\in\mathcal{S}(\mathbb{R}^{3})$, where $\mathcal{S}(\mathbb{R}^{3})$ is the Schwartz space of rapidly decaying $C^{\infty}$ function, $B_{\varepsilon}(x)$ denote an open ball of radius $r$ centered at $x$ and the normalization constant C_{s}=\Big{(}\int_{\mathbb{R}^{3}}\frac{1-\cos(\zeta_{1})}{|\zeta|^{3+2s}}\,{\rm d}\zeta\Big{)}^{-1}. For $u\in\mathcal{S}(\mathbb{R}^{3})$, the fractional Laplace operator $(-\Delta)^{s}$ can be expressed as an inverse Fourier transform

[TABLE]

where $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the Fourier transform and inverse transform, respectively.

Formally, system (1.1) consists of a fractional Schrödinger equation coupled with a fractional Poisson equation. It can be regarded as the associated fractional case of the following classical Schrödinger-Poisson system

[TABLE]

It is well known that system (1.2) has a strong physical meaning because it appears in quantum mechanics models (see e.g. [12, 28]) and in semiconductor theory [34]. In particular, systems like (1.2) have been introduced in [8] as a model to describe solitary waves. In (1.2), the first equation is a nonlinear stationary equation (where the nonlinear term simulates the interaction between many particles) that is coupled with a Poisson equation, to be satisfied by $\phi$, meaning that the potential is determined by the charge of the wave function. For this reason, (1.2) is referred to as a nonlinear Schrödinger-Poisson system. In recent years, there has been increasing attention to systems like (1.2) on the existence of positive solutions, ground state solutions, multiple solutions and semiclassical states; see for examples [3, 4, 8, 23, 39, 48, 53] and the references therein.

The other motivation to study the system (1.1) lies in the important roles that fractional equations involving fractional operators play in the problems of Physics, Chemistry and Geometry, and so on. Indeed, fractional operators appear in many problems, such as: fractional quantum mechanics [26, 27], anomalous diffusion [33], financial [15], obstacle problems [41], conformal geometry and minimal surfaces [11]. With the help of the harmonic extension technique developed by Caffarelli and Silvestre [13], the non-local problem can be reduced to a local one, choosing a weighted Sobolev space as the work space, the usual variational methods have been successfully applied to nonlinear problems involving fractional Laplacian, we refer to interesting readers to see the related works [2, 6, 9, 14, 17, 18, 21, 45] and so on. Another power technique is that one directly take the usual fractional Sobolev space as the work space so that the variational approaches can be applied, the related work can be referred to see [9, 16, 17, 19, 20, 40, 43, 44] and so on.

To the best of our knowledge, there are only few recent papers dealing with a similar system like (1.1). For example, in [46], we established the existence of positive ground state solution for a similar system involving a critical Sobolev exponent

[TABLE]

by using the Nehari-Pohozaev manifold combing monotone trick with global compactness Lemma. Using the similar methods, in [47], positive ground state solutions for subcritical problem, i.e., $|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u$ is replaced by $|u|^{p-1}u$ with $p\in(2,2_{s}^{\ast}-1)$, were established when $s=t$. In [50], the existence of infinitely many (but possibly sign changing) solutions by means of the Fountain Theorem under suitable assumptions on nonlinearity term. In [52], the authors studied the existence of radial solutions for system (2.1) with replacing $|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u$ by $f(u)$, where the nonlinearity $f(u)$ verifies the subcritical or critical assumptions of Berestycki-Lions type. In [35], the authors studied the semiclassical state of the following system

[TABLE]

where $s\in(0,1)$, $\alpha\in(0,N)$, $\theta\in(0,\alpha)$, $N\in(2s,2s+\alpha)$, $\gamma_{\alpha}$ is a positive constant, $f(u)$ satisfies the following subcritical growth assumptions: $0<KF(t)\leq f(t)t$ with some $K>4$ for all $t\geq 0$ and $\frac{f(t)}{t^{3}}$ is strictly increasing on $(0,+\infty)$. By adapting some ideas of Benci, Cerami and Passaseo [5, 7] and using the Ljusternick-Schnirelmann Theory, the authors obtained the multiplicity of positive solutions which concentrate on the minima of $V(x)$ as $\varepsilon\rightarrow 0$. Of course, recently, this methods have been successfully applied to other many problems, such as: Schrödinger-Poisson system [23], fractional Schrödinger equations [24], $p$-Laplacian problem [1], Kirchhoff type problems [25], and the references therein. In [32], by using the methods mentioned before, Liu and Zhang proved the existence and concentration of positive ground state solution for problem (1.1) when $K(x)\equiv 1$ and $Q(x)\equiv 1$, but they have not discuss the decay of solutions.

In the last decades, the existence, multiplicity and concentration behavior of solutions of nonlinear problems involving competing potential functions of the form

[TABLE]

where $h>0$, $1<q<p<\frac{N+2}{N-2}$, have been investigated by several scholars. Such as, Rabinowtiz [38] proved that if $V$ is coercive and $K$, $Q$ satisfy suitable assumptions, a result implies the existence of ground state solutions for problem (1.4), for any $h>0$. Wang and Zeng [49] assumed that $V(x)$ has a positive lower bound, $K(x)$ is bounded and positive, $Q(x)$ is bounded (and allowed to change sign), they proved the existence of a ground state solution of (1.4), for any $h>0$ small. Furthermore, they studied the concentration behaviour of such solutions and give a necessary condition for the location of the concentration points for positive bound states. Under the same assumptions on $V(x)$, $K(x)$ and $Q(x)$, Cingolani and Lazzo [10] used the Ljusternik-Schnirelman category to get the multiplicity of positive solutions for problem (1.4). When $V(x)$, $K(x)$ and $Q(x)$ are all bounded and positive functions, Zhao and Zhao [53] considered the critical Schrödinger-Poisson system (1.2) with $\varepsilon=1$ and $g(x,u)=K(x)|u|^{p-2}u+Q(x)|u|^{2^{\ast}-2}u$, and the existence of positive ground solutions was obtained. In [48], the authors proved the existence and concentration of positive solutions for system (1.2) with $g(x,u)=K(x)f(u)+Q(x)|u|^{2^{\ast}-2}u$, where $f$ is some superlinear-4 growth nonlinearity. In [31], the Kirchhoff type problem with competing potential was considered and the existence and concentration behavior of positive solution were established. In addition, fractional Schrödinger equations involving the competing potentials of the form

[TABLE]

with $2<p<2_{s}^{\ast}$, have been considered in [45], the existence of ground state solution was obtained.

Motivated by the above-cited works, the aim of this paper is to consider the existence and concentration of positive solutions for fractional Schrödinger-Poisson system with competing potentials. As far as we know, there are few results on the existence and concentration of positive solution for system (1.1), and even in the $s=t=1$ case. There are some difficulties compared with classical Schrodinger-Poisson system. One is the $L^{\infty}$-estimate, owing to the work of Dipierro, Medina and Valdinoci [17], similarly, we can get the $L^{\infty}$-estimate. The other is the decay estimate of solutions, with the help of works in [22], [2] and [24], we can establish the decay estimate at infinity. Below we give some assumptions on the nonlinearity $f$:

$(f_{0})$ $f\in C^{1}(\mathbb{R}^{3})$, $f(t)=o(t^{3})$, $f(t)t>0$ for $t>0$ and $f(t)=0$ for $t\leq 0$;

$(f_{1})$ $\frac{f(t)}{t^{3}}$ is strictly increasing on the interval $(0,\infty)$;

$(f_{2})$ $f(t)\geq c_{0}t^{q-1}$ and $|f^{\prime}(t)|\leq c_{1}(1+|t|^{p-2})$ for some constants $c_{0},c_{1}>0$, where $4<q<p<2_{s}^{\ast}$.

Our main result is stated as follows.

Theorem 1.1**.**

*Suppose that $(V)$, $(Q_{0})$, $(Q_{1})$, $(K)$, $(f_{0})$–$(f_{2})$ hold and let $s\in(\frac{3}{4},1)$. Then

$(i)$ there exists $\varepsilon_{0}>0$ such that system (1.1) possesses a positive solution $(u_{\varepsilon},\phi_{\varepsilon})\in H^{s}(\mathbb{R}^{3})\times\mathcal{D}^{t,2}(\mathbb{R}^{3})$, where $H^{s}(\mathbb{R}^{3})$ and $\mathcal{D}^{t,2}(\mathbb{R}^{3})$ are defined in Section 2;

$(ii)$ $u_{\varepsilon}$ possesses a maximum point $z_{\varepsilon}$ in $\mathbb{R}^{3}$ such that*

[TABLE]

and for any $z_{\varepsilon}\rightarrow x_{0}\in\Theta$ as $\varepsilon\rightarrow 0$, set $v_{\varepsilon}(x)=u_{\varepsilon}(\varepsilon x+z_{\varepsilon})$, the solution $(v_{\varepsilon},\phi_{\varepsilon})$ converges strongly in $H^{s}(\mathbb{R}^{3})$ to a solution $(v,\phi)$ of

[TABLE]

$(iii)$* there exist two constants $C>0$ and $C_{0}\in\mathbb{R}$ such that*

[TABLE]

We also obtain a supplementary result of a nonexistence of ground state solution for system (1.1).

Theorem 1.2**.**

*Assume that $(f_{0})$–$(f_{2})$ hold and the continuous functions $V(x)$, $K(x)$, $Q(x)$ satisfies

$(\mathcal{H})$ $V(x)\geq V_{\infty}=\lim\limits_{|x|\rightarrow\infty}V(x)=V_{0}$, $K(x)\leq K_{\infty}$ and $Q(x)\leq Q_{\infty}$, which one of the strictly inequality holds on a positive measure subset.

Then for any $\varepsilon>0$, system (1.1) has no ground state solution.*

The paper is organized as follows. In section 2, we present some preliminaries results. In section 3, we will prove the compactness condition. In section 4, the existence of positive ground state solutions of autonomous problem defined in section 2 and system (1.1) are established. Section 5 is devoted to proving the concentration of positive solutions. Section 6 is to prove the nonexistence of ground state solutions. In Appendix, we give some estimates for extremal function defined in Section 3.

2. Variational Setting

In this section, we outline the variational framework for problem (1.1) and give some preliminary Lemma. In the sequel, we denote by $\|\cdot\|_{p}$ the usual norm of the space $L^{p}(\mathbb{R}^{N})$, the letters $c_{i}$ ($i=1,2,\ldots$), $C_{i}$, $C$ will be indiscriminately used to denote various positive constants whose exact values are irrelevant. We denote $\widehat{u}$ the Fourier transform of $u$ for simplicity.

It is easily seen that, just performing the change of variables $u(x)\rightarrow u(x/\varepsilon)$ and $\phi(x)\rightarrow\phi(x/\varepsilon)$, and taking $z=x/\varepsilon$, problem (1.1) can be rewritten as the following equivalent form

[TABLE]

which will be referred from now on.

2.1. Work space stuff

For $\alpha\in(0,1)$, we define the homogeneous fractional Sobolev space $\mathcal{D}^{\alpha,2}(\mathbb{R}^{3})$ as follows

[TABLE]

which is the completion of $C_{0}^{\infty}(\mathbb{R}^{3})$ under the norm

[TABLE]

and for any $\alpha\in(0,1)$, there exists a best Sobolev constant $\mathcal{S}_{\alpha}>0$ such that

[TABLE]

The fractional Sobolev space $H^{\alpha}(\mathbb{R}^{3})$ can be described by means of the Fourier transform, i.e.

[TABLE]

In this case, the inner product and the norm are defined as

[TABLE]

[TABLE]

From Plancherel’s theorem we have $\|u\|_{2}=\|\widehat{u}\|_{2}$ and $\||\xi|^{\alpha}\widehat{u}\|_{2}=\|(-\Delta)^{\frac{\alpha}{2}}u\|_{2}$. Hence

[TABLE]

We denote $\|\cdot\|$ by $\|\cdot\|_{H^{\alpha}}$ in the sequel for convenience.

In terms of finite difference, the fractional Sobolev space $H^{\alpha}(\mathbb{R}^{3})$ also can be defined as follows

[TABLE]

endowed with the natural norm

[TABLE]

Indeed, in view of Proposition 3.4 and Proposition 3.6 in [36], we have

[TABLE]

It is well known that $H^{\alpha}(\mathbb{R}^{3})$ is continuously embedded into $L^{p}(\mathbb{R}^{3})$ for $2\leq r\leq 2_{\alpha}^{\ast}$ ($2_{\alpha}^{\ast}=\frac{6}{3-2\alpha}$), and is locally compactly embedded into $L_{loc}^{r}(\mathbb{R}^{3})$ for $1\leq r<2_{\alpha}^{\ast}$.

For any $\varepsilon>0$, let $H_{\varepsilon}=\{u\in H^{s}(\mathbb{R}^{3})\,|\,\int_{\mathbb{R}^{3}}V(\varepsilon x)|u|^{2}\,{\rm d}x<\infty\}$ be the Sobolev space endowed with the norm

[TABLE]

Clearly, by the assumption $(V)$, $\|\cdot\|_{\varepsilon}$ and $\|\cdot\|$ are equivalent norm on $H_{\varepsilon}$ uniformly for $\varepsilon>0$. Moreover, $H_{\varepsilon}$ is continuously embedded into $L^{r}(\mathbb{R}^{3})$ for $2\leq r\leq 2_{s}^{\ast}$ and locally compact embedded into $L_{loc}^{r}(\mathbb{R}^{3})$ for $1\leq r<2_{s}^{\ast}$.

2.2. Formulation of Problem (1.1) and preliminaries

By $(f_{0})$ and $(f_{1})$, for any $\varepsilon>0$, there exists $C_{\varepsilon}>0$ such that

[TABLE]

By $(f_{1})$ and $(f_{2})$, one can easily check that

[TABLE]

For problem (2.1), we first apply the usual ”reduction” argument to reduce it to a single equation involving just $u$.

From [47], the author has proved that if $4s+2t\geq 3$, for each $u\in H^{s}(\mathbb{R}^{3})$, the Lax-Milgram theorem implies that there exists a unique $\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$ such that

[TABLE]

that is $\phi_{u}^{t}$ is a weak solution of

[TABLE]

and the representation formula holds

[TABLE]

which is called $t$-Riesz potential.

Substituting $\phi_{u}^{t}$ in (2.1), it reduces to a single equation, i.e., the fractional Schrödinger equation with a non-local term $\phi_{u}^{t}u$:

[TABLE]

whose solutions can be obtained by looking for critical points of the functional $I_{\varepsilon}:H_{\varepsilon}\rightarrow\mathbb{R}$ defined by

[TABLE]

which is well defined in $H_{\varepsilon}$ and $I_{\varepsilon}\in C^{1}(H_{\varepsilon},\mathbb{R})$. Moreover,

[TABLE]

Definition 2.1**.**

*(1) We call $(u_{\varepsilon},\phi_{\varepsilon})\in H_{\varepsilon}\times\mathcal{D}^{t,2}(\mathbb{R}^{3})$ is a weak solution of system (1.1) if $u_{\varepsilon}$ is a weak solution of problem (2.5).

(2) We call $u\in H_{\varepsilon}$ is a weak solution of (2.5) if*

[TABLE]

for any $v\in H_{\varepsilon}$.

Obviously, the weak solutions of (2.5) are the critical points of $I_{\varepsilon}$. Now let us summarize some properties of $\phi_{u}^{t}$.

Lemma 2.2**.**

*For every $u\in H_{\varepsilon}$ with $4s+2t\geq 3$, define $\Phi(u)=\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$, where $\phi_{u}^{t}$ is the unique solution of equation $(-\Delta)^{t}\phi=u^{2}$. Then there hold:

$(i)$ If $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, then $\Phi(u_{n})\rightharpoonup\Phi(u)$ in $\mathcal{D}^{t,2}(\mathbb{R}^{3})$;

$(ii)$ $\Phi(tu)=t^{2}\Phi(u)$ for any $t\in\mathbb{R}$;

$(iii)$ For $u\in H_{\varepsilon}$, one has*

[TABLE]

*where constant $C$ is independent of $u$;

$(iv)$ Let $4s+2t>3$, if $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$ and $u_{n}\rightarrow u$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$, then*

[TABLE]

$(v)$* For $u,v\in H_{\varepsilon}$, there holds*

[TABLE]

Proof.

We only need to check that $(vi)$ and $(v)$ hold, the proof of others can be found in [47].

$(iv)$ By Hölder’s inequality and $4s+2t>3$ implying that $\frac{12}{3+2t}<\frac{6}{3-2s}$, we have that

[TABLE]

for any $v\in H_{\varepsilon}$. For the second part, using the similar argument, we have

[TABLE]

$(v)$ By the definition of $\phi_{u}^{t}$ and $\phi_{v}^{t}$, using Hölder’s inequality and $4s+2t\geq 3$ implying that $H_{\varepsilon}\hookrightarrow L^{\frac{12}{3+2t}}(\mathbb{R}^{3})$, we have that

[TABLE]

∎

The minimax level of the autonomous equation associated with equation (2.5)

[TABLE]

plays an important role in the proof of compactness of $(PS)$ sequence and concentration behavior of solutions, where $\mu,\nu,\kappa>0$ are arbitrary positive constants and $\phi_{u}^{t}=\int_{\mathbb{R}^{3}}\frac{u^{2}(y)}{|x-y|^{3-2t}}\,{\rm d}x$. For $\nu>0$, let

[TABLE]

be a Sobolev space endowed with the norm

[TABLE]

In fact, the Sobolev space $H_{\varepsilon}=E_{\nu}=H^{s}(\mathbb{R}^{3})$ for any $\varepsilon>0$ and $\nu>0$.

The energy functional $I_{\nu,\kappa,\mu}:E_{\nu}\rightarrow\mathbb{R}$ is given by

[TABLE]

It is easy to see that $I_{\nu,\kappa,\mu}\in C^{1}(E_{\nu},\mathbb{R})$ and

[TABLE]

for any $u,\varphi\in E_{\nu}$. Moreover, the critical points of $I_{\nu,\kappa,\mu}$ in $E_{\nu}$ are weak solutions of equation (2.6).

In section 3, we will apply the concentration-compactness principle of P. L. Lions [29, 30] and vanishing Lemma [40] to prove the compactness of $(PS)_{c}$ sequence of $I_{\varepsilon}$ on some low energy level. We first recall these results as follows.

Proposition 2.3**.**

Let $\rho_{n}(x)\in L^{1}(\mathbb{R}^{N})$ be a non-negative sequence satisfying

[TABLE]

*Then there exists a subsequence, still denoted by $\{\rho_{n}(x)\}$ such that one of the following cases occurs.

(i) (compactness) there exists $y_{n}\in\mathbb{R}^{N}$, such that for any $\varepsilon>0$, exists $R>0$ such that*

[TABLE]

(ii) (vanishing) for any fixed $R>0$, there holds

[TABLE]

(iii) (dichotomy) there exists $\alpha\in(0,l)$ such that for any $\varepsilon>0$, there exists $n_{0}\geq 1$, $\rho_{n}^{(1)}$, $\rho_{n}^{(2)}\in L^{1}(\mathbb{R}^{N})$, for $n\geq n_{0}$, there holds

[TABLE]

and

[TABLE]

Lemma 2.4**.**

Assume that $\{u_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{N})$ and it satisfies

[TABLE]

where $R>0$. Then $u_{n}\rightarrow 0$ in $L^{r}(\mathbb{R}^{N})$ for every $2<r<2_{s}^{\ast}$.

3. Compactness

Define the Nehari manifold associated to the functional $I_{\varepsilon}$ as

[TABLE]

where

[TABLE]

Thus, for any $u\in\mathcal{N}_{\varepsilon}$, we have that

[TABLE]

Remark 3.1**.**

Observing that $s>\frac{3}{4}$ implies that $4s+2t>3$ holds trivially.

Also, we define the Nehari manifold associated with functional $I_{\mu,\kappa,\nu}$ as follows

[TABLE]

where

[TABLE]

Particularly, when $\mu=V_{\infty}$, $\kappa=K_{\infty}$, $\mu=Q_{\infty}$, and $\mu=V_{0}$, $\kappa=K_{0}$, $\mu=Q_{0}$, the functional $I_{\infty}$ and $I_{0}$ are defined as

[TABLE]

and

[TABLE]

Also, we denote the Nehari manifolds by

[TABLE]

In order to find the least energy solutions of problem (2.5) and (2.6), we define the least energy levels as follows

[TABLE]

The following lemma describes some properties of the Nehari manifold $\mathcal{N}_{\varepsilon}$, $\mathcal{N}_{\nu,\kappa,\mu}$ and $I_{\varepsilon}$.

Lemma 3.2**.**

*Under the assumptions $(V)$, $(Q_{0})$, $(K)$ and $(f_{0})-(f_{2})$, the the following statements hold:

$(i)$ $\mathcal{N}_{\varepsilon}$ is a manifold of $C^{1}$-class diffeomorphic to the unite sphere of $H_{\varepsilon}$;

$(ii)$ For every $u\in H_{\varepsilon}\backslash\{0\}$, there exists a unique $t_{u}>0$ such that $t_{u}u\in\mathcal{N}_{\varepsilon}$ and*

[TABLE]

$(iii)$* For every $u\in\mathcal{N}_{\varepsilon}$, there exists $C>0$ such that $\|u\|_{\varepsilon}\geq C>0$;

$(iv)$ If $\{u_{n}\}$ is a $(PS)_{c}$ sequence in $H_{\varepsilon}$, i.e., $I_{\varepsilon}(u_{n})\rightarrow c$ and $I_{\varepsilon}^{\prime}(u_{n})\rightarrow 0$ in $(H_{\varepsilon})^{\prime}$ as $n\rightarrow\infty$, then there exists $u\in H_{\varepsilon}$ such that $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$ and $I_{\varepsilon}^{\prime}(u)=0$ if $2s+2t>3$;

$(v)$ Let $\{u_{n}\}\subset H_{\varepsilon}$ be a sequence satisfying*

[TABLE]

as $n\rightarrow\infty$, where $a$ is a positive constant. Then, up to a subsequence, there exists $t_{n}>0$ such that

[TABLE]

$(vi)$* Let $\{u_{n}\}$ be a sequence such that $u_{n}\in\mathcal{N}_{\varepsilon}$ and $I_{\varepsilon}(u_{n})\rightarrow m_{\varepsilon}$, then we may assume that $\{u_{n}\}$ is a $(PS)_{m_{\varepsilon}}$ sequence in $H_{\varepsilon}$.*

Proof.

The proof of $(ii)$ and $(iii)$ is standard, we only to verify the remain conclusions.

$(i)$ Let $u\in\mathcal{N}_{\varepsilon}$, by computation, using (2.4), we get

[TABLE]

$(iv)$ Let $\{u_{n}\}\subset H_{\varepsilon}$ be a $(PS)_{c}$ sequence, it is easy to show that $\{u_{n}\}$ is bounded in $H_{\varepsilon}$. By the reflexivity of $H_{\varepsilon}$ and using the Sobolev embedding property, up to a subsequence, still denoted by $\{u_{n}\}$, we may assume that there exists $u\in H_{\varepsilon}$ such that $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb{R}^{3})$ with $1\leq r<2_{s}^{\ast}$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$. By (2.3), we see that $f(u)\in L^{\frac{2_{s}^{\ast}-1}{p-1}}(\mathbb{R}^{3})$ because of $2<\frac{2(2_{s}^{\ast}-1)}{p-1}<2_{s}^{\ast}-1$. Therefore, $Q(\varepsilon x)^{\frac{2_{s}^{\ast}-1}{2_{s}^{\ast}}}|u_{n}|^{2_{s}^{\ast}-2}u_{n}$ and $K(\varepsilon x)f(u_{n})$ are bounded in $L^{\frac{2_{s}^{\ast}}{2_{s}^{\ast}-1}}(\mathbb{R}^{3})$ and $L^{\theta\frac{2_{s}^{\ast}-1}{p-1}}(\mathbb{R}^{3})$ with $\theta\in(\frac{p-1}{2_{s}^{\ast}-1}\frac{2_{s}^{\ast}}{2_{s}^{\ast}-1},\frac{2_{s}^{\ast}}{2_{s}^{\ast}-1})$, respectively. Here using $s>\frac{3}{4}$, we can choose $\theta\in(\frac{p-1}{2_{s}^{\ast}-1}\frac{2_{s}^{\ast}}{2_{s}^{\ast}-1},\frac{2_{s}^{\ast}}{2_{s}^{\ast}-1})$ such that $2<2\theta\frac{2_{s}^{\ast}-1}{p-1}<2_{s}^{\ast}$, $2<\theta(2_{s}^{\ast}-1)<2_{s}^{\ast}$ and $2<\frac{\theta(2_{s}^{\ast}-1)}{\theta(2_{s}^{\ast}-1)-(p-1)}<2_{s}^{\ast}$. Using the fact $u_{n}\rightarrow u$ a..e in $\mathbb{R}^{3}$, we obtain that

[TABLE]

and

[TABLE]

Next, we show that $\phi_{u_{n}}^{t}\rightarrow\phi_{u}^{t}$ a.e. in $\mathbb{R}^{3}$. In fact, using $2s+2t>3$ and choosing $\frac{3}{3-2t}<p<\frac{3}{2s}$, $q>\frac{3}{2s}>\frac{3}{3-2t}$ so that $2p^{\prime},2q^{\prime}\in(2,2_{s}^{\ast})$, using Hölder’s inequality, we deduce that

[TABLE]

concluding the pointwise convergence. Owing to $\phi_{u_{n}}^{t}u_{n}$ is bounded in $L^{\gamma}(\mathbb{R}^{3})$, where $\gamma$ satisfies $2\leq\frac{\gamma 2_{t}^{\ast}}{2_{t}^{\ast}-\gamma}\leq 2_{s}^{\ast}$ and $2\leq\gamma^{\prime}\leq 2_{s}^{\ast}$ (using $4s+2t\geq 3$). Using $\phi_{u_{n}}^{t}u_{n}\rightarrow\phi_{u}^{t}u$ a.e. in $\mathbb{R}^{3}$, we have that

[TABLE]

Therefore, combining with (3.1), (3.2), (3.3) and using the weakness convergence in $H_{\varepsilon}$, we infer that

[TABLE]

which leads to $I_{\varepsilon}^{\prime}(u)=0$.

$(v)$ By the assumptions, it is easy to see that $\|u_{n}\|_{\varepsilon}\neq 0$ for large $n$. Using the conclusion $(ii)$, there exists $t_{n}>0$ such that $t_{n}u_{n}\in\mathcal{N}_{\varepsilon}$ i.e., $\langle I_{\varepsilon}(t_{n}u_{n}),t_{n}u_{n}\rangle=0$. Now we prove that $t_{n}\rightarrow 1$ as $n\rightarrow\infty$. Set

[TABLE]

By assumption, we have that $c_{n}+d_{n}\rightarrow a>0$ as $n\rightarrow\infty$. Passing to a subsequence, we may assume that

[TABLE]

Thus, $a_{0}+b_{0}=c_{0}+d_{0}=a$ and $a_{0}>0$ (if not, contradiction with $a>0$). From $\langle I_{\varepsilon}(t_{n}u_{n}),t_{n}u_{n}\rangle=0$, by (2.3) and $(f_{2})$, we have that

[TABLE]

and

[TABLE]

which imply that there exist $T_{1},T_{2}>0$ such that $0<T_{1}\leq t_{n}\leq T_{2}$. Hence, up to a subsequence, still denoted by $\{t_{n}\}$, we may assume that $t_{n}\rightarrow T$ as $n\rightarrow\infty$. Since $\langle I_{\varepsilon}^{\prime}(t_{n}u_{n}),t_{n}u_{n}\rangle=0$ and $\langle I_{\varepsilon}^{\prime}(u_{n}),u_{n}\rangle\rightarrow 0$, we get

[TABLE]

which leads to

[TABLE]

Observing that $(\frac{1}{T^{2}}-1)a_{0}+(1-T^{2_{s}^{\ast}-4})d_{0}<0$ when $T>1$, but it follows from Fatou’s Lemma and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$ that

[TABLE]

which is impossible. Using similar argument, we can get a contradiction when $T<1$. Hence, it is only true that $T=1$.

$(vi)$ Suppose $\{u_{n}\}$ be a minimizing sequence of $I_{\varepsilon}$ constrained in $\mathcal{N}_{\varepsilon}$. By the Ekeland’s variational principle in [51] (Theorem 8.5, Page 122), there exists a sequence $\{v_{n}\}\subset\mathcal{N}_{\varepsilon}$ such that $I_{\varepsilon}(v_{n})\rightarrow m_{\varepsilon}$, $\|v_{n}-u_{n}\|_{\varepsilon}\rightarrow 0$ and $I_{\varepsilon}^{\prime}(v_{n})-\lambda_{n}G_{\varepsilon}^{\prime}(v_{n})\rightarrow 0$ as $n\rightarrow\infty$. Thus, in view of $v_{n}\in\mathcal{N}_{\varepsilon}$, we have that $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle=\lambda_{n}\langle G_{\varepsilon}(v_{n}),v_{n}\rangle+o_{n}(1)=o_{n}(1)$. We assume that $\lim\limits_{n\rightarrow\infty}\langle G_{\varepsilon}(v_{n}),v_{n}\rangle=\gamma\leq 0$, if $\gamma=0$, owing to the proof of $(i)$, we see that $\|v_{n}\|_{\varepsilon}\rightarrow 0$ as $n\rightarrow\infty$. This yields a contradiction with $m_{\varepsilon}>0$. Therefore, $\lambda_{n}\rightarrow 0$ as $n\rightarrow\infty$ and so $I_{\varepsilon}^{\prime}(v_{n})\rightarrow 0$ as $n\rightarrow\infty$. Hence, without loss of generalization, we may assume that $I_{\varepsilon}(u_{n})\rightarrow m_{\varepsilon}$ and $I_{\varepsilon}^{\prime}(u_{n})\rightarrow 0$ as $n\rightarrow\infty$, i.e., $\{u_{n}\}$ is a $(PS)_{m_{\varepsilon}}$ sequence for $I_{\varepsilon}$. ∎

The functional $I_{\varepsilon}$ satisfies the mountain pass geometry.

Lemma 3.3**.**

*Suppose $(V)$, $(K)$, $(Q_{0})$ and $(f_{0})-(f_{2})$ hold, then the functional $I_{\varepsilon}$ has the following properties:

$(i)$ there exists $\alpha,\rho>0$ such that $I_{\varepsilon}(u)\geq\alpha$ for $\|u\|_{\varepsilon}=\rho$;

$(ii)$ there exists $e\in H_{\varepsilon}$ satisfying $\|e\|_{\varepsilon}>\rho$ such that $I_{\varepsilon}(e)<0$.*

The proof of Lemma 3.3 is standard and hence is omitted. By Lemma 3.3 and Theorem 1.15 in [51] (Mountain pass theorem without Palais-Smale condition), it follows that there exists a $(PS)_{c_{\varepsilon}}$ sequence $\{u_{n}\}\subset H_{\varepsilon}$ such that

[TABLE]

where $c_{\varepsilon}=\inf_{\gamma\in\Gamma_{\varepsilon}}\max_{t\in[0,1]}I_{\varepsilon}(\gamma(t))$, where $\Gamma_{\varepsilon}=\{\gamma\in C([0,1],H_{\varepsilon})\,\,|\,\,\gamma(0)=0,I_{\varepsilon}(\gamma(1))<0\}$. Similarly to the arguments in [38] or [23], by (2.4), the equivalent characterization of $c_{\varepsilon}$ is given by

[TABLE]

The following Lemma gives the estimate of the critical value $c_{\varepsilon}$.

Lemma 3.4**.**

Suppose that $(V)$, $(K)$, $(Q_{0})$, $(Q_{1})$ and $(f_{0})-(f_{2})$ hold, then the infinimum $c_{\varepsilon}$ satisfies

[TABLE]

for $\varepsilon$ small enough, where $\mathcal{S}_{s}$ is the best Sobolev constant defined by (2.2).

Proof.

We define

[TABLE]

where $U_{\varepsilon}(x)=\varepsilon^{\frac{3-2s}{2}}u^{\ast}(x)$, $u^{\ast}(x)=\frac{\kappa}{(\varepsilon^{2}+|x-x_{0}/\varepsilon|^{2})^{\frac{3-2s}{2}}}$ (see Appendix), and $\psi\in C^{\infty}(\mathbb{R}^{3})$ such that $0\leq\psi\leq 1$ in $\mathbb{R}^{3}$, $\psi(x)\equiv 1$ in $B_{R}(0)$ and $\psi\equiv 0$ in $\mathbb{R}^{3}\backslash B_{2R}(0)$. From Lemma A.2 and Lemma A.3 in Appendix, we know that

[TABLE]

[TABLE]

where $\bar{u}=\frac{\kappa}{(1+|x|^{2})^{\frac{3-2s}{2}}}$ is such that $\mathcal{S}_{s}=\frac{|(-\Delta)^{\frac{s}{2}}\bar{u}\|_{2}^{2}}{\|\bar{u}\|_{2_{s}^{\ast}}^{2}}$, and

[TABLE]

By $(ii)$ of Lemma 3.2, there exists $t_{\varepsilon}>0$ such that $\sup\limits_{t\geq 0}I_{\varepsilon}(tu_{\varepsilon})=I_{\varepsilon}(t_{\varepsilon}u_{\varepsilon})$. Hence \frac{{\rm d}I_{\varepsilon}(tu_{\varepsilon})}{{\rm d}t}\Big{|}_{t=t_{\varepsilon}}=0, that is

[TABLE]

By $(f_{0})$, we have that

[TABLE]

It follows from $(iii)$ of Lemma 2.2 that

[TABLE]

(3.4), (3.5), (3.6) and (3.7) imply that $|t_{\varepsilon}|\leq C_{1}$, where $C_{1}$ is independent of $\varepsilon>0$ small. On the other hand, we may assume that there is a positive constant $C_{2}>0$ such that $t_{\varepsilon}\geq C_{2}>0$ for $\varepsilon>0$ small. Otherwise, we can find a sequence $\varepsilon_{n}\rightarrow 0$ as $n\rightarrow\infty$ such that $t_{\varepsilon_{n}}\rightarrow 0$ as $n\rightarrow\infty$. Therefore

[TABLE]

which is a contradiction.

Denote $g(t)=\frac{t^{2}}{2}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{s}{2}}u_{\varepsilon}|^{2}\,{\rm d}x-\frac{t^{2_{s}^{\ast}}}{2_{s}^{\ast}}\int_{\mathbb{R}^{3}}Q(x_{0})|u_{\varepsilon}|^{2_{s}^{\ast}}\,{\rm d}x$, by (3.4) and (3.5), it is easy to check that

[TABLE]

Thus

[TABLE]

where we have used (3.6) and $s>\frac{3}{4}$ which implies $2<\frac{3}{3-2s}$.

By (3.6), we have that

[TABLE]

Since $s>\frac{3}{4}$ and $q>\frac{3+2s}{3-2s}$, then $q+1>\frac{3}{3-2s}$, $2s-\frac{3-2s}{2}(q+1)<0$. Thus

[TABLE]

By the hypothesis $(Q_{1})$, we deduce that

[TABLE]

Since $\frac{3-2s}{2}\leq\alpha<\frac{3}{2}$, then

[TABLE]

Therefore, combining with (3), (3.12), (3.13) and (3.14), we conclude that

[TABLE]

for $\varepsilon$ small enough and thus the proof is completed. ∎

Lemma 3.5**.**

Assume that $(V)$, $(Q_{0})$, $(Q_{1})$, $(K)$ and $(f_{0})-(f_{2})$ hold. If $c_{\varepsilon}<\min\{m_{\infty},\frac{s}{3Q(x_{0})^{\frac{3-2s}{2s}}}\mathcal{S}_{s}^{\frac{3}{2s}}\}$, then $I_{\varepsilon}$ satisfies the $(PS)$ condition for $c_{\varepsilon}$.

Proof.

Let $\{u_{n}\}$ be a $(PS)$ sequence of $I_{\varepsilon}$ at the level $c_{\varepsilon}$, i.e.,

[TABLE]

It is easy to check that $\{u_{n}\}$ is bounded in $H_{\varepsilon}$. Thus, up to a subsequence, still denoted by $\{u_{n}\}$, we may assume that there exists $u\in H_{\varepsilon}$ such that $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$.

Next, we aim to show that $u_{n}\rightarrow u$ in $H_{\varepsilon}$. For this purpose, set

[TABLE]

Clearly, $\{\rho_{n}\}$ is bounded in $L^{1}(\mathbb{R}^{3})$. Hence, up to a subsequence, still denoted by $\{\rho_{n}\}$, we may assume that $\Psi(u_{n}):=\|\rho_{n}\|_{1}\rightarrow l$ as $n\rightarrow\infty$. Obviously, $l>0$, otherwise, we can get a contradiction with $c_{\varepsilon}>0$. In fact, $l=c_{\varepsilon}$.

Next, we apply Proposition 2.3 to $\{\rho_{n}\}$.

If $\{\rho_{n}\}$ vanishing, then $\{u_{n}^{2}\}$ also vanishing, i.e., there exists $R>0$ such that

[TABLE]

By Lemma 2.4, one has $u_{n}\rightarrow 0$ in $L^{r}(\mathbb{R}^{3})$, $2<r<2_{s}^{\ast}$. Thus, by (2.3) and $(iii)$ of Lemma 2.2, we can deduce that

[TABLE]

From (3.15), we get

[TABLE]

and

[TABLE]

We may assume that there exist $\mathcal{L}\geq 0$ such that

[TABLE]

Obviously, $\mathcal{L}>0$, otherwise, a contradiction with $c_{\varepsilon}>0$. By (2.2), we have that

[TABLE]

which implies that

[TABLE]

Combing (3.16), we can deduce that $c_{\varepsilon}=\frac{s}{3}\mathcal{L}\geq\frac{s}{3Q(x_{0})^{\frac{3-2s}{2s}}}\mathcal{S}_{s}^{\frac{3}{2s}}$, this contradicts with the assumption. Hence, vanishing does not occur.

Next, we show the dichotomy does not occur. Suppose by contradiction that there exist $\alpha\in(0,l)$ and $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\varepsilon_{n}\rightarrow 0$, we can choose $\{R_{n}\}\subset\mathbb{R}_{+}$ ($R_{n}>R_{0}/\varepsilon+R^{\prime}$, for any fixed $\varepsilon>0$, $R_{0},R^{\prime}$ are positive constants defined later) with $R_{n}\rightarrow\infty$ satisfying

[TABLE]

Let $\xi:\mathbb{R}^{+}\cup\{0\}\rightarrow\mathbb{R}^{+}$ be a cut-off function such that $0\leq\xi\leq 1$, $\xi(t)=1$ for $t\leq 1$, $\xi(t)=0$ for $t\geq 2$ and $|\xi^{\prime}(t)|\leq 2$. Set

[TABLE]

Then by (3.18), we see that

[TABLE]

Denote $\Omega_{n}=B_{2R_{n}}(y_{n})\backslash B_{R_{n}}(y_{n})$, by (3.18), then $\int_{\Omega_{n}}\rho_{n}(x)\,{\rm d}x\rightarrow 0$ as $n\rightarrow\infty$, which leads to

[TABLE]

and thus by interpolation inequality, we can infer that $\int_{\Omega_{n}}\phi_{u_{n}}^{t}u_{n}^{2}\,{\rm d}x\rightarrow 0$. Therefore, by similar arguments as Lemma 3.4 in [47], we have that

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

We need to check that (3.21) and (3.22) hold. Indeed, by $(K)$, (2.3) and Hölder’s inequality, we infer that

[TABLE]

as $n\rightarrow\infty$, where $\theta_{1}\in(0,1)$ and $\theta_{2}\in(0,1)$ satisfy $\frac{1}{4}=\frac{\theta_{1}}{2}+\frac{1-\theta_{1}}{2_{s}^{\ast}}$, $\frac{1}{p}=\frac{\theta_{2}}{2}+\frac{1-\theta_{2}}{2_{s}^{\ast}}$. Similarly, we can verify that (3.22) hold.

Hence, by (3.19)–(3.24), we get

[TABLE]

Then

[TABLE]

hence

[TABLE]

By (3.15), (3.19)-(3.24), we have

[TABLE]

We distinguish the following two cases:

Case 1. Up to a subsequence, we may assume that $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle\leq 0$ or $\langle I_{\varepsilon}^{\prime}(w_{n}),w_{n}\rangle\leq 0$.

Without loss of generality, we suppose that $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle\leq 0$, then

[TABLE]

By $(ii)$ of Lemma 3.2, for any $n$, there exists $t_{n}>0$ such that $t_{n}v_{n}\in\mathcal{N}_{\varepsilon}$ and then $\langle I_{\varepsilon}^{\prime}(t_{n}v_{n}),t_{n}v_{n}\rangle=0$, i.e.,

[TABLE]

Combining (3.27) and (3.28), we have

[TABLE]

which implies that $t_{n}\leq 1$ by using $(f_{1})$. Then, by $t_{n}v_{n}\in\mathcal{N}_{\varepsilon}$ and (2.4) (implies that $f(su)su-4F(su)$ is nondecreasing in $s\in(0,+\infty)$), we have that

[TABLE]

which is a contradiction.

Case 2. Up to a subsequence, we may assume that $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle>0$ and $\langle I_{\varepsilon}^{\prime}(w_{n}),w_{n}\rangle>0$.

By (3.26), we see that $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle=o_{n}(1)$ and $\langle I_{\varepsilon}^{\prime}(w_{n}),w_{n}\rangle=o_{n}(1)$. In view of (3.19)–(3.24), we have that

[TABLE]

If the sequence $\{y_{n}\}\subset\mathbb{R}^{3}$ is bounded, we will deduce a contradiction by comparing $I_{\varepsilon}(w_{n})$ and $m_{\infty}$. In fact, by the assumptions $(V)$, $(K)$ and $(Q)$, for any $\delta>0$, there exists $R_{0}>0$, such that

[TABLE]

By the boundedness of $\{y_{n}\}\subset\mathbb{R}^{3}$, there exists $R^{\prime}>0$ such that $|y_{n}|\leq R^{\prime}$. Thus, $\mathbb{R}^{3}\backslash B_{R_{n}}(y_{n})\subset\mathbb{R}^{3}\backslash B_{R_{n}-R^{\prime}}(0)\subset\mathbb{R}^{3}\backslash B_{R_{0}/\varepsilon}(0)$ for $n$ large enough. From (3.30), we can deduce that

[TABLE]

and by the arbitrariness of $\delta$, this leads to

[TABLE]

Moreover, it is easy to check that

[TABLE]

and

[TABLE]

It follows from (3.31)–(3.33) that

[TABLE]

If $\langle I_{\infty}^{\prime}(w_{n}),w_{n}\rangle\leq 0$ for $n$ large enough, similar to the proof of Case 1, we get that there exists $t_{n}\leq 1$ such that $t_{n}w_{n}\in\mathcal{N}_{\infty}$. Thus, by (3.31)–(3.33)

[TABLE]

which contradicts with the assumption $c_{\varepsilon}<m_{\infty}$.

Observing that $\langle I_{\infty}^{\prime}(w_{n}),w_{n}\rangle\rightarrow 0$ and $\langle I_{\varepsilon}^{\prime}(v_{n}),v_{n}\rangle\rightarrow 0$ as $n\rightarrow\infty$ (indeed, \int_{\mathbb{R}^{3}}\Big{(}K_{\infty}f(w_{n})w_{n}+Q_{\infty}|w_{n}|^{2_{s}^{\ast}}\Big{)}\,{\rm d}x\rightarrow A, \int_{\mathbb{R}^{3}}\Big{(}K(\varepsilon x)f(v_{n})v_{n}+Q(\varepsilon x)|v_{n}|^{2_{s}^{\ast}}\Big{)}\,{\rm d}x\rightarrow B, where $A,B>0$, otherwise, contradicts with (3.25)), by $(v)$ of Lemma 3.2, there exist two sequences $\{t_{n}\}\subset\mathbb{R}_{+}$ and $\{s_{n}\}\subset\mathbb{R}_{+}$ satisfying $t_{n}\rightarrow 1$ and $s_{n}\rightarrow 1$ as $n\rightarrow\infty$, respectively, such that $t_{n}w_{n}\in\mathcal{N}_{\infty}$, $s_{n}v_{n}\in\mathcal{N}_{\varepsilon}$. Hence, by (3.34), we get

[TABLE]

and

[TABLE]

Therefore, by (3.29), we have $c_{\varepsilon}\geq m_{\infty}+c_{\varepsilon}>m_{\infty}$, a contradiction.

If $\{y_{n}\}\subset\mathbb{R}^{3}$ is unbounded, we choose a subsequence, stilled denoted by $\{y_{n}\}$, such that $|y_{n}|\geq 2R_{n}$. Then $B_{2R_{n}}(y_{n})\subset\mathbb{R}^{3}\backslash B_{R_{n}}(0)\subset\mathbb{R}^{3}\backslash B_{R_{0}/\varepsilon}(0)$. Similarly to the proof (3.31)–(3.33), we can infer that

[TABLE]

and

[TABLE]

Similarly as to the case that $\{y_{n}\}$ is bounded, we can obtain a contradiction by comparing $I_{\varepsilon}(v_{n})$ with $m_{\infty}$. Thus, dichotomy does not happen.

According to the above arguments, the sequence $\{\rho_{n}\}$ must be compactness, i.e., there exists $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\hat{\delta}>0$, there exists $\widetilde{R}>0$, we have $\int_{\mathbb{R}^{3}\backslash B_{\widetilde{R}}(y_{n})}\rho_{n}(x)\,{\rm d}x<\hat{\delta}$. By the interpolation inequality, we have that

[TABLE]

where $m\in[2,2_{s}^{\ast}]$, $\theta_{3}\in[0,1]$ satisfies $\frac{1}{m}=\frac{\theta_{3}}{2}+\frac{1-\theta_{3}}{2_{s}^{\ast}}$. This means that the sequence $\{|u_{n}|^{m}\}$ with $2\leq m\leq 2_{s}^{\ast}$ is also compactness.

We claim that the sequence $\{y_{n}\}$ is bounded. If not, up to a subsequence, we can choose $r_{n}$ such that $|y_{n}|\geq r_{n}\geq\widetilde{R}+R_{0}/\varepsilon$ with $r_{n}\rightarrow+\infty$. For $n$ large enough, $B_{\widetilde{R}}(y_{n})\subset\mathbb{R}^{3}\backslash B_{r_{n}-\widetilde{R}}(0)\subset\mathbb{R}^{3}\backslash B_{R_{0}/\varepsilon}(0)$. By (3.35), we can infer that

[TABLE]

Similarly, we can obtain that

[TABLE]

and

[TABLE]

Thus, $I_{\varepsilon}(u_{n})\geq I_{\infty}(u_{n})+o_{n}(1)$ and $o_{n}(1)=\langle I_{\varepsilon}^{\prime}(u_{n}),u_{n}\rangle\geq\langle I_{\infty}^{\prime}(u_{n}),u_{n}\rangle+o_{n}(1)$.

If $\langle I_{\infty}^{\prime}(u_{n}),u_{n}\rangle\leq 0$, by the similar arguments as Case 1, there exists $t_{n}\leq 1$ for sufficiently large $n$ such that $t_{n}u_{n}\in N_{\infty}$. Hence,

[TABLE]

which is a contradiction.

If $\langle I_{\infty}^{\prime}(u_{n}),u_{n}\rangle=o_{n}(1)$, by $(v)$ of Lemma 3.2, there exists $t_{n}\rightarrow 1$ such that $t_{n}u_{n}\in\mathcal{N}_{\infty}$. Hence, for $n$ large enough, we have that

[TABLE]

which is a contradiction. Therefore, the claim is true.

In view of the boundedness of $\{y_{n}\}$ and $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$, using (3.35), it is easy to check that $u_{n}\rightarrow u$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$. Set $\widetilde{u_{n}}=u_{n}-u$, by the weakness convergence and Brezis-Lieb Lemma, one has

[TABLE]

and

[TABLE]

By Hölder’s inequality and using $u_{n}\rightarrow u$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$, it is easy to verify that

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Therefore, by $(iv)$ of Lemma 2.2 and (3.40)–(3.41), it is easy to see that $\langle I_{\varepsilon}^{\prime}(u_{n}),v\rangle\rightarrow\langle I_{\varepsilon}^{\prime}(u),v\rangle$ for any $v\in H_{\varepsilon}$, i.e., $I_{\varepsilon}^{\prime}(u)=0$ and then $u\in\mathcal{N}_{\varepsilon}$, so that $I_{\varepsilon}(u)\geq 0$. Consequently, by $(iv)$ of Lemma 2.2 and (3.36)–(3.41), we have that

[TABLE]

and

[TABLE]

By similar arguments as the proof of vanishing (see (3.16), (3.17)), it is easy to get a contradiction with the limit $\|\widetilde{u_{n}}\|_{\varepsilon}^{2}\rightarrow\mathcal{L}>0$. Thus, $\mathcal{L}=0$ and hence $u_{n}\rightarrow u$ in $H_{\varepsilon}$. ∎

4. Relation between $c_{\varepsilon}$ and $m_{0}$, $m_{\infty}$

In this section, we shall compare the energy level $c_{\varepsilon}$ of problem (2.3) with the energy level $m_{0}$ and $m_{\infty}$ of limit equation (2.6). For this purpose, we should prove the existence of positive ground state solutions for the autonomous problem (2.6). For reader’s convenience, we rewrite it as follows

[TABLE]

where $\mu,\nu,\kappa>0$ are arbitrary positive constants and $\phi_{u}^{t}=\int_{\mathbb{R}^{3}}\frac{u^{2}(y)}{|x-y|^{3-2t}}\,{\rm d}x$.

Similar to Section 3, we can easily prove that the functional $I_{\nu,\kappa,\mu}$ verifies the mountain pass geometry. Then, applying Theorem 1.15 in [51], there exists a $(PS)_{c_{\nu,\kappa,\mu}}$ sequence $\{u_{n}\}\subset E_{\nu}$ such that $I_{\nu,\kappa,\mu}(u_{n})\rightarrow c_{\nu,\kappa,\mu}$ and $I^{\prime}_{\nu,\kappa,\mu}(u_{n})\rightarrow 0$ as $n\rightarrow\infty$, where $c_{\nu,\kappa,\mu}$ can be characterized by the following relations

[TABLE]

where $\Gamma_{\nu,\kappa,\mu}=\{\gamma\in C([0,1],E_{\nu})\,\,|\,\,\gamma(0)=0,I_{\nu,\kappa,\mu}(\gamma(1))<0\}$. For obtaining a compactness of the above sequence, we give the following estimate for $c_{\nu,\kappa,\mu}$.

Lemma 4.1**.**

For any $\nu,\kappa,\mu>0$, the infinimum $c_{\nu,\kappa,\mu}$ satisfies

[TABLE]

where $\mathcal{S}_{s}$ is the best Sobolev constant defined by (2.2).

Proof.

We define

[TABLE]

where $U_{\varepsilon}(x)=\varepsilon^{-\frac{3-2s}{2}}u^{\ast}(x/\varepsilon)$, $u^{\ast}(x)=\frac{\widetilde{u}(x/\mathcal{S}_{s}^{\frac{1}{2s}})}{\|\widetilde{u}\|_{2_{s}^{\ast}}}$, $\kappa\in\mathbb{R}\backslash\{0\}$, $\mu>0$ and $x_{0}\in\mathbb{R}^{3}$ are fixed constants, $\widetilde{u}(x)=\kappa(\mu^{2}+|x-x_{0}|^{2})^{-\frac{3-2s}{2}}$, and $\psi\in C^{\infty}(\mathbb{R}^{3})$ such that $0\leq\psi\leq 1$ in $\mathbb{R}^{3}$, $\psi(x)\equiv 1$ in $B_{\delta}$ and $\psi\equiv 0$ in $\mathbb{R}^{3}\backslash B_{2\delta}$. From Proposition 21 and Proposition 22 in [42], Lemma 3.3 in [46], we know that

[TABLE]

[TABLE]

and

[TABLE]

Similar to the proof of $(ii)$ of Lemma 3.2, there exists $t_{\varepsilon}>0$ such that $\sup\limits_{t\geq 0}I_{\nu,\kappa,\mu}(tu_{\varepsilon})=I_{\nu,\kappa,\mu}(t_{\varepsilon}u_{\varepsilon})$. Hence \frac{{\rm d}I_{\nu,\kappa,\mu}(tu_{\varepsilon})}{{\rm d}t}\Big{|}_{t=t_{\varepsilon}}=0, that is

[TABLE]

By $(f_{0})$, we have that

[TABLE]

It follows from $(iii)$ of Lemma 2.2 that

[TABLE]

Therefore, (4.2), (4.3), (4.4) and (4.5) imply that $|t_{\varepsilon}|\leq C_{1}$, where $C_{1}$ is independent of $\varepsilon>0$ small. On the other hand, we may assume that there is a positive constant $C_{2}>0$ such that $t_{\varepsilon}\geq C_{2}>0$ for $\varepsilon>0$ small. Otherwise, we can find a sequence $\varepsilon_{n}\rightarrow 0$ as $n\rightarrow\infty$ such that $t_{\varepsilon_{n}}\rightarrow 0$ as $n\rightarrow\infty$. Therefore

[TABLE]

which is a contradiction.

Denote $g(t)=\frac{t^{2}}{2}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{s}{2}}u_{\varepsilon}|^{2}\,{\rm d}x-\frac{\mu t^{2_{s}^{\ast}}}{2_{s}^{\ast}}\int_{\mathbb{R}^{3}}|u_{\varepsilon}|^{2_{s}^{\ast}}\,{\rm d}x$, by (4.2) and (4.3), it is easy to check that

[TABLE]

Thus

[TABLE]

where we have used (4.4) and $s>\frac{3}{4}$ which implies $2<\frac{3}{3-2s}$.

Observing that

[TABLE]

Since $s>\frac{3}{4}$ and $q>3$, then $q+1>\frac{3}{3-2s}$, $2s-\frac{3-2s}{2}(q+1)<0$. Thus

[TABLE]

Therefore, we have proved that for $\varepsilon$ small enough, there holds

[TABLE]

and thus the proof is completed. ∎

Similar arguments to Lemma 3.5, we can obtain the compactness of the $(PS)_{c_{\nu,\kappa,\mu}}$ sequence. It is stated as follows.

Lemma 4.2**.**

Assume $c_{\nu,\kappa,\mu}<\frac{s}{3\mu^{\frac{3-2s}{2s}}}\mathcal{S}_{s}^{\frac{3}{2s}}$ and let $\{u_{n}\}$ be the $(PS)$ sequence at the level $c_{\nu,\kappa,\mu}$. Then there exists $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\xi>0$, there exists $\widehat{R}>0$ such that

[TABLE]

Proof.

It is easy to verify that the $(PS)_{c_{\nu,\kappa,\mu}}$ sequence $\{u_{n}\}$ which satisfying $I_{\nu,\kappa,\mu}(u_{n})\rightarrow c_{\nu,\kappa,\mu}$ and $I^{\prime}_{\nu,\kappa,\mu}(u_{n})\rightarrow 0$ as $n\rightarrow\infty$, is bounded in $E_{\nu}$.

Let $\rho_{n}(x)=\frac{1}{4}|(\Delta)^{\frac{s}{2}}u_{n}|^{2}+\frac{\nu}{4}|u_{n}|^{2}+\frac{\kappa}{4}(f(u_{n})u_{n}-4F(u_{n}))+\frac{4s-3}{12}\mu|u_{n}|^{2_{s}^{\ast}}$, then $\rho_{n}\in L^{1}(\mathbb{R}^{3})$ is also bounded.

Checking the proof of Lemma 3.5 line by line, we find that it is only to prove the Case 2.

Case 2. Up to a subsequence, we may assume that $\langle I_{\nu,\kappa,\mu}^{\prime}(v_{n}),v_{n}\rangle>0$ and $\langle I_{\nu,\kappa,\mu}^{\prime}(w_{n}),w_{n}\rangle>0$. Thus, $\langle I_{\nu,\kappa,\mu}^{\prime}(v_{n}),v_{n}\rangle=o_{n}(1)$ and $\langle I_{\nu,\kappa,\mu}^{\prime}(w_{n}),w_{n}\rangle=o_{n}(1)$. Similar to the proof of $(v)$ of Lemma 3.2, there exist two sequences $\{t_{n}\}\subset\mathbb{R}_{+}$ and $\{s_{n}\}\subset\mathbb{R}_{+}$ satisfying $t_{n}\rightarrow 1$ and $s_{n}\rightarrow 1$ as $n\rightarrow\infty$, respectively, such that $t_{n}w_{n}\in\mathcal{N}_{\nu,\kappa,\mu}$, $s_{n}v_{n}\in\mathcal{N}_{\nu,\kappa,\mu}$. Therefore,

[TABLE]

which leads to a contradiction that

[TABLE]

Hence, dichotomy does not happen. The sequence $\{\rho_{n}\}$ must be compactness, i.e., there exists $\{y_{n}\}\subset\mathbb{R}^{3}$ such that for every $\xi>0$, there exists $\widehat{R}>0$, we have $\int_{\mathbb{R}^{3}\backslash B_{\widehat{R}}(y_{n})}\rho_{n}(x)\,{\rm d}x<\xi$. The proof is completed. ∎

Remark 4.3**.**

By the interpolation inequality, we could show that the sequence $\{|u_{n}|^{m}\}$ with $2\leq m\leq 2_{s}^{\ast}$ is also compactness.

Proposition 4.4**.**

Assume that $(f_{0})$–$(f_{2})$ hold. Then problem (2.6) has at least a positive ground state solution in $E_{\nu}$ satisfying $\lim\limits_{|x|\rightarrow\infty}u(x)=0$.

Proof.

From the above arguments, we see that there exists a $(PS)_{c_{\nu,\kappa,\mu}}$ sequence for $I_{\nu,\kappa,\mu}$. From Lemma 4.2, the sequence $\{u_{n}\}$ is bounded and verifies the compactness in the sense of Proposition 2.3. Set $\widehat{u_{n}}(\cdot)=u_{n}(\cdot+y_{n})$. Using the invariance of $\mathbb{R}^{3}$ by translation, we see that $\{v_{n}\}$ is a bounded $(PS)_{c_{\nu,\kappa,\mu}}$ sequence and

[TABLE]

Since $\{\widehat{u_{n}}\}$ is bounded in $E_{\nu}$, up to a subsequence, still denoted by $\{\widehat{u_{n}}\}$, there exists $\widehat{u}\in E_{\nu}$ such that $\widehat{u_{n}}\rightharpoonup\widehat{u}$ in $E_{\nu}$, $\widehat{u_{n}}\rightarrow\widehat{u}$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$. From Remark 4.3, we see that $\int_{\mathbb{R}^{3}\backslash B_{\widehat{R}}(0)}|\widehat{u_{n}}|^{r}\,{\rm d}x\leq\xi$ and hence $\widehat{u_{n}}\rightarrow\widehat{u}$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$. Similar arguments to the proof of compactness in Lemma 3.5, we can conclude that $\widehat{u_{n}}\rightarrow\widehat{u}$ in $E_{\nu}$. Hence, $I_{\nu,\kappa,\mu}(\widehat{u})=c_{\nu,\kappa,\mu}$ and $I_{\nu,\kappa,\mu}^{\prime}(\widehat{u})=0$, that is, $\widehat{u}$ is a nontrivial critical point of $I_{\nu,\kappa,\mu}$. From the equivalent characterize of mountain value, we conclude that $\widehat{u}$ is a nontrivial ground state solution of problem (2.6).

Finally, we only need to show that $\widehat{u}$ is positive. For simplicity, we replace $\widehat{u}$ by $u$ in the following discussion. If we replace $I_{\nu,\kappa,\mu}$ by the following functional

[TABLE]

where $u^{\pm}=\max\{\pm u,0\}$, then we see that all the calculations above can be repeated word by word. So, there exists a nontrivial ground state critical point $u\in E_{\nu}$ of $I_{\nu,\kappa,\mu}^{+}$. Hence,

[TABLE]

which implies that

[TABLE]

However, by computation, we have that

[TABLE]

Hence,

[TABLE]

which leads to $u^{-}=0$, and thus $u\geq 0$ and $u\not\equiv 0$.

Let $f(x,u)=\mu|u|^{2_{s}^{\ast}-1}+\kappa f(u)-\nu u-\phi_{u}^{t}u$, by $(f_{0})$ and $(f_{2})$, it is easy to check that $f(x,u)\leq\mu u^{2_{s}^{\ast}-1}+1$, for any $u\geq 0$. Using Proposition 4.1.1 in [17], we see that $u\in L^{\infty}(\mathbb{R}^{3})$ and check the proof of Proposition 4.1.1 word by word, using 4.1.6 and 4.1.7, we can obtain that

[TABLE]

where $C>0$ independent of $u$, $\beta_{1}=\frac{2_{s}^{\ast}+1}{2}$. This yields $u\in L^{r}(\mathbb{R}^{3})$ for all $r\in[2,+\infty]$. Moreover, $\phi_{u}^{t}\in L^{\infty}(\mathbb{R}^{3})$. Therefore, according to Proposition 2.9 in [41], and $s>\frac{3}{4}$, we see that $u\in C^{1,\alpha}(\mathbb{R}^{3})$ for any $0<\alpha<2s-1$. Thus, by Lemma 3.2 in [20], we have that

[TABLE]

Assume that there exists $x_{0}\in\mathbb{R}^{3}$ such that $u(x_{0})=0$, then from $u\geq 0$ and $u\not\equiv 0$, we get

[TABLE]

However, observe that $(-\Delta)^{s}u(x_{0})=-\nu u(x_{0})-(\phi_{u}^{t}u)(x_{0})+\kappa f(u(x_{0}))+\mu u(x_{0})^{2_{s}^{\ast}-1}=0$, a contradiction. Hence, $u(x)>0$, for every $x\in\mathbb{R}^{3}$. Finally, the fact $u\in L^{r}(\mathbb{R}^{3})\cap C^{1,\alpha}(\mathbb{R}^{3})$ for $2\leq r\leq\infty$ implies that $\lim\limits_{|x|\rightarrow\infty}u(x)=0$. The proof is completed. ∎

Lemma 4.5**.**

There exists $\varepsilon^{\ast}>0$ such that $c_{\varepsilon}<m_{\infty}$ for all $\varepsilon\in(0,\varepsilon^{\ast})$.

Proof.

By the assumption $(V)$, there must exist a $w\in\mathbb{R}_{+}$ such that $V_{0}<w<V_{\infty}$. Hence, $m_{0}<m_{w,K_{0},Q_{0}}\leq m_{w,K_{\infty},Q_{\infty}}<m_{\infty}$ owing to $K_{0}\geq K_{\infty}$ and $Q_{0}\geq Q_{\infty}$. Indeed, using Proposition 4.4, choose $u$ be a ground state solution of problem (4.1) with $\nu=V_{\infty}$, $\kappa=K_{\infty}$ and $\mu=Q_{\infty}$, such that $I_{\infty}(u)=m_{\infty}$. Then there holds $I_{\infty}(u)=\max\limits_{t\geq 0}I_{\infty}(tu)$ and there exists $t_{0}>0$ such that $t_{0}u\in\mathcal{N}_{w,K_{\infty},Q_{\infty}}$ and $I_{w,K_{\infty},Q_{\infty}}(t_{0}u)=\max\limits_{t\geq 0}I_{w,K_{\infty},Q_{\infty}}(tu)$. Hence

[TABLE]

Similarly, we can show that $m_{0}<m_{w,K_{\infty},Q_{\infty}}$.

Taking $\nu=w$, $\kappa=K_{\infty}$ and $\mu=Q_{\infty}$, in view of Proposition 4.4, we know that there exists $v\in\mathcal{N}_{w,K_{\infty},Q_{\infty}}$ such that $I_{w,K_{\infty},Q_{\infty}}(v)=m_{w,K_{\infty},Q_{\infty}}$. Let $\eta\in C_{0}^{\infty}(\mathbb{R}^{3},[0,1])$ be such that $\eta(x)=1$ if $|x|\leq 1$, and $\eta(x)=0$ if $|x|\geq 2$. For $\theta>0$, set $u_{\theta}(x)=\eta(x/\theta)v(x)$. Similar to the proof of $(ii)$ of Lemma 3.2, there exists $t_{\theta}>0$ such that $t_{\theta}u_{\theta}\in\mathcal{N}_{w,K_{\infty},Q_{\infty}}$. We claim that there exists $\theta_{0}>0$ such that $I_{w,K_{\infty},Q_{\infty}}(t_{\theta_{0}}u_{\theta_{0}})<m_{\infty}$. For convenience, we denote $u=t_{\theta_{0}}u_{\theta_{0}}$. In fact, if $I_{w,K_{\infty},Q_{\infty}}(t_{\theta}u_{\theta})\geq m_{\infty}$, for all $\theta>0$. From the definition of $\eta(x)$, using Lemma 5 in [37], $u_{\theta}\rightarrow v$ in $H^{s}(\mathbb{R}^{3})$ as $\theta\rightarrow\infty$. Recall that $v\in\mathcal{N}_{w,K_{\infty},Q_{\infty}}$, $\mathcal{N}_{w,K_{\infty},Q_{\infty}}$ is closed in $H^{s}(\mathbb{R}^{3})$, thus we obtain that $t_{\theta}\rightarrow 1$. Thus

[TABLE]

which is impossible. Hence our claim is true. Since the compact support set of $u$ denoted by ${\rm supp}u$ is compact and $V(0)=V_{0}$, we can choose $\varepsilon^{\ast}>0$ small enough such that $V(\varepsilon x)\leq w$ for all $x\in{\rm supp}u$ and $\varepsilon\in(0,\varepsilon^{\ast})$. Thus, we have

[TABLE]

which implies that $c_{\varepsilon}<m_{\infty}$ for $\varepsilon\in(0,\varepsilon^{\ast})$.

∎

Proposition 4.6**.**

Suppose that $(V)$, $(Q_{0})$, $(Q_{1})$, $(K)$, $(f_{0})$–$(f_{2})$ hold and $s\in(\frac{3}{4},1)$. Then there exists $\widetilde{\varepsilon}_{0}>0$ small, such that for each $\varepsilon\in(0,\widetilde{\varepsilon}_{0})$, problem (2.5) has at least a positive ground state solution $u_{\varepsilon}$ satisfying $\lim\limits_{|x|\rightarrow\infty}u_{\varepsilon}(x)=0$.

Proof.

From Lemma 3.3, Lemma 3.4, Lemma 4.5 and Lemma 3.5, there exists a small $\widetilde{\varepsilon}_{0}>0$, such that $I_{\varepsilon}$ has a nontrivial critical point $u_{\varepsilon}\in H_{\varepsilon}$ for $\varepsilon\in(0,\widetilde{\varepsilon}_{0})$. Hence, $u_{\varepsilon}$ is a nontrivial ground state solution of problem (2.5). Similar arguments as Proposition 4.4, we can prove that for $\varepsilon\in(0,\widetilde{\varepsilon}_{0})$, $u_{\varepsilon}$ is a positive ground state solution for problem (2.5) with $\lim\limits_{|x|\rightarrow\infty}u_{\varepsilon}(x)=0$. ∎

In the end of this section, we will establish the relation between $\lim\limits_{\varepsilon\rightarrow 0}c_{\varepsilon}$ and $m_{0}$. We state the relation as the following Lemma.

Lemma 4.7**.**

$\lim\limits_{\varepsilon\rightarrow 0}c_{\varepsilon}=m_{0}$.

Proof.

First, we show that there exists $\varepsilon_{1}>0$ such that $c_{\varepsilon}\geq m_{0}$ for all $\varepsilon\in(0,\varepsilon_{1})$. We suppose by contradiction that for any given $\varepsilon_{1}>0$, there exists some $\varepsilon_{0}\in(0,\varepsilon_{1})$ such that $c_{\varepsilon_{0}}<m_{0}$. By Proposition 4.6, we can choose $u_{0}$ be a ground state solution of problem (2.5) such that $I_{\varepsilon_{0}}(u_{0})=c_{\varepsilon_{0}}=\max_{t\geq 0}I_{\varepsilon_{0}}(tu_{0})<m_{0}$. Similar to the proof of $(ii)$ of Lemma 3.2, there exists $t_{0}>0$ such that $t_{0}u_{0}\in\mathcal{N}_{0}$ such that $I_{0}(t_{0}u_{0})=\max\limits_{t\geq 0}I_{0}(tu_{0})$. Hence, by $V(\varepsilon_{0}x)\geq V_{0}$, $K(\varepsilon_{0}x)\leq K_{0}$ and $Q(\varepsilon_{0}x)\leq Q_{0}$, we get that

[TABLE]

which is a contradiction.

Next, we will prove that $\limsup\limits_{\varepsilon\rightarrow 0}c_{\varepsilon}\leq m_{0}$. Let $u_{0}$ be a nontrivial ground state solution of equation (4.1) with $\nu=V_{0}$, $\kappa=K_{0}$ and $\mu=Q_{0}$, that is $I_{0}(u_{0})=m_{0}$. For each $\theta>0$, set $u_{\theta}(x)=\eta(x/\theta)u_{0}(x)$, where $\eta\in C_{0}^{\infty}(\mathbb{R}^{3},[0,1])$ be such that $\eta(x)=1$ if $|x|\leq 1$, and $\eta(x)=0$ if $|x|\geq 2$. From the definition of $\eta(x)$, using Lemma 5 in [37], $u_{\theta}\rightarrow u_{0}$ in $H^{s}(\mathbb{R}^{3})$ as $\theta\rightarrow\infty$.

For each $\varepsilon>0$, $\theta>0$, there exists $t_{\varepsilon,\theta}>0$ such that $I_{\varepsilon}(t_{\varepsilon,\theta}u_{\theta})=\max_{t\geq 0}I_{\varepsilon}(tu_{\theta})$. Thus, $\langle I_{\varepsilon}^{\prime}(t_{\varepsilon,\theta}u_{\theta}),t_{\varepsilon,\theta}u_{\theta}\rangle=0$, that is,

[TABLE]

Similar to Lemma 3.5 or Lemma 4.2, we can deduce that for each $\theta>0$, $0<\lim\limits_{\varepsilon\rightarrow 0}t_{\varepsilon,\theta}=t_{\theta}<\infty$. Taking the limit as $\varepsilon\rightarrow 0$ in (4), we get

[TABLE]

This yields to $t_{\theta}u_{\theta}\in\mathcal{N}_{0}$, i.e., $I_{0}(t_{\theta}u_{\theta})=\max\limits_{t\geq 0}I_{0}(tu_{\theta})$. Recall that $u_{\theta}\rightarrow u_{0}$ in $H^{s}(\mathbb{R}^{3})$ as $\theta\rightarrow\infty$ and $u_{0}\in\mathcal{N}_{0}$, thus, using $(ii)$ of Lemma 3.2, we have $t_{\theta}\rightarrow 1$ as $\theta\rightarrow\infty$. By the definition of $c_{\varepsilon}$, we have that

[TABLE]

Let $\theta\rightarrow\infty$, we get that $\limsup\limits_{\varepsilon\rightarrow 0}c_{\varepsilon}\leq I_{0}(u_{0})=m_{0}$. The proof is completed. ∎

5. Concentration behavior

In this section, we study the concentration behavior of ground state solutions of system (1.1). In this section, we choose $H^{s}(\mathbb{R}^{3})$ as our work space since $H^{s}(\mathbb{R}^{3})=E_{\nu}=H_{\varepsilon}$ for any $\varepsilon>0$, $\nu>0$.

Proposition 4.6 tells us that there exists $\widetilde{\varepsilon}_{0}>0$, such that for each $\varepsilon\in(0,\widetilde{\varepsilon}_{0})$, problem (2.5) possesses a positive ground state solution $v_{\varepsilon}\in H^{s}(\mathbb{R}^{3})$ satisfying $I_{\varepsilon}(v_{\varepsilon})=c_{\varepsilon}$ and $I_{\varepsilon}^{\prime}(v_{\varepsilon})=0$. Now, we study the behavior of the family $\{v_{\varepsilon}\}$.

Lemma 5.1**.**

For the family $\{v_{\varepsilon}\}$ satisfying $I_{\varepsilon}(v_{\varepsilon})=c_{\varepsilon}$ and $I_{\varepsilon}^{\prime}(v_{\varepsilon})=0$, there exist $\widehat{\varepsilon}>0$, such that for all $\varepsilon\in(0,\widehat{\varepsilon})$, there exist a family $\{y_{\varepsilon}\}\subset\mathbb{R}^{3}$, and constants $R,\sigma>0$ such that

[TABLE]

Proof.

Suppose by contradiction that (5.1) does not happen. Then there exists a sequence $\varepsilon_{n}\rightarrow 0$ as $n\rightarrow\infty$ such that

[TABLE]

Taking a similar discussion as that in the proof of Lemma 3.5, we can easily obtain a contradiction. Hence, (5.1) holds. ∎

For simplicity, we denote

[TABLE]

By the fact that $I(v_{\varepsilon})=c_{\varepsilon}$ and $I_{\varepsilon}^{\prime}(v_{\varepsilon})=0$, so $w_{\varepsilon}$ is a positive ground state solution to the following equation

[TABLE]

Lemma 5.2**.**

The family $\{\varepsilon y_{\varepsilon}\}$ which obtained in Lemma 5.1 is bounded in $\mathbb{R}^{3}$ for any $\varepsilon\in(0,\widehat{\varepsilon})$.

Proof.

Suppose by contradiction that $\{\varepsilon y_{\varepsilon}\}$ is unbounded, then there exist two sequences $\varepsilon_{n}$ and $\{\varepsilon_{n}y_{\varepsilon_{n}}\}$ such that $\lim\limits_{n\rightarrow\infty}\varepsilon_{n}\rightarrow 0$ and $\lim\limits_{n\rightarrow\infty}|\varepsilon_{n}y_{\varepsilon_{n}}|=+\infty$. In the sequel, for simplicity, we denote $y_{n}=y_{\varepsilon_{n}}$ and $v_{n}=v_{\varepsilon_{n}}$. Set $w_{n}(\cdot)=v_{n}(\cdot+y_{n})$, then for each $n\in\mathbb{N}$, $w_{n}\geq 0$ satisfies that $I_{\varepsilon_{n}}(w_{n})=c_{\varepsilon_{n}}$ and $I_{\varepsilon_{n}}^{\prime}(w_{n})=0$, and from (5.1), we have that

[TABLE]

By Lemma 4.7, it is easy to check that $\{w_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$. Thus, up to a subsequence, still denoted by $\{w_{n}\}$, we assume that there exists $w\in H^{s}(\mathbb{R}^{3})$ such that $w_{n}\rightharpoonup w$ in $H^{s}(\mathbb{R}^{3})$, $w_{n}\rightarrow w$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for $2\leq r<2_{s}^{\ast}$ and $w_{n}\rightarrow w$ a.e. in $\mathbb{R}^{3}$. Obviously $w\geq 0$. Moreover, from (5.3), we see that $w\not\equiv 0$.

For each $n\in\mathbb{N}$, there exists $t_{n}>0$ such that $t_{n}w_{n}\in\mathcal{N}_{0}$. We claim that $\lim\limits_{n\rightarrow\infty}I_{0}(t_{n}w_{n})=m_{0}$. Since $t_{n}w_{n}\in\mathcal{N}_{0}$, then clearly $I_{0}(t_{n}w_{n})\geq m_{0}$. On the other hand, by Lemma 4.5, we have that

[TABLE]

which yields to $\limsup\limits_{n\rightarrow\infty}I_{0}(t_{n}w_{n})\leq m_{0}$ and hence the claim is true.

Since $\{w_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$, by (5), it is easy to get that $\{t_{n}\}$ is bounded. Thus, up to a subsequence, still denoted by $\{t_{n}\}$, we may assume that $\lim\limits_{n\rightarrow\infty}t_{n}=t\geq 0$. If $t=0$, in view of the boundedness of $\{w_{n}\}$ in $H^{s}(\mathbb{R}^{3})$, we see that $t_{n}w_{n}\rightarrow 0$ in $H^{s}(\mathbb{R}^{3})$, and thus $\lim\limits_{n\rightarrow\infty}I_{0}(t_{n}w_{n})=0$, a contradiction. Hence, $t>0$.

Observing that $\{t_{n}w_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$, up to a subsequence, still denoted by $\{t_{n}w_{n}\}$, we may assume that $t_{n}w_{n}\rightharpoonup\widehat{w}$ in $H^{s}(\mathbb{R}^{3})$. Since $w_{n}\rightharpoonup w$ in $H^{s}(\mathbb{R}^{3})$ and $t_{n}\rightarrow t$ as $n\rightarrow\infty$, then $t_{n}w_{n}\rightharpoonup tw$ in $H^{s}(\mathbb{R}^{3})$ as $n\rightarrow\infty$. By the uniqueness of weak limit, it yields to $\widehat{w}=tw$. Therefore, we obtain a bounded minimizing sequence $\{t_{n}w_{n}\}\subset\mathcal{N}_{0}$ as $n\rightarrow\infty$ and $\langle I_{0}^{\prime}(t_{n}w_{n}),t_{n}w_{n}\rangle=0$ for any $n\in\mathbb{N}$. Similar proof as that done in the proof of $(vi)$ of Lemma 3.2, we may assume that $\{t_{n}w_{n}\}$ is a $(PS)_{m_{0}}$ sequence for $I_{0}$. By Lemma 4.1, using similar argument as the proof of Proposition 4.4, we can conclude that $t_{n}w_{n}\rightarrow tw$ in $H^{s}(\mathbb{R}^{3})$. Moreover, $tw\in\mathcal{N}_{0}$. Thus,

[TABLE]

that is, $w_{n}\rightarrow w$ in $H^{s}(\mathbb{R}^{3})$ as $n\rightarrow\infty$. Therefore, by Fatou’s Lemma and $t_{n}w_{n}\in\mathcal{N}_{0}$, recalling that $\varepsilon_{n}\rightarrow 0$ and $|\varepsilon_{n}y_{n}|\rightarrow\infty$ as $n\rightarrow\infty$, we have that

[TABLE]

which yields a contradiction. Thus, $\{\varepsilon y_{\varepsilon}\}$ is bounded in $\mathbb{R}^{3}$. ∎

For any $\varepsilon_{n}\rightarrow 0$, the subsequence $\{\varepsilon_{n}y_{\varepsilon_{n}}\}$ of the family $\{\varepsilon y_{\varepsilon}\}$ is such that $\varepsilon_{n}y_{\varepsilon_{n}}\rightarrow x^{\ast}$ in $\mathbb{R}^{3}$, we will prove that $x^{\ast}\in\Theta$.

Lemma 5.3**.**

$x^{\ast}\in\Theta$.

Proof.

Set

[TABLE]

Suppose that $V(x^{\ast})>V_{0}$. Taking the similar arguments of Lemma 5.2 and replacing $I_{\infty}$ by $I^{x^{\ast}}$ in (5), we can obtain a contradiction. Hence, $x^{\ast}\in\Theta_{V}$. By similar discussion, we can obtain a contradiction in the case $x^{\ast}\in\Theta_{K}\cap\Theta_{Q}$. Therefore, $x^{\ast}\in\Theta=\Theta_{V}\cap\Theta_{K}\cap\Theta_{Q}$ and the proof is completed. ∎

Since $w_{\varepsilon}$ is a positive ground state solution of problem (5.2) and $I_{\varepsilon}(w_{\varepsilon})=c_{\varepsilon}$ (using the invariance of translation), by Lemma 4.7, it is easy to check that there exists $\widetilde{\varepsilon}>0$ such that for any $\varepsilon\in(0,\widetilde{\varepsilon})$, $w_{\varepsilon}$ is bounded in $H^{s}(\mathbb{R}^{3})$ by a constant which is independent of $\varepsilon$. Hence, for any $\varepsilon_{n}\rightarrow 0$, the subsequence $\{w_{\varepsilon_{n}}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$, we may assume that up to a subsequence, $w_{\varepsilon_{n}}\rightharpoonup w_{0}$ in $H^{s}(\mathbb{R}^{3})$ and by Lemma 5.2, up to a subsequence, we also may assume that $\varepsilon_{n}y_{\varepsilon_{n}}\rightarrow x_{0}\in\Theta$ as $n\rightarrow\infty$.

Lemma 5.4**.**

$w_{\varepsilon_{n}}\rightarrow w_{0}$* in $H^{s}(\mathbb{R}^{3})$ and $w_{0}$ is a positive ground state solution of the following problem*

[TABLE]

Proof.

Similar proof of Lemma 5.2, it is easy to check that $w_{\varepsilon_{n}}\rightarrow w_{0}$ in $H^{s}(\mathbb{R}^{3})$. Therefore, $\langle I_{\varepsilon_{n}}^{\prime}(w_{\varepsilon_{n}}),\varphi\rangle=\langle(I^{x_{0}})^{\prime}(w_{0}),\varphi\rangle$, for any $\varphi\in H^{s}(\mathbb{R}^{3})$. This means that $w_{0}$ is a nontrivial ground state solution of problem (5.6). By the similar argument of Proposition 4.4, we can complete the proof. ∎

Moreover, we have the following vanishing estimate of $\{w_{\varepsilon}\}$ at infinity.

Lemma 5.5**.**

$\lim\limits_{|x|\rightarrow\infty}w_{\varepsilon}(x)=0$* uniformly in $\varepsilon\in(0,\widetilde{\varepsilon})$.*

Proof.

For any $\varepsilon_{n}\rightarrow 0$, $w_{n}:=w_{\varepsilon_{n}}$ is a positive ground state solution of problem (5.2), then $f(x,w_{n}):=K(\varepsilon_{n}x+\varepsilon_{n}y_{\varepsilon_{n}})f(w_{n})+Q(\varepsilon_{n}x+\varepsilon_{n}y_{\varepsilon_{n}})|w_{n}|^{2_{s}^{\ast}-2}w_{n}-V(\varepsilon_{n}x+\varepsilon_{n}y_{\varepsilon_{n}})w_{\varepsilon_{n}}+\phi_{w_{n}}^{t}w_{n}\leq C(1+|w_{n}|^{2_{s}^{\ast}-1})$, where $C$ is independent of $n$ and $w_{n}$, by the estimate (4), we have that $\|w_{n}\|_{\infty}\leq C\|w_{n}\|^{\alpha}\leq C$, where $C>0$ is a constant independent of $n$.

Now we borrow the ideas in [2] to complete the proof. For this purpose, we rewrite problem (5.2) as follows

[TABLE]

where $g_{n}(x):=w_{n}+f(x,w_{n})$. Clearly, $g_{n}\in L^{\infty}(\mathbb{R}^{3})$ and is uniformly bounded. From Lemma 5.4, for $n\rightarrow\infty$, we have that $g_{n}\rightarrow g_{0}$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r\leq 2_{s}^{\ast}$, where $g_{0}=w_{0}+K(x_{0})f(w_{0})+Q(x_{0})|w_{0}|^{2_{s}^{\ast}-2}w_{0}-V(x_{0})w_{0}-\phi_{w_{0}}w_{0}$. Using some results found in [22], we see that

[TABLE]

where $\mathcal{K}$ is a Bessel potential, which possesses the following properties:

$(\mathcal{K}_{1})$ $\mathcal{K}$ is positive, radially symmetric and smooth in $\mathbb{R}^{3}\backslash\{0\}$;

$(\mathcal{K}_{2})$ there exists a constant $C>0$ such that $\mathcal{K}(x)\leq\frac{C}{|x|^{3+2s}}$ for all $x\in\mathbb{R}^{3}\backslash\{0\}$;

$(\mathcal{K}_{3})$ $\mathcal{K}\in L^{\tau}(\mathbb{R}^{3})$ for $\tau\in[1,\frac{3}{3-2s})$.

We define two sets $A_{\delta}=\{y\in\mathbb{R}^{3}\,\,|\,\,|x-y|\geq\frac{1}{\delta}\}$ and $B_{\delta}=\{y\in\mathbb{R}^{3}\,\,|\,\,|x-y|<\frac{1}{\delta}\}$. Hence,

[TABLE]

From the definition of $A_{\delta}$ and $(\mathcal{K}_{2})$, we have that for all $n\in\mathbb{N}$,

[TABLE]

On the other hand, by Hölder’s inequality and $(\mathcal{K}_{3})$, we deduce that

[TABLE]

where we have used the fact that $s>\frac{3}{4}$ so that $\frac{6}{3+2s}<\frac{3}{3-2s}$ and $2<\frac{3}{3-2s}$.

Since \Big{(}\int_{B_{\delta}}|g_{0}|^{2}\,{\rm d}y\Big{)}^{\frac{1}{2}}\rightarrow 0 as $|x|\rightarrow+\infty$, thus, we deduce that there exist $n_{0}\in\mathbb{N}$ and $R_{0}>0$ independence of $\varepsilon>0$ such that

[TABLE]

Hence,

[TABLE]

For each $n\in\{1,2,\cdots,n_{0}-1\}$, there exists $R_{n}>0$ such that \Big{(}\int_{B_{\delta}}|g_{n}|^{2}\,{\rm d}y\Big{)}^{\frac{1}{2}}<\delta as $|x|\geq R_{n}$. Thus, for $|x|\geq R_{n}$, we have that

[TABLE]

for each $n\in\{1,2,\cdots,n_{0}-1\}$. Therefore, taking $R=\max\{R_{0},R_{1},\cdots,R_{n_{0}-1}\}$, we infer that for any $n\in\mathbb{N}$, there holds

[TABLE]

implies that $\lim\limits_{|x|\rightarrow\infty}w_{n}(x)=0$ uniformly in $n\in\mathbb{N}$. As a result, the conclusion follows from the arbitrariness of $\varepsilon_{n}$. ∎

Now, we give the estimate of decay properties of solutions $u_{\varepsilon}$.

Lemma 5.6**.**

There exists a constant $C>0$ such that

[TABLE]

Proof.

We borrow some ideas of the proof of Theorem 1.1 in [24] to give the proof of Lemma 5.6. By Lemma 4.2 and Lemma 4.3 in [22], by scaling, there exists a continuous function $W$ such that

[TABLE]

and

[TABLE]

for some suitable $R>0$. By Lemma 5.5, there exists $R_{1}>0$ (we can choose $R_{1}>R$) large enough such that

[TABLE]

for any $x\in\mathbb{R}^{3}\backslash B_{R_{1}}(0)$. Therefore, we have obtained that

[TABLE]

Let $\mathbb{A}=\inf\limits_{B_{R_{1}}(0)}W>0$, $Z_{\varepsilon}(x)=(\mathbb{B}+1)W-\mathbb{A}w_{\varepsilon}$, where $\mathbb{B}=\sup\limits_{0<\varepsilon<\widetilde{\varepsilon}}\|w_{\varepsilon}\|_{\infty}\leq C<\infty$, where $C$ is some positive constant independent of $\varepsilon$. We claim that $Z_{\varepsilon}(x)\geq 0$ for all $x\in\mathbb{R}^{3}$ and $\varepsilon\in(0,\widetilde{\varepsilon})$. If the claim is true, we have that

[TABLE]

and the conclusion is proved.

Suppose by contradiction that there exist $\varepsilon_{0}\in(0,\widetilde{\varepsilon}_{1})$ and $x_{\varepsilon_{0}}^{n}\in\mathbb{R}^{3}$ such that

[TABLE]

Since $\lim\limits_{|x|\rightarrow\infty}W(x)=\lim\limits_{|x|\rightarrow\infty}w_{\varepsilon}(x)=0$ uniformly for $\varepsilon\in(0,\widetilde{\varepsilon}_{1})$, then $\lim\limits_{|x|\rightarrow\infty}Z_{\varepsilon_{0}}(x)=0$. Hence, the sequence $\{x_{\varepsilon_{0}}^{n}\}$ is bounded and then up to a subsequence, we may assume that $x_{\varepsilon_{0}}^{n}\rightarrow\widetilde{x}_{\varepsilon_{0}}$. From (5.9), we have that

[TABLE]

By (5.10), we get

[TABLE]

Note that $Z_{\varepsilon_{0}}(x)\geq\mathbb{A}\mathbb{B}+W-\mathbb{A}\mathbb{B}>0$ on $B_{R_{1}}(0)$, hence $\widetilde{x}_{\varepsilon_{0}}\in\mathbb{R}^{3}\backslash B_{R_{1}}(0)$. From (5.8), by computation, we have that

[TABLE]

which is a contradiction. Thus, the claim holds true and the proof is completed.

∎

Proof of Theorem 1.1. $(i)$ Taking $\varepsilon_{0}=\min\{\widetilde{\varepsilon}_{0},\widetilde{\varepsilon},\widehat{\varepsilon}\}$. By Proposition 4.6, for each $\varepsilon\in(0,\varepsilon_{0})$, problem (2.5) has at least a positive ground state solution $v_{\varepsilon}$. Hence, let $\phi_{\varepsilon}=\phi_{v_{\varepsilon}}^{t}$, then $(v_{\varepsilon},\phi_{\varepsilon})$ is a positive solution for system (1.1) for $\varepsilon\in(0,\varepsilon_{0})$.

$(ii)$ From Proposition 4.2 and Lemma 5.5, there exists $R>0$ such that the global maximum point of $w_{\varepsilon}(x)=v_{\varepsilon}(x+y_{\varepsilon})$, denoted by $\widetilde{x}_{\varepsilon}$, is located in $B_{R}(0)$. Thus, the global maximum point of $v_{\varepsilon}$ is given by $x_{\varepsilon}=\widetilde{x}_{\varepsilon}+y_{\varepsilon}$. Observing that $u_{\varepsilon}(x)=v_{\varepsilon}(x/\varepsilon)$, then we have that $(u_{\varepsilon}(x),\phi_{u_{\varepsilon}}^{t}(x))$ is a positive ground state solution of system (1.1) and $u_{\varepsilon}$ has a global maximum point $z_{\varepsilon}=\varepsilon x_{\varepsilon}$. It follows from Lemma 5.2, Lemma 5.3 and $\widetilde{x}_{\varepsilon}\in B_{R}(0)$ that $\lim\limits_{\varepsilon\rightarrow 0}V(z_{\varepsilon})=V_{0}$, $\lim\limits_{\varepsilon\rightarrow 0}K(z_{\varepsilon})=K_{0}$ and $\lim\limits_{\varepsilon\rightarrow 0}Q(z_{\varepsilon})=Q_{0}$. Moreover, in view of Lemma 5.4, we see that $z_{\varepsilon}\rightarrow x_{0}$ if $\varepsilon\rightarrow 0$, then $u_{\varepsilon}(\varepsilon x+z_{\varepsilon})$ converges to $u$ and $u$ is a solution for problem (5.6). As a result, $(u,\phi)$ is a solution of system (1.5).

$(iii)$ By Lemma 5.6, we have that

[TABLE]

where $C_{0}=1-R^{3+2s}$.

6. Nonexistence of ground states

In this section, our goal is to show the nonexistence of ground state solution to system (1.1), that is, for each $\varepsilon>0$, the ground energy $c_{\varepsilon}$ is not attained.

Lemma 6.1**.**

Assume the continuous functions $V(x)$, $K(x)$, $Q(x)$ satisfies $(\mathcal{H})$ and $(f_{0})$–$(f_{2})$ hold. Then for each $\varepsilon>0$, $c_{\varepsilon}=m_{\infty}$.

Proof.

Noting that $H_{\varepsilon}=H^{s}(\mathbb{R}^{3})$, for any $\varepsilon>0$. By $(\mathcal{H})$, we have $I_{\infty}(u)\leq I_{\varepsilon}(u)$, for all $u\in H^{s}(\mathbb{R}^{3})$. By $(ii)$ of Lemma 3.2, for each $u\in\mathcal{N}_{\infty}$, there exists $t_{u}>0$ such that $t_{u}u\in\mathcal{N}_{\varepsilon}$. Hence, for each $u\in\mathcal{N}_{\infty}$, we have that

[TABLE]

By $(i)$ of Lemma 3.2, one has

[TABLE]

So, it suffices to show that $c_{\varepsilon}\leq m_{\infty}$.

By Proposition 4.4, there exists $u_{\infty}\in\mathcal{N}_{\infty}$ is a ground state solution of (2.6) with $\nu=V_{\infty}$, $\kappa=K_{\infty}$ and $\mu=Q_{\infty}$. Set $e_{n}(x)=u_{\infty}(x-y_{n})$ where $y_{n}\in\mathbb{R}^{3}$ and $|y_{n}|\rightarrow\infty$ as $n\rightarrow\infty$. Then, there exists $t_{n}(e_{n})>0$ such that $t_{n}e_{n}\in\mathcal{N}_{\varepsilon}$, that is,

[TABLE]

This implies that $t_{n}$ cannot converge to zero and infinity. Suppose that $t_{n}\rightarrow t$ as $n\rightarrow\infty$. Letting $n\rightarrow\infty$ in (6), we have that

[TABLE]

In view of $u_{\infty}\in\mathcal{N}_{\infty}$, we conclude that $t=1$. Since

[TABLE]

By the assumption on $V$, for any $\delta>0$, there exists $R>0$ such that

[TABLE]

By $|y_{n}|\rightarrow\infty$, according to Lebesgue s theorem, we have

[TABLE]

Thus,

[TABLE]

Similarly, we deduce that

[TABLE]

Therefore, using $t_{n}\rightarrow 1$ and letting $n\rightarrow\infty$ in (6), we infer that $c_{\varepsilon}\leq m_{\infty}$ and the proof is completed. ∎

Proof of Theorem 1.2. Suppose by contradiction that there exist some $\varepsilon_{0}>0$ and $u_{0}\in\mathcal{N}_{\varepsilon_{0}}$ such that $I_{\varepsilon_{0}}(u_{0})=c_{\varepsilon_{0}}$. From Lemma 6.1, $c_{\varepsilon_{0}}=m_{\infty}$. In view of Lemma 3.2, there exists $t_{0}>0$ such that $t_{0}u_{0}\in\mathcal{N}_{\infty}$. Thus, using the fact $u_{0}\in\mathcal{N}_{\varepsilon_{0}}$, we have that

[TABLE]

Thus, $m_{\infty}=I_{\infty}(t_{0}u_{0})=I_{\varepsilon_{0}}(t_{0}u_{0})$. However,

[TABLE]

which is a contradiction. The proof is completed.

7. Appendix

In this section, By a similar argument of Section 4 in [42], we give the estimate of (3.4), (3.5) and (3.6). We prove them in the general case.

Let $\mathcal{S}_{s}=\mathcal{S}_{s}(\widetilde{u})$, $\widetilde{u}=\kappa(\mu^{2}+|x-x_{0}|^{2})^{-\frac{N-2s}{2}}$, $x\in\mathbb{R}^{N}$ with $\kappa\in\mathbb{R}\backslash\{0\}$, $\mu>0$ and $x_{0}\in\mathbb{R}^{N}$ are fixed constant. Using scaling and translation transform, we see that

[TABLE]

Set

[TABLE]

It is easy to check that

[TABLE]

Let $\eta\in C_{0}^{\infty}(\mathbb{R}^{N})$ be such that $0\leq\eta\leq 1$, \eta\Big{|}_{B_{R}(0)}=1, ${\rm supp}\eta\subset B_{2R}(0)$. Set $u_{\varepsilon}(x)=\eta(x-x_{0}/\varepsilon)U_{\varepsilon}(x)$, for any $\varepsilon>0$ and $x\in\mathbb{R}^{N}$. By computation, we have the following estimate for $u_{\varepsilon}$.

Lemma A.1. $(i)$ Let $\rho>0$. If $x\in\mathbb{R}^{N}\backslash B_{\rho}(x_{0}/\varepsilon)$, then

[TABLE]

for any $\varepsilon>0$ and for some positive constants $C_{1}$ and $C_{2}$, possibly depending on $N,s,\rho$.

$(ii)$ For any $x\in\mathbb{R}^{N}$ and $y\in\mathbb{R}^{N}\backslash B_{R}(x_{0}/\varepsilon)$, with $|x-y|\leq\frac{R}{2}$,

[TABLE]

$(iii)$ For any $x,y\in\mathbb{R}^{N}\backslash B_{R}(x_{0}/\varepsilon)$,

[TABLE]

for any $\varepsilon>0$ and for some positive constants $C_{3}$ and $C_{4}$, possibly depending on $N,s,\rho$.

Proof.

$(i)$ By the definition of $u_{\varepsilon}$, for any $x\in\mathbb{R}^{N}\backslash B_{\rho}(x_{0}/\varepsilon)$, we have

[TABLE]

and

[TABLE]

$(ii)$ Let $x\in\mathbb{R}^{N}$, $y\in\mathbb{R}^{N}\backslash B_{R}(x_{0}/\varepsilon)$, with $|x-y|\leq\frac{R}{2}$, and let $\xi$ be any point on the segment joining $x$ and $y$. Then $\xi=tx+(1-t)y$, $t\in[0,1]$. Hence

[TABLE]

Taking $\rho=\frac{R}{2}$ in $(i)$, implies that $|\nabla u_{\varepsilon}(\xi)|\leq C\varepsilon^{\frac{N-2s}{2}}$ and so by Taylor expansion,

[TABLE]

$(iii)$ Let $x,y\in\mathbb{R}^{N}\backslash B_{R}(x_{0}/\varepsilon)$. If $|x-y|\leq\frac{R}{2}$, then the second conclusion of $(ii)$ follows from (7.2). If $|x-y|>\frac{R}{2}$, using the conclusion $(i)$, we have that

[TABLE]

The proof is completed. ∎

Lemma A.2. Let $s\in(0,1)$ and $n>2s$. Then the following estimate holds true

[TABLE]

as $\varepsilon\rightarrow 0$.

Proof.

By the definition of $u_{\varepsilon}$, we can rewrite $\|(-\Delta)^{\frac{s}{2}}u_{\varepsilon}\|_{2}^{2}$ as follows

[TABLE]

where the sets $\mathbb{D}$ and $\mathbb{E}$ are given by

[TABLE]

and

[TABLE]

Similar to the proof of Proposition 21 in [42], using Lemma A.1, it is easy to show that, as $\varepsilon\rightarrow 0$, there hold

[TABLE]

and

[TABLE]

To complete the proof, checking the proof of Proposition 21 in [42], it is sufficient to verify the following estimate holds

[TABLE]

as $\varepsilon\rightarrow 0$.

In fact, by $(i)$ of Lemma A.1, making the change of variables $\xi=x-y$, we deduce that

[TABLE]

where $\mathbb{D}_{\varepsilon}$ is defined as

[TABLE]

Finally, combing with (7.2), (7.3), we conclude that

[TABLE]

as $\varepsilon\rightarrow 0$. ∎

Lemma A.3. Let $s\in(0,1)$ and $N>2s$, $p\in(1,2_{s}^{\ast})$. Then the following estimates hold.

[TABLE]

and

[TABLE]

as $\varepsilon\rightarrow 0$.

Proof.

By the definition of $u_{\varepsilon}$, we have

[TABLE]

for any $0<\delta<R/\varepsilon$. Therefore,

[TABLE]

On the other hand, arguing in the same way, we infer that

[TABLE]

where $0<\delta_{1}<2R/\varepsilon$. Hence, (7.4) holds true.

$(ii)$

[TABLE]

as $\varepsilon\rightarrow 0$. ∎

Acknowledgements. This work was done when the first author visited Department of Mathematics, Texas A$\&$M University-Kingsville under the support of China Scholarship Council (201508140053), and he thanks Department of Mathematics, Texas A$\&$M University-Kingsville for their kind hospitality. The first author is also supported by NSFC grant 11501403.