Concentrating bounded states for fractional Schr\"{o}dinger-Poisson system involving critical Sobolev exponent

TL;DR

This paper investigates the existence and multiplicity of positive solutions to a fractional Schr"odinger-Poisson system with critical Sobolev exponent, showing solutions concentrate near potential minima as the parameter approaches zero.

Contribution

It establishes the concentration of solutions around local minima of the potential and employs topological methods to prove multiple solutions exist.

Findings

Solutions concentrate near local minima of V as epsilon approaches zero.

Multiple solutions are obtained using Ljusternik-Schnirelmann theory.

The results extend understanding of fractional Schr"odinger-Poisson systems with critical exponents.

Abstract

In this paper, we study the concentration and multiplicity of solutions to the following fractional Schr\"{o}dinger-Poisson system \begin{equation*} \left\{ \begin{array}{ll} \varepsilon^{2s}(-\Delta)^su+V(x)u+\phi u=f(u)+u^{2_s^{\ast}-1} & \hbox{in $\mathbb{R}^3$,} \varepsilon^{2t}(-\Delta)^t\phi=u^2, u>0& \hbox{in $\mathbb{R}^3$,} \end{array} \right. \end{equation*} where $s>\frac{3}{4}$, $s,t\in(0,1)$, $\varepsilon>0$ is a small parameter, $f\in C^1(\mathbb{R}^{+},\mathbb{R})$ is subcritical, $V:\mathbb{R}^3\rightarrow\mathbb{R}$ is a continuous bounded function. We establish a family of positive solutions $u_{\varepsilon}\in H_{\varepsilon}$ which concentrates around the local minima of $V$ in $\Lambda$ as $\varepsilon\rightarrow0$. With Ljusternik-Schnirelmann theory, we also obtain multiple solutions by employing the topology construct of the set where the potential $V$ attains its minimum.

Equations

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=f(u)+u^{2_{s}^{\ast}-1}&\hbox{in $\mathbb{R}^{3}$,}\\ \varepsilon^{2t}(-\Delta)^{t}\phi=u^{2},u>0&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$$V_0=\underset{\Lambda }{\inf }V(x)<\underset{\partial \Lambda }{\min }V(x).$$

$$(-\Delta {)}^{s}u(x)=C_{s}P.V.{\int }_{{\mathbb{R}}^3}\frac{u(x)-u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y=C_s\underset{\epsilon \to 0}{\lim }{\int }_{{\mathbb{R}}^3\backslash B_{\epsilon }(x)}\frac{u(x)-u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y$$

$$(-\Delta)^{s}u=\mathcal{F}^{-1}\Big{(}(2\pi|\xi|)^{2s}\mathcal{F}u(\xi)\Big{)},$$

$$(-\Delta {)}^{s}u(x)=-\frac{1}{2}{C}_s{\int }_{{\mathbb{R}}^3}\frac{u(x+y)+u(x-y)-2u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(x)u+\phi u=f(u)&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=f(u)&\hbox{in $\mathbb{R}^{N}$,}\\ \varepsilon^{\theta}(-\Delta)^{\frac{\alpha}{2}}\phi=\gamma_{\alpha}u^{2}&\hbox{in $\mathbb{R}^{N}$,}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=K(x)f(u)+Q(x)|u|^{2_{s}^{\ast}-2}u&\hbox{in $\mathbb{R}^{3}$,}\\ \varepsilon^{2t}(-\Delta)^{t}\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}-\varepsilon^{2}\Delta u+V(x)u+K(x)\phi u=u^{p}&\hbox{in $\mathbb{R}^{3}$,}\\ -\Delta\phi=K(x)u^{2}&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}-\varepsilon^{2}\Delta u+V(x)u+\lambda\phi u=f(x,u)&\hbox{in $\mathbb{R}^{3}$,}\\ -\varepsilon^{2}\Delta\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$u_{\epsilon }(x)\le \frac{C{\epsilon }^{3+2s}}{C_0{\epsilon }^{3+2s}+|x-x_{\epsilon }{|}^{3+2s}}x\in {\mathbb{R}}^3,\text{and}\epsilon \in (0,{\epsilon }_0)$$

$$(V_2)\mathcal{M}=\{x\in \Lambda |V(x)=\underset{{\mathbb{R}}^3}{\inf }V\}\ne \varnothing .$$

$${\mathcal{M}}_{\delta }=\{x\in {\mathbb{R}}^3|\mathrm{dist}(x,\mathcal{M})\le \delta \}.$$

$$u_{\epsilon }(x)\le \frac{C{\epsilon }^{3+2s}}{C_0{\epsilon }^{3+2s}+|x-x_{\epsilon }{|}^{3+2s}}x\in {\mathbb{R}}^3,\text{and}\epsilon \in (0,{\epsilon }_{\delta })$$

$$\mathcal{D}^{s,2}(\mathbb{R}^{3})=\Big{\{}u\in L^{2_{s}^{\ast}}(\mathbb{R}^{3})\,\,\Big{|}\,\,|\xi|^{s}\widehat{u}(\xi)\in L^{2}(\mathbb{R}^{3})\Big{\}}$$

$$\|u\|_{\mathcal{D}^{s,2}}=\Big{(}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{s}{2}}u|^{2}\,{\rm d}x\Big{)}^{\frac{1}{2}}=\Big{(}\int_{\mathbb{R}^{3}}|\xi|^{2s}|\widehat{u}(\xi)|^{2}\,{\rm d}\xi\Big{)}^{\frac{1}{2}}$$

$$H^{s}(\mathbb{R}^{3})=\Big{\{}u\in L^{2}(\mathbb{R}^{3})\,\,\Big{|}\,\,\int_{\mathbb{R}^{3}}(|\xi|^{2s}|\widehat{u}(\xi)|^{2}+|\widehat{u}(\xi)|^{2})\,{\rm d}\xi<+\infty\Big{\}}.$$

$$(u,v)={\int }_{{\mathbb{R}}^3}(|\xi {|}^{2s}\hat{u}(\xi )\overline{\hat{v}(\xi )}+\hat{u}(\xi )\overline{\hat{v}(\xi )})\mathrm{d}\xi$$

$$\|u\|_{H^{s}}=\bigg{(}\int_{\mathbb{R}^{3}}(|\xi|^{2s}|\widehat{u}(\xi)|^{2}+|\widehat{u}(\xi)|^{2})\,{\rm d}\xi\bigg{)}^{\frac{1}{2}},$$

$$\|u\|_{H^{s}}=\bigg{(}\int_{\mathbb{R}^{3}}(|(-\Delta)^{\frac{s}{2}}u(x)|^{2}+|u(x)|^{2})\,{\rm d}x\bigg{)}^{\frac{1}{2}},\quad\forall u\in H^{s}(\mathbb{R}^{3}).$$

$$\|u\|_{H_{\varepsilon}}=\Big{(}\int_{\mathbb{R}^{3}}\Big{(}|(-\Delta)^{\frac{s}{2}}u|^{2}+V(\varepsilon x)u^{2}\Big{)}\,{\rm d}x\Big{)}^{\frac{1}{2}}.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(\varepsilon z)u+\phi u=f(u)+u^{2_{s}^{\ast}-1}&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2},u>0&\hbox{in $\mathbb{R}^{3}$}\end{array}\right.$$

$$f^{\prime }(\tau )\tau -3f(\tau )>0\text{and}f(\tau )\tau -4F(\tau )>0\text{for any}\tau >0.$$

$${\tilde{\mathcal{L}}}_u(v)={\int }_{{\mathbb{R}}^3}{u}^{2}v\mathrm{d}z.$$

$${\int }_{{\mathbb{R}}^3}(-\Delta {)}^{\frac{t}{2}}{\varphi }_u^t(-\Delta {)}^{\frac{t}{2}}v\mathrm{d}z={\int }_{{\mathbb{R}}^3}{u}^{2}v\mathrm{d}z,\forall v\in {\mathcal{D}}^{t,2}({\mathbb{R}}^3),$$

$${\varphi }_u^t(z)=c_t{\int }_{{\mathbb{R}}^3}\frac{u^2(y)}{|z-y{|}^{3-2t}}\mathrm{d}y,z\in {\mathbb{R}}^3,c_t={\pi }^{-\frac{3}{2}}{2}^{-2t}\frac{\Gamma (\frac{3-2t}{2})}{\Gamma (t)}.$$

$$(-\Delta {)}^{s}u+V(\epsilon z)u+{\varphi }_u^{t}u=f(u)+u^{2_s^{\ast }-1},z\in {\mathbb{R}}^3.$$

$$J_{\varepsilon}(u)=\frac{1}{2}\int_{\mathbb{R}^{3}}\Big{(}|(-\Delta)^{\frac{s}{2}}u|^{2}+V(\varepsilon z)u^{2}\Big{)}\,{\rm d}z+\frac{1}{4}\int_{\mathbb{R}^{3}}\phi_{u}^{t}u^{2}\,{\rm d}z-\int_{\mathbb{R}^{3}}F(u)\,{\rm d}z-\int_{\mathbb{R}^{3}}(u^{+})^{2_{s}^{\ast}}\,{\rm d}z$$

$$\Vert \Phi (u){\Vert }_{{\mathcal{D}}^{t,2}}\le C\Vert u{\Vert }_{\frac{12}{3+2t}}^2\le C\Vert u{\Vert }_{H_{\epsilon }}^2,{\int }_{{\mathbb{R}}^3}\Phi (u)u^2\mathrm{d}x\le C\Vert u{\Vert }_{\frac{12}{3+2t}}^4\le C\Vert u{\Vert }_{H_{\epsilon }}^4,$$

Full text

[TABLE]

where $s>\frac{3}{4}$, $s,t\in(0,1)$, $\varepsilon>0$ is a small parameter, $f\in C^{1}(\mathbb{R}^{+},\mathbb{R})$ is subcritical, $V:\mathbb{R}^{3}\rightarrow\mathbb{R}$ is a continuous bounded function. We establish a family of positive solutions $u_{\varepsilon}\in H_{\varepsilon}$ which concentrates around the local minima of $V$ in $\Lambda$ as $\varepsilon\rightarrow 0$. With Ljusternik-Schnirelmann theory, we also obtain multiple solutions by employing the topology construct of the set where the potential $V$ attains its minimum.

1. Introduction

In this paper, we study the following fractional Schrödinger-Poisson system

[TABLE]

where $s>\frac{3}{4}$, $s,t\in(0,1)$, $\varepsilon>0$ is a small parameter. The potential $V:\mathbb{R}^{3}\rightarrow\mathbb{R}$ is a continuous bounded function satisfying

$(V_{0})$ $\inf\limits_{x\in\mathbb{R}^{3}}V(x)=\alpha_{0}>0$;

$(V_{1})$ There is a bounded domain $\Lambda\subset\mathbb{R}^{3}$ such that

[TABLE]

This kind of hypothesis was first introduced by del Pino and Felmer in [16]. The nonlinearity $f:\mathbb{R}\rightarrow\mathbb{R}$ is of $C^{1}$-class function. Since we are looking for positive solutions, we may assume that $f(\tau)=0$ for $\tau<0$. Furthermore, we need the following conditions:

$(f_{0})$ $\lim\limits_{\tau\rightarrow 0^{+}}\frac{f(\tau)}{\tau^{3}}=0$;

$(f_{1})$ there exist $\lambda>0$ and $C>0$ such that $f(\tau)\geq\lambda\tau^{q-1}$ for some $4\leq q<2_{s}^{\ast}$ and $|f^{\prime}(\tau)|\leq C(1+|\tau|^{p-2})$, where $4<p<2_{s}^{\ast}$ ;

$(f_{2})$ $\frac{f(\tau)}{\tau^{3}}$ is non-decreasing in $\tau\in(0,+\infty)$.

The non-local operator $(-\Delta)^{s}$ ($s\in(0,1)$), which is called fractional Laplacian operator, can be defined by

[TABLE]

for $u\in\mathcal{S}(\mathbb{R}^{3})$, where $\mathcal{S}(\mathbb{R}^{3})$ is the Schwartz space of rapidly decaying $C^{\infty}$ function, $B_{\varepsilon}(x)$ denote an open ball of radius $r$ centered at $x$ and the normalization constant C_{s}=\Big{(}\int_{\mathbb{R}^{3}}\frac{1-\cos(\zeta_{1})}{|\zeta|^{3+2s}}\,{\rm d}\zeta\Big{)}^{-1}. For $u\in\mathcal{S}(\mathbb{R}^{3})$, the fractional Laplace operator $(-\Delta)^{s}$ can be expressed as an inverse Fourier transform

[TABLE]

where $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the Fourier transform and inverse transform, respectively. If $u$ is sufficiently smooth, it is known that (see [34]) it is equivalent to

[TABLE]

By a classical solution of (1.1), we mean two continuous functions that $(-\Delta)^{s}u$ is well defined for all $x\in\mathbb{R}^{3}$ and satisfies (1.1) in pointwise sense.

In the last several years, nonlinear equations involving fractional Laplacian is much of interest, and attracts much attention by many scholars. One of the main reason is that fractional operators appear in many mathematical and physical problems, such as: fractional quantum mechanics [27, 28], Financial modelling [14], anomalous diffusion [31], obstacle problems [41], conformal geometry and minimal surfaces [12] and so on. Another main reason is that the fractional Laplacian $(-\Delta)^{s}$ ($s\in(0,1)$) is a non-local operator comparing with the classical Laplacian $-\Delta$ which is a local one, the methods which were previously developed, maybe not be applied directly. We refer the interesting readers to see the recent progresses such as [9, 13, 15, 17, 20, 19, 32, 33, 34, 41, 39, 42, 44, 45, 46, 48] and the references therein.

In the very recent, fractional Schrödinger-Poisson system

[TABLE]

has been investigated by some scholars. When $f(u)=|u|^{p-1}u$ for $2<p<2_{s}^{\ast}-1$ or $f(u)=\mu|u|^{q-1}+|u|^{2_{s}^{\ast}-2}u$ with $q\in(\frac{3s+t}{s+t},2_{s}^{\ast})$ and $\mu>0$ which maybe large for some $q$, in [44, 45], we established the existence of positive ground state solution by using the Nehari-Pohozaev manifold combing monotone trick with global compactness Lemma, respectively. In [48], the authors studied the existence of radial solutions for system (2.1) with the nonlinearity $f(u)$ verifying the subcritical or critical assumptions of Berestycki-Lions type.

For the semiclassical state, in [33], the authors studied the semiclassical state of the following system

[TABLE]

where $s\in(0,1)$, $\alpha\in(0,N)$, $\theta\in(0,\alpha)$, $N\in(2s,2s+\alpha)$, $\gamma_{\alpha}$ is a positive constant, $f(u)$ satisfies the following subcritical growth assumptions: $0<KF(t)\leq f(t)t$ with some $K>4$ for all $t\geq 0$ and $\frac{f(t)}{t^{3}}$ is strictly increasing on $(0,+\infty)$. By adapting some ideas of Benci, Cerami and Passaseo [7, 8] and using the Ljusternick-Schnirelmann Theory, the authors obtained the multiplicity of positive solutions which concentrate on the minima of $V(x)$ as $\varepsilon\rightarrow 0$. Using the similar methods, the authors in [30] studied the system (1.1) and established the multiplicity and concentration behavior of solutions. In [46], we also studied the concentration of positive ground state solution via the Nehari manifold for the following system

[TABLE]

where $V(x)$, $K(x)$, $Q(x)$ is a bounded continuous potential and $f$ satisfies the subcritical growth and some monotone condition. In the above works, the potential $V(x)$ is either constant or some bounded potentials possessing some global minimum points, the main purpose of this paper is devoted to studying the case that $V(x)$ possesses some local minimum points.

In the last several years, the semiclassical state of the nonlinear Schrödinger-Poisson system has been object of interest for many authors. Ruiz and Vaira [38] proved the existence of multi-bump solutions of system

[TABLE]

with $K(x)\equiv 1$ and these bumps concentrate around a local minimum of the potential $V$. Ruiz [37] and D’Aprile and Wei [18] showed that system (1.3) with $V(x)\equiv K(x)\equiv 1$ possesses a family of solutions concentrating around a sphere when $\varepsilon\rightarrow 0$ for $p\in(2,\frac{18}{7})$. Their results were generalized in [24, 25] for the radial $V$ and $K$. Ianni and Vaira [26] obtained the existence of positive bound state solutions which concentrate on a non-degenerate local minimum or maximum of $V$ by using a Lyapunov-Schmitt reduction method. Seok [40] proved that system (1.3) has single and multi-peak solutions which concentrate around the local minimum of $V$ with the Berestycki-Lions conditions. After that, Zhang [51] considered the critical Berestycki-Lions conditions and obtained the single solutions which concentrate around the local minimum of $V$. Liu at el [29] proved the multi-semiclassical states which concentrates around its corresponding global minimum point of $V$. The concentration phenomenon of solutions for the following Schrödinger-Poisson system also investigated by some authors:

[TABLE]

For the subcritical case, When $f(x,u)=b(x)f(u)$ and $f$ satisfies super-4 growth condition and some monotone condition, Wang et al. [49] studied the existence of ground state solutions for system (1.4) and the concentration behavior of least energy solutions was obtained. For the critical growth case, He and Zou [23] considered the existence and concentration behavior of ground state solutions for (1.4) with $f(x,u)=u^{5}+f(u)$ and the subcritical term $f$ the so-called Ambrosetti-Rabinowitz condition, they proved that system (1.4) has a ground state solution concentrating around a global minimum $V$ as $\varepsilon\rightarrow 0$. When $f(x,u)=\lambda|u|^{p-2}u+|u|^{4}u$ with $3<p\leq 4$, which it does not satisfy the monotone assumption or Ambrosetti-Rabinowtiz condition, He and Li [21] construct a family of positive solution which concentrates around a local minimum of $V$ as $\varepsilon\rightarrow 0$.

But, to the best of knowledge, for the local case, only in [21, 40, 51], the authors considered the the potential $V(x)$ possessing a local minimum point. They followed the method developed by Byeon-Jeanjean [10, 11] to construct a peak or multiple-peak solution which concentrates on the local minimum of $V(x)$ as $\varepsilon\rightarrow 0$. For the nonlocal case, there are no papers to consider the potential $V(x)$ possessing a local minimum point. Motivated by some related works, the aim of this paper is to study the existence of concentration solutions in the case that $V(x)$ has local minimum points. We will take the penalization arguments due to del Pino and Felmer [35] to investigate system (1.1). As we know, this kind of penalization method has been successfully applied to study the multiplicity and concentration of solutions for other problems, such as: Kirchhoff type problems [22], fractional Schrödinger equations [1], quasilinear problem involving $N$-Laplacian [2], quasilinear Choquard equation [3] and so on. However, now we are working with a class of non-local problems, there are some new difficulties in dealing with the system (1.1). One of the main difficulties is that we consider the system (1.1) being with critical Sobolev exponent, it is required to use the concentration-compactness principle to return the compactness, but in [36], the authors provided a version of concentration-compactness principle which is useful for the bounded domain, and here our problem is set on the whole space $\mathbb{R}^{3}$. This is the first obstacle to be killed. Another is the decay estimate of solution sequence at infinity, this is different from the classical one, such as (1.4). These difficulties make us more careful analysis, which permit us to use the penalization method.

Our main results are as follows.

Theorem 1.1**.**

Let $2s+2t>3$, $s,t\in(0,1)$ and $s>\frac{3}{4}$. Suppose that $V$ satisfies $(V_{0})$, $(V_{1})$ and $f\in C(\mathbb{R}^{+},\mathbb{R})$ satisfies $(f_{0})$-$(f_{2})$. Then there exists an $\varepsilon_{0}>0$ such that system (1.1) possesses a positive solution $(u_{\varepsilon},\phi_{\varepsilon})\in H_{\varepsilon}\times\mathcal{D}^{t,2}(\mathbb{R}^{3})$ for all $\varepsilon\in(0,\varepsilon_{0})$. Moreover, $u_{\varepsilon}$ possesses a maximum $x_{\varepsilon}\in\Lambda$ such that $V(x_{\varepsilon})\rightarrow\inf\limits_{\Lambda}V$ as $\varepsilon\rightarrow 0$, and

[TABLE]

for some constants $C>0$ and $C_{0}\in\mathbb{R}$.

By using the Lyusternik-Schnirelmann category, we can obtain the multiplicity of positive solutions. For this purpose, we also need the assumption that

[TABLE]

We denote the closed $\delta$-neighborhood of the set $\mathcal{M}$ by

[TABLE]

Also we recall that, if $Y$ is a closed set of a topological space $X$, ${\rm cat}_{X}(Y)$ is the Ljusternik-Schnirelmann category of $Y$ in $X$, namely the least number of closed and contractible sets in $X$ which cover $Y$. We shall prove the following multiplicity result.

Theorem 1.2**.**

Let $2s+2t>3$, $s,t\in(0,1)$ and $s>\frac{3}{4}$. Suppose that $V$ satisfies $(V_{0})$, $(V_{1})$, $(V_{2})$ and $f\in C(\mathbb{R}^{+},\mathbb{R})$ satisfies $(f_{0})$-$(f_{2})$. Then, for any $\delta>0$ given, there exists $\varepsilon_{\delta}>0$ such that, for any $\varepsilon\in(0,\varepsilon_{\delta})$, system (1.1) has at least ${\rm cat}_{\mathcal{M}_{\delta}}(\mathcal{M})$ solutions. Furthermore, if one of these solutions $u_{\varepsilon}$ possesses a maximum $x_{\varepsilon}\in\Lambda$, then $\lim\limits_{\varepsilon\rightarrow 0}V(x_{\varepsilon})=\inf\limits_{\Lambda}V$ and

[TABLE]

for some constants $C>0$ and $C_{0}\in\mathbb{R}$.

This paper is organized as follows, in section 2, we give some preliminary results and a version of concentration-compactness principle. In section 3, we will prove the penalization problem has a positive solution. In section 4, we will prove the limiting problem has a positive ground state solution. In section 5, we will give a uniform estimate for solution sequences. In section 6, we complete the proof of Theorem 1.1. Section 7 is devote to prove Theorem 1.2 by using the Lyusternik-Schnirelmann category.

2. Variational Setting

In this section, we outline the variational framework for studying problem (1.1) and list some preliminary Lemma which used later. In the sequel, we denote by $\|\cdot\|_{p}$ the usual norm of the space $L^{p}(\mathbb{R}^{3})$, the letter $c_{i}$ ($i=1,2,\ldots$) or $C$ denote by some positive constants. We denote $\widehat{u}$ the Fourier transform of $u$.

2.1. Work space stuff

We define the homogeneous fractional Sobolev space $\mathcal{D}^{s,2}(\mathbb{R}^{3})$ as follows

[TABLE]

which is the completion of $C_{0}^{\infty}(\mathbb{R}^{3})$ under the norm

[TABLE]

The fractional Sobolev space $H^{s}(\mathbb{R}^{3})$ can be described by means of the Fourier transform, i.e.

[TABLE]

In this case, the inner product and the norm are defined as

[TABLE]

[TABLE]

From Plancherel’s theorem we have $\|u\|_{2}=\|\widehat{u}\|_{2}$ and $\||\xi|^{s}\widehat{u}\|_{2}=\|(-\Delta)^{\frac{s}{2}}u\|_{2}$. Hence

[TABLE]

We denote $\|\cdot\|$ by $\|\cdot\|_{H^{s}}$ in the sequel for convenience.

We define the Sobolev space $H_{\varepsilon}=\{u\in H^{s}(\mathbb{R}^{3})\,\,|\,\,\int_{\mathbb{R}^{3}}V(\varepsilon x)u^{2}\,{\rm d}x<\infty\}$ endowed with the norm

[TABLE]

It is well known that $H^{s}(\mathbb{R}^{3})$ is continuously embedded into $L^{r}(\mathbb{R}^{3})$ for $2\leq r\leq 2_{\alpha}^{\ast}$ ($2_{\alpha}^{\ast}=\frac{6}{3-2\alpha}$). Obviously, the conclusion also holds for $H_{\varepsilon}$.

2.2. Formulation of Problem (1.1)

It is easily seen that, just performing the change of variables $u(x)\rightarrow u(x/\varepsilon)$ and $\phi(x)\rightarrow\phi(x/\varepsilon)$, and taking $z=x/\varepsilon$, problem (1.1) can be rewritten as the following equivalent form

[TABLE]

which will be referred from now on.

From $(f_{2})$, it is easy to verify that

[TABLE]

Observe that if $4s+2t\geq 3$, there holds $2\leq\frac{12}{3+2t}\leq\frac{6}{3-2s}$ and thus $H_{\varepsilon}\hookrightarrow L^{\frac{12}{3+2t}}(\mathbb{R}^{3})$. Considering $u\in H_{\varepsilon}$, the linear functional $\widetilde{\mathcal{L}}_{u}:\mathcal{D}^{t,2}(\mathbb{R}^{3})\rightarrow\mathbb{R}$ is defined by

[TABLE]

Using the Lax-Milgram theorem, there exists a unique $\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$ such that

[TABLE]

that is $\phi_{u}^{t}$ is a weak solution of $(-\Delta)^{t}\phi_{u}^{t}=u^{2}$ and so the representation formula holds

[TABLE]

Substituting $\phi_{u}^{t}$ in (2.1), it reduces to a single fractional Schrödinger equation

[TABLE]

The solvation of (2.2) can be looking for the critical points of the associated energy functional $J_{\varepsilon}:H_{\varepsilon}\rightarrow\mathbb{R}$ defined by

[TABLE]

and $J_{\varepsilon}\in C^{1}(H_{\varepsilon},\mathbb{R})$. Let us summarize some properties of the function $\phi_{u}^{t}$, the proof can be found in [46].

Lemma 2.1**.**

*For every $u\in H_{\varepsilon}$ with $4s+2t\geq 3$, define $\Phi(u)=\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$, where $\phi_{u}^{t}$ is the unique solution of equation $(-\Delta)^{t}\phi=u^{2}$. Then there hold:

$(i)$ If $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, then $\Phi(u_{n})\rightharpoonup\Phi(u)$ in $\mathcal{D}^{t,2}(\mathbb{R}^{3})$;

$(ii)$ $\Phi(tu)=t^{2}\Phi(u)$ for any $t\in\mathbb{R}$;

$(iii)$ For $u\in H_{\varepsilon}$, one has*

[TABLE]

*where constant $C$ is independent of $u$;

$(iv)$ Let $2s+2t>3$, if $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$, then for any $v\in H_{\varepsilon}$,*

[TABLE]

and

[TABLE]

In the end of this section, we will give a version of concentration-compactness on whole space $\mathbb{R}^{3}$ which is sufficient to prove our main results. We define

[TABLE]

Lemma 2.2**.**

*(Theorem 5 [36], Lemma 3.5 and 3.7 [52]) Let $\{u_{n}\}\subset\mathcal{D}^{s,2}(\mathbb{R}^{3})$ be such that $u_{n}\rightharpoonup u$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$, $u_{n}\rightarrow u$ in $L^{2}(\mathbb{R}^{3})$, $|(-\Delta)^{\frac{s}{2}}u_{n}|^{2}\rightharpoonup\mu$ and $|u_{n}|^{2_{s}^{\ast}}\rightharpoonup\nu$ weakly$-\ast$ in $\mathcal{M}(\mathbb{R}^{3})$ as $n\rightarrow\infty$. Here $\mathcal{M}(\mathbb{R}^{3})$ is the space of finite nonnegative Borel measures on $\mathbb{R}^{3}$. Then

$(i)$ $u_{n}\rightarrow u$ in $L_{loc}^{2_{s}^{\ast}}$ or there exists a (at most countable) set of distinct points $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$ and positive number $\{\nu_{j}\}_{j\in J}$ such that*

[TABLE]

$(ii)$* Then $\mu_{\infty}$ and $\nu_{\infty}$ are well defined satisfy*

[TABLE]

$(iii)$**

[TABLE]

Proof.

The conclusion $(i)$ comes from Theorem 5 in [36], $(ii)$ comes from Lemma 3.5 in [52]. We only need to show that $(iii)$ holds.

1. Take $\psi\in C_{0}^{\infty}(\mathbb{R}^{3})$ such that $\psi=1$ on $B_{1}(0)$, $\psi=0$ on $\mathbb{R}^{3}\backslash B_{2}(0)$, $0\leq|\psi\leq 1$ and $|\nabla\psi|\leq C$. For any $\rho>0$, define $\psi_{\rho}(x)=\psi(\frac{x-x_{j}}{\rho})$, where $j\in J$. It follows from Sobolev inequality that

[TABLE]

Since $|u_{n}|^{2_{s}^{\ast}}\rightharpoonup\nu$ in $\mathcal{M}(\mathbb{R}^{3})$, we have

[TABLE]

Using the nonlocal Leibniz rule:

[TABLE]

where $B(u_{n},\psi_{\rho})=C_{s}{\rm P.V.}\int_{\mathbb{R}^{3}}\frac{(\psi_{\rho}(x)-\psi_{\rho}(y))(u_{n}(x)-u_{n}(y))}{|x-y|^{3+s}}\,{\rm d}y$. Then it is easy to obtain

[TABLE]

Next, we will show that $\lim\limits_{\rho\rightarrow 0^{+}}\limsup\limits_{n\rightarrow\infty}A_{i}=0$, $i=1,2,3,4,5$. If these are proved, using the assumption $|(-\Delta)^{\frac{s}{2}}u_{n}|^{2}\rightharpoonup\mu$ in $\mathcal{M}(\mathbb{R}^{3})$, then

[TABLE]

and the fist conclusion of $(iii)$ is established.

Note that using Hölder’s inequality, we get that

[TABLE]

[TABLE]

and

[TABLE]

Hence, we only need to show that

[TABLE]

and

[TABLE]

In fact, using the assumption that $u_{n}\rightarrow u$ in $L^{2}(\mathbb{R}^{3})$, similar arguments as Lemma 2.8 and 2.9 in [4], we can conclude that (2.3) and (2.4) hold true.

2. Let $\phi\in C^{\infty}(\mathbb{R}^{3})$ such that $\phi=0$ on $B_{1}(0)$, $\phi=1$ on $\mathbb{R}^{3}\backslash B_{2}(0)$, $1\leq\phi\leq 1$ and $|\nabla\phi|\leq C$. Set $\phi_{R}(x)=\phi(\frac{x}{R})$. It follows from Sobolev inequality that

[TABLE]

It is easy to check that

[TABLE]

Similar argument to the proof of the first conclusion, we only need to show that

[TABLE]

and

[TABLE]

$\bullet$ Estimate of $(-\Delta)^{\frac{s}{2}}\phi_{R}$.

[TABLE]

where $\xi=y+\tau(x-y)$ with $\tau\in(0,1)$.

$\bullet$ Estimate of $I(u_{n},\phi_{R})$. By Hölder’s inequality, we have that

[TABLE]

Therefore, by (2.2), one has

[TABLE]

which implies that (2.5) holds. From (2.2) and Proposition 3.4 in [34], we deduce that

[TABLE]

which yields (2.6). Thus we complete the proof of the second conclusion. ∎

3. The penalization problem

For the bounded domain $\Lambda$ given in $(V_{1})$, $k>2$, $a>0$ such that $f(a)+a^{2_{s}^{\ast}-1}=\frac{\alpha_{0}}{k}a$ where $\alpha_{0}$ is mentioned in $(V_{0})$, we consider a new problem

[TABLE]

where $g(z,\tau)=\chi(z)(f(\tau)+(\tau^{+})^{2_{s}^{\ast}-1})+(1-\chi(z))\tilde{f}(\tau)$ with

[TABLE]

and $\chi(z)$ is a smooth function such that $\chi(z)=1$ on $\Lambda$, $0\leq\chi(z)\leq 1$ on $\Lambda\backslash\Lambda^{\prime}$, $\chi(z)=0$ on $\mathbb{R}^{3}\backslash\Lambda^{\prime}$, where $\Lambda^{\prime}$ is a suitable open set satisfying $\bar{\Lambda}\subset\Lambda^{\prime}$ and $V(z)>\inf_{\xi\in\Lambda}V(\xi)$ for all $z\in\bar{\Lambda^{\prime}}\backslash\Lambda$. It is easy to see that under the assumptions $(f_{1})$-$(f_{3})$, $g(z,\tau)$ is a Caratheodory function and satisfies the following assumptions:

$(g_{1})$ $g(z,\tau)=o(\tau^{3})$ as $\tau\rightarrow 0$ uniformly on $z\in\mathbb{R}^{3}$;

$(g_{2})$ $g(z,\tau)\leq f(\tau)+\tau^{2_{s}^{\ast}-1}$ for all $\tau\in\mathbb{R}^{+}$ and $z\in\mathbb{R}^{3}$, $g(z,\tau)=0$ for all $z\in\mathbb{R}^{3}$ and $\tau<0$;

$(g_{3})$ $0<2\tilde{F}(\tau)\leq\tilde{f}(\tau)\tau\leq\frac{\alpha_{0}}{k}\tau^{2}\leq\frac{V(x)}{k}\tau^{2}$ for all $s\geq 0$ with the number $k>2$, where $\tilde{F}(\tau)$ is a prime function of $\tilde{f}$;

$(g_{4})$ $0<4G(z,\tau)\leq g(z,\tau)\tau$ for all $z\in\Lambda$, $\tau>0$ or $z\in\mathbb{R}^{3}\backslash\Lambda$, $\tau\leq a$ and $g(z,\tau)\tau+\frac{V(z)}{4}\tau^{2}\geq 4G(z,\tau)>0$ for all $z\in\mathbb{R}^{3}$ and $\tau>0$, where $G(z,\tau)$ is a prime function of $g(z,\tau)$;

$(g_{5})$ $\frac{g(z,s\tau)}{s}$ is nondecreasing in $\tau\in\mathbb{R}^{+}$ uniformly for $z\in\mathbb{R}^{3}$, $\frac{g(z,s\tau)}{\tau^{3}}$ is nondecreasing in $\tau\in\mathbb{R}^{+}$ and $z\in\Lambda$, $\frac{g(z,s\tau)}{\tau^{3}}$ is nondecreasing in $\tau\in(0,a)$ and $z\in\mathbb{R}^{3}\backslash\Lambda^{\prime}$.

The energy functional corresponding to (3.1) is defined as

[TABLE]

and $J_{\varepsilon}\in C^{1}(H_{\varepsilon},\mathbb{R})$.

By standard argument, the functional $J_{\varepsilon}$ satisfies the mountain pass geometry.

Lemma 3.1**.**

*Suppose $(V_{0})$, $(V_{1})$ and $(f_{0})-(f_{2})$ hold, then the functional $J_{\varepsilon}$ has the following properties:

$(i)$ there exist $\alpha,\rho>0$ such that $J_{\varepsilon}(u)\geq\alpha$ for $\|u\|_{H_{\varepsilon}}=\rho$;

$(ii)$ there exists $e_{0}\in H_{\varepsilon}$ satisfying $\|e_{0}\|_{H_{\varepsilon}}>\rho$ such that $J_{\varepsilon}(e_{0})<0$.*

By Lemma 3.1 and Theorem 1.15 in [47] (Mountain pass theorem without Palais-Smale condition), it follows that there exists a $(PS)_{c_{\varepsilon}}$ sequence $\{u_{n}\}\subset H_{\varepsilon}$ such that

[TABLE]

where $c_{\varepsilon}=\inf\limits_{\gamma\in\Gamma}\max\limits_{t\in[0,1]}J_{\varepsilon}(\gamma(t))>0$. Here

[TABLE]

Similarly to the argument in [35, 47], by $(g_{5})$, the equivalent characterization of $c_{\varepsilon}$ is given by

[TABLE]

where $\mathcal{N}_{\varepsilon}$ is the Nehari manifold defined as

[TABLE]

For author’s convenience, we give the rough proof. We state it as the following Proposition.

Proposition 3.2**.**

[TABLE]

Proof.

1. For each $u\in H_{\varepsilon}\backslash\{0\}$, we claim that there exists a unique $t_{u}>0$ such that $t_{u}u\in\mathcal{N}_{\varepsilon}$. Indeed, set $h(t)=J_{\varepsilon}(tu)$, by $(g_{1})$ and $(g_{2})$, it is easy to check that $h(t)>0$ when $t>0$ small and $h(t)<0$ when $t>0$ large. Since $h\in C^{1}(\mathbb{R}^{+},\mathbb{R})$ and $h(0)=0$, there is $t_{u}>0$ global maximum point of $h(t)$ and $h^{\prime}(t)=0$. Thus, $\langle J_{\varepsilon}(t_{u}u),t_{u}u\rangle=0$, and $t_{u}u\in\mathcal{N}_{\varepsilon}$. We see that $t_{u}>0$ is the unique positive number such that $h^{\prime}(t_{u})=0$. Indeed, suppose by contradiction that there exist $t_{1}>t_{2}>0$ such that $h^{\prime}(t_{1})=h^{\prime}(t_{2})=0$. Then for $i=1,2$,

[TABLE]

Therefore

[TABLE]

$\bullet$ If ${\rm supp}(u^{+})\subset\Lambda/\varepsilon$, then $g(\varepsilon z,u)=f(u)+(u^{+})^{2_{s}^{\ast}-1}$, the uniqueness of $t_{u}$ follows from the hypothesis $(f_{2})$.

$\bullet$ If ${\rm supp}(u^{+})\subset\mathbb{R}^{3}\backslash(\Lambda^{\prime}/\varepsilon)$, then $g(\varepsilon z,u)=\tilde{f}(u)$. By the definition of $\tilde{f}$, we have that

[TABLE]

Multiplying both sides by $\frac{1}{\frac{1}{t_{1}^{2}}-\frac{1}{t_{2}^{2}}}$ and using the hypothesis $t_{1}>t_{2}>0$, we get

[TABLE]

Since $u\neq 0$, we have that $k\leq 1$, but this is a contradiction. Thus, the uniqueness of $t_{u}$ follows for ${\rm supp}(u^{+})\subset\mathbb{R}^{3}\backslash(\Lambda^{\prime}/\varepsilon)$.

$\bullet$ If ${\rm supp}(u^{+})\subset(\mathbb{R}^{3}\backslash(\Lambda/\varepsilon))\backslash(\mathbb{R}^{3}\backslash(\Lambda^{\prime}/\varepsilon))$, by the definition of $g$ and hypothesis $(f_{2})$, we have that

[TABLE]

Multiplying both sides by $\frac{1}{\frac{1}{t_{1}^{2}}-\frac{1}{t_{2}^{2}}}$ and using the hypothesis $t_{1}>t_{2}>0$, we get

[TABLE]

which implies a contradiction with $k>2$.

$\bullet$ If ${\rm supp}(u^{+})\subset(\mathbb{R}^{3}\backslash(\Lambda/\varepsilon))\backslash(\mathbb{R}^{3}\backslash(\Lambda^{\prime}/\varepsilon))\cup\mathbb{R}^{3}\backslash(\Lambda^{\prime}/\varepsilon)$, similar argument as the above, we can deduce that $k\leq 2$, this is a contradiction.

Therefore, whatever any cases, the claim holds true. Thus

[TABLE]

By $(ii)$ of Lemma 3.1, using standard argument, we get that

[TABLE]

For any $\gamma\in\Gamma$, $\gamma([0,1])\cap\mathcal{N}_{\varepsilon}\neq\emptyset$. Indeed, if $u\in H_{\varepsilon}\backslash\{0\}$ is interior to or on $\mathcal{N}_{\varepsilon}$, then

[TABLE]

and

[TABLE]

Hence $\gamma$ crosses $\mathcal{N}_{\varepsilon}$ since $\gamma(0)=0$, $J_{\varepsilon}(\gamma(1))\leq 0$ and $\gamma(1)\neq 0$. Therefore

[TABLE]

∎

The following Lemma gives the estimate of the critical value $c_{\varepsilon}$.

Lemma 3.3**.**

Suppose that $(V_{0})$, $(V_{1})$ and $(f_{0})-(f_{2})$ hold, then the infinimum $c_{\varepsilon}$ satisfies

[TABLE]

for $\varepsilon$ small enough, where $\mathcal{S}_{s}$ is the best Sobolev constant for the embedding $\mathcal{D}^{s,2}(\mathbb{R}^{3})\hookrightarrow L^{2_{s}^{\ast}}(\mathbb{R}^{3})$.

Proof.

Without loss of generalization, we assume that $0\in\Lambda$. Choose $R>0$ such that $B_{2R}(0)\subset\Lambda/\varepsilon$ and $\psi\in C_{0}^{\infty}(B_{2R}(0))$ satisfying $\psi=1$ on $B_{R}(0)$ and $0\leq\psi\leq 1$ on $B_{2R}(0)$. Given $\varepsilon>0$, we define

[TABLE]

where $U_{\varepsilon}(z)=\varepsilon^{-\frac{3-2s}{2}}u^{\ast}(z/\varepsilon)$, $u^{\ast}(z)=\frac{\widetilde{u}(z/\mathcal{S}_{s}^{\frac{1}{2s}})}{\|\widetilde{u}\|_{2_{s}^{\ast}}}$, $\kappa\in\mathbb{R}\backslash\{0\}$, $\mu>0$ and $x_{0}\in\mathbb{R}^{3}$ are fixed constants, $\widetilde{u}(z)=\kappa(\mu^{2}+|z-x_{0}|^{2})^{-\frac{3-2s}{2}}$. From Proposition 21 and Proposition 22 in [43], Lemma 3.3 in [44], we know that

[TABLE]

[TABLE]

and

[TABLE]

Since ${\rm supp}(v_{\varepsilon})\subset\Lambda/\varepsilon$, $g(\varepsilon z,v_{\varepsilon})=f(v_{\varepsilon})+v_{\varepsilon}^{2_{s}^{\ast}-1}$. There exists $t_{\varepsilon}>0$ such that $\sup\limits_{t\geq 0}J_{\varepsilon}(tv_{\varepsilon})=J_{\varepsilon}(t_{\varepsilon}v_{\varepsilon})$. Hence \frac{{\rm d}J_{\varepsilon}(tv_{\varepsilon})}{{\rm d}t}\Big{|}_{t=t_{\varepsilon}}=0, that is

[TABLE]

By $(f_{0})$, we have that

[TABLE]

It follows from $(iii)$ of Lemma 2.1 that

[TABLE]

Thus, (3.3)-(3.6) imply that $t_{\varepsilon}\leq C_{1}$, where $C_{1}$ is independent of $\varepsilon>0$ small. On the other hand, we may assume that there is a positive constant $C_{2}>0$ such that $t_{\varepsilon}\geq C_{2}>0$ for $\varepsilon>0$ small. Otherwise, we can find a sequence $\varepsilon_{n}\rightarrow 0$ as $n\rightarrow\infty$ such that $t_{\varepsilon_{n}}\rightarrow 0$ as $n\rightarrow\infty$. Therefore

[TABLE]

which is a contradiction.

Denote $g(t)=\frac{t^{2}}{2}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{s}{2}}v_{\varepsilon}|^{2}\,{\rm d}z-\frac{t^{2_{s}^{\ast}}}{2_{s}^{\ast}}\int_{\mathbb{R}^{3}}|v_{\varepsilon}|^{2_{s}^{\ast}}\,{\rm d}z$, by (3.3) and (3.4), it is easy to check that

[TABLE]

where we have used the elementary inequality $(a+b)^{\alpha}\leq a^{\alpha}+\alpha(a+b)^{\alpha-1}b$, $\alpha\geq 1$, $a,b>0$. Thus, using the fact that $\max\limits_{z\in B_{2R}(0)}V(\varepsilon z)\leq C$ for some $C>0$ independent of $\varepsilon$ and $s>\frac{3}{4}\Rightarrow 2<\frac{3}{3-2s}$, by (3.5), we deduce that

[TABLE]

By (3.5), we have that

[TABLE]

Since $s>\frac{3}{4}$ and $q\geq 4$, then $q>\frac{3}{3-2s}$, $2s-\frac{3-2s}{2}q<0$. Thus

[TABLE]

Therefore, combining with (3), (3.11) and (3.12), we conclude that

[TABLE]

for $\varepsilon$ small enough and thus the proof is completed. ∎

Now we study the $(PS)_{c_{\varepsilon}}$ sequence given in (3.2).

Lemma 3.4**.**

Sequence $\{u_{n}\}$ given in (3.2) is bounded in $H_{\varepsilon}$.

Proof.

By (3.2) and $(g_{2})$-$(g_{3})$, we have that

[TABLE]

By the choice of $k$, we get the boundedness of $\{u_{n}\}$ in $H_{\varepsilon}$. ∎

Next, we show the bounded sequence $\{u_{n}\}$ is nonvanishing, that is

Lemma 3.5**.**

There exist a sequence $\{z_{n}\}\subset\mathbb{R}^{3}$ and $R>0$, $\beta>0$ such that

[TABLE]

where $\{u_{n}\}$ is the sequence given by Lemma 3.4.

Proof.

Suppose by contradiction that the Lemma does not hold. Thus by the vanishing Lemma, it follows that

[TABLE]

From $(f_{1})$-$(f_{2})$, $(iii)$ of Lemma 2.1 and (3.13), it is easy to check that

[TABLE]

and

[TABLE]

By the definition $\tilde{f}$ and (3.13), we have that

[TABLE]

and

[TABLE]

From (3.2), it follows that

[TABLE]

and

[TABLE]

We may assume that

[TABLE]

and

[TABLE]

Observe that

[TABLE]

thus it is easy to check that $l>0$, if not, $\|u_{n}\|_{H_{\varepsilon}}\rightarrow 0$ as $n\rightarrow\infty$, which contradicts with $c_{\varepsilon}>0$.

By (3), we get

[TABLE]

In view of the definition of $\mathcal{S}_{s}$, we see that

[TABLE]

which achieves that

[TABLE]

which contradicts with Lemma 3.3. ∎

Lemma 3.6**.**

The sequence $\{z_{n}\}$ obtained in Lemma 3.4 is bounded in $\mathbb{R}^{3}$.

Proof.

For each $\rho>0$, consider a smooth cut-off function $0\leq\psi_{\rho}\leq 1$ such that $\psi_{\rho}=1$ on $|z|\geq 2\rho$, $\psi_{\rho}=0$ on $|z|\leq\rho$ and $|\nabla\psi_{\rho}|\leq\frac{C}{\rho}$. Clearly, $\psi_{\rho}u_{n}\in H_{\varepsilon}$ for each $\rho>0$. Using $\langle J_{\varepsilon}^{\prime}(u_{n}),\psi_{\rho}u_{n}\rangle=o(1)$, we obtain

[TABLE]

Choose $\rho>0$ large enough such that $\Lambda^{\prime}/\varepsilon\subset B_{\rho}(0)$, then $\varepsilon z\in\Lambda^{\prime}$, by $(g_{3})$, we have

[TABLE]

Now, by the nonlocal Leibniz rule, (2.2), (2.2), using Hölder’s inequality and Proposition 3.4 in [34], we have that

[TABLE]

In view of (3), we have

[TABLE]

Hence, we get

[TABLE]

If $\{z_{n}\}$ is unbounded, by Lemma 3.5 and (3.17), we have

[TABLE]

which achieves a contradiction for large $\rho$. ∎

Proposition 3.7**.**

The functional $J_{\varepsilon}$ possesses a nontrivial critical point $u_{\varepsilon}\in H_{\varepsilon}$ such that

[TABLE]

Proof.

From Lemma 3.4, up to a subsequence, we may assume that there is $u:=u_{\varepsilon}\in H_{\varepsilon}$ such that $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for all $1\leq r<2_{s}^{\ast}$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$. Lemma 3.5 and Lemma 3.6 imply that $u$ is nontrivial. Moreover, by $(iv)$ of Lemma 2.1, it is easy to check that for any $\varphi\in H_{\varepsilon}$, $\langle J_{\varepsilon}^{\prime}(u_{n}),\varphi\rangle\rightarrow\langle J_{\varepsilon}^{\prime}(u),\varphi\rangle=0$ as $n\rightarrow\infty$, that is $u$ is a nontrivial critical point of $J_{\varepsilon}$. Next, we show that $J_{\varepsilon}(u)=c_{\varepsilon}$. Indeed, using the fact that $u\in\mathcal{N}_{\varepsilon}$, Fatou’s Lemma, $(g_{4})$ and (3.2), we have

[TABLE]

The proof is completed.

∎

Remark 3.8**.**

From (3), it is not difficult to verify that $\|u_{n}\|_{H_{\varepsilon}}\rightarrow\|u\|_{H_{\varepsilon}}$ as $n\rightarrow\infty$. Using the Brezis-Lieb Lemma, we conclude that $u_{n}\rightarrow u$ in $H_{\varepsilon}$.

4. The limiting problem

In this section, we consider the limiting problem

[TABLE]

where $\mu$ is a positive constant. The energy functional corresponding to problem (3.1) is

[TABLE]

The main result of this section is stated as follows.

Proposition 4.1**.**

Suppose that $f$ satisfies $(f_{0})$-$(f_{2})$, then autonomous problem (4.1) has a positive ground state solution $u\in H^{s}(\mathbb{R}^{3})\cap C^{2,\alpha}(\mathbb{R}^{3})$, such that $\mathcal{I}_{\mu}(u)=c_{\mu}$, where

[TABLE]

where

[TABLE]

and

[TABLE]

For proving Proposition 4.1, we will shoe the following preliminary results.

Lemma 4.2**.**

$\mathcal{I}_{\mu}$* possesses the mountain pass geometry:

$(i)$ there exist $\rho_{0},\alpha_{0}>0$ such that $\mathcal{I}_{\mu}(u)\geq\alpha_{0}$ for all $u\in H^{s}(\mathbb{R}^{3})$ with $\|w\|=\rho_{0}$;

$(ii)$ there exists $u_{0}\in H^{s}(\mathbb{R}^{3})$ such that $\mathcal{I}_{\mu}(u_{0})<0$.*

Similar argument to Lemma 3.3, we can obtain the estimate of $c_{\mu}$.

Lemma 4.3**.**

Suppose $(f_{0})-(f_{2})$ hold, then $c_{\mu}$ satisfies

[TABLE]

where $\mathcal{S}_{s}$ is the best Sobolev constant for the embedding $\mathcal{D}^{s,2}(\mathbb{R}^{3})\hookrightarrow L^{2_{s}^{\ast}}(\mathbb{R}^{3})$.

We can prove the following compactness condition.

Lemma 4.4**.**

Let $\{u_{n}\}$ be a $(PS)_{c}$ sequence for functional $\mathcal{I}_{\mu}$, then for any $c\in(0,\frac{s}{3}\mathcal{S}_{s}^{\frac{3}{2s}})$, under a translation, the sequence $\{u_{n}\}$ strongly convergence in $H^{s}(\mathbb{R}^{3})$.

Proof.

Suppose $\{u_{n}\}\subset H^{s}(\mathbb{R}^{3})$ satisfies

[TABLE]

It follows from $(f_{1})$-$(f_{2})$ and (4.2) that

[TABLE]

Thus $\{u_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$. Similar argument to the proof of Lemma 3.5, we can obtain the bounded sequence $\{u_{n}\}$ is nonvanishing, i.e., there exist a sequence $\{x_{n}\}\subset\mathbb{R}^{3}$ and $R_{0}>0$, $\beta_{0}>0$ such that

[TABLE]

Up to a subsequence, we may assume that there is $u\in H^{s}(\mathbb{R}^{3})$ such that $u_{n}\rightharpoonup u$ in $H^{s}(\mathbb{R}^{3})$. By (4.3), we can assume that $u\neq 0$. Indeed, if $u=0$, then $u_{n}\rightharpoonup 0$ in $H^{s}(\mathbb{R}^{3})$ and $u_{n}\rightarrow 0$ (otherwise, contradicts with $c>0$). Set $v_{n}(x)=u_{n}(x+x_{n})$, then $\{v_{n}\}$ is also a $(PS)_{c}$ sequence for $\mathcal{I}_{\mu}$, so $\{v_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$ and there exists $v\in H^{s}(\mathbb{R}^{3})$ such that $v_{n}\rightharpoonup v$ in $H^{s}(\mathbb{R}^{3})$, $v_{n}\rightarrow v$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for all $1\leq r<2_{s}^{\ast}$ and $v_{n}\rightarrow v$ a.e. in $\mathbb{R}^{3}$. By $(iv)$ of Lemma 2.1, we have that $\langle\mathcal{I}_{\mu}^{\prime}(v_{n}),\varphi\rangle\rightarrow\langle\mathcal{I}_{\mu}^{\prime}(v),\varphi\rangle=0$ for any $\varphi\in H^{s}(\mathbb{R}^{3})$. Clearly, $\langle\mathcal{I}_{\mu}^{\prime}(v),v\rangle=0$.

Next we only need to show that $v_{n}\rightarrow v$ in $H^{s}(\mathbb{R}^{3})$. By the weakly semi-lower continuity of norm, we have that

[TABLE]

In order to prove that (4.4) hold, we must show the equality holds in (4.4). Otherwise, by Fatou’s Lemma, we get

[TABLE]

which is a contradiction. Thus, up to a subsequence, using Brezis-Lieb Lemma, we conclude that $v_{n}\rightarrow v$ in $H^{s}(\mathbb{R}^{3})$.

Proof of Proposition 4.1. From Lemma 4.2, and using the mountain-pass Lemma without $(PS)$ condition, we get a $(PS)_{c_{\mu}}$ sequence $\{u_{n}\}\subset H^{s}(\mathbb{R}^{3})$. By Lemma 4.3 and Lemma 4.4, under a translation, still denoted by $\{u_{n}\}$, there is $u\in H^{s}(\mathbb{R}^{3})$ such that $u_{n}\rightarrow u$ in $H^{s}(\mathbb{R}^{3})$. Therefore, $\mathcal{I}_{\mu}(u)=c_{\mu}$ and $\mathcal{I}_{\mu}^{\prime}(u)=0$, i.e., $u\in H^{s}(\mathbb{R}^{3})$ is a weak solution of problem (4.1). By standard argument to the proof Proposition 4.4 in [46], we have that $u\in C^{2,\alpha}(\mathbb{R}^{3})$ for some $\alpha\in(0,1)$. The remain proof is to show $u$ is positive. Using $-u^{-}$ as a testing function, it is easy to see that $u\geq 0$. Since $u\in C^{2,\alpha}(\mathbb{R}^{3})$, by Lemma 3.2 in [34], we have that

[TABLE]

Assume that there exists $x_{0}\in\mathbb{R}^{3}$ such that $u(x_{0})=0$, then from $u\geq 0$ and $u\not\equiv 0$, we get

[TABLE]

However, observe that $(-\Delta)^{s}u(x_{0})=-\mu u(x_{0})-(\phi_{u}^{t}u)(x_{0})+f(u(x_{0}))+u(x_{0})^{2_{s}^{\ast}-1}=0$, a contradiction. Hence, $u(x)>0$, for every $x\in\mathbb{R}^{3}$. The proof is completed. ∎

For $V_{0}=\min\limits_{\Lambda}V$, let $w$ be a ground state solution to the following problem

[TABLE]

satisfying $\mathcal{I}_{V_{0}}(w)=\inf\limits_{v\in H^{s}(\mathbb{R}^{3})\backslash\{0\}}\max\limits_{\tau\geq 0}\mathcal{I}_{V_{0}}(\tau v):=c_{V_{0}}$.

Lemma 4.5**.**

[TABLE]

Proof.

Let $z_{0}\in\Lambda$ be such that $V(z_{0})=V_{0}$ and so there is $R_{0}>0$ such that$B_{2R_{0}}(z_{0})\subset\Lambda$. Let $v_{\varepsilon}(z)=\eta(\varepsilon z-z_{0})w(\frac{\varepsilon z-z_{0}}{\varepsilon})$, where $\eta$ is smooth cut-off function satisfying $0\leq\eta\leq 1$, $\eta=1$ on $B_{R_{0}}(0)$, $\eta=0$ on $\mathbb{R}^{3}\backslash B_{2R_{0}}(0)$, $|\nabla\eta|\leq C$. By using the change of variables $\varepsilon z=\varepsilon z^{\prime}-z_{0}$, we can write

[TABLE]

Since $w>0$, by standard argument, there is a unique $t_{\varepsilon}>0$ such that $\sup\limits_{t>0}J_{\varepsilon}(tv_{\varepsilon})=J_{\varepsilon}(t_{\varepsilon}v_{\varepsilon})$ and $\frac{dJ_{\varepsilon}(tv_{\varepsilon})}{dt}|_{t=t_{\varepsilon}}=0$, i.e.,

[TABLE]

which means that $t_{\varepsilon}v_{\varepsilon}\in\mathcal{N}_{\varepsilon}$. We claim that there exist $T_{1},T_{2}>0$ such that $0<T_{1}\leq t_{\varepsilon}\leq T_{2}$. Observe that

[TABLE]

[TABLE]

and

[TABLE]

If $t_{\varepsilon}\rightarrow 0$ as $\varepsilon\rightarrow 0$, by $(f_{0})$ and $(f_{2})$, we have that

[TABLE]

which leads to a contradiction using (4.7)-(4.9). If $t_{\varepsilon}\rightarrow\infty$ as $\varepsilon\rightarrow 0$, then

[TABLE]

Hence, by (4.7)-(4.9), it is easy to achieve a contradiction. Thus the claim holds. Going if necessary to a subsequence, we may assume that $t_{\varepsilon}\rightarrow T>0$ as $\varepsilon\rightarrow 0$, then using (4.7)-(4.9), we get

[TABLE]

Since $w$ is a solution of problem (4.1), we have that

[TABLE]

By $(f_{2})$, we see that $t_{\varepsilon}\rightarrow T=1$. Therefore, by (4.7)-(4.9), we get

[TABLE]

Thus (4.6) follows. ∎

5. Uniformly estimate of solution sequence.

In this section, we consider the following problem

[TABLE]

where $\{V_{n}\}$ satisfies $V_{n}(x)\geq\alpha_{0}>0$ for all $x\in\mathbb{R}^{3}$ and $f_{n}(x,\tau)$ is a Carathedory function satisfying that for any $\delta>0$, there exists $C_{\varepsilon}>0$ such that

[TABLE]

Proposition 5.1**.**

Assume that $u_{n}$ are nonnegative weak solution of (5.1) satisfying $u_{n}$ convergence strongly in $H^{s}(\mathbb{R}^{3})$. Then there exists $C>0$ such that

[TABLE]

Proof.

Define

[TABLE]

Clearly $\psi$ is a convex and differentiable function, and $(-\Delta)^{s}\psi_{T,k}(u)\leq\psi_{T,k}^{\prime}(u)(-\Delta)^{s}u$. Moreover, $\|\psi_{T,k}(u)\|_{\mathcal{D}^{s,2}}\leq kT^{k-1}\|u\|_{\mathcal{D}^{s,2}}$. Taking $\psi_{T,k}(u_{n})\psi_{T,k}^{\prime}(u_{n})$ as a test function in (5.1), by (5.2), we get

[TABLE]

Taking $\delta=\alpha_{0}$, using Sobolev inequality, Hölder’s inequality and the fact $t\psi_{T,k}^{\prime}(t)\leq k\psi_{T,k}(t)$ for $t\geq 0$, we have that

[TABLE]

Take $k=k_{1}=\frac{2_{s}^{\ast}}{2}$, we get

[TABLE]

Since $u_{n}$ convergence strongly in $H^{s}(\mathbb{R}^{3})$, then $u_{n}$ convergence strongly in $L^{2_{s}^{\ast}}(\mathbb{R}^{3})$. Thus, we can take $A_{0}$ large enough such that

[TABLE]

Thus, from $\psi_{T,k}(t)\leq t^{k}$ for $t\geq 0$ and (5.4), we deduce that

[TABLE]

Letting $T\rightarrow+\infty$ in (5.5), we get

[TABLE]

Letting $T\rightarrow+\infty$ in (5), we have

[TABLE]

For $m\geq 1$, we define $k_{m+1}$ inductively so that $2k_{m+1}+2_{s}^{\ast}-2=2_{s}^{\ast}k_{m}$ and $k_{1}=\frac{2_{s}^{\ast}}{2}$, using (5.6), it is easy to check that

[TABLE]

letting $m\rightarrow\infty$ in the above inequality, we conclude that

[TABLE]

∎

6. Proof of Theorem 1.1.

For $\varepsilon>0$, let $u_{\varepsilon}$ be the mountain-pass solution to (3.1) given by Proposition 3.7. For any sequence $\{\varepsilon_{n}\}$ satisfying $\varepsilon_{n}\rightarrow 0^{+}$, denote by $u_{n}:=u_{\varepsilon_{n}}$, $J_{n}:=J_{\varepsilon_{n}}$ and $H_{n}:=H_{\varepsilon_{n}}$. Then $u_{n}$ satisfies

[TABLE]

Here $u_{n}$ is a critical point of $J_{n}$ and $J_{n}(u_{n})=c_{\varepsilon_{n}}$. Using Lemma 4.5, and similar argument to the proof of Lemma 3.4, we have that $\{u_{n}\}$ is bounded in $H_{n}$. Similar to Lemma 3.5, we have

Lemma 6.1**.**

There exist a sequence $\{y_{n}\}\subset\mathbb{R}^{3}$ and $R>0$, $\beta>0$ such that

[TABLE]

Lemma 6.2**.**

$\{\varepsilon_{n}y_{n}\}$* is bounded in $\mathbb{R}^{3}$. Moreover, ${\rm dist}(\varepsilon_{n}y_{n},\Lambda^{\prime})\leq\varepsilon_{n}R$.*

Proof.

For $\delta>0$, define $K_{\delta}=\{z\in\mathbb{R}^{3}\,\,|\,\,{\rm dist}(z,\Lambda^{\prime})\leq\delta\}$. Let $\varphi_{\varepsilon_{n}}(z)=\varphi(\varepsilon_{n}z)$ satisfy that $\varphi\in C^{\infty}(\mathbb{R}^{3},[0,1])$, $\varphi=1$ on $\mathbb{R}^{3}\backslash K_{\delta}$, $\varphi=0$ on $\Lambda^{\prime}$ and $|\nabla\varphi|\leq\frac{C}{\delta}$. Noting that ${\rm supp}\varphi_{\varepsilon_{n}}\cap(\Lambda^{\prime}/\varepsilon_{n})=\emptyset$, then $g(\varepsilon_{n}z,u_{n})u_{n}\varphi_{\varepsilon_{n}}=\tilde{f}(u_{n})u_{n}\varphi_{\varepsilon_{n}}$. Taking $u_{n}\varphi_{\varepsilon_{n}}$ as a test function in (6.1), by $(g_{3})$, similar argument to Lemma 3.6, we have that

[TABLE]

We claim that for small $\varepsilon_{n}>0$, there exists $y_{n}^{\prime}$ such that $\varepsilon_{n}y_{n}^{\prime}\in K_{\delta}$ and $|y_{n}^{\prime}-y_{n}|\leq R$. Otherwise, there exists a subsequence $\varepsilon_{n_{j}}\rightarrow 0$ such that $|z-y_{n_{j}}|\leq R$ and $\varepsilon_{n_{j}}z\not\in K_{\delta}$, that is, $B_{R}(y_{n_{j}})\cap\{z\in\mathbb{R}^{3}\,|\,\varepsilon_{n_{j}}z\in K_{\delta}\}={\O}$. Thus,

[TABLE]

which contradicts with Lemma 6.1. Thus the claim follows. Moreover,

[TABLE]

By the arbitrariness of $\delta$, we complete the proof. ∎

By Lemma 6.2, we see that $\lim\limits_{n\rightarrow\infty}{\rm dist}(\varepsilon_{n}y_{n},\Lambda^{\prime})=0$, hence, there is a subsequence of $\{\varepsilon_{n}y_{n}\}$, still denoted by $\varepsilon_{n}y_{n}$ and $x_{0}\in\overline{\Lambda^{\prime}}$ such that $\lim\limits_{n\rightarrow\infty}\varepsilon_{n}y_{n}=x_{0}$. Set

[TABLE]

where $G(u)=\int_{0}^{u}g(s)\,{\rm d}s$ and $g(u)=\chi(x_{0})(f(u)+(u^{+})^{2_{s}^{\ast}-1})+(1-\chi(x_{0}))\tilde{f}(u)$.

Lemma 6.3**.**

$x_{0}\in\Lambda$.

Proof.

It suffices to show that $V(x_{0})=V_{0}$. If this fact is proved, by $(V_{1})$ and the definition of $\Lambda^{\prime}$, we see that $x_{0}\not\in\partial\Lambda$ and $x_{0}\not\in\overline{\Lambda^{\prime}}/\Lambda$, then $x_{0}\in\Lambda$.

Now, clearly, $V(x_{0})\geq V_{0}$. The remain is to prove $V(x_{0})\leq V_{0}$. Set $v_{n}(z)=u_{n}(z+y_{n})$, then $v_{n}$ satisfies

[TABLE]

[TABLE]

and $\|v_{n}\|=\|u_{n}\|$ is bounded. Up to a subsequence, there exists $v\in H^{s}(\mathbb{R}^{3})\backslash\{0\}$ such that $v_{n}\rightharpoonup v$ in $H^{s}(\mathbb{R}^{3})$, $v_{n}\rightarrow v$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for all $1\leq r<2_{s}^{\ast}$ and $v_{n}\rightarrow v$ a.e. in $\mathbb{R}^{3}$.

Therefore, by $(iv)$ of Lemma 2.1, it is easy to show that

[TABLE]

for any $\varphi\in H^{s}(\mathbb{R}^{3})$. Thus, we get that $\langle\widetilde{J}_{V(x_{0})}^{\prime}(v),v\rangle=0$.

Let $c_{x_{0}}$ be the mountain-pass energy of $\widetilde{J}_{V(x_{0})}$. Since $\widetilde{J}_{V(x_{0})}(v)\geq\mathcal{I}_{V(x_{0})}(v)$, then $c_{x_{0}}\geq c_{V(x_{0})}$. Thus, similar argument to (3), we get

[TABLE]

Assume by the contrary that $V(x_{0})>V_{0}$. Denote $w$ be a ground state critical point of $\mathcal{I}_{V(x_{0})}$. By standard argument, there exists $\tau_{0}>0$ such that $\mathcal{I}_{V_{0}}(\tau_{0}w)=\sup_{\tau>0}\mathcal{I}_{V_{0}}(\tau w)$. Hence,

[TABLE]

which contradicts with (6). Thus $V(x_{0})=V_{0}$. ∎

From $V(x_{0})=V_{0}$, we see that $c_{V(x_{0})}=c_{V_{0}}$. It follows from (6) that $\|v_{n}\|\rightarrow\|v\|$. Using Brezis-Lieb Lemma, we conclude that $v_{n}\rightarrow v$ in $H^{s}(\mathbb{R}^{3})$.

Lemma 6.4**.**

Let $v_{n}(z)=u_{n}(z+y_{n})$ satisfies (6.2). Then

[TABLE]

Proof.

By Proposition 5.1, we see that there exists $C>0$ independent of $n$ such that $\|v_{n}\|_{L^{\infty}}\leq C$. Now, we rewrite the reduced form of problem (6.1) as follows

[TABLE]

where $h_{n}(z):=v_{n}-V(\varepsilon_{n}z+\varepsilon_{n}y_{n})v_{n}-\phi_{v_{n}}^{t}v_{n}+g(\varepsilon_{n}z+\varepsilon_{n}y_{n},v_{n})$. Clearly, $h_{n}\in L^{\infty}(\mathbb{R}^{3})$ and is uniformly bounded. By interpolation inequality and $v_{n}\rightarrow v$ in $H^{s}(\mathbb{R}^{3})$, for $n\rightarrow\infty$, we have that $h_{n}\rightarrow h$ in $L^{r}(\mathbb{R}^{3})$ for $2\leq r<+\infty$, where $h(z)=v(z)-V(x_{0})v(z)-\phi_{v}^{t}(z)v(z)+g(x_{0},v(z))$. Using some results found in [20], we see that

[TABLE]

where $\mathcal{K}$ is a Bessel potential, which possesses the following properties:

$(\mathcal{K}_{1})$ $\mathcal{K}$ is positive, radially symmetric and smooth in $\mathbb{R}^{3}\backslash\{0\}$;

$(\mathcal{K}_{2})$ there exists a constant $C>0$ such that $\mathcal{K}(x)\leq\frac{C}{|x|^{3+2s}}$ for all $x\in\mathbb{R}^{3}\backslash\{0\}$;

$(\mathcal{K}_{3})$ $\mathcal{K}\in L^{\tau}(\mathbb{R}^{3})$ for $\tau\in[1,\frac{3}{3-2s})$.

We define two sets $A_{\delta}=\{y\in\mathbb{R}^{3}\,\,|\,\,|z-y|\geq\frac{1}{\delta}\}$ and $B_{\delta}=\{y\in\mathbb{R}^{3}\,\,|\,\,|z-y|<\frac{1}{\delta}\}$. Hence,

[TABLE]

From the definition of $A_{\delta}$ and $(\mathcal{K}_{2})$, we have that for all $n\in\mathbb{N}$,

[TABLE]

On the other hand, by Hölder’s inequality and $(\mathcal{K}_{3})$, we deduce that

[TABLE]

where we have used the fact that $s>\frac{3}{4}$ so that $\frac{6}{3+2s}<\frac{3}{3-2s}$ and $2<\frac{3}{3-2s}$.

Since \Big{(}\int_{B_{\delta}}|h|^{2}\,{\rm d}y\Big{)}^{\frac{1}{2}}\rightarrow 0 as $|z|\rightarrow+\infty$, thus, we deduce that there exist $n_{0}\in\mathbb{N}$ and $R_{0}>0$ independence of $\delta>0$ such that

[TABLE]

Hence,

[TABLE]

For each $n\in\{1,2,\cdots,n_{0}-1\}$, there exists $R_{n}>0$ such that \Big{(}\int_{B_{\delta}}|h_{n}|^{2}\,{\rm d}y\Big{)}^{\frac{1}{2}}<\delta as $|z|\geq R_{n}$. Thus, for $|z|\geq R_{n}$, we have that

[TABLE]

for each $n\in\{1,2,\cdots,n_{0}-1\}$. Therefore, taking $R=\max\{R_{0},R_{1},\cdots,R_{n_{0}-1}\}$, we infer that for any $n\in\mathbb{N}$, there holds

[TABLE]

implies that $\lim\limits_{|z|\rightarrow\infty}v_{n}(z)=0$ uniformly in $n\in\mathbb{N}$. ∎

Lemma 6.5**.**

There is $n_{0}>0$ such that $u_{n}(z)=v_{n}(z-y_{n})<a$, for all $n\geq n_{0}$ and all $z\in\mathbb{R}^{3}\backslash(\Lambda/\varepsilon_{n})$. Hence, $v_{n}$ is a solution of problem (3.1) for $n\geq n_{0}$.

Proof.

By Lemma 6.3, we see that $\varepsilon_{n}y_{n}\rightarrow x_{0}$ and $x_{0}\in\Lambda$. Thus, there exists $R^{\prime}>0$ such that for some subsequence, still denoted by itself, $B_{R^{\prime}}(\varepsilon_{n}y_{n})\subset\Lambda\subset\Lambda^{\prime}$ for all $n\in\mathbb{N}$. Hence, $B_{R^{\prime}/\varepsilon_{n}}(y_{n})\subset\Lambda/\varepsilon_{n}$, $\forall n\in\mathbb{N}$. Moreover, by Lemma 6.4, there is $R_{1}>0$ such that $v_{n}(z)<a$ for $|z|\geq R_{1}$ and $\forall n\in\mathbb{N}$. Thus,

[TABLE]

Hence, there exists $n_{0}\in\mathbb{N}$ such that

[TABLE]

and then

[TABLE]

∎

Proof of Theorem 1.1. By Proposition 3.7, we see that problem (3.1) has a nonnegative solution $v_{\varepsilon}$ for all $\varepsilon>0$. From Lemma 6.5, there exists $\varepsilon_{0}>0$ such that

[TABLE]

which implies that $g(\varepsilon z,v_{\varepsilon})=f(v_{\varepsilon})+v_{\varepsilon}^{2_{s}^{\ast}-1}$. Thus, $v_{\varepsilon}$ is a solution of problem

[TABLE]

for all $\varepsilon\in(0,\varepsilon_{0})$. Let $u_{\varepsilon}(x)=v_{\varepsilon}(x/\varepsilon)$ for every $\varepsilon\in(0,\varepsilon_{0})$, it follows that $u_{\varepsilon}$ must be a solution to original problem (1.1) for $\varepsilon\in(0,\varepsilon_{0})$.

If $z_{\varepsilon}$ denotes a global maximum point of $v_{\varepsilon}$, then

[TABLE]

Suppose that $v_{\varepsilon}(z_{\varepsilon})<a$, taking $v_{\varepsilon}$ as a text function for (6.6), we get

[TABLE]

Hence we get a contradiction owing to the choosing $k>2$. In view of Lemma 6.4, we see that $\{z_{\varepsilon}\}$ is bounded for $\varepsilon\in(0,\varepsilon_{0})$.

In what follows, setting $x_{\varepsilon}=\varepsilon z_{\varepsilon}+\varepsilon y_{\varepsilon}$, where $\{y_{\varepsilon}\}$ is given in Lemma 6.1. Since $u_{\varepsilon}(x)=v_{\varepsilon}(\frac{x}{\varepsilon}-y_{\varepsilon})$, then $x_{\varepsilon}$ is a global maximum point of $u_{\varepsilon}$ and $u_{\varepsilon}(x_{\varepsilon})\geq a$ for all $\varepsilon\in(0,\varepsilon_{0})$.

Now, we claim that $\lim\limits_{\varepsilon\rightarrow 0^{+}}V(x_{\varepsilon})=V_{0}$. Indeed, if the above limit does not hold, there is $\varepsilon_{n}\rightarrow 0^{+}$ and $\gamma_{0}>0$ such that

[TABLE]

By Lemma 6.4, we know that $\lim\limits_{|z|\rightarrow\infty}v_{\varepsilon_{n}}(z)=0$ uniformly in $n\in\mathbb{N}$. From (6.7), thus $\{z_{\varepsilon_{n}}\}$ is a bounded sequence. Up to a subsequence, using Lemma 6.3, we know that there is $x_{0}\in\Lambda$ such that $V(x_{0})=V_{0}$ and $\varepsilon_{n}y_{\varepsilon_{n}}\rightarrow x_{0}$. Hence, $x_{\varepsilon_{n}}=\varepsilon_{n}z_{\varepsilon_{n}}+\varepsilon_{n}y_{\varepsilon_{n}}\rightarrow x_{0}$ which implies that $V(x_{\varepsilon_{n}})\rightarrow V(x_{0})=V_{0}$ contradicting with (6.8).

To complete the proof, we only need to prove the decay properties of $u_{\varepsilon}$. Similar argument to the proof of Lemma 5.6 in [46], we can obtain that

[TABLE]

Thus, by the boundedness of $\{z_{\varepsilon}\}$, i.e., there exists $C_{0}>0$ such that $|z_{\varepsilon}|\leq C_{0}$, we have

[TABLE]

7. Multiplicity of solutions to (1.1)

In this section, we will use the following two abstract Propositions to get the multiplicity of solutions.

Proposition 7.1**.**

([50]) Let $I$ be a $C^{1}$-functional defined on a $C^{1}$-Finsler manifold $M$. If $I$ is bounded from below and satisfies the $(PS)$ condition, then $I$ possesses at least ${\rm cat}_{M}(M)$ distinct critical point.

Let us consider $\delta>0$ such that $\mathcal{M}_{\delta}\subset\Lambda$ and a smooth cut-off function with $0\leq\eta\leq 1$, $\eta=1$ on $B_{\delta/2}(0)$, $\eta=0$ on $\mathbb{R}^{3}\backslash B_{\delta}(0)$, $|\nabla\eta|\leq C$. For any $y\in\mathcal{M}$, set

[TABLE]

where $w\in H^{s}(\mathbb{R}^{3})$ is a solution of (4.1) with $\mu=V_{0}$ such that $\mathcal{I}_{V_{0}}^{\prime}(w)=0$ and $\mathcal{I}_{V_{0}}=c_{V_{0}}$. Thus, there exists $t_{\varepsilon}>0$ such that $\max\limits_{t>0}J_{\varepsilon}(t\psi_{\varepsilon,y})=J_{\varepsilon}(t_{\varepsilon}\psi_{\varepsilon,y})$. We define $\Phi_{\varepsilon}:\mathcal{M}\rightarrow\mathcal{N}_{\varepsilon}$ by

[TABLE]

For the $\delta>0$ given by above, we choose $\rho=\rho(\delta)>0$ such that $\mathcal{M}_{\delta}\subset B_{\rho}(0)$. Define $\Upsilon:\mathbb{R}^{3}\rightarrow\mathbb{R}$ as $\Upsilon(z)=z$ for $|z|\leq\rho$ and $\Upsilon(z)=\frac{\rho z}{|z|}$ for $|z|\geq\rho$, and consider the map $\beta_{\varepsilon}:\mathcal{N}_{\varepsilon}\rightarrow\mathbb{R}^{3}$ given by

[TABLE]

Define

[TABLE]

where $h(\varepsilon)=\sup\limits_{y\in\mathcal{M}}|J_{\varepsilon}(\Phi_{\varepsilon}(y))-c_{V_{0}}|$.

Proposition 7.2**.**

([6], Lemma 4.3) Let $\Phi_{\varepsilon}:\mathcal{M}\rightarrow\widetilde{\mathcal{N}_{\varepsilon}}$, $\beta_{\varepsilon}:\widetilde{\mathcal{N}_{\varepsilon}}\rightarrow\mathcal{M}_{\delta}$ be two continuous maps defined as above. If $\beta_{\varepsilon}\circ\Phi_{\varepsilon}$ is homotopically equivalent to the embedding ${\rm id}:\mathcal{M}\rightarrow\mathcal{M}_{\delta}$. Then ${\rm cat}_{\widetilde{\mathcal{N}_{\varepsilon}}}(\widetilde{\mathcal{N}_{\varepsilon}})\geq{\rm cat}_{\mathcal{M}_{\delta}}(\mathcal{M})$.

Therefore, in Proposition 7.1, we choose the Finsler manifold $M$ as $\widetilde{\mathcal{N}_{\varepsilon}}$. It is standard to show the following result.

Proposition 7.3**.**

For any $\delta>0$, there exists $\varepsilon_{\delta}>0$ such that for any $\varepsilon\in(0,\varepsilon_{\delta})$, the system (3.1) has at least ${\rm cat}_{\mathcal{M}_{\delta}}(\mathcal{M})$ solutions, where $\mathcal{M}$ and $\mathcal{M}_{\delta}$ defined in Introduction.

The remain is to verify the $(PS)$ condition and the homopotically equivalent of $\beta_{\varepsilon}\circ\Phi_{\varepsilon}$ with the embedding ${\rm id}:\mathcal{M}\rightarrow\mathcal{M}_{\delta}$. The proof is standard, we are only to verify the $(PS)$ condition. The other detailed proof can be consulted in the papers [2, 3, 22] and the references therein.

Lemma 7.4**.**

Let $\{u_{n}\}\subset H_{\varepsilon}$ be a $(PS)_{c}$ sequence for $c\in(0,\frac{s}{3}\mathcal{S}_{s}^{\frac{3}{2s}})$. Then for each $\delta>0$, there exists $R>0$ such that

[TABLE]

Proof.

From Lemma 3.4, we know that $\{u_{n}\}$ is bounded in $H_{\varepsilon}$ and up to a subsequence, we may assume that there exists $u\in H_{\varepsilon}$ such that $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, $u_{n}\rightarrow u$ in $L_{loc}^{r}(\mathbb{R}^{3})$ for $1\leq r<2_{s}^{\ast}$ and $u_{n}\rightarrow u$ a.e. $\mathbb{R}^{3}$. First we may assume that $R$ is chosen such that $\Lambda^{\prime}/\varepsilon\subset B_{R/2}(0)$. Let $\eta_{R}$ be a smooth cut-off function so that $\eta_{R}=0$ on $B_{R/2}(0)$, $\eta_{R}=1$ on $\mathbb{R}^{3}\backslash B_{R}(0)$, $0\leq\eta_{R}\leq 1$ and $|\nabla\eta_{R}|\leq\frac{C}{R}$. Since $\{u_{n}\}$ is a bounded $(PS)_{c}$ sequence, we have

[TABLE]

Similar arguments to (2.2) and (2.2), we deduce that

[TABLE]

which implies that (7.1) holds. ∎

By the well known argument, we see that the Nehari manifold $\mathcal{N}_{\varepsilon}$ is a $C^{1}$-manifold.

Lemma 7.5**.**

The functional $J_{\varepsilon}$ restricted to $\mathcal{N}_{\varepsilon}$ satisfies $(PS)_{c}$ condition for each $c\in(0,\frac{s}{3}\mathcal{S}_{s}^{\frac{3}{2s}})$.

Proof.

Let $\{u_{n}\}\subset\mathcal{N}_{\varepsilon}$ be such that

[TABLE]

where $\|J_{\varepsilon}^{\prime}(u)$ denotes the norm of the derivative of $J_{\varepsilon}$ restricted to $\mathcal{N}_{\varepsilon}$ at the point $u\in\mathcal{N}_{\varepsilon}$. Similar arguments to the proof of Lemma 3.4, we obtain that $\{u_{n}\}$ is bounded in $H_{\varepsilon}$. Thus, up to a subsequence, we may assume that there is $u\in H_{\varepsilon}$ such that

[TABLE]

By standard computation, we can assume that there exists $\{\lambda_{n}\}\subset\mathbb{R}$ such that

[TABLE]

where $\mathcal{G}_{\varepsilon}(u)=\langle J_{\varepsilon}^{\prime}(u),u\rangle$. Moreover, since $u_{n}\in\mathcal{N}_{\varepsilon}$, we know that

[TABLE]

Next we will show that $\lambda_{n}\rightarrow 0$ as $n\rightarrow\infty$. Indeed, by the fact that $(\frac{f(\tau)}{\tau^{3}})^{\prime}>0$, $(\frac{\tilde{f}(\tau)}{\tau})^{\prime}>0$ for all $\tau\geq 0$, we deduce that

[TABLE]

Thus, we may assume that $\langle\mathcal{G}_{\varepsilon}^{\prime}(u_{n}),u_{n}\rangle\rightarrow l<0$. It follows from (7.4) that $\lambda_{n}\rightarrow 0$ as $n\rightarrow\infty$ and then we see that $J_{\varepsilon}^{\prime}(u_{n})\rightarrow 0$ as $n\rightarrow\infty$ in the dual space of $H_{\varepsilon}$. Hence, $\{u_{n}\}$ is a $(PS)_{c}$ sequence for $J_{\varepsilon}$.

We claim that

[TABLE]

In fact, by (7), we can also obtain that

[TABLE]

By interpolation inequality, we have that for any $\frac{1}{r}=\frac{\theta}{2}+\frac{1-\theta}{2_{s}^{\ast}}$ with $0<\theta\leq 1$,

[TABLE]

which yields

[TABLE]

By Sobolev inequality, (7.6) and using similar arguments to Lemma 3.6, we have that

[TABLE]

Therefore, we get

[TABLE]

which together with (7.3) implies that

[TABLE]

By the estimate (7.7), it is easy to check that

[TABLE]

On the other hand, using (7.3) and Lebesgue dominated convergence theorem, it is easy to show that

[TABLE]

and

[TABLE]

In order to prove (7.5), it is only need to show that

[TABLE]

Now we apply Lemma 2.2 to establish (7.9). Since $\{u_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$, by Phrokorov s theorem (Theorem 8.6.2 in [5]), there exist $\mu,\nu\in\mathcal{M}(\mathbb{R}^{3})$ such that

[TABLE]

By (7.8) and using Lemma 2.2, there exist an at most countable index set $J$, sequence $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$ and $\{\mu_{j}\}$, $\{\nu_{j}\}\subset(0,\infty)$ such that

[TABLE]

[TABLE]

and

[TABLE]

It is suffices to show that $\{x_{j}\}_{j\in J}\cap\{z|\chi(\varepsilon z)>0\}=\emptyset$. Suppose by contradiction that $\chi(\varepsilon x_{j})>0$ for some $j\in J$. Define the function $\psi_{\rho}(z)=\psi(\frac{z-x_{j}}{\rho})$ for $\rho>0$, where $\psi$ is a smooth cut-off function such that $\psi=1$ on $B_{1}(0)$, $\psi=0$ on $\mathbb{R}^{3}\backslash B_{2}(0)$, $0\leq\psi\leq 1$ and $|\nabla\psi|\leq C$. Suppose that $\rho$ is chosen in such a way that the support of $\psi_{\rho}$ is contained in $\{z|\chi(\varepsilon z)>0\}$. Since

[TABLE]

i.e.,

[TABLE]

By (7.3), we can deduce that

[TABLE]

[TABLE]

and similarly,

[TABLE]

[TABLE]

which leads to

[TABLE]

where we have used (7.10). Since for any $u,v\in H^{s}(\mathbb{R}^{3})$, there holds

[TABLE]

then it is easy to check that

[TABLE]

By nonlocal Leibniz rule, we have

[TABLE]

Using (2.3) and (2.4), it is easy to verify that

[TABLE]

and

[TABLE]

Therefore, by (7)–(7.16), we have that

[TABLE]

Combining with (7.12), we have

[TABLE]

Considering $\eta_{R}(z):=\eta(\frac{z}{R})$ for $R>0$, where $\eta$ is a smooth cut-off function such that $\eta=0$ on $B_{1}(0)$, $\eta=1$ on $\mathbb{R}^{3}\backslash B_{2}(0)$, $0\leq\eta\leq 1$ and $|\nabla\eta|\leq C$. Suppose that $R$ is chosen in such a way that $\Lambda^{\prime}/\varepsilon\subset B_{R}(0)$. By $(g_{3})$, we have that

[TABLE]

Since

[TABLE]

and using the similar argument to (2.2) and (2.2) to deduce that

[TABLE]

thus, by (7) and (7.12), we conclude that $\mu_{\infty}=\nu_{\infty}=0$.

On the other hand, by $(g_{2})$, (7.12) and $\mu_{\infty}=\nu_{\infty}=0$, we have that

[TABLE]

Thus, using (7.12) and (7.17), one has

[TABLE]

which leads to a contradiction. Hence (7.9) holds, then the claim (7.5) is true. Combining $\langle J_{\varepsilon}^{\prime}(u_{n}),u_{n}\rangle=0$ with (7.3) and (7.5), by standard argument, we can get $u_{n}\rightarrow u$ in $H_{\varepsilon}$. ∎

Acknowledgements. The work is supported by NSFC grant 11501403.