Concentrating phenomenon for fractional nonlinear Schr\"{o}dinger-Poisson system with critical nonlinearity

TL;DR

This paper investigates the concentration behavior of positive solutions to a fractional Schrödinger-Poisson system with critical nonlinearity, showing solutions localize near potential minima as the parameter approaches zero.

Contribution

It constructs a family of solutions that concentrate around the global minima of the potential, extending understanding of fractional systems with critical nonlinearities.

Findings

Solutions concentrate near potential minima as epsilon approaches zero.

Constructed positive solutions in fractional Sobolev space.

Analyzed effects of critical nonlinearity on solution behavior.

Abstract

In this paper, we study the following fractional Schr\"{o}dinger-Poisson system \begin{equation*} \left\{ \begin{array}{ll} \varepsilon^{2s}(-\Delta)^su+V(x)u+\phi u=g(u) & \hbox{in $\mathbb{R}^3$,} \varepsilon^{2t}(-\Delta)^t\phi=u^2,\,\, u>0& \hbox{in $\mathbb{R}^3$,} \end{array} \right. \end{equation*} where $s,t\in(0,1)$, $\varepsilon>0$ is a small parameter. Under some suitable assumptions on potential function $V(x)$ and critical nonlinearity term $g(u)$, we construct a family of positive solutions $u_{\varepsilon}\in H^s(\mathbb{R}^3)$ which concentrates around the global minima of $V$ as $\varepsilon\rightarrow0$.

Equations

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=g(u)&\hbox{in $\mathbb{R}^{3}$,}\\ \varepsilon^{2t}(-\Delta)^{t}\phi=u^{2},\,\,u>0&\hbox{in $\mathbb{R}^{3}$,}\end{array}\right.$$

$$V_0<\underset{\partial \Lambda }{\min }V(x),\mathcal{M}=\{x\in \Lambda |V(x)=V_0\}\ne \text{O}.$$

$$(-\Delta {)}^{s}u(x)=C_{s}P.V.{\int }_{{\mathbb{R}}^3}\frac{u(x)-u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y=C_s\underset{\epsilon \to 0}{\lim }{\int }_{{\mathbb{R}}^3\backslash B_{\epsilon }(x)}\frac{u(x)-u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y$$

$$(-\Delta)^{s}u=\mathcal{F}^{-1}\Big{(}(2\pi|\xi|)^{2s}\mathcal{F}u(\xi)\Big{)},$$

$$(-\Delta {)}^{s}u(x)=-\frac{1}{2}{C}_s{\int }_{{\mathbb{R}}^3}\frac{u(x+y)+u(x-y)-2u(y)}{|x-y{|}^{3+2s}}\mathrm{d}y.$$

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=f(u)+u^{2_{s}^{\ast}-1}&\hbox{in $\mathbb{R}^{3}$,}\\ \varepsilon^{2t}(-\Delta)^{t}\phi=u^{2},\,\,u>0&\hbox{in $\mathbb{R}^{3}$}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(x)u+\phi u=|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}\varepsilon^{2s}(-\Delta)^{s}u+V(x)u+\phi u=f(u)&\hbox{in $\mathbb{R}^{N}$,}\\ \varepsilon^{\theta}(-\Delta)^{\frac{\alpha}{2}}\phi=\gamma_{\alpha}u^{2}&\hbox{in $\mathbb{R}^{N}$,}\end{array}\right.$$

$$V_0=\underset{\Lambda }{\inf }V(x)<\underset{\partial \Lambda }{\min }V(x);$$

$$\left\{\begin{array}[]{ll}-\varepsilon^{2}\Delta u+V(x)u+\phi u=g(x,u)&\hbox{in $\mathbb{R}^{3}$,}\\ -\varepsilon^{2}\Delta\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$.}\end{array}\right.$$

$$\left\{\begin{array}[]{ll}-\varepsilon^{2}\Delta u+V(x)u+K(x)\phi u=u^{p}&\hbox{in $\mathbb{R}^{3}$,}\\ -\Delta\phi=u^{2}&\hbox{in $\mathbb{R}^{3}$}\end{array}\right.$$

$$u_{\epsilon }(x)\le \frac{C{\epsilon }^{3+2s}}{C_0{\epsilon }^{3+2s}+|x-x_{\epsilon }{|}^{3+2s}}x\in {\mathbb{R}}^3,\text{and}\epsilon \in (0,{\epsilon }_0)$$

$$\underset{\Lambda }{\inf }V(x)<\underset{\partial \Lambda }{\inf }V(x)$$

$$\mathcal{D}^{\alpha,2}(\mathbb{R}^{3})=\Big{\{}u\in L^{2_{\alpha}^{\ast}}(\mathbb{R}^{3})\,\,\Big{|}\,\,|\xi|^{\alpha}(\mathcal{F}u)(\xi)\in L^{2}(\mathbb{R}^{3})\Big{\}}$$

$$\|u\|_{\mathcal{D}^{\alpha,2}}=\Big{(}\int_{\mathbb{R}^{3}}|(-\Delta)^{\frac{\alpha}{2}}u|^{2}\,{\rm d}x\Big{)}^{\frac{1}{2}}=\Big{(}\int_{\mathbb{R}^{3}}|\xi|^{2\alpha}|(\mathcal{F}u)(\xi)|^{2}\,{\rm d}\xi\Big{)}^{\frac{1}{2}}$$

$$H^{\alpha}(\mathbb{R}^{3})=\Big{\{}u\in L^{2}(\mathbb{R}^{3})\,\,\Big{|}\,\,\int_{\mathbb{R}^{3}}(|\xi|^{2\alpha}|(\mathcal{F}u)(\xi)|^{2}+|(\mathcal{F}u)(\xi)|^{2})\,{\rm d}\xi<+\infty\Big{\}}.$$

$$(u,v)={\int }_{{\mathbb{R}}^3}(|\xi {|}^{2\alpha }(\mathcal{F}u)(\xi )\overline{(\mathcal{F}v)(\xi )}+(\mathcal{F}u)(\xi )\overline{(\mathcal{F}v)(\xi )})\mathrm{d}\xi$$

$$\|u\|_{H^{\alpha}}=\bigg{(}\int_{\mathbb{R}^{3}}(|\xi|^{2\alpha}|(\mathcal{F}u)(\xi)|^{2}+|(\mathcal{F}u)(\xi)|^{2})\,{\rm d}\xi\bigg{)}^{\frac{1}{2}}.$$

$$\|u\|_{H^{\alpha}}=\bigg{(}\int_{\mathbb{R}^{3}}(|(-\Delta)^{\frac{\alpha}{2}}u(x)|^{2}+|u(x)|^{2})\,{\rm d}x\bigg{)}^{\frac{1}{2}},\quad\forall u\in H^{\alpha}(\mathbb{R}^{3}).$$

$$H^{\alpha}(\mathbb{R}^{3})=\Big{\{}u\in L^{2}(\mathbb{R}^{3})\,\,\Big{|}\,\,D_{\alpha}u\in L^{2}(\mathbb{R}^{3})\Big{\}},\quad|D_{\alpha}u|^{2}=\int_{\mathbb{R}^{3}}\frac{|u(x)-u(y)|^{2}}{|x-y|^{3+2\alpha}}\,{\rm d}y$$

$$\|u\|_{H^{\alpha}}=\bigg{(}\int_{\mathbb{R}^{3}}|u|^{2}\,{\rm d}x+\int_{\mathbb{R}^{3}}|D_{\alpha}u|^{2}\,{\rm d}x\bigg{)}^{\frac{1}{2}}.$$

$$\Vert (-\Delta {)}^{\frac{\alpha }{2}}u{\Vert }_2^2={\int }_{{\mathbb{R}}^3}|\xi {|}^{2\alpha }|(\mathcal{F}u)(\xi ){|}^2\mathrm{d}\xi =\frac{1}{C_{\alpha }}{\int }_{{\mathbb{R}}^3}|D_{\alpha }u{|}^2\mathrm{d}x.$$

$$\|u\|_{H_{\varepsilon}}=\Big{(}\int_{\mathbb{R}^{3}}(|D_{s}u|^{2}+V(\varepsilon x)u^{2})\,{\rm d}x\Big{)}^{\frac{1}{2}}.$$

$$\left\{\begin{array}[]{ll}(-\Delta)^{s}u+V(\varepsilon z)u+\phi u=f(u)+u^{2_{s}^{\ast}-1}&\hbox{in $\mathbb{R}^{3}$,}\\ (-\Delta)^{t}\phi=u^{2},\,\,u>0&\hbox{in $\mathbb{R}^{3}$}\end{array}\right.$$

$$C_s{\int }_{{\mathbb{R}}^3\times {\mathbb{R}}^3}\frac{({\varphi }_u^t(z)-{\varphi }_u^t(y))(v(z)-v(y))}{|z-y{|}^{3+2s}}\mathrm{d}y\mathrm{d}z$$

$$={\int }_{{\mathbb{R}}^3}{u}^{2}v\mathrm{d}z,\forall v\in {\mathcal{D}}^{t,2}({\mathbb{R}}^3),$$

$${\varphi }_u^t(x)=c_t{\int }_{{\mathbb{R}}^3}\frac{u^2(y)}{|x-y{|}^{3-2t}}\mathrm{d}y,x\in {\mathbb{R}}^3,c_t={\pi }^{-\frac{3}{2}}{2}^{-2t}\frac{\Gamma (\frac{3-2t}{2})}{\Gamma (t)}.$$

$$(-\Delta {)}^{s}u+V(\epsilon z)u+{\varphi }_u^{t}u=f(u)+(u^{+}{)}^{2_s^{\ast }-1}z\in {\mathbb{R}}^3.$$

$$J_{\epsilon }(u)$$

$$-\frac{1}{2_s^{\ast }}{\int }_{{\mathbb{R}}^3}(u^{+}{)}^{2_s^{\ast }}\mathrm{d}z.$$

Full text

[TABLE]

where $s,t\in(0,1)$, $\varepsilon>0$ is a small parameter. Under some suitable assumptions on potential function $V(x)$ and critical nonlinearity term $g(u)$, we construct a family of positive solutions $u_{\varepsilon}\in H^{s}(\mathbb{R}^{3})$ which concentrates around the global minima of $V$ as $\varepsilon\rightarrow 0$.

1. Introduction

In this paper, we study the following fractional Schrödinger-Poisson system

[TABLE]

where $s,t\in(0,1)$ and $\varepsilon>0$ is a small parameter. The potential $V:\mathbb{R}^{3}\rightarrow\mathbb{R}$ is a bounded continuous function satisfying

$(V_{0})$ $\inf\limits_{x\in\mathbb{R}^{3}}V(x)=V_{0}>0$;

$(V_{1})$ There is a bounded domain $\Lambda\subset\mathbb{R}^{3}$ such that

[TABLE]

Without of loss of generality, we may assume that $0\in\mathcal{M}$. The nonlinearity $g:\mathbb{R}\rightarrow\mathbb{R}$ is of $C^{1}$-class function. The non-local operator $(-\Delta)^{s}$ ($s\in(0,1)$), which is called fractional Laplace operator, can be defined by

[TABLE]

for $u\in\mathcal{S}(\mathbb{R}^{3})$, where $\mathcal{S}(\mathbb{R}^{3})$ is the Schwartz space of rapidly decaying $C^{\infty}$ function, $B_{\varepsilon}(x)$ denote an open ball of radius $r$ centered at $x$ and the normalization constant C_{s}=\Big{(}\int_{\mathbb{R}^{3}}\frac{1-\cos(\zeta_{1})}{|\zeta|^{3+2s}}\,{\rm d}\zeta\Big{)}^{-1}. Fractional Laplacian appears in lots of real world, such as: fractional quantum mechanics [33, 34], anomalous diffusion [39], financial [15], obstacle problems [48], conformal geometry and minimal surfaces [12]. In the very recent years, the progress of nonlinear equations involving fractional Lapalcian can be found in [1, 5, 9, 13, 14, 15, 16, 18, 19, 20, 22, 28, 42, 43, 48, 49, 51, 52] and so on.

For $u\in\mathcal{S}(\mathbb{R}^{3})$, the fractional Laplace operator $(-\Delta)^{s}$ can be expressed as an inverse Fourier transform

[TABLE]

where $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the Fourier transform and inverse transform, respectively. If $u$ is sufficiently smooth, it is known that (see [42]) it is equivalent to

[TABLE]

By a classical solution of (1.1), we mean two continuous functions that $(-\Delta)^{s}u$ is well defined for all $x\in\mathbb{R}^{3}$ and satisfies (1.1) in pointwise sense.

Since we are looking for positive solutions, we may assume that $g(s)=0$ for $s<0$. Furthermore, we need the following conditions:

$(g_{0})$ $\lim\limits_{\tau\rightarrow 0^{+}}\frac{g(\tau)}{\tau}=0$;

$(g_{1})$ $\lim\limits_{\tau\rightarrow+\infty}\frac{g(\tau)}{\tau^{2_{s}^{\ast}-1}}=\kappa>0$;

$(g_{2})$ there exists $\lambda>0$ such that $g(\tau)\geq\lambda\tau^{q-1}+\tau^{2_{s}^{\ast}-1}$ for some $\frac{4s+2t}{s+t}<q<2_{s}^{\ast}$ and all $\tau\geq 0$.

The hypotheses $(g_{0})$-$(g_{2})$ are so-called the critical Berestycki-Lions type conditions, which was introduced in [63]. For simplicity, we may assume that $\kappa=1$ and $g(\tau)=f(\tau)+|\tau|^{2_{s}^{\ast}-2}\tau$, for $\tau>0$. Then system (1.1) is equivalent to the following one

[TABLE]

where $f$ satisfies

$(f_{0})$ $\lim\limits_{\tau\rightarrow 0^{+}}\frac{f(\tau)}{\tau}=0$;

$(f_{1})$ $\liminf\limits_{\tau\rightarrow+\infty}\frac{f^{\prime}(\tau)}{\tau^{2_{s}^{\ast}-2}}=0$;

$(f_{2})$ there exists $\lambda>0$ such that $f(\tau)\geq\lambda\tau^{q-1}$, for $\tau>0$ and some $q\in(\frac{4s+2t}{s+t},2_{s}^{\ast})$.

In the very recent years, the study of the existence, concentration and multiplicity of positive solutions for fractional Schrödinger-Poisson system (1.1) is just starting. When $\varepsilon=1$, by using the Nehari-Pohozaev manifold combing monotone trick with global compactness Lemma, Teng [55] studied the existence of positive ground state solution for the system

[TABLE]

Using the similar methods, in [56], positive ground state solutions for problem (1.3) with $|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u$ replaced by $|u|^{p-1}u$ with $p\in(2,2_{s}^{\ast}-1)$, were established when $s=t$. In [62], the authors studied the existence of radial solutions for system (1.3) with $|u|^{p-1}u+|u|^{2_{s}^{\ast}-2}u$ replaced by $f(u)$, where the nonlinearity $f(u)$ verifies the subcritical or critical assumptions of Berestycki-Lions type. When $0<\varepsilon<1$ small, in [40], the authors studied the semiclassical state of the following system

[TABLE]

where $s\in(0,1)$, $\alpha\in(0,N)$, $\theta\in(0,\alpha)$, $N\in(2s,2s+\alpha)$, $\gamma_{\alpha}$ is a positive constant, $f(u)$ satisfies the following subcritical growth assumptions: $0<KF(t)\leq f(t)t$ with some $K>4$ for all $t\geq 0$ and $\frac{f(t)}{t^{3}}$ is strictly increasing on $(0,+\infty)$. In [38], by using the methods mentioned before, Liu and Zhang proved the existence and concentration of positive ground state solution for problem (1.2). When the system (1.2) verifying that $s=t$ and $f(u)+u^{2_{s}^{\ast}-1}$ replaced by $K(x)|u|^{p-2}u$ which $V$ has positive global minimum and $K(x)$ has global maximum, in [60], the authors prove the existence of a positive ground state for $\varepsilon>0$ sufficiently small and concentration behavior of these ground state solutions as $\varepsilon\rightarrow 0$. In [55], we studied the system (1.1) with competing potential, i.e., $g(u)=K(x)f(u)+Q(x)|u|^{2_{s}^{\ast}-2}u$, where $f$ is a function of $C^{1}$ class, superlinear and subcritical nonlinearity, $V(x)$, $K(x)$ and $Q(x)$ are positive continuous functions. Under some suitable assumptions on $V$, $K$ and $Q$, we prove that there is a family of positive ground state solutions which concentrate on the set of minimal points of $V(x)$ and the sets of maximal points of $K(x)$ and $Q(x)$. For the local assumption on the potential $V(x)$, Teng [56] firstly applied the penalization methods developed by [17] to study the concentration phenomenon of system (1.4) under the hypotheses made on $V(x)$ and $f$:

• •

$\inf\limits_{x\in\mathbb{R}^{3}}V(x)=\alpha_{0}>0$ and there is a bounded domain $\Lambda\subset\mathbb{R}^{3}$ such that

[TABLE]

• •

$\lim\limits_{\tau\rightarrow 0^{+}}\frac{f(\tau)}{\tau^{3}}=0$, there exist $\lambda>0$ and $C>0$ such that $f(\tau)\geq\lambda\tau^{q-1}$ for some $4\leq q<2_{s}^{\ast}$ and $|f^{\prime}(\tau)|\leq C(1+|\tau|^{p-2})$, where $4<p<2_{s}^{\ast}$;

• •

$\frac{f(\tau)}{\tau^{3}}$ is non-decreasing in $\tau\in(0,+\infty)$.

The penalization methods were applied to fractional Schrödinger equations, please see [1, 3, 28]. For extending our result in [56], through modifying the penalization methods developed by Byeon and Wang [10], Teng [57] studied the concentration behavior of system (1.1) with $V(x)$ satisfying $(V_{0})$-$(V_{1})$ and $g$ verifying

• •

$\lim\limits_{\tau\rightarrow 0^{+}}\frac{g(\tau)}{\tau}=0$, $\lim\limits_{\tau\rightarrow+\infty}\frac{g^{\prime}(\tau)}{\tau^{2_{s}^{\ast}-2}}=0$;

• •

there exists $\lambda>0$ such that $g(\tau)\geq\lambda\tau^{q-1}$ for some $\frac{4s+2t}{s+t}<q<2_{s}^{\ast}$ and all $\tau\geq 0$;

• •

$\frac{g(\tau)}{\tau^{q-1}}$ is non-decreasing in $\tau\in(0,+\infty)$.

When $s=1$, system (1.1) reduces to the following Schrödinger-Poisson system

[TABLE]

In recent years, there has been increasing attention to (1.4) on the existence of positive solutions, ground state solutions, multiple solutions and semiclassical states, see for example [2, 6, 21, 27, 45, 46, 64] and the references therein. It is well known that system (1.4) appearing in quantum mechanics models (see e.g. [37]) and in semiconductor theory [41]. Especially, systems like (1.4) have been introduced in [6] as a model to describe solitary waves. Regarding the concentration phenomenon of solutions for Schrödinger-Poisson systems like (1.4), there has been the object of interest for many scholars. In [26], the author studied the system (1.4) with $g(x,v)=f(v)$ satisfying

• •

$\frac{f(t)}{t^{3}}$ increasing in $(0,\infty)$, $\exists\theta>4$ such that $0<\theta F(t)=\int_{0}^{t}f(s)\,{\rm d}s\leq tf(t)$ for all $t>0$,

• •

$f^{\prime}(t)t^{2}-3f(t)t\geq Ct^{\sigma}$, $\sigma\in(4,6)$, and $f(t)=o(t^{3})$ as $t\rightarrow 0$.

By using Ljusternik-Schnirelmann theory and minimax method, the author obtained the multiplicity of positive solutions for $\varepsilon>0$ small which concentrate on the minima of $V(x)$. In [58], Wang et al. studied the existence and concentration of positive ground state solutions for system (1.4) with $g(x,v)=b(x)f(v)$ satisfying

• •

$\frac{f(t)}{t^{3}}$ increasing in $(0,\infty)$, $\frac{F(t)}{t^{4}}\rightarrow+\infty$ as $t\rightarrow\infty$,

• •

$|f(t)|\leq c(1+|t|^{p-1})$, $p\in(4,6)$, and $f(t)=o(t^{3})$ as $t\rightarrow 0$.

In the critical case, He and Zou [27] considered system (1.4) with $g(x,v)=v^{5}+f(v)$, where $f$ satisfies the similar hypotheses as [26], proved that system (1.4) has a ground state solution concentrating around a global minimum of $V(x)$ as $\varepsilon\rightarrow 0$. In [59], the authors studied the system (1.4) with $g(x,v)=b(x)f(v)+|v|^{4}v$, where $f$ satisfies

• •

$\frac{f(t)}{t^{3}}$ increasing in $(0,\infty)$, $f(t)=o(t^{3})$ as $t\rightarrow 0$,

• •

$f(t)\geq ct^{q-1}$, $|f(t)|\leq c(1+|t|^{p-1})$, $4<q\leq p<6$.

Under some suitable assumptions on $V(x)$ and $b(x)$, Wang et al. [59] proved the existence of least energy solution $(u_{\varepsilon},\phi_{\varepsilon})$ and then showed that $u_{\varepsilon}$ converges to the least energy solution of the associated limit problem and concentrates to some set in $\mathbb{R}^{3}$ depending on the potentials $V$ and $b$. The above assumptions made on the potential $V(x)$ is global, for the local assumption, there are few results obtained in the literature. As far as we know, only in [29] studied the Schrödinger-Poisson system (1.4) with $V(x)$ satisfying the local condition $\inf\limits_{\Lambda}V(x)<\inf\limits_{\partial\Lambda}V(x)$ and $g(x,v)=\lambda|v|^{p-2}v+|v|^{4}v$ for $3<p\leq 4$, where $\Lambda$ is an open set of $\mathbb{R}^{3}$ and $\lambda>0$, the authors constructed a family of positive solutions which concentrates around a local minimum of $V$ as $\varepsilon\rightarrow 0$.

The semiclassical state of the following Schrödinger-Poisson system

[TABLE]

has attracted many scholars’ attention. When $p\in(1,5)$, Ruiz and Vaira [47] proved the existence of multi-bump solutions of system and these bumps concentrate around a local minimum of the potential $V$. Ianni and Vaira [31] obtained the existence of positive bound state solutions which concentrate on a non-degenerate local minimum or maximum of $V$ by using a Lyapunov-Schmitt reduction method. Ianni and Vaira [30] also showed the existence of radially symmetric solutions, which concentrate on the spheres. For the critical case, for system (1.5) with $u^{p}$ replaced by $f(u)+u^{5}$, in [36], the authors proved the multiplicity of positive solutions and the number of positive solutions depends on the profile of the potential and that each solution concentrates around its corresponding global minimum point of the potential in the semiclassical limit. For the local assumptions on the potential $V(x)$, Seok [48] studied the system (1.5) with $u^{p}$ replaced by $f(u)$ satisfying

• •

$f(t)=o(t)$ as $t\rightarrow 0$, $\lim\limits_{t\rightarrow\infty}\frac{f(t)}{t^{p}}<\infty$ for some $p\in(1,5)$,

• •

$\exists T>0$ such that $\frac{1}{2}mT^{2}<F(T)$, $F(t)=\int_{0}^{t}f(s)\,{\rm d}s$

and proved the existence of the spike solutions through following a variational approach developed by Byeon-Jeanjean [7, 8]. Using the similar ideas as Byeon-Jeanjean [7], Zhang [61] considered the system (1.5) with $u^{p}$ replaced by a general nonlinearity $f(u)$ satisfying the critical growth assumptions

• •

$f(t)=o(t)$ as $t\rightarrow 0$, $\lim\limits_{t\rightarrow\infty}\frac{f(t)}{t^{5}}=\kappa>0$,

• •

$\exists C>0$ and $p<6$ such that $f(t)\geq\kappa t^{5}+Ct^{p-1}$ for $t\geq 0$

and constructed a solution $(u_{\varepsilon},\phi_{\varepsilon})$, which concentrates at an isolated component of positive locally minimum points of $V$ as $\varepsilon\rightarrow 0$.

From the above known results, we see that the monotonic hypothesis $\frac{f(t)}{t^{3}}$ is necessary to study the concentration behavior of system (1.4) whatever critical case or subcritical case. The purpose of this paper is to weak this monotonic hypothesis to the following one:

$(f_{3})$ $\frac{f(\tau)}{\tau^{q-1}}$ is non-decreasing in $\tau\in(0,+\infty)$, where $q\in(\frac{4s+2t}{s+t},2_{s}^{\ast})$.

To the best of our knowledge, except [29], there are few papers to study the concentration phenomenon of Schrödinger-Poisson system (1.4) with local assumption on the potential $V(x)$, not mention to the fractional Schrödinger-Poisson system (1.1). Motivated by the above cited papers, the goal of this paper is to study the existence and concentration of positive bound state solutions for system (1.2) under $(V_{0})$-$(V_{1})$ and $(f_{0})$-$(f_{3})$.

Our main results is as follows.

Theorem 1.1**.**

Let $2s+2t>3$, $s,t\in(0,1)$. Suppose that $V$ satisfies $(V_{0})$, $(V_{1})$ and $g\in C(\mathbb{R}^{+},\mathbb{R})$ satisfies $(g_{0})$–$(g_{3})$. Then there exists an $\varepsilon_{0}>0$ such that system (1.1) possesses a positive solution $(u_{\varepsilon},\phi_{\varepsilon})\in H_{\varepsilon}\times\mathcal{D}^{t,2}(\mathbb{R}^{3})$ for all $\varepsilon\in(0,\varepsilon_{0})$. Moreover, there exists a maximum point $x_{\varepsilon}$ of $u_{\varepsilon}$ such that $\lim\limits_{\varepsilon\rightarrow 0}{\rm dist}(x_{\varepsilon},\mathcal{M})=0$ and

[TABLE]

for some constants $C>0$ and $C_{0}\in\mathbb{R}$.

We will give some comments on our main result.

Remark 1.2**.**

The hypothesis $(V_{1})$ is a special case of the local assumption

[TABLE]

which first introduced by M. del Pino and P. L. Felmer [17], because there have not some local priori estimates like Theorem 8.17 in [25].

Remark 1.3**.**

Comparing with the results in [26, 27, 58, 59], the monotone hypothesis $(f_{3})$ is weaker even in the case $s=t=1$ ($q-1>\frac{4s+2t}{s+t}-1=2$).

Remark 1.4**.**

The condition $(V_{2})$ is local and the $(AR)$-condition for fractional Schrödinger-Poisson system is not satisfied, we need to modify the penalization methods developed by J. Byeon, Z. Q. Wang [7, 10] and combine the penalization methods introduced by M. del Pino, P. L. Felmer [17], for overcoming the obstacle caused by the non-compactness due to the unboundedness of the domain and the lack of $(AR)$ condition.

The paper is organized as follows, in Section 2, we give some preliminary results. In Section 3, we prove the existence of positive ground state solutions for ”limit problem”. In Section 4, we prove the main result Theorem 1.1.

2. Variational Setting

In this section, we outline the variational framework for studying problem (1.2) and list some preliminary Lemma which used later. In the sequel, we denote by $\|\cdot\|_{p}$ the usual norm of the space $L^{p}(\mathbb{R}^{3})$, the letter $c_{i}$ ($i=1,2,\ldots$) or $C$ denote by some positive constants.

2.1. Work space stuff

We define the homogeneous fractional Sobolev space $\mathcal{D}^{\alpha,2}(\mathbb{R}^{3})$ as follows

[TABLE]

which is the completion of $C_{0}^{\infty}(\mathbb{R}^{3})$ under the norm

[TABLE]

The fractional Sobolev space $H^{\alpha}(\mathbb{R}^{3})$ can be described by means of the Fourier transform, i.e.

[TABLE]

In this case, the inner product and the norm are defined as

[TABLE]

and

[TABLE]

From Plancherel’s theorem we have $\|u\|_{2}=\|\mathcal{F}u\|_{2}$ and $\||\xi|^{\alpha}\mathcal{F}u\|_{2}=\|(-\Delta)^{\frac{\alpha}{2}}u\|_{2}$. Hence

[TABLE]

We denote $\|\cdot\|$ by $\|\cdot\|_{H^{\alpha}}$ in the sequel for convenience.

In terms of finite differences, the fractional Sobolev space $H^{\alpha}(\mathbb{R}^{3})$ also can be defined as follows

[TABLE]

endowed with the natural norm

[TABLE]

Also, in view of Proposition 3.4 and Proposition 3.6 in [42], we have

[TABLE]

We define the Sobolev space $H_{\varepsilon}=\{u\in H^{s}(\mathbb{R}^{3})\,\,|\,\,\int_{\mathbb{R}^{3}}V(\varepsilon x)u^{2}\,{\rm d}x<\infty\}$ endowed with the norm

[TABLE]

It is well known that (see [35]) $H^{s}(\mathbb{R}^{3})$ is continuously embedded into $L^{r}(\mathbb{R}^{3})$ for $2\leq r\leq 2_{s}^{\ast}$ ($2_{s}^{\ast}=\frac{6}{3-2s}$). Obviously, the conclusion also holds for $H_{\varepsilon}$.

2.2. Formulation of Problem (1.2)

It is easily seen that, just performing the change of variables $u(x)\rightarrow u(x/\varepsilon)$ and $\phi(x)\rightarrow\phi(x/\varepsilon)$, and taking $z=x/\varepsilon$, problem (1.2) can be rewritten as the following equivalent form

[TABLE]

which will be referred from now on. Observe that if $4s+2t\geq 3$, there holds $2\leq\frac{12}{3+2t}\leq\frac{6}{3-2s}$ and thus $H_{\varepsilon}\hookrightarrow L^{\frac{12}{3+2t}}(\mathbb{R}^{3})$. Considering $u\in H_{\varepsilon}$, the linear functional $\widetilde{\mathcal{L}}_{u}:\mathcal{D}^{t,2}(\mathbb{R}^{3})\rightarrow\mathbb{R}$ is defined by $\widetilde{\mathcal{L}}_{u}(v)=\int_{\mathbb{R}^{3}}u^{2}v\,{\rm d}x$. Using the Lax-Milgram theorem, there exists a unique $\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$ such that

[TABLE]

that is $\phi_{u}^{t}$ is a weak solution of $(-\Delta)^{t}\phi_{u}^{t}=u^{2}$ and so the representation formula holds

[TABLE]

Substituting $\phi_{u}^{t}$ in (2.2), it reduces to a single fractional Schrödinger equation

[TABLE]

The solvation of (2.3) can be looking for the critical points of the associated energy functional $J_{\varepsilon}:H_{\varepsilon}\rightarrow\mathbb{R}$ defined by

[TABLE]

Let us summarize some properties of the function $\phi_{u}^{t}$. By using simple computation, it is easy to check the following conclusions.

Lemma 2.1**.**

*For every $u\in H_{\varepsilon}$ with $4s+2t\geq 3$, define $\Phi(u)=\phi_{u}^{t}\in\mathcal{D}^{t,2}(\mathbb{R}^{3})$, where $\phi_{u}^{t}$ is the unique solution of equation $(-\Delta)^{t}\phi=u^{2}$. Then there hold:

$(i)$ If $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$, then $\Phi(u_{n})\rightharpoonup\Phi(u)$ in $\mathcal{D}^{t,2}(\mathbb{R}^{3})$;

$(ii)$ $\Phi(tu)=t^{2}\Phi(u)$ for any $t\in\mathbb{R}$;

$(iii)$ For $u\in H_{\varepsilon}$, one has*

[TABLE]

*where constant $C$ is independent of $u$;

$(iv)$ Let $2s+2t>3$, if $u_{n}\rightharpoonup u$ in $H_{\varepsilon}$ and $u_{n}\rightarrow u$ a.e. in $\mathbb{R}^{3}$, then for any $v\in H_{\varepsilon}$,*

[TABLE]

and

[TABLE]

and thus $u$ is a solution for problem (2.3).

In the following, we collect some useful Lemma. We define

[TABLE]

Lemma 2.2**.**

*([43, 65]) Let $\{u_{n}\}\subset H^{s}(\mathbb{R}^{3})$ be such that $u_{n}\rightharpoonup u$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$, $|D_{s}u_{n}|^{2}\rightharpoonup\mu$ and $|u_{n}|^{2_{s}^{\ast}}\rightharpoonup\nu$ weakly$-\ast$ in $\mathcal{M}(\mathbb{R}^{3})$ as $n\rightarrow\infty$. Here $\mathcal{M}(\mathbb{R}^{3})$ is the space of finite nonnegative Borel measures on $\mathbb{R}^{3}$. Then

$(i)$ there exist a (at most countable) set of distinct points $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$, $\mu_{j}\geq 0$, $\nu_{j}\geq 0$ with $\mu_{j}+\nu_{j}>0$ ($j\in J$) such that*

[TABLE]

$(ii)$* Then $\mu_{\infty}$ and $\nu_{\infty}$ are well defined satisfy*

[TABLE]

$(iii)$**

[TABLE]

Proposition 2.3**.**

([53]) Let $\{u_{n}\}$ be a bounded sequence in $H^{s}(\mathbb{R}^{3})$. If

[TABLE]

where $R$ is a positive number, then $u_{n}\rightarrow 0$ in $L^{2_{s}^{\ast}}(\mathbb{R}^{3})$ as $n\rightarrow\infty$.

Proposition 2.4**.**

Let $\{u_{k}\}\subset\mathcal{D}^{s,2}(\mathbb{R}^{3})$ be a bounded sequence such that $u_{k}\rightharpoonup 0$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$. Suppose that there exists a bounded open set $Q\subset\mathbb{R}^{3}$ and a positive number $\gamma>0$ such that

[TABLE]

Moreover, suppose that

[TABLE]

where $\chi_{k}\in(H^{s}(\mathbb{R}^{3}))^{\prime}$, and $|\langle\chi_{k},\varphi\rangle|\leq\varepsilon_{k}\|\varphi\|$ for any $\varphi\in C_{0}^{\infty}(V)$, where $V$ is an open neighborhood of $Q$ and $\varepsilon_{k}\rightarrow 0$ as $k\rightarrow\infty$. Then there exist a sequence of points $\{z_{k}\}\in\mathbb{R}^{3}$ and a sequence of positive numbers $\{\sigma_{k}\}$ such that $v_{k}(x)=\sigma_{k}^{\frac{3-2s}{2}}u_{k}(\sigma_{k}x+z_{k})$ converges weakly in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$ to a nontrivial solution $v$ of

[TABLE]

Moreover, $z_{k}\rightarrow z\in\overline{Q}$ and $\sigma_{k}\rightarrow 0$ as $k\rightarrow\infty$.

Proof.

Since $\{u_{k}\}$ is bounded in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$ and $u_{k}\rightharpoonup 0$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$, by Phrokorov s theorem (Theorem 8.6.2 in [4]), there exist $\mu,\nu\in\mathcal{M}(\mathbb{R}^{3})$ such that

[TABLE]

By Lemma 2.2, there exist an at most countable index set $J$, sequence $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$ and $\{\nu_{j}\}\subset(0,\infty)$ such that

[TABLE]

We claim that there is at least one $j_{0}\in J$ such that $x_{j_{0}}\in\overline{Q}$ with $\nu_{j_{0}}>0$. If not, for all $j\in J$, $x_{j}\not\in\overline{Q}$ with $\nu_{j}>0$, then

[TABLE]

Taking ${\rm supp}(\varphi)=\overline{Q}$, we see that $\int_{Q}|u_{k}|^{2_{s}^{\ast}}\,{\rm d}x\rightarrow 0$, contradicts with (2.4). Thus, the claim is true.

We define the Lévy concentration function

[TABLE]

then $Q_{k}$ is a non-decreasing and bounded function. Fixing a small $\tau\in(0,\mathcal{S}_{s}^{\frac{3}{2s}})$, we can find $\sigma_{k}:=\sigma_{k}(\tau)\in\mathbb{R}_{+}$, $z_{k}\in\overline{Q}$ such that

[TABLE]

Set $v_{k}(x)=\sigma_{k}^{\frac{3-2s}{2}}u_{k}(\sigma_{k}x+z_{k})$, we have that

[TABLE]

where $\bar{Q}_{k}=\{x\in\mathbb{R}^{3}\,\,|\,\,\sigma_{k}x+z_{k}\in\overline{Q}\}$. Hence, we obtain that

[TABLE]

Now, we prove that there is a small $\tau_{0}\in(0,\mathcal{S}_{s}^{\frac{3}{2s}})$ such that $\sigma_{k}(\tau_{0})\rightarrow 0$ as $k\rightarrow\infty$. Otherwise, for any $\varepsilon>0$, there exists $r_{\varepsilon}>0$ such that $\sigma_{k}(\varepsilon)>r_{\varepsilon}$. Hence, for any $x\in\bar{\Omega}$, there holds

[TABLE]

Furthermore,

[TABLE]

Let $k\rightarrow+\infty$ and then $\varepsilon\rightarrow 0$, we get $\nu_{j_{0}}\leq 0$, which achieves a contradiction. For the above $\tau_{0}$, we still denote $\sigma_{k}:=\sigma_{k}(\tau_{0})$ and the corresponding sequence $z_{k}\in\overline{Q}$. Thus $v_{k}(x)=\sigma_{k}^{\frac{3-2s}{2}}u_{k}(\sigma_{k}x+z_{k})$ satisfies

[TABLE]

Note that

[TABLE]

by the boundedness of $\{u_{k}\}$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$, up to a subsequence, we may assume that there exists $v\in\mathcal{D}^{s,2}(\mathbb{R}^{3})$ such that $v_{k}\rightharpoonup v$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$.

For any $\phi\in C_{0}^{\infty}(\mathbb{R}^{3})$, denote $\phi_{k}(x)=\phi((x-z_{k})/\sigma_{k})$. By the fact that $z_{k}\in\overline{Q}$ and $\sigma_{k}\rightarrow 0$, we see that for $k$ large enough, ${\rm supp}\phi_{k}\subset B_{\sigma_{k}}(z_{k})\subset V$, then $\phi_{k}\in C_{0}^{\infty}(V)$. From (2.5), we have that

[TABLE]

Thus, $v$ is a solution of equation (2.6). Next, we will prove that $v$ is nontrivial. By virtue of (2.7), we only need to show that

[TABLE]

If (2.8) holds true, from (2.7), we know that

[TABLE]

which implies that $v$ is nontrivial.

By the boundedness of $\{v_{k}\}$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$ and $v_{k}\rightharpoonup v$ in $\mathcal{D}^{s,2}(\mathbb{R}^{3})$, by Phrokorov s theorem (Theorem 8.6.2 in [4]), there exist $\mu,\nu\in\mathcal{M}(\mathbb{R}^{3})$ such that

[TABLE]

By Lemma 2.2, there exist an at most countable index set $J$, sequence $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$ and $\{\nu_{j}\}\subset(0,\infty)$ such that

[TABLE]

and

[TABLE]

Next, we show that $\{x_{j}\}_{j\in J}\cap\overline{B_{1}(0)}=\emptyset$. Suppose by contradiction that there exists $j_{0}\in J$ such that $x_{j_{0}}\in\overline{B_{1}(0)}$, and define the function $\phi_{\rho}=:\phi(\frac{x-x_{j_{0}}}{\rho})$, where $\phi$ is a smooth cut-off function such that $\phi=1$ on $B_{1}(0)$, $\phi=0$ on $\mathbb{R}^{3}\backslash B_{2}(0)$, $0\leq\phi\leq 1$ and $|\nabla\phi|\leq C$. Denote $\phi_{k,\rho}(x)=\phi_{\rho}(\frac{x-z_{k}}{\sigma_{k}})$, by the fact that $z_{k}\in\overline{Q}$, $x_{j_{0}}\in\overline{B_{1}(0)}$ and $\sigma_{k}\rightarrow 0$ as $k\rightarrow\infty$, we see that for $k$ large, ${\rm supp}\phi_{k,\rho}\subset B_{2\sigma_{k}\rho}(z_{k}+\sigma_{k}x_{j_{0}})\subset V$. Direct computation, it can be checked that $\phi_{k,\rho}u_{k}\in H^{s}(\mathbb{R}^{3})$. Indeed, by Hölder’s inequality, we have that

[TABLE]

and directly computations, we get

[TABLE]

which implies that $\phi_{\rho,k}u_{k}\in H^{s}(\mathbb{R}^{3})$.

[TABLE]

Since

[TABLE]

and

[TABLE]

then we claim that

[TABLE]

Indeed, since

[TABLE]

where $B_{\rho}^{c}(x_{j})=\mathbb{R}^{3}\backslash B_{\rho}(x_{j})$ and $B_{2\rho}^{c}(x_{j})=\mathbb{R}^{3}\backslash B_{2\rho}(x_{j})$. Next we will discuss the six cases on the above domains, respectively.

$\bullet$ $(y,z)\in B_{2\rho}^{c}(x_{j})\times B_{2\rho}^{c}(x_{j})$. Clearly $|\phi_{\rho}(z)-\phi_{\rho}(y)|=0$ and so

[TABLE]

$\bullet$ $(y,z)\in B_{2\rho}(x_{j})\times B_{2\rho}(x_{j})$. Since $|\phi_{\rho}(z)-\phi_{\rho}(y)|\leq\frac{C}{\rho}|z-y|$ and $|y-z|\leq|y-x_{j}|+|z-x_{j}|\leq 4\rho$, we have

[TABLE]

$\bullet$ $(y,z)\in B_{\rho}(x_{j})\times B_{2\rho}^{c}(x_{j})$. There holds $|z-y|\geq|z-x_{j}|-|y-x_{j}|\geq\rho$ and thus

[TABLE]

$\bullet$ $(y,z)\in B_{2\rho}^{c}(x_{j})\times B_{\rho}(x_{j})$. Obviously, $|y-z|\geq\rho$. Observe that for any fixed $K\geq 4$, $B_{2\rho}^{c}(x_{j})\times B_{\rho}(x_{j})\subset B_{K\rho}(x_{j})\times B_{\rho}(x_{j})\cup B_{K\rho}^{c}(x_{j})\times B_{\rho}(x_{j})$. Hence, if $|y-z|>\rho$ and $(y,z)\in B_{K\rho}(x_{j})\times B_{\rho}(x_{j})$, we have

[TABLE]

If $(y,z)\in B_{K\rho}^{c}(x_{j})\times B_{\rho}(x_{j})$, $|y-z|\geq|y-x_{j}|-|z-x_{j}|\geq\frac{3|y-x_{j}|}{4}+\frac{K}{4}\rho-\rho>\frac{3|y-x_{j}|}{4}$. By Hölder’s inequality, we have

[TABLE]

$\bullet$ $(y,z)\in B_{2\rho}^{c}(x_{j})\times B_{2\rho}(x_{j})\backslash B_{\rho}(x_{j})$. If $|y-z|\leq\rho$, then $|y-x_{j}|\leq|y-z|+|z-x_{j}|\leq 3\rho$ and thus

[TABLE]

Observe that for any fixed $K\geq 4$, $B_{2\rho}^{c}(x_{j})\times B_{2\rho}(x_{j})\backslash B_{\rho}(x_{j})A\subset B_{K\rho}(x_{j})\times B_{2\rho}(x_{j})\cup B_{K\rho}^{c}(x_{j})\times B_{2\rho}(x_{j})$. Hence, if $|y-z|>\rho$ and $(y,z)\in B_{K\rho}(x_{j})\times B_{2\rho}(x_{j})$, we have

[TABLE]

If $(y,z)\in B_{K\rho}^{c}(x_{j})\times B_{2\rho}(x_{j})$, $|y-z|\geq|y-x_{j}|-|z-x_{j}|\geq\frac{|y-x_{j}|}{2}+\frac{K}{2}\rho-2\rho\geq\frac{|y-x_{j}|}{2}$. By Hölder’s inequality, we have

[TABLE]

$\bullet$ $(y,z)\in B_{2\rho}(x_{j})\backslash B_{\rho}(x_{j})\times B_{2\rho}^{c}(x_{j})$. If $|y-z|\leq\rho$, we have

[TABLE]

If $|y-z|>\rho$, then $|y-z|\geq\frac{|y-x_{j}|}{2}$. One has

[TABLE]

From all the above estimates and using Hölder’s inequality, we get that

[TABLE]

Letting $\rho\rightarrow 0^{+}$ and then letting $K\rightarrow+\infty$, (2.12) follows. Thus, from (2.2), (2.9) and (2.10), we get that

[TABLE]

which leads to

[TABLE]

But, by (2.7), we have that

[TABLE]

which contradicts with $\tau_{0}<\mathcal{S}_{s}^{\frac{3}{2s}}$. Hence, $\{x_{j}\}_{j\in J}\cap\overline{B_{1}(0)}=\emptyset$ and then (2.8) holds. We complete the proof. ∎

Proposition 2.5**.**

([56], Proposition 5.1) Assume that $u_{n}$ are nonnegative weak solution of

[TABLE]

where $\{V_{n}\}$ satisfies $V_{n}(x)\geq V_{0}>0$ for all $x\in\mathbb{R}^{3}$ and $f_{n}(x,\tau)$ is a Carathedory function satisfying that for any $\delta>0$, there exists $C_{\varepsilon}>0$ such that

[TABLE]

satisfying $u_{n}$ convergence strongly in $H^{s}(\mathbb{R}^{3})$ or $u_{n}$ convergence strongly in $L^{2_{s}^{\ast}}(\mathbb{R}^{3})$. Then there exists $C>0$ such that

[TABLE]

Lemma 2.6**.**

*([48], Proposition 2.9) Let $w=(-\Delta)^{s}u$. Assume $w\in L^{\infty}(\mathbb{R}^{n})$ and $u\in L^{\infty}(\mathbb{R}^{n})$ for $s>0$.

If $2s\leq 1$, then $u\in C^{0,\alpha}(\mathbb{R}^{n})$ for any $\alpha\leq 2s$. Moreover*

[TABLE]

*for some constant $C$ depending only on $n$, $\alpha$ and $s$.

If $2s>1$, then $u\in C^{1,\alpha}(\mathbb{R}^{n})$ for any $\alpha<2s-1$. Moreover*

[TABLE]

for some constant $C$ depending only on $n$, $\alpha$ and $s$.

3. Limiting problem

In this section, we consider the ”limiting problem” associated with problem (2.2)

[TABLE]

for $\mu>0$. The energy functional for the limiting problem (3.1) is given by

[TABLE]

Let

[TABLE]

and

[TABLE]

We define the Nehari-Pohozaev manifold

[TABLE]

and set $b_{\mu}=\inf\limits_{u\in\mathcal{M}_{\mu}}\mathcal{I}_{\mu}(u)$. By standard arguments, we can show the following properties of $\mathcal{M}_{\mu}$.

Proposition 3.1**.**

*The set $\mathcal{M}_{\mu}$ possesses the following properties:

$(i)$ $0\not\in\partial\mathcal{M}_{\mu}$;

$(ii)$ for any $u\in H^{s}(\mathbb{R}^{3})\backslash\{0\}$, there exists a unique $\tau_{0}:=\tau(u)>0$ such that $u_{\tau_{0}}\in\mathcal{M}_{\mu}$, where $u_{\tau}=\tau^{s+t}u(\tau x)$. Moreover,*

[TABLE]

Now, it is easy to check that $\mathcal{I}_{\mu}$ satisfies the mountain pass geometry.

Lemma 3.2**.**

$(i)$* there exist $\rho_{0},\beta_{0}>0$ such that $\mathcal{I}_{\mu}(u)\geq\beta_{0}$ for all $u\in H^{s}(\mathbb{R}^{3})$ with $\|u\|=\rho_{0}$;

$(ii)$ there exists $u_{0}\in H^{s}(\mathbb{R}^{3})$ such that $\mathcal{I}_{\mu}(u_{0})<0$.*

From Lemma 3.2, the mountain-pass level of $\mathcal{I}_{\mu}$ defined as follows

[TABLE]

where

[TABLE]

satisfies that $c_{\mu}>0$. Furthermore, by $(f_{3})$, it is easy to verify that

[TABLE]

By using Lemma 3.2 and (3.2), we can show the equivalent characterization of mountain-pass level $c_{\mu}$.

Lemma 3.3**.**

[TABLE]

In order to obtain the boundedness of $(PS)$ sequence, we will construct a $(PS)$ sequence $\{u_{n}\}$ for $\mathcal{I}_{\mu}$ at the level $c_{\mu}$ that satisfies $\mathcal{G}_{\mu}(u_{n})\rightarrow 0$ as $n\rightarrow+\infty$ i.e.,

Lemma 3.4**.**

There exists a sequence $\{u_{n}\}$ in $H^{s}(\mathbb{R}^{3})$ such that as $n\rightarrow+\infty$,

[TABLE]

Lemma 3.5**.**

Every sequence $\{u_{n}\}\subset H^{s}(\mathbb{R}^{3})$ satisfying (3.3) is bounded in $H^{s}(\mathbb{R}^{3})$.

Proof.

By (3.3), (3.2) and $q>\frac{4s+2t}{s+t}$, we have that

[TABLE]

Hence, sequence $\{u_{n}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$. ∎

For obtaining the compactness of the above bounded sequence $\{u_{n}\}$, we need the estimate of the Mountain-Pass level $c_{\mu}$ which is given as the following Lemma.

Lemma 3.6**.**

[TABLE]

if in the case $s>\frac{3}{4}$, $q\in(2_{s}^{\ast}-2,2_{s}^{\ast})$ for all $\lambda>0$ or $q\in(\frac{4s+2t}{s+t},2_{s}^{\ast}-2]$ for $\lambda>0$ large; if in the case $\frac{1}{2}<s\leq\frac{3}{4}$, $q\in(\frac{4s+2t}{s+t},2_{s}^{\ast})$ for any $\lambda>0$, where $\mathcal{S}_{s}$ is the best Sobolev consatnt for the embedding $\mathcal{D}^{s,2}(\mathbb{R}^{3})\hookrightarrow L^{2_{s}^{\ast}}(\mathbb{R}^{3})$.

Proof.

Let

[TABLE]

where $U_{\delta}(x)=\delta^{-\frac{3-2s}{2}}u^{\ast}(x/\delta)$, $u^{\ast}(x)=\frac{\widetilde{u}(x/\mathcal{S}_{s}^{\frac{1}{2s}})}{\|\widetilde{u}\|_{2_{s}^{\ast}}}$, $\kappa\in\mathbb{R}\backslash\{0\}$, $\mu>0$ and $x_{0}\in\mathbb{R}^{3}$ are fixed constants, $\widetilde{u}(x)=\kappa(\mu^{2}+|x-x_{0}|^{2})^{-\frac{3-2s}{2}}$, and $\psi\in C^{\infty}(\mathbb{R}^{3})$ such that $0\leq\psi\leq 1$ in $\mathbb{R}^{3}$, $\psi(x)\equiv 1$ in $B_{R}$ and $\psi\equiv 0$ in $\mathbb{R}^{3}\backslash B_{2R}$. From Proposition 21 and Proposition 22 in [50], Lemma 3.3 in [53], we know that

[TABLE]

[TABLE]

and

[TABLE]

Here $a_{\delta}=O(b_{\delta})$ means that $C_{1}\leq\frac{a_{\delta}}{b_{\delta}}\leq C_{2}$ for some $C_{1},C_{2}>0$, independent of $\delta$.

Set $u_{\delta}^{\tau}(x)=\tau^{s+t}u_{\delta}(\tau x)$ for any $\tau\geq 0$, by $(f_{2})$, we deduce that

[TABLE]

Since $h_{\delta}(\tau)\rightarrow-\infty$ as $\tau\rightarrow+\infty$, we have that $\sup\{h_{\delta}(\tau):\,\,\tau\geq 0\}=h_{\delta}(\tau_{\delta})$ for some $\tau_{\delta}>0$. Hence, $\tau_{\delta}$ verifies the following equality:

[TABLE]

We claim that $\{\tau_{\delta}\}$ is bounded from below by a positive constant for $\delta$ small. Otherwise, there exists a sequence $\delta_{n}\rightarrow 0$ such that $\tau_{\delta_{n}}\rightarrow 0$ as $n\rightarrow+\infty$. Thus $0<c_{\mu}\leq\sup_{\tau\geq 0}\mathcal{I}_{\mu}(u_{\delta_{n}}^{\tau})\leq\sup_{\tau\geq 0}h_{\delta_{n}}(\tau)=h_{\delta_{n}}(\tau_{\delta_{n}})\rightarrow 0$ as $n\rightarrow\infty$, a contradiction. So there exists a constant $C_{0}>0$ independent of $\delta$ such that $\tau_{\delta}\geq C_{0}$. Using the similar argument in (3), we can show that the sequence $\{\tau_{\delta}\}$ is bounded from above by a constant $C$ independent of $\delta$. Thus $0<C_{0}\leq\tau_{\delta}\leq C$ for $\delta$ small.

Let $g_{\delta}(\tau)=\frac{\tau^{4s+2t-3}}{2}\int_{\mathbb{R}^{3}}|D_{s}u_{\delta}|^{2}\,{\rm d}x-\frac{\tau^{2_{s}^{\ast}(s+t)-3}}{2_{s}^{\ast}}\int_{\mathbb{R}^{3}}|u_{\delta}|^{2_{s}^{\ast}}\,{\rm d}x$, then we get for some universal constant $C>0$ so that

[TABLE]

Directly computation, we get that $\frac{4s+2t-3}{2}\frac{2_{s}^{\ast}}{2_{s}^{\ast}(s+t)-3}=1$ and $\frac{2_{s}^{\ast}(s+t)-3}{(2_{s}^{\ast}-4)s+(2_{s}^{\ast}-2)t}=\frac{3}{2s}$. Thus, by (3.4), we deduce that

[TABLE]

where

[TABLE]

By $(i)$ of Lemma 2.1, (3.4) and (3), using the elementary inequality $(a+b)^{\alpha}\leq a^{\alpha}+\alpha(a+b)^{\alpha-1}b$, $\alpha\geq 1$ and $a,b\geq 0$, we deduce that

[TABLE]

$\bullet$ In the case $s>\frac{3}{4}$, by (3.4), we deduce that

[TABLE]

In view of (3.4), we have that

[TABLE]

and noting that $2s-\frac{3-2s}{2}q<0$ if $\frac{4s}{3-2s}<q<\frac{6}{3-2s}$, we have

[TABLE]

We can choosing $\lambda$ large enough such that the above three limit equal to $+\infty$, for instance, $\lambda=\delta^{-2s}$.

$\bullet$ If $s=\frac{3}{4}$, it follows from (3.4) that

[TABLE]

Since $\frac{12}{3+2t}>2=\frac{3}{3-2s}$, we get that

[TABLE]

and owing to $\frac{3}{3-2s}=2<\frac{4s+2t}{s+t}<q$, then for any $\lambda>0$, we obtain that

[TABLE]

$\bullet$ In the case $\frac{1}{2}<s<\frac{3}{4}$, by means of (3.4), we get

[TABLE]

Observing that $\frac{3}{3-2s}\in(\frac{3}{2},2)$, thus $\frac{12}{3+2t}>\frac{3}{3-2s}$ and $\frac{3}{3-2s}<\frac{4s+2t}{s+t}<q<\frac{6}{3-2s}$. Hence

[TABLE]

and for any $\lambda>0$, we have

[TABLE]

From the above arguments, we conclude the proof. ∎

From the estimate of mountain pass level, using the Vanishing Lemma, it is not difficult to deduce that the bounded sequence $\{u_{n}\}\subset H^{s}(\mathbb{R}^{3})$ given in (3.3) is non-vanishing. That is,

Lemma 3.7**.**

There exists a sequence $\{x_{n}\}\subset\mathbb{R}^{3}$ and $R>0$, $\beta>0$ such that $\int_{B_{R}(x_{n})}|u_{n}|^{2}\,{\rm d}x\geq\beta$.

Combining with Lemma 3.4 and Lemma 3.7, we can show the existence of positive ground state solution for the limiting problem (3.1).

Proposition 3.8**.**

Problem (3.1) possesses a positive ground state solution $u\in H^{s}(\mathbb{R}^{3})$.

Proof.

Let $\{u_{n}\}$ be the sequence given in (3.3). Set $\widehat{u}_{n}(x)=u_{n}(x+x_{n})$, where $\{x_{n}\}$ is the sequence obtained in Lemma 3.7. Thus $\{\widehat{u}_{n}\}$ is still bounded in $H^{s}(\mathbb{R}^{3})$ and so up to a subsequence, still denoted by $\{\widehat{u}_{n}\}$, we may assume that there exists $\widehat{u}\in H^{s}(\mathbb{R}^{3})$ such that

[TABLE]

It follows from Lemma 3.7 that $\widehat{u}$ is nontrivial. Moreover, using $(iv)$ of Lemma 2.1, it is not difficult to verify that $\widehat{u}$ is a nontrivial solution of problem (3.1), and since $f\in C^{1}(\mathbb{R}^{3})$, standard arguments lead to $\mathcal{G}_{\mu}(\widehat{u})=0$. By Fatou’s Lemma and (3.3), we have

[TABLE]

which implies that $\widehat{u}_{n}\rightarrow\widehat{u}$ in $H^{s}(\mathbb{R}^{3})$. Indeed, from the above inequality, we get that

[TABLE]

By virtue of the Brezis-Lieb Lemma and interpolation argument, we conclude that

[TABLE]

Hence, from the standard arguments, it follows that $\widehat{u}_{n}\rightarrow\widehat{u}$ in $H^{s}(\mathbb{R}^{3})$. Therefore, by Lemma 3.3, we conclude that $\mathcal{I}_{\mu}(\widehat{u})=c_{\mu}$ and $\mathcal{I}^{\prime}_{\mu}(\widehat{u})=0$.

Next, we show that the ground state solution of (3.1) is positive. Indeed, by standard argument to the proof Proposition 4.4 in [55], using Lemma 2.6 two times, we have that $\widehat{u}\in C^{2,\alpha}(\mathbb{R}^{3})$ for some $\alpha\in(0,1)$ for $s>\frac{1}{2}$. Using $-\widehat{u}^{-}$ as a testing function, it is easy to see that $\widehat{u}\geq 0$. Since $\widehat{u}\in C^{2,\alpha}(\mathbb{R}^{3})$, by Lemma 3.2 in [42], we have that

[TABLE]

Assume that there exists $x_{0}\in\mathbb{R}^{3}$ such that $\widehat{u}(x_{0})=0$, then from $\widehat{u}\geq 0$ and $\widehat{u}\not\equiv 0$, we get

[TABLE]

However, observe that $(-\Delta)^{s}\widehat{u}(x_{0})=-\mu\widehat{u}(x_{0})-(\phi_{\widehat{u}}^{t}\widehat{u})(x_{0})+f(\widehat{u}(x_{0}))+\widehat{u}(x_{0})^{2_{s}^{\ast}-1}=0$, a contradiction. Hence, $\widehat{u}(x)>0$, for every $x\in\mathbb{R}^{3}$. The proof is completed.

∎

Let $\mathcal{L}_{\mu}$ be the set of ground state solutions $W$ of (3.1) satisfying $W(0)=\max\limits_{\mathbb{R}^{3}}W(x)$. By similar proof of Proposition 3.8 in [57], we can establish the following compactness of $\mathcal{L}_{\mu}$.

Proposition 3.9**.**

$(i)$* For each $\mu>0$, $\mathcal{L}_{\mu}$ is compact in $H^{s}(\mathbb{R}^{3})$.

$(ii)$ $0<W(x)\leq\frac{C}{1+|x|^{3+2s}}$, for any $x\in\mathbb{R}^{3}$.*

4. The penalization scheme

For the bounded domain $\Lambda$ given in $(V_{1})$, $k>2$, $a>0$ such that $f(a)+a^{2_{s}^{\ast}-1}=\frac{V_{0}}{k}a$ where $\alpha_{0}$ is mentioned in $(V_{0})$, we consider a new problem

[TABLE]

where $g(\varepsilon z,\tau)=\chi_{\Lambda_{\varepsilon}}(\varepsilon z)(f(\tau)+(\tau^{+})^{2_{s}^{\ast}-1})+(1-\chi_{\Lambda_{\varepsilon}}(\varepsilon z))\tilde{f}(\tau)$ with

[TABLE]

and $\chi_{\Lambda_{\varepsilon}}(\varepsilon z)=1$ if $z\in\Lambda_{\varepsilon}$, $\chi(z)=0$ if $z\not\in\Lambda_{\varepsilon}$, where $\Lambda_{\varepsilon}=\Lambda/\varepsilon$. It is easy to see that under the assumptions $(f_{1})$-$(f_{3})$, $g(z,\tau)$ is a Caratheodory function and satisfies the following assumptions:

$(g_{1})$ $g(z,\tau)=o(\tau^{3})$ as $\tau\rightarrow 0$ uniformly on $z\in\mathbb{R}^{3}$;

$(g_{2})$ $g(z,\tau)\leq f(\tau)+\tau^{2_{s}^{\ast}-1}$ for all $\tau\in\mathbb{R}^{+}$ and $z\in\mathbb{R}^{3}$, $g(z,\tau)=0$ for all $z\in\mathbb{R}^{3}$ and $\tau<0$, $g(z,\tau)=f(\tau)+(\tau^{+})^{2_{s}^{\ast}-1}$ for $z\in\mathbb{R}^{3}$, $\tau\in[0,a]$;

$(g_{3})$ $0<2\tilde{F}(\tau)\leq\tilde{f}(\tau)\tau\leq\frac{V_{0}}{k}\tau^{2}\leq\frac{V(x)}{k}\tau^{2}$ for all $s\geq 0$ with the number $k>2$, where $\tilde{F}(\tau)$ is a prime function of $\tilde{f}$;

$(g_{4})$ $0<qG(z,\tau)\leq g(z,\tau)\tau$ for all $z\in\Lambda$, $\tau>0$ or $z\in\mathbb{R}^{3}\backslash\Lambda$, $\tau\leq a$, where $G(z,\tau)$ is a prime function of $g(z,\tau)$;

$(g_{5})$ $\frac{g(z,s\tau)}{s}$ is nondecreasing in $\tau\in\mathbb{R}^{+}$ uniformly for $z\in\mathbb{R}^{3}$, $\frac{g(z,s\tau)}{\tau^{q}}$ is nondecreasing in $\tau\in\mathbb{R}^{+}$ and $z\in\Lambda$, $\frac{g(z,s\tau)}{\tau^{q}}$ is nondecreasing in $\tau\in(0,a)$ and $z\in\mathbb{R}^{3}\backslash\Lambda$.

Obviously, if $u_{\varepsilon}$ is a solution of (4.1) satisfying $u_{\varepsilon}(z)\leq a$ for $z\in\mathbb{R}^{3}$, then $u_{\varepsilon}$ is indeed a solution of the original problem (2.3).

For $u\in H_{\varepsilon}$, let

[TABLE]

and

[TABLE]

Let us define the functional $\mathcal{J}_{\varepsilon}:H_{\varepsilon}\rightarrow\mathbb{R}$ as follows

[TABLE]

Clearly, $\mathcal{J}_{\varepsilon}\in C^{1}(H_{\varepsilon},\mathbb{R})$. To find solutions of (4.1) which concentrates in $\Lambda$ as $\varepsilon\rightarrow 0$, we shall search critical points of $\mathcal{J}_{\varepsilon}$ such that $Q_{\varepsilon}$ is zero.

Set

[TABLE]

Fix a cut-off function $\varphi\in C_{0}^{\infty}(\mathbb{R}^{3})$ such that $0\leq\varphi\leq 1$, $\varphi=1$ for $|z|\leq\beta$, $\varphi=0$ for $|z|\geq 2\beta$ and $|\nabla\varphi|\leq C/\beta$. Set $\varphi_{\varepsilon}(z)=\varphi(\varepsilon z)$, for any $W\in\mathcal{L}_{V_{0}}$ and any point $y\in\mathcal{M}^{\beta}=\{y\in\mathbb{R}^{3}\,\,|\,\,\inf\limits_{z\in\mathcal{M}}|y-z|\leq\beta\}$, we define

[TABLE]

For $A\subset H_{\varepsilon}$, we use the notation

[TABLE]

We want to find a solution near the set

[TABLE]

for $\varepsilon>0$ sufficiently small.

Similar arguments as the proof of Lemma 4.1, Lemma 4.2 and Lemma 4.3 in [57], we can show that

• •

$\mathcal{N}_{\varepsilon}$ is uniformly bounded in $H_{\varepsilon}$ and it is compact in $H_{\varepsilon}$ for any $\varepsilon>0$;

• •

[TABLE]

• •

[TABLE]

where

[TABLE]

$\mathcal{A}_{\varepsilon}=\{\gamma\in C([0,1],H_{\varepsilon})\,\,|\,\,\gamma(0)=0,\,\,\gamma(1)=U_{\varepsilon,\tau_{0}}\}$, $\gamma_{\varepsilon}(\tau)=W_{\varepsilon,\tau\tau_{0}}$ for $\tau\in[0,1]$ and $c_{V_{0}}=\mathcal{I}_{V_{0}}(W^{\ast})$ for $W^{\ast}\in\mathcal{L}_{V_{0}}$. Moreover, $\mathcal{J}_{\varepsilon}(U_{\varepsilon,\tau})$ possesses the mountain-pass geometry.

Lemma 4.1**.**

There exists a small $d_{0}>0$ such that for any $\{\varepsilon_{i}\}$, $\{u_{\varepsilon_{i}}\}$ satisfying $\lim\limits_{i\rightarrow\infty}\varepsilon_{i}\rightarrow 0$, $u_{\varepsilon_{i}}\in\mathcal{N}_{\varepsilon_{i}}^{d_{0}}$ and

[TABLE]

there exist, up to a subsequence, $\{x_{i}\}\subset\mathbb{R}^{3}$, $x_{0}\in\mathcal{M}$, $W\in\mathcal{L}_{V_{0}}$ such that

[TABLE]

Proof.

In the proof we will drop the index $i$ and write $\varepsilon$ instead of $\varepsilon_{i}$ for simplicity, and we still use $\varepsilon$ after taking a subsequence. By the definition of $\mathcal{N}_{\varepsilon}^{d_{0}}$, there exist $\{W_{\varepsilon}\}\subset\mathcal{L}_{V_{0}}$ and $\{x_{\varepsilon}\}\subset\mathcal{M}^{\beta}$ such that

[TABLE]

Since $\mathcal{L}_{V_{0}}$ and $\mathcal{M}^{\beta}$ are compact, there exist $W_{0}\in\mathcal{L}_{V_{0}}$, $x_{0}\in\mathcal{M}^{\beta}$ such that $W_{\varepsilon}\rightarrow W_{0}$ in $H^{s}(\mathbb{R}^{3})$ and $x_{\varepsilon}\rightarrow x_{0}$ as $\varepsilon\rightarrow 0$. Thus, for $\varepsilon>0$ small,

[TABLE]

Step 1. We claim that

[TABLE]

where $A_{\varepsilon}=B_{3\beta/\varepsilon}(x_{\varepsilon}/\varepsilon)\backslash B_{\beta/2\varepsilon}(x_{\varepsilon}/\varepsilon)$. Suppose by contradiction that there exists $r>0$ such that

[TABLE]

Thus, there exists $y_{\varepsilon}\in A_{\varepsilon}$ such that $\int_{B_{1}(y_{\varepsilon})}|u_{\varepsilon}|^{2_{s}^{\ast}}\,{\rm d}z\geq r>0$ for $\varepsilon>0$ small. Since $y_{\varepsilon}\in A_{\varepsilon}$, there exists $y^{\ast}\in\mathcal{M}^{4\beta}\subset\Lambda$ such that $\varepsilon y_{\varepsilon}\rightarrow y^{\ast}$ as $\varepsilon\rightarrow 0$. Set $v_{\varepsilon}(z)=u_{\varepsilon}(z+y_{\varepsilon})$, then for $\varepsilon>0$ small,

[TABLE]

Thus, up to a subsequence, we may assume that there exists $v\in H^{s}(\mathbb{R}^{3})$ such that $v_{\varepsilon}\rightharpoonup v$ in $H^{s}(\mathbb{R}^{3})$, $v_{\varepsilon}\rightarrow v$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$ and $v_{\varepsilon}\rightarrow v$ a.e. in $\mathbb{R}^{3}$. It is easy to check that $v$ satisfies

[TABLE]

Indeed, by the definition of weakly convergence, we have

[TABLE]

for any $\varphi\in C_{0}^{\infty}(\mathbb{R}^{3})$. Now given $\varphi\in C_{0}^{\infty}(\mathbb{R}^{3})$, we have $\|\varphi(\cdot-y_{\varepsilon})\|_{H_{\varepsilon}}\leq C$ and so $\langle\mathcal{J}_{\varepsilon}^{\prime}(u_{\varepsilon}),\varphi(\cdot-y_{\varepsilon})\rangle\rightarrow 0$ as $\varepsilon\rightarrow 0$. Using the fact that $v_{\varepsilon}\rightarrow v$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$, the Lebesgue dominated convergence Theorem, the boundedness of ${\rm supp}(\varphi)$ and $(g_{0})$–$(g_{1})$, it follows that

[TABLE]

[TABLE]

and

[TABLE]

for any $\varphi\in C_{0}^{\infty}(\mathbb{R}^{3})$. Therefore, we get that

[TABLE]

for any $\varphi\in C_{0}^{\infty}(\mathbb{R}^{3})$. Since $\varphi$ is arbitrary and $C_{0}^{\infty}(\mathbb{R}^{3})$ is dense in $H_{\varepsilon}$, it follows that $v$ satisfies (4.6).

Case 1. If $v\neq 0$, then

[TABLE]

Hence, for sufficiently large $R>0$, by Fatou’s Lemma, we have that

[TABLE]

On the other hand, by the Sobolev embedding theorem and (4.3), one has

[TABLE]

Observing that $y_{\varepsilon}\in A_{\varepsilon}$, implies that $|y_{\varepsilon}-\frac{x_{\varepsilon}}{\varepsilon}|\geq\frac{\beta}{2\varepsilon}$, then for $\varepsilon>0$ small enough, there hold

[TABLE]

where $o(1)\rightarrow 0$ as $\varepsilon\rightarrow 0$. Thus, we have proved that

[TABLE]

This leads to a contradiction if $d_{0}$ is small enough.

Case 2. If $v=0$, i.e., $v_{\varepsilon}\rightharpoonup 0$ in $H^{s}(\mathbb{R}^{3})$, $v_{\varepsilon}\rightarrow 0$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$ and $v_{\varepsilon}\rightarrow 0$ a.e. in $\mathbb{R}^{3}$. Now we claim that

[TABLE]

where \rho_{\varepsilon}=-(-\Delta)^{s}v_{\varepsilon}+(v_{\varepsilon}^{+})^{2_{s}^{\ast}-1}\in\Big{(}H^{s}(\mathbb{R}^{3})\Big{)}^{\prime}. It is easy to check that for $\varepsilon>0$ small, $\int_{\mathbb{R}^{3}\backslash\Lambda_{\varepsilon}}u_{\varepsilon}\varphi(z-y_{\varepsilon})\,{\rm d}z=0$ uniformly for any $\varphi\in C_{0}^{\infty}(B_{2}(0))$. Thus, for any $\varphi\in C_{0}^{\infty}(B_{2}(0))$ with $\|\varphi\|=1$, we deduce that

[TABLE]

By the facts that $\lim\limits_{\varepsilon\rightarrow 0}\mathcal{J}^{\prime}_{\varepsilon}(u_{\varepsilon})=0$, ${\rm supp}\varphi\subset B_{2}$, $\sup\limits_{z\in B_{2}(0)}V(\varepsilon z+\varepsilon y_{\varepsilon})\leq C$ uniformly for all $\varepsilon>0$ small, $v_{\varepsilon}\rightarrow 0$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$, $\frac{12}{3+2t}<2_{s}^{\ast}$, and using Hölder’s inequality, we deduce that

[TABLE]

[TABLE]

[TABLE]

uniformly for $\varphi\in C_{0}^{\infty}(B_{2}(0))$ with $\|\varphi\|=1$. By $(f_{0})$ and $(f_{1})$, for any $\eta>0$, there exists $C_{\eta}>0$ such that

[TABLE]

and

[TABLE]

uniformly for $z\in B_{2}(0)$ and for $\varepsilon$ small enough and then, we have

[TABLE]

for $\varepsilon$ sufficiently small. Letting $\varepsilon\rightarrow 0$ and then $\eta\rightarrow 0$ in the above inequality, we see that $A_{4}\rightarrow 0$ as $\varepsilon\rightarrow 0$ uniformly for $\varphi\in C_{0}^{\infty}(B_{2}(0))$ with $\|\varphi\|=1$. Hence (4.7) holds.

By Proposition 2.4, taking $Q=B_{1}(0)$ and $V=B_{2}(0)$, from (4.5) and (4.7), it follows that there exists $\widetilde{z}_{\varepsilon}\in\mathbb{R}^{3}$ and $\sigma_{\varepsilon}>0$ with $\widetilde{z}_{\varepsilon}\rightarrow\widetilde{z}\in\overline{B_{1}(0)}$, $\sigma_{\varepsilon}\rightarrow 0$ as $\varepsilon\rightarrow 0$, such that

[TABLE]

and $\widetilde{v}\geq 0$ is a nontrivial solution of

[TABLE]

By Theorem 1.1 in [11] and Claim 6 in [50], we see that

[TABLE]

for some $\kappa>0$, $\mu>0$, $x_{0}\in\mathbb{R}^{3}$, and

[TABLE]

Thus, there exists $R>0$ such that

[TABLE]

On the other hand, using the facts that $\sigma_{\varepsilon}\rightarrow 0$ and $\widetilde{z}_{\varepsilon}\rightarrow\widetilde{z}\in\overline{B_{1}(0)}$ (imply that $B_{\sigma_{\varepsilon}R}(\widetilde{z}_{\varepsilon}+z_{\varepsilon})\subset B_{2}(z_{\varepsilon})$ for $\varepsilon$ small), we have that

[TABLE]

But, by the Sobolev imbedding Theorem and (4.3), we get

[TABLE]

since $|y_{\varepsilon}-\frac{x_{\varepsilon}}{\varepsilon}|\geq\frac{\beta}{2\varepsilon}$. This leads to a contradiction with (4) for $d_{0}>0$ small. Hence the claim (4.4) holds.

From (4.4), by Proposition 2.3, we conclude that

[TABLE]

where $A_{\varepsilon}^{1}=B_{2\beta/\varepsilon}(\frac{x_{\varepsilon}}{\varepsilon})\backslash B_{\beta/\varepsilon}(\frac{x_{\varepsilon}}{\varepsilon})$. Indeed, taking a smooth cut-off function $\psi_{\varepsilon}\in C_{0}^{\infty}(\mathbb{R}^{3})$ such that $\psi_{\varepsilon}=1$ on $B_{2\beta/\varepsilon}(\frac{x_{\varepsilon}}{\varepsilon})\backslash B_{\beta/\varepsilon}(\frac{x_{\varepsilon}}{\varepsilon})$, $\psi_{\varepsilon}=0$ on $A_{\varepsilon}^{2}=B_{3\beta/\varepsilon-1}(\frac{x_{\varepsilon}}{\varepsilon})\backslash B_{\beta/2\varepsilon+1}(\frac{x_{\varepsilon}}{\varepsilon})$. Since $u_{\varepsilon}\in H_{\varepsilon}$ and using $(V_{0})$, it is easy to check that $u_{\varepsilon}\psi_{\varepsilon}\in H^{s}(\mathbb{R}^{3})$. Moreover,

[TABLE]

By Proposition 2.3, we have

[TABLE]

Since $A_{\varepsilon}^{1}\subset A_{\varepsilon}^{2}$ for $\varepsilon>0$ small, so (4.11) holds. Therefore, by the interpolation inequality, it is not difficult to verify that

[TABLE]

Set $u_{\varepsilon,1}(z)=\varphi(\varepsilon z-x_{\varepsilon})u_{\varepsilon}(z)$, $u_{\varepsilon,2}(z)=(1-\varphi(\varepsilon z-x_{\varepsilon}))u_{\varepsilon}(z)$. As the same proof of (26) in [56], we obtain that

[TABLE]

By (4.12), we deduce that

[TABLE]

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

Hence, we get

[TABLE]

where $o(1)\rightarrow 0$ as $\varepsilon\rightarrow 0$.

We now estimate $P_{\varepsilon}(u_{\varepsilon,2})$. It follows from (4.3) that

[TABLE]

where $o(1)\rightarrow 0$ as $\varepsilon\rightarrow 0$ and using the similar arguments as (4), we can prove that

[TABLE]

and

[TABLE]

Furthermore, the above inequality implies that

[TABLE]

Then, we get

[TABLE]

In particular, taking $d_{0}>0$ small enough, we can assume that $P_{\varepsilon}(u_{\varepsilon,2})\geq 0$. Hence, from (4.14), it holds

[TABLE]

Furthermore, by (4.12), it is easy to check that

[TABLE]

and

[TABLE]

Hence, using the facts that $\langle\mathcal{J}_{\varepsilon}^{\prime}(u_{\varepsilon}),u_{\varepsilon,2}\rangle\rightarrow 0$ as $\varepsilon\rightarrow 0$, $\langle Q_{\varepsilon}^{\prime}(u_{\varepsilon}),u_{\varepsilon,2}\rangle\geq 0$, we have that

[TABLE]

which yields to

[TABLE]

Combining with (4.15), we get that for $d>0$ sufficiently small,

[TABLE]

We next estimate $P_{\varepsilon}(u_{\varepsilon,1})$. Denote $\widehat{u}_{\varepsilon}(z)=u_{\varepsilon,1}(z+\frac{x_{\varepsilon}}{\varepsilon})=\varphi(\varepsilon z)u_{\varepsilon}(z+\frac{x_{\varepsilon}}{\varepsilon})$, then $\{\widehat{u}_{\varepsilon}\}$ is bounded in $H^{s}(\mathbb{R}^{3})$ by virtue of $(V_{0})$. Thus, up to a subsequence, we may assume that there exists a $\widehat{u}\in H^{s}(\mathbb{R}^{3})$ such that $\widehat{u}_{\varepsilon}\rightharpoonup\widehat{u}$ in $H^{s}(\mathbb{R}^{3})$, $\widehat{u}_{\varepsilon}\rightarrow\widehat{u}$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$ and $\widehat{u}_{\varepsilon}\rightarrow\widehat{u}$ a.e. in $\mathbb{R}^{3}$. We now claim that

[TABLE]

In view of Proposition 2.3, suppose the contrary that there exists $r>0$ such that

[TABLE]

Thus, for $\varepsilon>0$ small, there exists $\widehat{y}_{\varepsilon}\in\mathbb{R}^{3}$ such that

[TABLE]

$\bullet$ $\{\widehat{y}_{\varepsilon}\}$ is bounded in $\mathbb{R}^{3}$, then there exists $r_{0}>0$ such that $|\widehat{y}_{\varepsilon}|\leq r_{0}$. Let $\widehat{v}_{\varepsilon}=\widehat{u}_{\varepsilon}-\widehat{u}$, then $\widehat{v}_{\varepsilon}\rightharpoonup 0$ in $H^{s}(\mathbb{R}^{3})$, for $\varepsilon>0$ small, by (4.20), it holds

[TABLE]

We now are to prove that

[TABLE]

where $\widehat{\rho}_{\varepsilon}=-(-\Delta)^{s}\widehat{v}_{\varepsilon}+(\widehat{v}_{\varepsilon}^{+})^{2_{s}^{\ast}-1}\in(H^{s}(\mathbb{R}^{3}))^{\prime}$. For $\varepsilon>0$ small, $\int_{\mathbb{R}^{3}\backslash\Lambda_{\varepsilon}}u_{\varepsilon}\widehat{\varphi}(z-\frac{x_{\varepsilon}}{\varepsilon})\,{\rm d}z=0$ uniformly for all $\widehat{\varphi}\in C_{0}^{\infty}(B_{r_{0}+2}(0))$. Hence, by virtue of $\lim\limits_{\varepsilon\rightarrow 0}\|u_{\varepsilon,2}\|_{H_{\varepsilon}}=0$, we have

[TABLE]

Combining the above estimate with $x_{\varepsilon}\rightarrow x_{0}\in\mathcal{M}^{\beta}$ , we see that $\widehat{u}\geq 0$ is a solution of

[TABLE]

On the other hand, the following Brezis-Lieb splitting properties hold:

[TABLE]

[TABLE]

[TABLE]

and

[TABLE]

uniformly for all $\widehat{\varphi}\in C_{0}^{\infty}(B_{r_{0}+2}(0))$ with $\|\widehat{\varphi}\|=1$. Thus, (4.22) is proved.

By Proposition 2.4, there exist $\widehat{z}_{\varepsilon}\in\mathbb{R}^{3}$ and $\widehat{\sigma}_{\varepsilon}>0$ such that $\widehat{z}_{\varepsilon}\rightarrow\widehat{z}\in B_{r_{0}+1}(0)$, $\widehat{\sigma}_{\varepsilon}\rightarrow 0$ and

[TABLE]

where $\overline{w}\geq 0$ is a nontrivial solution of (4.8) and satisfies (4.9).

Since

[TABLE]

but by (4.3), we get

[TABLE]

Thus,

[TABLE]

On the other hand, by (4.3), we have

[TABLE]

[TABLE]

hence, we get

[TABLE]

From the $W_{0}\in\mathcal{L}_{V_{0}}$, and by (4.24), (4.25), we have that

[TABLE]

where we have used the fact that $\frac{(q-4)s+(q-2)t}{2(q(s+t)-3)}+\frac{(2_{s}^{\ast}-q)(s+t)}{2_{s}^{\ast}(q(s+t)-3)}=\frac{s}{3}$. For $d_{0}$ sufficient small, we get a contradiction with Lemma 3.6.

$\bullet$ $\{\widehat{z}_{\varepsilon}\}$ is unbounded. Without loss of generality, we may assume that $\lim\limits_{\varepsilon\rightarrow 0}|\widehat{z}_{\varepsilon}|=+\infty$. Then, by (4.20), we have that

[TABLE]

i.e.,

[TABLE]

Since $\varphi(z)=0$ for $|z|\geq 2\beta$, so $|\widehat{z}_{\varepsilon}|\leq\frac{3\beta}{\varepsilon}$ for $\varepsilon$ small. If $|\widehat{z}_{\varepsilon}|\geq\frac{\beta}{2\varepsilon}$, then $\widehat{z}_{\varepsilon}\in B_{3\beta/\varepsilon}(0)\backslash B_{\beta/2\varepsilon}(0)$, and by (4.4), we get

[TABLE]

which contradicts with (4.26). Thus $|\widehat{z}_{\varepsilon}|\leq\frac{\beta}{2\varepsilon}$ for $\varepsilon>0$ small. Without loss of generality, we may assume that $\varepsilon\widehat{z}_{\varepsilon}\rightarrow z_{0}\in\overline{B_{\beta/2}(0)}$ and $\widetilde{u}_{\varepsilon}\rightharpoonup\widetilde{u}$ in $H^{s}(\mathbb{R}^{3})$, where $\widetilde{u}_{\varepsilon}(z):=\widehat{u}_{\varepsilon}(z+\widehat{z}_{\varepsilon})$. If $\widetilde{u}\neq 0$, it is easy to check that $\widetilde{u}$ satisfies that

[TABLE]

Similarly as in the proof of the case $v\neq 0$ of the claim (4.4), we can get a contradiction for $d_{0}$ sufficient small. Thus $\widetilde{u}=0$. Similarly as the proof of the case $u=0$ of the claim (4.4) (where using Proposition (2.4)), we find that there exist $\widetilde{x}_{\varepsilon}\in\mathbb{R}^{3}$ and $\widetilde{\sigma}_{\varepsilon}>0$ such that $\widetilde{x}_{\varepsilon}\rightarrow\widetilde{x}\in\overline{B_{1}(0)}$, $\widetilde{\sigma}_{\varepsilon}\rightarrow 0$ and

[TABLE]

where $u^{\ast}$ is a nontrivial of solution of (4.7) and satisfies (4.8). Thus, there exists $R>0$ such that

[TABLE]

On the other hand, we have that

[TABLE]

which contradicts to (4.3) for $d_{0}$ small enough. Hence, the claim (4.19) holds and so using the interpolation inequality, we deduce that

[TABLE]

By (4.17), recalling that $\widehat{u}_{\varepsilon}(z)=u_{\varepsilon,1}(z+\frac{x_{\varepsilon}}{\varepsilon})$, we have

[TABLE]

Letting $\varepsilon\rightarrow 0$, and using (4.27), $(V_{0})$, we get

[TABLE]

On the other hand, in view of $\langle\mathcal{J}^{\prime}_{\varepsilon}(u_{\varepsilon}),u_{\varepsilon,1}\rangle\rightarrow 0$ and (4.18), and $\langle Q_{\varepsilon}^{\prime}(u_{\varepsilon}),u_{\varepsilon,1}\rangle=0$, we deduce that

[TABLE]

then by Fatou’s Lemma, (4.27) and (4.23), we have that

[TABLE]

which implies that

[TABLE]

and

[TABLE]

Hence, by $(V_{0})$, we can deduce that

[TABLE]

By (4.3), (4.27), it is easy to check that $\widehat{u}\neq 0$. It follows from (4.23) that $\mathcal{I}_{V(x_{0})}(\widehat{u})\geq c_{V(x_{0})}$. Hence, $\mathcal{I}_{V(x_{0})}(\widehat{u})=c_{V(x_{0})}$ is proved. In view of $x_{0}\in\mathcal{M}^{\beta}\subset\Lambda$, we have that $V(x_{0})=V_{0}$ and $x_{0}\in\mathcal{M}$. As a consequence, $\widehat{u}$ is, up to a translation in the $x-$variable, an element of $\mathcal{L}_{V_{0}}$, namely there exists $W\in\mathcal{L}_{V_{0}}$ and $z_{0}\in\mathbb{R}^{3}$ such that $\widehat{u}(z)=W(z-z_{0})$. Consequently, from (4.3), (4.18) and (4.28), we have that

[TABLE]

Observing that $\varepsilon(\frac{x_{\varepsilon}}{\varepsilon}+z_{0})\rightarrow x_{0}\in\mathcal{M}$ as $\varepsilon\rightarrow 0$, so the proof is completed. ∎

For $a\in\mathbb{R}$ we define the sublevel set of $\mathcal{J}_{\varepsilon}$ as follows

[TABLE]

We observe that the result of Lemma 4.1 holds for $d_{0}>0$ sufficiently small independently of the sequences satisfying the assumptions.

Lemma 4.2**.**

Let $d_{0}$ be the number given in Lemma 4.1. Then for any $d\in(0,d_{0})$, there exist positive constants $\varepsilon_{d}>0$, $\rho_{d}>0$ and $\alpha_{d}>0$ such that

[TABLE]

We recall the definition (4.2) of $\gamma_{\varepsilon}(\tau)$. The following Lemma holds.

Lemma 4.3**.**

There exists $M_{0}>0$ such that for any $\delta>0$ small, there exists $\alpha_{\delta}>0$ and $\varepsilon_{\delta}>0$ such that if $\mathcal{J}_{\varepsilon}(\gamma_{\varepsilon}(\tau))\geq c_{V_{0}}-\alpha_{\delta}$ and $\varepsilon\in(0,\varepsilon_{\delta})$, then $\gamma_{\varepsilon}(\tau)\in\mathcal{N}_{\varepsilon}^{M_{0}\delta}$.

We are now ready to show that the penalized functional $\mathcal{J}_{\varepsilon}$ possesses a critical point for every $\varepsilon>0$ sufficiently small. Choose $\delta_{1}>0$ such that $M_{0}\delta_{1}<\frac{d_{0}}{4}$ in Lemma 4.3, and fixing $d=\frac{d_{0}}{4}:=d_{1}$ in Lemma 4.2. Similar to the proof of Lemma 4.6 in [29], we can prove the following result.

Lemma 4.4**.**

There exists $\overline{\varepsilon}>0$ such that for each $\varepsilon\in(0,\overline{\varepsilon})$, there exists a sequence $\{u_{\varepsilon,n}\}\subset\mathcal{J}_{\varepsilon}^{\widetilde{\mathcal{C}}_{\varepsilon}+\varepsilon}\cap\mathcal{N}_{\varepsilon}^{d_{0}}$ such that $\mathcal{J}^{\prime}_{\varepsilon}(u_{\varepsilon,n})\rightarrow 0$ in $(H_{\varepsilon})^{\prime}$ as $n\rightarrow\infty$.

Lemma 4.5**.**

$\mathcal{J}_{\varepsilon}$* possesses a nontrivial critical point $u_{\varepsilon}\in\mathcal{N}_{\varepsilon}^{d_{0}}\cap\mathcal{J}_{\varepsilon}^{\mathcal{D}_{\varepsilon}+\varepsilon}$ for $\varepsilon\in(0,\bar{\varepsilon}]$.*

Proof.

By Lemma 4.4, there exists $\bar{\varepsilon}>0$ such that for each $\varepsilon\in(0,\bar{\varepsilon}]$, there exists a sequence $\{u_{\varepsilon,n}\}\subset\mathcal{J}_{\varepsilon}^{\mathcal{D}_{\varepsilon}+\varepsilon}\cap\mathcal{N}_{\varepsilon}^{d_{0}}$ such that $\mathcal{J}_{\varepsilon_{n}}^{\prime}(u_{\varepsilon,n})\rightarrow 0$ as $n\rightarrow\infty$ in $(H_{\varepsilon})^{\prime}$. Since $\mathcal{N}_{\varepsilon}^{d_{0}}$ is bounded, then $\{u_{\varepsilon,n}\}$ is bounded in $H_{\varepsilon}$ and up to a subsequence, we may assume that there exists $u_{\varepsilon}\in H_{\varepsilon}$ such that $u_{\varepsilon,n}\rightharpoonup u_{\varepsilon}$ in $H_{\varepsilon}$, $u_{\varepsilon,n}\rightarrow u_{\varepsilon}$ in $L_{loc}^{p}(\mathbb{R}^{3})$ for $1\leq p<2_{s}^{\ast}$ and $u_{\varepsilon,n}\rightarrow u_{\varepsilon}$ a.e. in $\mathbb{R}^{3}$. It is easy to check that $u_{\varepsilon}$ satisfies

[TABLE]

where \mu_{\varepsilon,n}=\Big{(}\int_{\mathbb{R}^{3}\backslash\Lambda_{\varepsilon}}u_{\varepsilon,n}^{2}\,{\rm d}z-\varepsilon\Big{)}_{+}\rightarrow\mu_{\varepsilon} as $n\rightarrow\infty$.

We claim that

[TABLE]

Indeed, Choosing a cutoff function $\psi_{\rho}\in C^{\infty}(\mathbb{R}^{3})$ such that $\psi_{\rho}(z)=1$ on $\mathbb{R}^{3}\backslash B_{2\rho}(0)$, $\psi_{\rho}(z)=0$ on $B_{\rho}(0)$, $0\leq\psi_{\rho}\leq 1$ and $|\nabla\psi_{\rho}|\leq\frac{C}{\rho}$. Since $\psi_{\rho}u_{\varepsilon,n}\in H_{\varepsilon}$, then $\langle\mathcal{J}_{\varepsilon_{n}}^{\prime}(u_{\varepsilon,n}),\psi_{\rho}u_{\varepsilon,n}\rangle\rightarrow 0$ as $n\rightarrow\infty$. Thus, for sufficiently large $\rho$ such that $\Lambda_{\varepsilon}\subset B_{\rho}(0)$, we have

[TABLE]

In view of the fact that $|D_{s}\psi_{\rho}|^{2}\leq\frac{C}{\rho^{2s}}$ for any $z\in\mathbb{R}^{3}$ and Hölder’s inequality, we deduce that

[TABLE]

Therefore, from the estimates above, we obtain

[TABLE]

Thus, the claim follows. From (4.30), we see that $u_{\varepsilon,n}\rightarrow u_{\varepsilon}$ in $L^{2}(\mathbb{R}^{3})$.

Next, we claim that $u_{\varepsilon,n}\rightarrow u_{\varepsilon}$ in $L^{2_{s}^{\ast}}(\mathbb{R}^{3})$ as $n\rightarrow\infty$. Indeed, from Lemma 2.2, we may assume that

[TABLE]

and there exist a (at most countable) set of distinct points $\{x_{j}\}_{j\in J}\subset\mathbb{R}^{3}$, $\mu_{j}\geq 0$, $\nu_{j}\geq 0$ with $\mu_{j}+\nu_{j}>0$ ($j\in J$) such that

[TABLE]

We are going to show that $J={\O}$. Suppose by contradiction that $J\neq{\O}$, i.e., there exists $x_{j_{0}}\in\mathbb{R}^{3}$ for some $j_{0}\in J$. Similar to the arguments in Proposition 2.4, we get $\nu_{j_{0}}\geq\mathcal{S}_{s}^{\frac{3}{2s}}$. On the other hand, since $\{u_{\varepsilon,n}\}\subset\mathcal{N}_{\varepsilon}^{d_{0}}$, by the definition of $\mathcal{N}_{\varepsilon}^{d_{0}}$, there exist $\{W_{n}\}\subset\mathcal{L}_{V_{0}}$, $\{x_{n}\}_{n=1}^{\infty}\subset\mathcal{M}^{\beta}$ such that

[TABLE]

Since $\mathcal{L}_{V_{0}}$ and $\mathcal{M}^{\beta}$ are compact, there exist $W_{0}\in\mathcal{L}_{V_{0}}$, $x^{\prime}\in\mathcal{M}^{\beta}$ such that $W_{n}\rightarrow W_{0}$ in $H^{s}(\mathbb{R}^{3})$ and $x_{n}\rightarrow x^{\prime}$ as $n\rightarrow\infty$. Thus, for $\varepsilon>0$ small,

[TABLE]

It follows from (4.31), (4.32), $(f_{3})$, $v_{j_{0}}\geq\mathcal{S}_{s}^{\frac{3}{2s}}$ and $W_{0}\in\mathcal{L}_{V_{0}}$ that

[TABLE]

where $o(1)\rightarrow 0$ as $\varepsilon\rightarrow 0$. Taking $\varepsilon\rightarrow 0$ and $d_{0}\rightarrow 0$, we have that $c_{V_{0}}\geq\frac{s}{3}\mathcal{S}_{s}^{\frac{3}{2s}}$, contradicts with Lemma 3.6. Therefore, $u_{\varepsilon,n}^{+}\rightarrow u_{\varepsilon}^{+}$ in $L^{2_{s}^{\ast}}(\mathbb{R}^{3})$. Together with $u_{\varepsilon,n}\rightarrow u_{\varepsilon}$ in $L^{2}(\mathbb{R}^{3})$, Lebesgue Dominated Convergence Theorem implies that for $\varepsilon>0$ small,

[TABLE]

From (4.29) and $\mathcal{J}_{\varepsilon}^{\prime}(u_{\varepsilon,n})\rightarrow 0$ as $n\rightarrow\infty$, we get that

[TABLE]

and

[TABLE]

where $o(1)\rightarrow 0$ as $n\rightarrow\infty$. From (4.33), (4) and (4), we can deduce that

[TABLE]

Since $0\not\in\mathcal{N}_{\varepsilon}^{d_{0}}$, $u_{\varepsilon}\neq 0$ and $u_{\varepsilon}\in\mathcal{N}_{\varepsilon}^{d_{0}}\cap\mathcal{J}_{\varepsilon}^{\mathcal{D}_{\varepsilon}+\varepsilon}$. The proof is completed. ∎

Proof of Theorem 1.1. Using Lemma 4.5 and by similar arguments as the proof the Theorem 1.1 in [57], we can complete the proof of Theorem 1.1.

Acknowledgements. The work is supported by NSFC grant 11501403.