Nonlinear Modulational Instability of Dispersive PDE Models

TL;DR

This paper establishes nonlinear modulational instability for various dispersive PDEs, including KDV, Schrödinger, and BBM equations, using Hamiltonian structures and advanced energy estimates.

Contribution

It introduces new methods to prove nonlinear instability for a broad class of dispersive PDEs, handling derivative loss and low regularity nonlinearities.

Findings

Proves nonlinear modulational instability for multiple dispersive PDEs.

Develops semigroup estimates using Hamiltonian structures.

Overcomes derivative loss with higher order approximations and bootstrap arguments.

Abstract

We prove nonlinear modulational instability for both periodic and localized perturbations of periodic traveling waves for several dispersive PDEs, including the KDV type equations (e.g. the Whitham equation, the generalized KDV equation, the Benjamin-Ono equation), the nonlinear Schr\"odinger equation and the BBM equation. First, the semigroup estimates required for the nonlinear proof are obtained by using the Hamiltonian structures of the linearized PDEs; Second, for KDV type equations the loss of derivative in the nonlinear term is overcome in two complementary cases: (1) for smooth nonlinear terms and general dispersive operators, we construct higher order approximation solutions and then use energy type estimates; (2) for nonlinear terms of low regularity, with some additional assumption on the dispersive operator, we use a bootstrap argument to overcome the loss of derivative.

Equations

$${\partial }_{t}u+{\partial }_x(\mathcal{M}u+f(u))=0,$$

$$\text{(A2a) }{c}_1{\left|\xi \right|}^m\le \alpha \left(\xi \right)\le c_2{\left|\xi \right|}^m,\text{for large }\xi ,$$

$$\text{(A2b) }{c}_1{\left|\xi \right|}^{-m}\le \alpha \left(\xi \right)\le c_2{\left|\xi \right|}^{-m},\text{for large }\xi .$$

$${\partial }_{t}u+{\partial }_x({\mathcal{M}}_{1}u+f(u))=0.$$

$$\mathcal{M}{u}_c-c{u}_c+f\left(u_c\right)=a,$$

$${\partial }_{t}U-c{\partial }_{x}U+{\partial }_x(\mathcal{M}U+f(U))=0.$$

$${\partial }_{t}U=-{\partial }_x\left(\mathcal{M}-c+f^{\prime }(u_c)\right)U=JLU,$$

$$J=-{\partial }_x,L=\mathcal{M}-c+f^{\prime }(u_c).$$

$$JL{e}^{ikx}{v}_k(x)=\lambda (k)e^{ikx}{v}_k(x),$$

$$J_k={\partial }_x+ik,L_k={\mathcal{M}}_k-c+f^{\prime }(u_c).$$

$$\underset{y\in \mathbb{T}}{\inf }\Vert U_{\delta }(T^{\delta },x)-u_c(x+y){\Vert }_{L^2({\mathbb{T}}_{2\pi q})}⩾{\theta }_0,$$

$$f\in C^{2n+2}\left(\mathbf{R}\right),\text{where }n⩾\frac{1}{2}\max \{1+m,1\}\text{is an integer,}$$

$$c_1{\left|\xi \right|}^m\le \alpha \left(\xi \right)\le c_2{\left|\xi \right|}^m,m\ge 1,c_1,c_2>0,\text{for large }\xi .$$

$$\left|f\left(u+v\right)-f\left(v\right)-f^{\prime }\left(v\right)u\right|\le C\left({\left|u\right|}_{\infty },{\left|v\right|}_{\infty }\right){\left|u\right|}^{p_1},$$

$$\left|F\left(u+v\right)-F\left(v\right)-f\left(v\right)u-\frac{1}{2}{f}^{\prime }\left(v\right)u^2\right|\le C\left({\left|u\right|}_{\infty },{\left|v\right|}_{\infty }\right){\left|u\right|}^{p_2},$$

$$u(x)={\int }_0^1{e}^{i\xi x}{u}_{\xi }(x)d\xi ,\text{where }{u}_{\xi }(x)={\Sigma }_{n\in \mathbf{Z}}{e}^{inx}\hat{u}(n+\xi )\in H^s({\mathbb{T}}_{2\pi }),$$

$$\|u(x)\|_{H^{s}{(\mathbf{R})}}^{2}\thickapprox\int_{0}^{1}\|u_{\xi}\left(x\right)\|_{H^{s}(\mathbb{T}_{2\pi})}^{2}\,d\xi,\$$

$$e^{tJL}u(x)={\int }_0^1{e}^{i\xi x}{e}^{t{J}_{\xi }{L}_{\xi }}{u}_{\xi }\left(x\right)d\xi ,$$

$${\Vert {\partial }_{x}f\left(u\right)\Vert }_{H^{-1}}\approx {\Vert f\left(u\right)\Vert }_{L^2},$$

$$J_k{L}_k{v}_k=\lambda \left(k\right)v_k,v_k(x)\in L^2({\mathbb{T}}_{2\pi }),$$

$$\mathrm{Re}⟨\lambda v_k,L_k{v}_k⟩=\mathrm{Re}⟨J_k{L}_k{v}_k,L_k{v}_k⟩=0.$$

$$(\mathrm{Re}\lambda )⟨v_k,L_k{v}_k⟩=0.$$

$$⟨v_k,L_k{v}_k⟩=⟨v_k(x),(c-{\mathcal{M}}_k-f^{\prime }(u_c))v_k(x)⟩=0,$$

$$c{\int }_{{\mathbb{T}}_{2\pi }}{v}_k(x)\overline{v_k(x)}dx-{\int }_{{\mathbb{T}}_{2\pi }}{v}_k(x)\overline{{\mathcal{M}}_k{v}_k(x)}dx-{\int }_{{\mathbb{T}}_{2\pi }}{v}_k(x)\overline{f^{\prime }(u_c)v_k(x)}dx=0.$$

$$|⟨{\mathcal{M}}_k{v}_k,v_k⟩|$$

$$\le (c+\Vert f^{\prime }(u_c){\Vert }_{L^{\infty }({\mathbb{T}}_{2\pi })})\Vert v_k(x){\Vert }_{L^2({\mathbb{T}}_{2\pi })}^2,$$

$${\mathcal{M}}_k{J}_k{L}_k{v}_k=\lambda {\mathcal{M}}_k{v}_k.$$

$$\mathrm{Re}⟨{\mathcal{M}}_k{J}_k{L}_k{v}_k,L_k{v}_k⟩=\mathrm{Re}⟨\lambda {\mathcal{M}}_k{v}_k,L_k{v}_k⟩.$$

$$\mathrm{Re}⟨{\mathcal{M}}_k{J}_k{L}_k{v}_k,L_k{v}_k⟩=0,$$

$$\mathrm{Re}⟨\lambda {\mathcal{M}}_k{v}_k,L_k{v}_k⟩=0.$$

$$(\mathrm{Re}\lambda )⟨{\mathcal{M}}_k{v}_k,{\mathcal{M}}_k{v}_k⟩=\mathrm{Re}⟨\lambda {\mathcal{M}}_k{v}_k,\left(c-f^{\prime }(u_c)\right)v_k⟩,$$

Full text

1 Introduction

The modulational instability, also called Benjamin-Feir or side-band instability in the literature, is a very important instability mechanism in lots of dispersive and fluid models. It has been used to explain the instability of periodic wave trains to self modulation and the development of large-scale structures such as envelope solitons. The modulational instability has been observed in experiments and in nature, for many physical systems. The first theoretical understanding of modulational instability arose in 1960s, in the works of Benjamin and Feir ([4]) for water waves and independently by Lighthill ([26]), Whitham ([31]), Zakharov ([32]) for various dispersive wave equations. We refer to the review ([33]) for more details on the history and physical applications of modulational instability. In recent years, there have been lots of mathematical work on the rigorous justification of linear modulational instability for various dispersive wave models including the KDV type equations, the nonlinear Schrödinger equation, the BBM equation etc. In particular, the modulational instability conditions for perturbations of long wavelength (i.e. frequencies near zero) were derived in lots of works ([7] [8] [16] [11] [15] [17] [19] [21]). We refer to the recent survey ([6]) for more details and references. The modulational instability for perturbations of high frequencies (i.e. not near zero) was also considered in some papers ([9] [20]). However, there has been no proof of modulational instability under the nonlinear dynamics of the PDE models. The purpose of this paper is to provide a proof of nonlinear modulational instability under both multi-periodic and localized perturbations, for a large class of dispersive wave models.

We mainly consider the KDV type equations,

[TABLE]

where $\mathcal{M}$ is a Fourier multiplier operator satisfying $\widehat{\mathcal{M}u}(\xi)=\alpha(\xi)\widehat{u}(\xi)$ and $f(s)\in C^{1}(\mathbf{R},\mathbf{R})$. We make the following assumptions on the operator $\mathcal{M}$:

(A1) $\mathcal{M}$ is a self-adjoint operator, and the symbol $\alpha:\mathbf{R}\mapsto\mathbf{R}^{+}$ is even and regular near [math].

(A2) There exist constants $m,c_{1},c_{2}>0$, such that

[TABLE]

or

[TABLE]

The assumption (1.2) implies that $\mathcal{M}$ is an “differential” operator with $\|\mathcal{M}(\cdot)\|_{L^{2}}\sim\|\cdot\|_{H^{m}}$, and (1.3) implies that $\mathcal{M}$ is a “smoothing” operator with $\|\mathcal{M}(\cdot)\|_{H^{m}}\sim\|\cdot\|_{L^{2}}$.  For the classical KDV equation, $\mathcal{M=-\partial}_{x}^{2}$ and $f\left(u\right)=u^{2}$. Other examples include: Benjamin-Ono equation, Whitham equation and intermediate long-wave (ILW) equation, which are all of KDV type with $\alpha(\xi)=\left|\xi\right|,$ $\sqrt{\frac{\tanh\xi}{\xi}}$ and $\xi\coth\left(\xi H\right)-H^{-1}$ respectively.

For convenience, we assume $\min\alpha(\xi)>0$. Since otherwise, we can always break $\mathcal{M=M}_{1}+c_{1}$, where $\mathcal{M}_{1}$ has a positive symbol and $c_{1}$ is a constant. Then in the traveling frame $\left(x-c_{1}t,t\right)$, the equation (1.1) becomes

[TABLE]

A periodic traveling wave (TW) of (1.1) is of the form $u\left(x,t\right)=u_{c}\left(x-ct\right)$, where $c\in\mathbf{R}$ is the traveling speed and $u_{c}$ satisfies the equation

[TABLE]

for a constant $a$. The existence of the periodic TWs of (1.4) had been well studied in the literature, and we refer to the book ([2]) and the references therein. In general, the periodic TWs are a three-parameter family of solutions depending on period $T$, traveling speed $c$ and the constant $a$. The stability of TWs to perturbations of the same period has been studied a lot in recent years (e.g. [2] [1] [18] [27] [21]). The modulational instability is to study the instability of periodic TWs for perturbations of different periods and even for localized perturbations in $\mathbf{R}$. The equation (1.1) in the traveling frame $\left(x-ct,t\right)$ becomes

[TABLE]

The linearized equation of (1.5) near $u_{c}$ can be written in the Hamiltonian form

[TABLE]

where

[TABLE]

Without lost of generality, we take the minimal period $T=2\pi$. By the standard Floquet-Bloch theory, any bounded eigenfunction $\phi(x)$ of $JL$ takes the form $\phi(x)=e^{ikx}v_{k}(x)$, where $k\in\left[0,1\right]$ is a parameter and $v_{k}\in L^{2}(\mathbb{T}_{2\pi})$. It leads us to the one-parameter family of eigenvalue problems

[TABLE]

or equivalently $J_{k}L_{k}v_{k}=\lambda\left(k\right)v_{k}$, where

[TABLE]

Here, $\mathcal{M}_{k}$ is the Fourier multiplier operator with the symbol $\alpha(\xi+k)$.

Definition 1.1

We say that $u_{c}$ is linearly modulationally unstable if there exists $k\in\left[0,1\right]$ such that the operator $J_{k}L_{k}$ has an unstable eigenvalue $\lambda(k)$ with $\operatorname{Re}\lambda(k)>0$ in the space $L^{2}(\mathbb{T}_{2\pi})$.

By above definition and the analytic perturbation theory of the spectra of linear operators, if $k_{0}$ is an unstable frequency, then all $k$ near $k_{0}$ are also unstable. So there exist intervals of unstable frequencies in $\left[0,1\right]$. For periodic waves which are orbitally stable under co-periodic perturbations (i.e. same period), it is shown in Proposition 11.3 of [27] that when $k$ is small (i.e. long wavelength), the possible unstable eigenvalues of $J_{k}L_{k}$ can only be perturbed from the zero eigenvalue of $JL$ in $L^{2}\left(\mathbb{T}_{2\pi}\right)$. The conditions of linear modulational instability for such long wavelength perturbations had been studied in lots of papers for various dispersive models (see the references cited at the beginning). In Section 8, we give some examples for which the linear modulational instability condition is satisfied.

Our first main result is the proof of nonlinear modulational instability under both multi-periodic and localized perturbations, for a smooth nonlinear term $f\left(u\right)$ and $\mathcal{M}$ with a general symbol.

Theorem 1.1

Assume (A1)-(A2a) or (A1)-(A2b) and (2.3), f\in C^{\infty}\left(\mathbf{R}\right)\and $u_{c}$ is linearly modulationally unstable. Then

i) $u_{c}$ is nonlinearly orbitally unstable to (1.5) for multi-periodic perturbations in the following sense: there exists $q\in\mathbb{N},$ $\theta_{0}>0$, such that for any $s\in\mathbb{N}$ and arbitrarily small $\delta>0$, there exists a solution $U_{\delta}(t,x)$ to (1.5) satisfying \|U_{\delta}(0,x)-u_{c}(x)\|_{H^{s}({\mathbb{T}_{2\pi q}})}<\delta\and

[TABLE]

where $T^{\delta}\sim\left|\ln\delta\right|$.

ii) $u_{c}$ is nonlinearly unstable to (1.5) for localized perturbations in the following sense: there exists $\theta_{0}>0$, such that for any $s\in\mathbb{N}$ and arbitrarily small $\delta>0$, there exists a solution $U_{\delta}(t,x)$ to (1.5) satisfying \|U_{\delta}(0,x)-u_{c}(x)\|_{H^{s}(\mathbf{R})}<\delta\and$\ \|U(T^{\delta},x)-u_{c}(x)\|_{L^{2}(\mathbf{R})}\geqslant\theta_{0}$, where $T^{\delta}\sim|\ln\delta|$.

For some examples, $f\left(u\right)$ is not smooth. Our second result is complementary to Theorem 1.1, about nonlinear modulational instability for non-smooth $f$ with some additional assumptions.

Theorem 1.2

Assume

[TABLE]

the symbol \alpha\left(\xi\right)\of $\mathcal{M}$ satisfies the condition

[TABLE]

Suppose $u_{c}$ is linearly modulationally unstable. Then $u_{c}$ is nonlinearly unstable to (1.5) for both multi-periodic and localized perturbations in the sense of Theorem 1.1, with the initial perturbation arbitrarily small in $H^{2n}\left({\mathbb{T}_{2\pi q}}\right)$ or $H^{2n}\left(\mathbf{R}\right)$.

Remark 1.1

In Theorem 1.2, the regularity assumption (1.9) on $f$ is only used to prove that the equation (1.5) is locally well-posed in $H^{2n}\left({\mathbb{T}_{2\pi q}}\right)$ and $u_{c}+H^{2n}\left(\mathbf{R}\right)$ by Kato’s approach (see Lemma 5.2). Assuming the local well-posedness of (1.5) in the energy space $H^{\frac{m}{2}}$, we only need the following much weaker assumptions on $f$ to prove nonlinear instability:

f\in C^{1}\left(\mathbf{R}\right)\and there exist p_{1}>1,\ p_{2}>2,\such that

[TABLE]

[TABLE]

where $F\left(u\right)=\int_{0}^{u}f\left(s\right)ds$. The conditions (1.11)-(1.12) are automatically satisfied when $f\in C^{2}\left(\mathbf{R}\right)$.

In above Theorems, the nonlinear instability for multi-periodic perturbations is proved in the orbital distance since (1.5) is translation invariant. For localized perturbations, we study the equation (1.5) in the space $u_{c}+H^{s}\left(\mathbf{R}\right)$ which is not translation invariant. Therefore, we do no use the orbital distance for nonlinear instability under localized perturbations.

Below we discuss main ingredients in the proof of Theorems 1.1 and 1.2. For the proof of nonlinear instability, first we need to establish the semigroup estimates for the linearized equation (1.6), more specifically, to show that the growth of solutions of (1.6) is bounded by the maximal growth rate of unstable eigenvalues of the linearized operator. To get such estimates, we strongly use the Hamiltonian structure of the linearized equation (1.6). For multi-periodic perturbations, since $L$ has only finitely many negative modes, this fits into the general theory developed by Lin and Zeng ([27]) and the semigroup estimates follow directly from the exponential trichotomy Theorem 3.1. For localized perturbations, the quadratic form of $L$ has infinitely many negative modes and we cannot use Theorem 3.1 directly. By observing that any function $u\in H^{s}{(\mathbf{R)}}$ can be written as

[TABLE]

and

[TABLE]

[TABLE]

the estimate of $e^{tJL}|_{H^{s}{(\mathbf{R})}}$ is reduced to estimate $e^{tJ_{\xi}L_{\xi}}|_{H^{s}(\mathbb{T}_{2\pi})}$ uniformly for $\xi\in\left[0,1\right]$. This is proved in [27] for the case when $\mathcal{M}$ is “differential” (i.e. (1.2)) and in Lemma 3.5 when $\mathcal{M}$ is “smoothing” (i.e. (1.3)).

With the semigroup estimates, to prove nonlinear instability we still need to overcome the loss of derivative of the nonlinear term in (1.5). We use two different approaches to handle two complementary cases. For the case of smooth nonlinear term and general $\mathcal{M}$ including both “differential” and “smoothing” cases, we basically adapt Grenier’s approach in [12] developed for proving nonlinear instability of shear flows of the 2D Euler equation. The idea is to construct higher order approximate solutions of (1.5) and then use the energy estimates to overcome the loss of derivative. When the nonlinear term is smooth, the approximate solution can be constructed to sufficiently high order to compensate for the roughness of the energy estimates. For the multi-periodic case, the initial perturbation is chosen to be along the most unstable eigenfunction. For the localized case, since there is no genuine eigenfunction of $JL$ in $L^{2}\left(\mathbf{R}\right)$, the initial perturbation is constructed as a wave packet concentrated near the most unstable frequency.

When the nonlinear term is not smooth, we cannot use the approach of higher order approximate solutions. Instead, a totally different approach of bootstrap estimates is used to overcome the loss of derivative when $f$ is $C^{1}$ with the growth conditions (1.11)-(1.12) and $\mathcal{M}$ is “differential” with the condition (1.2). First, the invariance of the energy functional is used to show that $H^{\frac{m}{2}}$norm of the unstable solution has the same growth as the $L^{2}$ norm. Then we estimate the growth of $H^{-1}$ norm with the help of the semigroup estimate $e^{tJL}|_{H^{-1}}$. The estimates are closed by interpolating $H^{-1}$ and $H^{\frac{m}{2}}$ to get back to $L^{2}$. The loss of derivative in the nonlinear term \partial_{x}f\left(u\right)\is overcome by observing that

[TABLE]

which is controllable in $H^{\frac{m}{2}}$. To get the crucial semigroup estimate $e^{tJL}|_{H^{-1}}$ used in the above bootstrap process, by duality it is equivalently to estimate $e^{tLJ}|_{H^{1}}$, which is then related to $e^{tJL}$ by certain conjugate transforms. The proof is much more involved for the localized case. By using the norm equivalence (1.13), this is reduced to estimate $e^{tL_{\xi}J_{\xi}}|_{H^{1}(\mathbb{T}_{2\pi})}$ uniformly for $\xi\in\left[0,1\right]$. This is done by a careful decomposition of the spectral projections of $L_{\xi}$ near [math] and away from [math]. We note that the idea of overcoming the loss of derivative by bootstrapping the growth of higher order norms from a lower order one was originated in [14] for the Vlasov-Poisson system. This approach was later extended to treat other problems including 2D Euler equation ([3] [28]) and Vlasov-Maxwell systems ([29]). Here, our approach of bootstrapping the lower order norm \left(H^{-1}\right)\from a higher order norm $\left(H^{\frac{m}{2}}\right)$ and then closing by interpolation seems to be new. This idea coupled with the $H^{-1}$ semigroup estimates could be useful in other problems involving the loss of derivative.

Besides the KDV type equations, modulational instability also appears in semilinear models such as BBM and Schrödinger equations. Since there is no loss of derivative, the nonlinear instability can be proved by ODE arguments. The required semigroup estimates can be obtained similarly by using the Hamiltonian structures. As an example, we consider BBM equation in Section 7.

This paper is organized as follows. In Section 2, we study the regularity of unstable eigenfunctions. In Section 3, we gather and prove the semigroup estimates used in the proof of nonlinear instability. In Sections 4 and 5, the nonlinear instability for multi-periodic and localized cases is proved by constructing higher order approximate solutions. In Section 6, we prove nonlinear instability by bootstrap arguments. In Section 7, we prove nonlinear instability for BBM equation. In the final Section 8, we list some models for which our theorems are applicable.

2 Linear Modulational Instability

In this section, we prove the regularity of the unstable eigenfunctions of $J_{k}L_{k}$. In the proof below and throughout this paper, we use $g\lesssim h$ \left(g,h\geq 0\right)\to denote $g\leq Ch$,   for a generic constant $C>0$, which may differ from one inequality to another. First, we consider the case when $\mathcal{M}$ is a ”differential” operator as in the case of the KDV, the Benjamin-Ono and the ILW equations.

Lemma 2.1

Assume (1.2). If  $f\in C^{\infty}\left(\mathbf{R}\right)$ and $v_{k}(x)\in L^{2}(\mathbb{T}_{2\pi})$ is an unstable eigenfunction to $J_{k}L_{k}$ with $k\in\left[0,1\right]$, then $v_{k}\in H^{s}(\mathbb{T}_{2\pi})$ for every $s\in\mathbb{N}$.

Proof. By assumption, there exists $\lambda\left(k\right)$ with $\operatorname{Re}\lambda\left(k\right)>0$ such that

[TABLE]

where $J_{k},L_{k}$ are defined in (1.8). It is easy to see that $J_{k}$ is a skew-adjoint operator and $L_{k}$ is a self-adjoint operator. Taking the real part of the $L^{2}$ inner product of (2.1) with $L_{k}v_{k}$, we get the following “conservation law”:

[TABLE]

Since $L_{k}$ is self-adjoint, $\left\langle v_{k}(x),L_{k}v_{k}(x)\right\rangle$ is real. It follows that

[TABLE]

Noting that $\operatorname{Re}\lambda>0$, we have

[TABLE]

i.e.

[TABLE]

It follows immediately that

[TABLE]

Applying \mathcal{M}_{k}\to (2.1), we obtain

[TABLE]

Taking the real part of inner product of (2.2) with $L_{k}v_{k}$, one has

[TABLE]

Note that $\mathcal{M}_{k}$ is self-adjoint and $J_{k}$ is skew-adjoint, also $\mathcal{M}_{k}$ and $J_{k}$ are commutable, therefore $\mathcal{M}_{k}J_{k}$ is skew-adjoint. It follows that

[TABLE]

which implies

[TABLE]

Then, we obtain

[TABLE]

which implies that

[TABLE]

and

[TABLE]

In the above, we use the estimate

[TABLE]

Similarly, one can show that

[TABLE]

for any $s\in\mathbb{N}$.

In the next Lemma, we prove the regularity of unstable eigenfunctions when $\mathcal{M}$ is a smoothing operator satisfying (1.3) as in the case of Whitham equation. We need the following assumption on the periodic TWs of (1.1):

[TABLE]

which was assumed in [17] and [10] to show the regularity of TWs of Whitham equation. This assumption is satisfied for small amplitude waves of Whitham equation (see Section 8.1). By a similar proof as in [17] and [10], we can show that $u_{c}\in C^{\infty}$ under the assumption (2.3) when $f\in C^{\infty}\left(\mathbf{R}\right).$

Lemma 2.2

Assume (1.3) and (2.3). If  $f\in C^{\infty}\left(\mathbf{R}\right)$ and $v_{k}\in L^{2}(\mathbb{T}_{2\pi})$ is an unstable eigenfunction of $J_{k}L_{k}$ with $k\in\left[0,1\right]$, then $v_{k}\in H^{s}(\mathbb{T}_{2\pi})$ for every $s\in\mathbb{N}$.

Proof. Step 1: We first prove that under the assumption (2.3), for any integer $s\geq 0$, there exists a constant $C(s)$, such that for any $\phi\in H^{s}(\mathbb{T}_{2\pi}),$

[TABLE]

where $D=-(\partial_{x}+ik)\left(c-f^{\prime}(u_{c})\right)$.

Define an inner product $[\cdot,\cdot]$ by

[TABLE]

One can check that

[TABLE]

i.e. $D$ is skew-adjoint in the inner product $[\cdot,\cdot]$. For any $u\in Dom(D)=H^{1}(\mathbb{T}_{2\pi})$, denote $v=(\lambda+D)u\in L^{2}(\mathbb{T}_{2\pi})$, then one has

[TABLE]

Also, by the Schwartz inequality, one has

[TABLE]

It follows that

[TABLE]

where $\|\cdot\|_{{\bar{L}^{2}(\mathbb{T}_{2\pi})}}:=[\cdot,\cdot]^{1/2}$. Note that $c-\|f^{\prime}(u_{c})\|_{L^{\infty}(\mathbb{T}_{2\pi})}\geq\delta_{0}>0$, so $\bar{L}^{2}(\mathbb{T}_{2\pi})\sim L^{2}(\mathbb{T}_{2\pi})$. Thus, $\lambda+D$ is invertible and $(\lambda+D)^{-1}$ is bounded from $L^{2}(\mathbb{T}_{2\pi})$ to $L^{2}(\mathbb{T}_{2\pi})$. Taking the inner product of the equation $v=(\lambda+D)u$ with $Du$, we obtain

[TABLE]

which implies that $\|Du\|_{L^{2}(\mathbb{T}_{2\pi})}\leq C\|v\|_{L^{2}(\mathbb{T}_{2\pi})}$ for some constant $C$. Hence, we have that $(\lambda+D)^{-1}$ is bounded from $L^{2}(\mathbb{T}_{2\pi})$ to $H^{1}(\mathbb{T}_{2\pi})$, from which it follows immediately that $(\lambda+D)^{-1}D$ is bounded from $H^{1}(\mathbb{T}_{2\pi})$ to $H^{1}(\mathbb{T}_{2\pi})$. Also, $(\lambda+D)^{-1}D=I-\lambda(\lambda+D)^{-1}$ is bounded from $L^{2}$ to $L^{2}$.

We now prove (2.4) by induction. Suppose it is true for $0\leq s\leq l$. Let $\psi=(\lambda+D)^{-1}D\phi$, then

[TABLE]

From

[TABLE]

we get

[TABLE]

It is easy to check the following commutator estimate

[TABLE]

Therefore,

[TABLE]

Also,

[TABLE]

Thus, we obtain

[TABLE]

by the induction assumption. This finishes the proof of (2.4).

Step 2: From (2.2), one has

[TABLE]

Let $B=(\lambda+D)^{-1}D(c-f^{\prime}(u_{c}))^{-1}$, then we have $v_{k}(x)=B\mathcal{M}_{k}v_{k}(x)$. Since $f\left(u_{c}\right)\in C^{\infty}$ and $c-f^{\prime}(u_{c})\geqslant\delta_{0}>0$, by (2.4) $B$ is bounded from $H^{s}(\mathbb{T}_{2\pi})$ to $H^{s}(\mathbb{T}_{2\pi})$, for any integer $s\geq 0$. By using the interpolation theory, $B$ is bounded from $H^{s}(\mathbb{T}_{2\pi})$ to $H^{s}(\mathbb{T}_{2\pi})$ for any $s\geq 0$. So we have

[TABLE]

Repeatedly using the identity $v_{k}(x)=B\mathcal{M}_{k}v_{k}(x)$, we arrive at

[TABLE]

which implies that

[TABLE]

Since $n\in\mathbb{N}$ is arbitrary, this finishes the proof of the lemma.

3 Semigroup estimates

In this section, we consider semigroup estimates for $e^{tJL}$ (equivalently, for the linearized equation (1.6)), which will be used in later sections to prove nonlinear instability. First, we consider the estimates in both multi-periodic space H^{s}\left({\mathbb{T}_{2\pi q}}\right)\and localized spaces $H^{s}\left(\mathbf{R}\right)$, with $s\geq\frac{m}{2}$. Such estimates are given in Section 11.4 of [27] and we only sketch it here. It is obtained by using the theory in ([27]) for general linear Hamiltonian PDEs which we describe below. Consider a linear Hamiltonian system

[TABLE]

where $X$ is a Hilbert space. Assume that:

(H1) $J:X^{\ast}\rightarrow X$ is a skew-adjoint operator.

(H2) The operator $L:X\rightarrow X^{\ast}$ generates a bounded bilinear symmetric form $\left\langle L\cdot,\cdot\right\rangle$ on $X$.  There exists a decomposition $X=X_{-}\oplus\ker L\oplus X_{+}$ satisfying that $\left\langle L\cdot,\cdot\right\rangle|_{X_{-}}<0,\ \dim X_{-}=n^{-}\left(L\right)<\infty$, and there exists $\delta_{1}>0$ such that

[TABLE]

(H3) The above $X_{\pm}$ satisfy

[TABLE]

where $i_{X_{+}\oplus X_{-}}^{\ast}:X^{\ast}\rightarrow(X_{+}\oplus X_{-})^{\ast}$ is the dual operator of the embedding $i_{X_{+}\oplus X_{-}}$.

The assumption (H3) is automatically satisfied when $\dim\ker L<\infty$, as in the case of this paper.

Theorem 3.1

[27]** Under assumptions (H1)-(H3), $JL$ generates a $C^{0}$ group $e^{tJL}$ of bounded linear operators on $X$ and there exists a decomposition

[TABLE]

*satisfying:

i) $E^{u},E^{s}$ and $E^{c}$ are invariant under $e^{tJL}$;

ii) $E^{c}=\{u\in X\mid\langle Lu,v\rangle=0,\ \forall v\in E^{s}\oplus E^{u}\}$;

iii) let $\lambda_{u}=\max\{\operatorname{Re}\lambda\mid\lambda\in\sigma(JL|_{E^{u}})\}$, there exist $M>0$ and an integer $k_{0}\geq 0$, such that*

[TABLE]

[TABLE]

and

[TABLE]

Moreover, for $k\geq 1$, define the space $X^{k}\subset X$ to be

[TABLE]

and

[TABLE]

Assume $E^{u,s}\subset X^{k}$, then the exponential trichotomy for $X^{k}$ holds true: $X^{k}$ is decomposed as a direct sum

[TABLE]

and the estimates (3.1) and (3.2) still hold in the norm $X^{k}$.

By using above Theorem, we can prove the following estimates for the linearized equation (1.6).

Lemma 3.1

Consider the semigroup $e^{tJL}$ associated with the solutions of (1.6), where $J,L$ are given in (1.7).

i) (KDV type) Assume (1.2), the exponential trichotomy in the sense of (3.1) and (3.2) holds true in the spaces $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ $\left(s\geq\frac{m}{2},q\in\mathbb{N}\right)$

ii) (Whitham type) Assume (1.3) and (2.3), then the exponential trichotomy of $e^{tJL}$ holds true in the spaces $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ $\left(s\geq 0,q\in\mathbb{N}\right).$

Proof. It suffices to check the assumption (H2) in Theorem 3.1, since (H1) is obvious and (H3) is automatic due to $\dim\ker L<\infty$.

For i), the quadratic form $\left\langle L\cdot,\cdot\right\rangle$ is bounded in the space $H^{\frac{m}{2}}({\mathbb{T}_{2\pi q}})$. The operator $L$ is a compact perturbation of $\mathcal{M}$, whose spectrum in $H^{m}\left({\mathbb{T}_{2\pi q}}\right)$ are positive and discrete. Therefore, $L$ has at most a finite number of negative eigenvalues, that is, $n^{-}\left(L\right)<\infty$. Thus, the exponential trichotomy of $e^{tJL}$ is true in $H^{\frac{m}{2}}({\mathbb{T}_{2\pi q}})$. By the proof of Lemma 2.1, any stable or unstable eigenfunction of $JL$ in $L^{2}({\mathbb{T}_{2\pi q}})$ lies in $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ for any $s>0$. Therefore, the exponential trichotomy of $e^{tJL}$ is also true in $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ for any $s\geq\frac{m}{2}$.

For ii), the quadratic form $\left\langle L\cdot,\cdot\right\rangle$ is bounded in the space $L^{2}({\mathbb{T}_{2\pi q}})$. Under the condition (2.3), the operator $-L$ is a compact perturbation of the positive operator $c-f^{\prime}\left(u_{c}\right),$ thus $n^{-}\left(-L\right)<\infty$. Applying Theorem 3.1 to $JL=\left(-J\right)\left(-L\right)$, we get the exponential trichotomy of $e^{tJL}$ in $L^{2}({\mathbb{T}_{2\pi q}})$ and in $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ $\left(s\geq 0\right)$ by the regularity of stable and unstable eigenfunctions of $JL$ in $L^{2}({\mathbb{T}_{2\pi q}})$ as in Lemma 2.2.

As an immediate corollary of the above lemma, we get the following upper bound on the growth of the semigroup $e^{tJL}$.

Corollary 3.1

Let $\lambda_{0}$ be the growth rate of the most unstable eigenvalue of $JL$. Then under the conditions of Lemma 3.1, for any $\varepsilon>0$, there exists constant $C_{\varepsilon}$ such that

[TABLE]

where $q\in\mathbb{N}$ , $s\geq s_{0}$ with $s_{0}=\frac{m}{2}$ for case i) and $s_{0}=0$ for case ii).

The above semigroup estimate implies the following lemma for the inhomogeneous equation. For convenience, we use $\mathbb{T}$ for $\mathbb{T}_{2\pi q}$.

Lemma 3.2

If $\|g\left(t\right)\|_{H^{s}(\mathbb{T})}\leqslant C_{g}e^{wt}$, for some $s\geq s_{0}$ and $w>\lambda_{0}$, then the solution to the equation

[TABLE]

satisfies

[TABLE]

Proof. Using Corollary 3.1 with $\epsilon=\frac{1}{2}\left(w-\lambda_{0}\right)$, we have

[TABLE]

Then

[TABLE]

To prove nonlinear instability for localized perturbations, we need to study the semigroup $e^{tJL}$ on the space $H^{s}\left(\mathbf{R}\right)$ $\left(s\geq\frac{m}{2}\right)$. In general, the operator $L$ has negative continuous spectra in $H^{s}\left(\mathbf{R}\right)$. For example, when $\mathcal{M=-\partial}_{x}^{2}$, the spectrum of $L=$ $\mathcal{-\partial}_{x}^{2}$ $+V\left(x\right)$ with a periodic potential $V\left(x\right)$ is well studied in the literature and is known to have bands of continuous spectrum. So Theorem 3.1 does not apply. However, we can prove upper bound estimate of $e^{tJL}$ on $H^{s}\left(\mathbf{R}\right)$, which suffices for proving nonlinear localized instability. We will need the following lemma to estimate $e^{tJL}$ on $H^{s}\left(\mathbf{R}\right)$.

Lemma 3.3

Suppose $h(k,x)\in H_{x}^{s}(\mathbb{T})$ for any $k\in I$, where $I$ is a measurable set contained in an interval with length less than or equal to $1$, then $\int_{I}h(k,x)e^{ikx}\,dk\in H_{x}^{s}(\mathbf{R})$ if and only if $\|h(k,x)\|_{H_{x}^{s}(\mathbb{T})}\in L_{k}^{2}\left(I\right)$. More precisely, there exist constants $C_{1}(s),C_{2}(s)>0$, such that

[TABLE]

and

[TABLE]

Proof. First, we write $h(k,x)$ as a Fourier series

[TABLE]

By direct computations, we have

[TABLE]

where

[TABLE]

Note that I\is contained in an interval with length no more than $1$, therefore I_{j}\are disjoint with each other, which implies that \mathcal{X}_{I_{j_{1}}}\mathcal{X}_{I_{j_{2}}}=0\almost everywhere, if $j_{1}\neq j_{2}$. Then by Parseval’s identity, we have

[TABLE]

Then the desired results follow directly.

Now, we are ready to prove the upper bound estimate of $e^{tJL}$ on $H^{s}\left(\mathbf{R}\right)$. The following semigroup estimates were proved in [27] for the “differential” case (1.2).

Lemma 3.4

[27*]*Assume (1.2) and

[TABLE]

Let $\lambda_{0}\geq 0$ be such that

[TABLE]

Then for every $s\geq\frac{m}{2}$, \varepsilon>0\there exist $C(s,\varepsilon)>0$ such that

[TABLE]

for any $u\in H^{s}{(\mathbf{R})}$.

Remark 3.1

The assumption (3.4) can be replaced by a weaker assumption

[TABLE]

For the Whitham type equation, we have the following similar result.

Lemma 3.5

Assume (1.3) and (2.3), then for every $s\geq 0$, \varepsilon>0\there exists $C(s,\varepsilon)>0$ such that

[TABLE]

for any $u\in H^{s}{(\mathbf{R})}$. Here, $\lambda_{0}$ is the largest growth rate as defined in (3.5).

Proof. The proof is similar to that of Lemma 3.4 (or Lemma 11.2 in [27]). We sketch it here. First, for any $u\in H^{s}{(\mathbf{R)}}$,

[TABLE]

and $\hat{u}$ is the Fourier transform of $u$. By Lemma 3.3, there exists $C>0$ such that

[TABLE]

Note that

[TABLE]

and thus

[TABLE]

So to prove (3.6), it suffices to show that: for any $\varepsilon,s\geq 0$, there exists $C(s,\varepsilon)>0$ such that

[TABLE]

It suffices to prove the lemma for $s=0$ since the estimates for general $s\geq 0$ can be obtained by applying $J_{\xi}L_{\xi}$ repeatedly to the estimates for $s=0$ (and interpolation for the case when $s$ is not an integer). Due to the compactness of $[0,1]$, it suffices to prove that for any $\xi_{0}\in[0,1]$, there exist $C,\epsilon>0$ such that (3.9) holds for $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. We first note that each $\lambda\in\sigma(J_{\xi_{0}}L_{\xi_{0}})$ is an isolated eigenvalue with finite algebraic multiplicity and $L_{\xi_{0}}$ is non-degenerate on $E_{\lambda}$ when $\lambda\neq 0$ and on $E_{0}/(E_{0}\cap\ker L_{\xi_{0}})$, where $E_{\lambda}$ is the generalized eigenspace of the eigenvalue $\lambda$ of $J_{\xi_{0}}L_{\xi_{0}}$. By (2.3), $n^{-}\left(-L_{\xi}\right)<\infty$. Let

[TABLE]

By the instability index formula (Proposition 11.2 in [27]), $\sigma(J_{\xi_{0}}L_{\xi_{0}})\backslash\Lambda$ is finite and thus

[TABLE]

Moreover, there exists $\varepsilon_{0}>0$ such that $\Omega\cap\Lambda=\emptyset$, where

[TABLE]

Assuming that:

[TABLE]

we now prove (3.9) for $\xi$ in a small interval near $\xi_{0}$. Indeed, by (3.10), there exists $\epsilon>0$ such that $\partial\Omega\cap\sigma(J_{\xi}L_{\xi})=\emptyset$ for any $\xi\in[\xi_{0}-\epsilon,\xi_{0}+\epsilon]$. For such $\xi$, let

[TABLE]

which are continuous in $\xi$ and invariant under $e^{tJ_{\xi}L_{\xi}}$. Therefore $\dim Z_{\xi}=n$. By the definition of $\Omega$, we know that $-L_{\xi_{0}}|_{Y_{\xi_{0}}}$ is positive definite. Then the continuity of $L_{\xi}$ in $\xi$ implies that there exists $\delta_{0}>0$ such that

[TABLE]

So for any $\xi\in[\xi_{0}-\epsilon,\xi_{0}+\epsilon]$, there exists a generic constant $C>0$ independent of $\xi$, such that for any $v\in L^{2}\left(\mathbb{T}_{2\pi}\right)$,

[TABLE]

Along with the compactness of $[0,1]$, this implies estimates (3.9) and (3.6).

It remains to prove (3.10) about the continuity of the resolvent. Fix $k\in[0,1]$. For $k^{\prime}$ near $k$, we have

[TABLE]

Let $D=(\partial_{x}+ik)\left(c-f^{\prime}(u_{c})\right)$, then by (2.3) and the proof of Lemma 2.2, for any $a_{0}>0$, $\left(a_{0}+D\right)^{-1}:L^{2}\rightarrow H^{1}$ is bounded. So

[TABLE]

Moreover,

[TABLE]

is compact in $L^{2}$. Therefore $A=\left(a_{0}+D\right)^{-1}(\lambda-J_{k}L_{k})$ is a Fredholm operator of index [math]. Suppose $\lambda\notin\sigma(J_{k}L_{k})$, then $A$ is injective and thus $A^{-1}$ is bounded on $L^{2}$. Along with (3.11), we obtain

[TABLE]

as $k^{\prime}\rightarrow k$. From

[TABLE]

we obtain the continuity of the resolvent $(\lambda-J_{k}L_{k})^{-1}$ in $k\in[0,1]$. This finishes the proof of the lemma.

The following is an analogue of Lemma 3.2.

Lemma 3.6

If

[TABLE]

for some $b>0$, $s\geq s_{0}$ and $w>\lambda_{0}$, then the solution to the equation

[TABLE]

satisfies

[TABLE]

Proof. Choose $\epsilon=\frac{1}{2}\left(w-\lambda_{0}\right)$ in Lemma 3.5, then $\left\|e^{tJL}\right\|_{H^{s}}\lesssim e^{\frac{1}{2}\left(\lambda_{0}+w\right)t}$. So we have

[TABLE]

since $w>\lambda_{0}$.

Lastly, we prove the semigroup estimates in the space $H^{-1}$, which will be used in the proof of nonlinear instability by bootstrap arguments. First, we consider the estimates for periodic perturbations.

Lemma 3.7

Consider the semigroup $e^{tJL}$ associated with the solutions of (1.6), where $J,L$ are given in (1.7). Assume (1.2) or (1.3) and (2.3), then for any \varepsilon>0\there exist $C(\varepsilon)>0$ such that

[TABLE]

for any $u\in H^{-1}{(\mathbb{T}_{2\pi q})}$.

Proof. Since $\left(JL\right)^{\ast}=-LJ$, by duality it suffices to show that

[TABLE]

Denote $P^{0}$ and $P^{1}=1-P^{0}$ to be the projection operators to $\ker L$ and $\left(\ker L\right)^{\perp}=R\left(L\right)$ respectively. For any $v\in H^{1}{(\mathbb{T}_{2\pi q})}$, let $v=P^{0}v+P^{1}v=v_{1}+v_{2}.$ Then the equation $\partial_{t}v=LJv$ can be written as

[TABLE]

Since $L_{1}=L|_{R\left(L\right)}:R\left(L\right)\rightarrow R\left(L\right)$ has a bounded inverse and

[TABLE]

by Lemma 3.1 we have

[TABLE]

for the case of (1.2), and

[TABLE]

for the case of (1.3) and (2.3). By (3.13), we have $\left\|v_{1}\left(t\right)\right\|_{H^{1}}=\left\|v_{1}\left(0\right)\right\|_{H^{1}}$ and

[TABLE]

which implies (3.12) and the lemma.

In the next lemma, we consider localized perturbations.

Lemma 3.8

Consider the semigroup $e^{tJL}$ associated with the solutions of (1.6), where $J,L$ are given in (1.7). Assume (1.2) or (1.3) and (2.3), then for any \varepsilon>0\there exist $C(\varepsilon)>0$ such that

[TABLE]

for any $u\in H^{-1}{(}\mathbf{R}{)}$.

Proof. By duality, it suffices to show that

[TABLE]

As in the proof of Lemmas 3.4 and 3.5, it is enough to show that for any $\varepsilon>0$, there exists $C\left(\varepsilon\right)>0$ such that

[TABLE]

is true for any $\xi\in[0,1]$ and $u\in H^{1}(\mathbb{T}_{2\pi})$. By compactness of $\left[0,1\right]$, again it suffices to prove that for any $\xi_{0}\in[0,1]$, there exist $C,\epsilon>0$ such that (3.14) holds for $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. We consider two cases below.

Case 1 ($L_{\xi_{0}}$ is invertible): In this case, there exists $\epsilon>0$ such that $L_{\xi}$ is invertible for $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. So we have

[TABLE]

for the case of (1.2), and

[TABLE]

for the case of (1.3) and (2.3). In the above, we use the estimate (3.9) which is true for both cases of (1.2) and (1.3)-(2.3).

Case 2 ($L_{\xi_{0}}$ is not invertible): In this case, $\ker L_{\xi_{0}}\neq\left\{0\right\}$. It is possible that $L_{\xi}$ is invertible for $\xi$ near $\xi_{0}$. For example, when $\mathcal{M=-\partial}_{x}^{2}$, it was shown in Remark 11.1 of [27] that $L_{\xi}$ has zero eigenvalue if and only if $\xi=0,1$. However, for $\xi$ near $\xi_{0}$, there is no uniform (in $\xi$) estimate for $L_{\xi}^{-1}$ and we cannot argue as in Case 1. We will separate the eigenspaces of $L_{\xi}$ (\xi\near $\xi_{0}$) for eigenvalues near [math] and away [math]. Since [math] is an isolated eigenvalue of $L_{\xi_{0}}$, so

[TABLE]

Let $\epsilon>0$ be small enough such that when $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$,

[TABLE]

Denote $P_{\xi}^{0}=\oint_{\Gamma}\left(z-L_{\xi}\right)^{-1}dz$ to be the Riesz projection associated with the eigenvalues of $L_{\xi}$ inside $\Gamma,$ and $P_{\xi}^{1}=1-P_{\xi}^{0}$. In particular, $P_{\xi_{0}}^{0},\ P_{\xi_{0}}^{1}$ are the projection operators to $\ker L_{\xi_{0}}$ and $R\left(L_{\xi_{0}}\right)$ respectively. By choosing $\epsilon$ small, we can assume that: $\dim R\left(P_{\xi}^{0}\right)=\dim\ker L_{\xi_{0}},$

[TABLE]

and

[TABLE]

with $a\left(\epsilon\right)\rightarrow 0$ when $\epsilon\rightarrow 0$. Denote

[TABLE]

and

[TABLE]

It is easy to show that $E_{1}^{\xi}$ can be written as a graph of a $O\left(\epsilon\right)$-bounded operator $S_{\xi}:E_{1}\rightarrow E_{0}$. That is, let $\tilde{S}_{\xi}=I+S_{\xi}$, then $E_{1}^{\xi}=\tilde{S}_{\xi}\left(E_{1}\right)$. For any $u\in H^{1}(\mathbb{T}_{2\pi})$, let

[TABLE]

then the equation $\partial_{t}u=L_{\xi}J_{\xi}u$ becomes

[TABLE]

[TABLE]

We will show that: For any \varepsilon>0\there exist $C(\varepsilon),$ \epsilon>0\such that

[TABLE]

holds for $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. Assuming (3.17), we now show (3.14) for $\xi$ $\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. First, by (3.16) we have

[TABLE]

Since the operator $P_{\xi}^{0}L_{\xi}J_{\xi}$ is finite ranked and $\left\|P_{\xi}^{0}L_{\xi}J_{\xi}\right\|_{H^{1}}\leq Ca\left(\epsilon\right)$ for some constant $C$, so from (3.15) we have

[TABLE]

We choose $\epsilon$ small enough such that $Ca\left(\epsilon\right)<\frac{\varepsilon}{2}$. Plugging above into (3.18), we get

[TABLE]

where $C^{\prime},C^{\prime\prime}$ are some constants independent of $\epsilon$. Define

[TABLE]

then above inequality becomes

[TABLE]

Choose $\epsilon$ further small such that $Ca\left(\epsilon\right)C(\varepsilon)<\frac{\varepsilon}{2}$. Then by Gronwall’s inequality, we have

[TABLE]

that is,

[TABLE]

Plugging above estimate into (3.19), we also get

[TABLE]

Combining above, we have

[TABLE]

and thus (3.14) is proved. It remains to prove (3.17). Since

[TABLE]

and

[TABLE]

to prove (3.17) it suffices to show that there exist $C(\varepsilon),$ \epsilon>0\such that

[TABLE]

for $\xi\in(\xi_{0}-\epsilon,\xi_{0}+\epsilon)$. Again, it is enough to estimate $e^{tP_{\xi}^{1}J_{\xi}L_{\xi}|_{E_{1}^{\xi}}}$ on the energy space $H^{\frac{m}{2}}$ and then apply $P_{\xi}^{1}J_{\xi}L_{\xi}|_{E_{1}^{\xi}}$ repeatedly (and by interpolation) to get the estimates for $s>\frac{m}{2}$. We will study the semigroup generated by $P_{\xi}^{1}J_{\xi}L_{\xi}|_{E_{1}^{\xi}}$ on H^{\frac{m}{2}}\via the perturbation of the semigroup generated by $P_{\xi_{0}}^{1}J_{\xi_{0}}L_{\xi_{0}}|_{E_{1}}$. First, we use the transform $\tilde{S}_{\xi}:E_{1}\rightarrow E_{1}^{\xi}$ to study the conjugated operators on the same space $E_{1}$. Notice that $\left(\tilde{S}_{\xi}\right)^{-1}:E_{1}^{\xi}\rightarrow E_{1}$ is exactly the projection operator $P_{\xi_{0}}^{1}$. Therefore the $\tilde{S}_{\xi}-$conjugated operator can be written in a Hamiltonian form

[TABLE]

where

[TABLE]

and

[TABLE]

are anti-selfadjoint and self-adjoint respectively. We also write

[TABLE]

where

[TABLE]

We note that the spectrum of $\tilde{J}_{\xi}\tilde{L}_{\xi}$ is discrete, $n^{-}\left(\tilde{L}_{\xi}\right)\leq n^{-}\left(L_{\xi}\right)<\infty$. Moreover, the maximal growth rate of the eigenvalues of P_{\xi_{0}}^{1}J_{\xi_{0}}L_{\xi_{0}}$$|$${}_{E^{1}} is still $\lambda_{0}$. Therefore by the similar proof as in Lemma 3.5 or Lemma 11.2 in [27], to prove the estimate (3.20) in $H^{\frac{m}{2}}$, it suffices to show that the resolvent (\lambda-\tilde{J}_{\xi}\tilde{L}_{\xi})^{-1}\is continuous for $\xi$ near $\xi_{0}$. We have

[TABLE]

In the above,

[TABLE]

since $J_{\xi}-J_{\xi_{0}}=O\left(\left|\xi-\xi_{0}\right|\right),$

[TABLE]

Also,

[TABLE]

where

[TABLE]

Thus by similar arguments as in the proof of Lemma 3.5 or Lemma 11.2 in [27], we can show the continuity of the resolvent (\lambda-\tilde{J}_{\xi}\tilde{L}_{\xi})^{-1}\for $\xi$ near $\xi_{0}$. This finishes the proof of the Lemma.

4 Nonlinear Modulational Instability (multi-periodic)

In this section, we prove that linearly modulationally unstable traveling waves are nonlinearly orbitally unstable under multi-periodic perturbations. First, by the definition (1.1) of linear modulational instability and the remark thereafter, there exists an interval $I_{0}\subset\left[0,1\right]$ such that for any $k\in I_{0}$, there exists an unstable solution $e^{\lambda\left(k\right)t}e^{ikx}v_{k}\left(x\right)$ with $\operatorname{Re}\lambda\left(k\right)>0$ and $2\pi-$periodic $v_{k}\left(x\right)$ to the linearized equation (1.6). So we can pick an rational number $k_{0}=\frac{p}{q}\in I_{0}$ with $p,q\in\mathbb{N}$. Then $e^{ik_{0}x}v_{k_{0}}\left(x\right)$ is a $2\pi q$-periodic unstable eigenfunction to the operator $JL$ in $L^{2}{(\mathbb{T}_{2\pi q})}$. It leads us to consider the nonlinear instability of $u_{c}$ in ${L^{2}(\mathbb{T}_{2\pi q})}$.

The proof of Theorem 1.1 i) uses the strategy in [12], by constructing higher order approximation solutions and then using the energy estimates to overcome the loss of derivative.

The following energy estimate will be used in the proof later. We use $\mathbb{T}$ for ${\mathbb{T}_{2\pi q}}$ below.

Lemma 4.1

Consider the solution of the following equation

[TABLE]

[TABLE]

where $U\left(t,\cdot\right)\in H^{4}(\mathbb{T})$ and $R\left(t,\cdot\right)\in H^{2}(\mathbb{T})$ are given and $f\in C^{\infty}(\mathbf{R})$.  Assume that

[TABLE]

then there exists a constant $C\left(\beta\right)$ such that for $0\leq t\leq T,$

[TABLE]

Proof. We write

[TABLE]

First, taking the inner product of (4.1) with $v$ and integrating by parts, we have

[TABLE]

where in the above we use the fact that $\partial_{x}\mathcal{M}$ is anti-selfadjoint and

[TABLE]

Thus

[TABLE]

Next, applying $\partial_{x}^{2}$ to (4.1) and then taking the inner product with $\partial_{x}^{2}v$, we get

[TABLE]

By direct computation and integration by parts, we can show that for $0<t\leq T$, there exists a constant $C(\beta)$, such that

[TABLE]

We only sketch the estimates of the terms involving $\partial_{x}^{3}v$. One such term is

[TABLE]

and another term

[TABLE]

can be handled similarly. Thus by (4.4), we have

[TABLE]

and combined with (4.3) this proves (4.2).

Now we are ready to prove nonlinear modulational instability for multi-periodic perturbations.

Proof of Theorem 1.1 i). Let v_{g}\left(x\right)\be the eigenfunction associated with the most unstable eigenvalue \lambda\of JL\in $L^{2}\left(\mathbb{T}\right)$. By Lemmas 2.1 and 2.2, v_{g}\in H^{s}\left(\mathbb{T}\right)\for any $s\geq 0$. We construct an approximate solution U^{app}\to (1.5) of the form

[TABLE]

where

[TABLE]

is the most rapidly growing real-valued $2\pi q$-periodic solution of the linearized equation (1.6). The integer $N$ is chosen such that $\left(N+1\right)\operatorname{Re}\lambda>C\left(1\right)$, where the constant $C\left(1\right)$ is the one in the energy estimate (4.2) with $\beta=1$.

Now we construct the terms $U_{2},\cdots,U_{N}$. By the Taylor expansion formula,

[TABLE]

Since $u_{c}$ is a stationary solution to (1.5) and $U_{1}$ satisfies the linearized equation

[TABLE]

by using (4.7) we have

[TABLE]

where

[TABLE]

and $P_{j},\ Q_{j}$ are polynomials of U_{1},\cdots,U_{N}\with degree j\such that

[TABLE]

For $j=2,\cdots,N$, we define $U_{j}$ be the solution of

[TABLE]

Now we estimate $U_{j}$ for $j\geq 2$. First, by Lemmas 2.1 and 2.2, one has

[TABLE]

where $l=s+N$. By (4.8), $U_{2}$ satisfies the equation

[TABLE]

where $P_{2}(U_{1})=\frac{1}{2}f^{\prime\prime}(u_{c})U_{1}^{2}$. By (4.9), we have

[TABLE]

Then, it follows from Lemma 3.2 that

[TABLE]

By induction, for each $2<j\leqslant N$, we have

[TABLE]

and then by Lemma 3.2

[TABLE]

Therefore, there exists a constant $C(N,s)$, such that

[TABLE]

By the construction of $U^{app}$, we have

[TABLE]

where

[TABLE]

Let $0<\theta<1$ to be determined and define $T^{\delta}$ by $\delta e^{\operatorname{Re}\lambda T^{\delta}}=\theta$. Then $T^{\delta}=O\left(\left|\ln\delta\right|\right)$. Choose $s\geq 4$ and recall that $l-N=s$. Then by (4.11), for any $N+1\leq j\leq N^{N}$, we have

[TABLE]

and thus by (4.13)

[TABLE]

Let $U_{\delta}(t,x)$ be the solution to (1.5) with initial value $u_{c}(x)+\delta U_{1}(0,x)$, and let $v=U_{\delta}-U^{app}$. Then by using (4.12), one finds that $v$ satisfies the equation

[TABLE]

Define $T_{1}$ to be the maximal time such that

[TABLE]

We claim that $T_{1}>T^{\delta}$ when $\theta$ is chosen to be small enough. Suppose otherwise, $T_{1}\leq T^{\delta}$. Then for $0\leq t\leq T_{1}$, we have

[TABLE]

when$\ \theta$ is small. Thus we have

[TABLE]

By using Lemma 4.1 for the equation (4.15), we have

[TABLE]

Recall that $\left(N+1\right)\operatorname{Re}\lambda>C\left(1\right)$. So by using (4.14) and the Gronwall’s inequality, we obtain from (4.16) that for $0\leq t\leq T_{1},$

[TABLE]

Thus

[TABLE]

when $\theta$ is small. This is in contradiction to the definition of $T_{1}$ and the claim is proved. Moreover, for $0\leq t\leq T^{\delta}<T_{1}$, when $\theta$ is small enough the estimate (4.17) is true by above arguments. So there exist $C_{1},C_{2}>0$ such that

[TABLE]

when \theta\is small enough.

It remains to show that above nonlinear instability is also true in the orbital distance. This can be done by using the argument in ([13]). By the previous estimates, there exists a constant $\widetilde{C}$, such that

[TABLE]

where \widetilde{C}\may depend on $\theta$, but is independent of $\delta$. Denote

[TABLE]

then it is easy to see that for any $s\geq 0$, there exist two constants c_{1}\left(s\right),c_{2}\left(s\right)>0\such that

[TABLE]

Let V_{1}^{\bot}(t,x)\be the projection of V_{1}(t,x)\into Z^{\bot}\in the L^{2}\inner product, where

[TABLE]

Let h(t)\be such that

[TABLE]

Then for $0<t\leqslant T^{\delta}$, we have

[TABLE]

which implies $|h(t)|=O(\theta)$. So we can write

[TABLE]

This implies that

[TABLE]

for some $c_{0}>0$, when \theta\is small enough. On the other hand, we have

[TABLE]

which implies that

[TABLE]

for some $C^{\prime}>0$. This finishes the proof of Theorem 1.1 i).

5 Localized Nonlinear Modulational Instability

In this section, we prove nonlinear instability for localized perturbations. Since the linearized operator JL\(defined in (1.7)) does not have an unstable eigenvalue in $H^{s}\left(\mathbf{R}\right)$, we will construct unstable initial data in the form of a wave package of unstable eigenfunctions of J_{k}L_{k}\where k\is near the most unstable frequency $k_{0}$. Without loss of generality, we can assume that $k_{0}\in\left[0,\frac{1}{2}\right]$. Indeed, if k\in\left[0,1\right]\is an unstable frequency in the sense that J_{k}L_{k}\has an unstable eigenvalue, then -k,1-k\are also unstable frequencies. So we can always pick k_{0}\in\left[0,\frac{1}{2}\right]\such that J_{k_{0}}L_{k_{0}}\has the most unstable eigenvalue $\lambda\left(k_{0}\right)$. More precisely, for any $k\in\left[0,1\right]$, if J_{k}L_{k}\has an unstable eigenvalue \lambda\then $\operatorname{Re}\lambda\leq\operatorname{Re}\lambda\left(k_{0}\right)$. To construct the unstable wave package, we choose a small interval I\subset\left[0,\frac{1}{2}\right]\and I\is near $k_{0}$. If \left|I\right|\is small enough, then any k\in I\is still an unstable frequency since J_{k}L_{k}\depends on k\smoothly. In the case when \lambda_{k_{0}}\is a simple eigenvalue of $J_{k_{0}}L_{k_{0}}$, then by the analytic perturbation theory ([24]) of linear operators, there is a smooth curve of unstable eigenvalue \lambda\left(k\right)\of $J_{k}L_{k}$, with $k\in I$. Since \operatorname{Re}\lambda(k)\is smooth in the vicinity of $k_{0}$, and \operatorname{Re}\lambda\left(k\right)\obtains its maximum at $k_{0}$, there exists an even number $l\geqslant 2$, such that

[TABLE]

Now consider the general case when $\lambda_{k_{0}}$ is a multiple eigenvalue of $J_{k_{0}}L_{k_{0}}$. Since the eigenvalues of $J_{k}L_{k}$ are all discrete, we can use the analytic perturbation theory ([24]) of eigenvalues of matrices to study the eigenvalues of $J_{k}L_{k}$ near $k_{0}$. In this case, the eigenvalues of $J_{k}L_{k}$ near $k_{0}$ can be grouped in the manner

[TABLE]

such that each group constitutes a branch of an analytic function (defined near $k_{0}$) with a branch point (if $p_{i}\geq 2$) at $k=k_{0}$. Assume $p_{1}\geq 2$, then we have the following Puiseux series (see p. 65 of [24]) for the first group $\left\{\lambda_{1}\left(k\right),\cdots,\lambda_{p_{1}}\left(k\right)\right\}$

[TABLE]

where $\omega=\exp\left(2\pi i/p_{1}\right)$ and $h=0,1,\cdots,p_{1}-1$. In the next lemma, we show that the leading order term of $\lambda_{h+1}\left(k\right)$ in (5.2) is still given by $\left(k-k_{0}\right)^{l}$ for an even integer $l$.

Lemma 5.1

Let $p_{1}\geq 2$, consider the Puiseux series (5.2) near $k_{0}$. If

[TABLE]

for $k$ in a neighborhood of $k_{0}$, then there exists an even integer $l$ such that

[TABLE]

Proof. Let $m_{n}$ be the first coefficient in (5.2) such that $\operatorname{Re}m_{n}\neq 0$. Then by (5.3), we have

[TABLE]

This implies that:

[TABLE]

and

[TABLE]

for $h=0,1,\cdots,p_{1}-1$. So

[TABLE]

If $n/p_{1}$ is not an integer, then we must have $m_{n}=0$. Since otherwise if $m_{n}\neq 0$, it is clearly impossible for all the $2p_{1}$ points

[TABLE]

to stay in the left half complex plane when $n/p_{1}$ is not an integer. If $n/p_{1}$ is odd, then for (5.4) to hold true we must have $\operatorname{Re}m_{n}=0$. So for $\operatorname{Re}m_{n}\neq 0$, we must have $n/p_{1}=l$ to be even. In this case, (5.4) implies that $\operatorname{Re}m_{lp_{1}}<0.$

Let $I\subset\left[0,\frac{1}{2}\right]$ be a small interval with $k_{0}$ being its right end point. Let $\lambda\left(k\right),\ k\in I$ be a curve of unstable eigenvalues of $J_{k}L_{k}$ ending on the right at $\lambda\left(k_{0}\right)$, as determined by one of the functions in (5.2) when $\lambda\left(k_{0}\right)$ is a multiple eigenvalue. Then by (5.1) when $\lambda\left(k_{0}\right)$ is simple or by Lemma 5.1 when $\lambda\left(k_{0}\right)$ is multiple, we have

[TABLE]

where $a_{0}<0$ and $l$ is even. Let $v_{1}(k,x)$ be the corresponding eigenfunction of $\lambda\left(k\right)$ for $J_{k}L_{k}$, which depends on $k$ continuously. By Lemmas 2.1 and 2.2, $v_{1}(k,x)\in H_{x}^{s}\left(\mathbb{T}\right)$ for any $s\geq 0$ when $f$ is smooth.

Define the following wave packet consisting of unstable eigenfunctions with frequencies in $I$,

[TABLE]

Since $I\cup-I\subset\left[-\frac{1}{2},\frac{1}{2}\right]$, so by Lemma 3.3,

[TABLE]

We will choose initial data $U_{\delta}\left(0\right)=u_{c}+\delta u_{1}$ to show nonlinear localized instability. First, we follow the arguments in Section 8.5 of [30] to prove the well-posedness of (1.5) in the space $u_{c}+H^{s}\left(R\right)$. The arguments can be also found in [25] [23].

Lemma 5.2** (Well Posedness)**

Assuming that \mathcal{M}\in L\big{(}H^{\beta}(\mathbf{R}),L^{2}(\mathbf{R})\big{)} ($\beta$ may be negative ) and $f\in C^{s+2}\left(\mathbf{R}\right)$, where $s\geqslant\max\{1+\beta,1\}$ is an even integer. Then for every u_{0}\in B^{s}(\mathbf{R}):=\big{\{}u_{c}+w:w\in H^{s}(\mathbf{R})\big{\}}, there exists $T>0$, such that the Cauchy problem

[TABLE]

has a unique solution $u\in C([0,T],B^{s}(\mathbf{R}))\cap C^{1}([0,T],B^{0}(\mathbf{R})).$

Proof. It is equivalent to prove that the following problem

[TABLE]

has an unique solution $w\in C([0,T],H^{s}(\mathbf{R}))\cap C^{1}([0,T],L^{2}(\mathbf{R})).$

Rewrite the equation (5.7) as

[TABLE]

Let $A_{0}=-c\partial_{x}+\partial_{x}\mathcal{M}.$ It is clear that $D(A_{0})=H^{\sigma}(\mathbf{R})$, where $\sigma=\max\{1+\beta,1\}$.

For any $v\in H^{s}$ with $s\geq\sigma$, define $A_{1}(v):H^{1}(\mathbf{R})\rightarrow L^{2}(\mathbf{R})$ as

[TABLE]

Following the arguments in Section 8.5 of [30], we consider the equation

[TABLE]

where $A(v)=A_{0}+A_{1}(v)$.

Let $B_{r}$ be the ball of radius $r>0$ in $H^{\sigma}(\mathbf{R})$. According to Theorem 6.4.6 and Section 8.5 in [30], the following four conditions guarantee the well-posedness of (5.7):

(C1) There exists a constant $k$, such that if $\|w_{0}\|_{H^{s}(\mathbf{R})}\leq r$, then

[TABLE]

for every $v\in B_{r}$;

(C2) The family $A(v)$, $v\in B_{r}$ is a stable family in $L^{2}(\mathbf{R})$ (see Definition 6.4.1 in P. 200 of [30]);

(C3) There is an isomorphism of $H^{s}(\mathbf{R})$ onto $L^{2}(\mathbf{R})$ such that for every $v\in B_{r}$, $SA(v)S^{-1}-A(v)$ is a bounded operator in $L^{2}(\mathbf{R})$ and $\|SA(v)S^{-1}-A(v)\|\leqslant C_{1}$;

(C4) For each $v\in B_{r}$, $D(A(v))\supset H^{s}(\mathbf{R})$, $A(v)$ is a bounded linear operator from $H^{s}(\mathbf{R})$ into $L^{2}(\mathbf{R})$ and

[TABLE]

Since $\|w_{0}\|_{H^{s}(\mathbf{R})}<r$ and $\|v\|_{H^{s}(\mathbf{R})}<r$, it is straightforward to show that

[TABLE]

where

[TABLE]

Thus (C1) holds.

Note that $A_{0}$ is skew-adjoint, therefore one has $\langle A_{0}w,w\rangle=0$. Also, it is easy to check that

[TABLE]

Therefore A(v)\generates a C_{0}\semigroup from L^{2}(\mathbf{R})\to L^{2}(\mathbf{R})\and A(v)\is stable for $v\in B_{r}$.

Following the similar argument as in the proof of Lemma 5.5 in [30], one can verify (C3) by letting $S=\Lambda^{s}$, where \Lambda^{s}\is an operator with Fourier symbol $(1+\xi^{2})^{s/2}$. We only consider $s=2n$, where $n$ is any positive integer. It is easy to check that

[TABLE]

where $C_{k}^{n}$ is the number of $k$-combinations.

Then one can check that

[TABLE]

It follows that

[TABLE]

where

[TABLE]

Moreover, it is easy to check that

[TABLE]

Thus, (C3) holds. It is trivial to verify (C4). So we complete the proof of this lemma.

Now we are ready to show nonlinear localized instability. Let

[TABLE]

It is easy to see that U_{1}\left(t,x\right)\is a real-valued solution to (1.6) with initial data $U_{1}\left(0,x\right)=u_{1}\left(x\right)$ (defined in (5.6)). Denote $\lambda_{0}=\operatorname{Re}\lambda\left(k_{0}\right)$.

Lemma 5.3

There exist $c_{1}>c_{2}>0$ such that

[TABLE]

Proof. By Lemma 3.3, we have

[TABLE]

Denote $I=\left[k_{0}-\eta,k_{0}\right]$, $\eta>0$. By (5.5), when $\eta$ is small enough, for  any $k\in I,$ we have

[TABLE]

So letting $k_{1}=k-k_{0}$, then

[TABLE]

When $0\leq t\leq 1$, it is easy to estimate that

[TABLE]

When $t>1$, by direct calculations we have

[TABLE]

where

[TABLE]

Similarly,

[TABLE]

Combining above, we get the estimate (5.9).

Proof of Theorem 1.1 ii). Following the same way as in the periodic case, we construct an approximate solution $U^{app}$ to (1.5) of the form

[TABLE]

where $U_{1}$ is defined in (5.8). By Lemma 5.3

[TABLE]

Following the same arguments as in the proof of Theorem 1.1 i), for $j=1,2,\cdots,N$, we solve $U_{j}$ by the equation

[TABLE]

By Lemma 3.6, we obtain

[TABLE]

Define $T_{\delta}$ by the equation

[TABLE]

where $\theta$ is to be determined. Then $T_{\delta}=O\left(\left|\ln\delta\right|\right)$. The energy estimate in Lemma 4.1 is still true in $H^{2}\left(\mathbf{R}\right)$. Let $U_{\delta}\left(x,t\right)$ be the solution of (1.5) with initial data $U_{\delta}\left(x,0\right)=u_{c}+\delta u_{1}\left(x\right)$. Then by the same arguments as in the periodic case, when $\theta$ is small enough, we have

[TABLE]

This proves the nonlinear instability in the localized space.

6 Nonlinear instability by bootstrap arguments

The proof of nonlinear instability by constructing higher order approximate solutions requires the nonlinear term $f\left(s\right)$ in (1.5) to be in $C^{\infty}(\mathbf{R})$. In this section, we give a different proof by using bootstrap arguments, for the case when $f$ is not smooth. We assume that (1.5) is locally well-posed in the energy space $H^{\frac{m}{2}}$, which is certainly satisfied under the assumption (1.9) (see Lemma 5.2). We will prove nonlinear instability for the nonlinear term $f\in C^{1}(\mathbf{R})$ with the growth conditions (1.11) and (1.12). The bootstrap arguments are done in three steps. First, we use the energy conservation to control the growth of the energy norm in $H^{\frac{m}{2}}$ from the assumed $L^{2}$ growth. Then we use the semigroup estimates in $H^{-1}$ to control the growth of $H^{-1}$ norm of the nonlinear part of the solution. Lastly, the estimates are closed by using the interpolation of $L^{2}$ by $H^{\frac{m}{2}}$ and $H^{-1}$.

Proof of Theorem 1.2. We only give the proof for localized perturbations since it is similar for multiple periodic perturbations.

Step 1. (bootstrap from $L^{2}$ to $H^{\frac{m}{2}}$).

The nonlinear equation for the perturbation $u$ of u_{c}\in the traveling frame $\left(x-ct,t\right)$ is

[TABLE]

where $J,L$ are defined in (1.7). For any $\delta>0$, we choose the initial data $u_{\delta}\left(0\right)=\delta u_{1}$, where $u_{1}$ is defined in (5.6). Then by Lemma 5.3,

[TABLE]

for some $C_{0},C_{1}>0$, $l\in\mathbf{N}$, where $\lambda_{0}$ is the largest growth rate defined in (3.5). Define $T_{1}>0$ to be the maximal time such that

[TABLE]

where $u_{\delta}\left(t\right)$ is the solution of (6.1) with the initial data $u_{\delta}\left(0\right)$. Define $T_{\delta}$ by

[TABLE]

where $\theta>0$ is to be determined. We will show that $T_{1}>T_{\delta}$ when $\theta$ is small. Suppose otherwise $T_{1}\leq T_{\delta}$. The equation (6.1) has the conserved energy-momentum functional

[TABLE]

since (6.1) can be written in the Hamiltonian form $\partial_{t}u=\partial_{x}H^{\prime}\left(u\right)$. By the assumption (1.10), there exists $c_{0}>0$ such that

[TABLE]

Let $T_{2}$ be the maximal time such that

[TABLE]

where

[TABLE]

with

[TABLE]

We claim that $T_{2}>T_{1}$. Suppose otherwise $T_{2}\leq T_{1}$. Then by the energy conservation $H\left(u_{\delta}\left(t\right)\right)=H\left(u_{\delta}\left(0\right)\right)$ and the assumption (1.12), we have

[TABLE]

for any $0\leq t\leq T_{2}$. Here, we use the fact that $L^{P_{2}}\left(\mathbf{R}\right)\hookrightarrow H^{\frac{m}{2}}\left(\mathbf{R}\right)$ when $\frac{m}{2}\geq\frac{1}{2}$. For any $t\leq T_{2}\leq T_{1}\leq T_{\delta}$, by (6.2) we have

[TABLE]

Therefore (6.3) implies that for $0\leq t\leq T_{2}$, we have

[TABLE]

and thus by choosing $\theta$ small enough

[TABLE]

for $0\leq t\leq T_{2}.$This is in contradiction to the definition of $C_{2}$ and shows that $T_{2}>T_{1}$.

Step 2 (bootstrap from $L^{2}$ to $H^{-1}$).

The solution $u_{\delta}\left(t\right)$ to (6.1) can be written as

[TABLE]

By (6.2), Lemma 3.7 and the assumption (1.11), when $0\leq t\leq T_{1}$ we have

[TABLE]

by choosing $\varepsilon<\left(p_{1}-1\right)\lambda_{0}$ and using Lemma 3.6.

Step 3 (Interpolation and closing of the estimates).

For $0\leq t\leq T_{1},$ by interpolation we have

[TABLE]

where we use

[TABLE]

Noticing that $p_{3}=\alpha p_{1}+1-\alpha_{1}>1$, so when 0\leq t\leq T_{1}\leq T_{\delta}\we have

[TABLE]

by choosing $\theta$ to be small enough. This is in contradiction to the definition of $T_{1}$. Thus we must have $T_{1}>T$. At $t=T_{\delta}$, by using (6.4) we have

[TABLE]

when $\theta$ is chosen to be small. This finishes the proof of nonlinear instability for localized perturbations.

Remark 6.1

The assumption (1.10) could be weakened to $0<m<1$ depending on the nonlinearity. In the proof, we only need the embedding of $L^{p}$ into the energy space $H^{\frac{m}{2}}$, where $p>1$ is the highest power of the nonlinear term $f\left(u\right)$ and its anti-derivative $F\left(u\right)$.

7 Semilinear equations

In this section, we consider the nonlinear modulational instability of the generalized BBM equation

[TABLE]

The BBM equation can be viewed as an ordinary differential equation in $H^{1}$

[TABLE]

Assume that (7.1) admits a $T$-periodic traveling solution $u_{c}(t,x)=u_{c}(x-ct)$. Writing (7.1) in the traveling frame $u(t,x)=U(t,x-ct)$, we arrive at

[TABLE]

Linearizing (7.2) at $u_{c}$, we obtain the linearized equation in the Hamiltonian form

[TABLE]

where

[TABLE]

Assume $T=2\pi$. For any $k\in\left[0,1\right]$, define

[TABLE]

As for the KDV type equations, the linear modulational instability of $u_{c}$ means that $J_{k}L_{k}$ has an unstable eigenvalue for some $k\in\left[0,1\right]$. Denote $\lambda_{0}$ to be the maximal growth rate of $e^{tJ_{k}L_{k}},\ k\in\left[0,1\right]$. By the same proof of Lemmas 3.1 and 3.4, we have the semigroup estimates for (7.3).

Lemma 7.1

Suppose $u_{c}$ is modulationally unstable. Consider the semigroup $e^{tJL}$ associated with the solutions of (7.3), where $J,L$ are given in (7.4). Then

i) the exponential trichotomy in the sense of (3.1) and (3.2) holds true in the spaces $H^{s}\left({\mathbb{T}_{2\pi q}}\right)$ $\left(s\geq 1,q\in\mathbb{N}\right)$.

ii) for every $s\geq 1$, \varepsilon>0\there exist $C(s,\varepsilon)>0$ such that

[TABLE]

for any $u\in H^{s}{(\mathbf{R})}$.

For (7.2), there is no loss of derivative in the nonlinear term. Therefore, we can use the semigroup estimates in Lemma 7.1 to prove nonlinear modulational instability directly by ODE arguments. We consider localized perturbations below.

Theorem 7.1

Assume $f\in C^{1}\left(\mathbf{R}\right)$ and there exists $p_{1}>1$, such that

[TABLE]

Let $u_{c}\left(x-ct\right)$ be a traveling wave solution of (7.1) which is assumed to be linearly modulationally unstable. Then $u_{c}$ is nonlinearly unstable under localized perturbations in the following sense: there exists $\theta_{0}>0$, such that for any $s\in\mathbb{N}$ and arbitrarily small $\delta>0$, there exists a time $T^{\delta}=O\left(\left|\ln\delta\right|\right)$ and a solution $U_{\delta}(t,x)$ to (7.2) satisfying \|U_{\delta}(0,x)-u_{c}(x)\|_{H^{s}(\mathbf{R})}<\delta\and$\ \|U_{\delta}(T^{\delta},x)-u_{c}(x)\|_{L^{2}(\mathbf{R})}\geqslant\theta_{0}.$

Proof. For any $\delta>0$, choose the initial perturbation $u_{\delta}(0)=\delta u_{1}$, where $u_{1}$ is defined as in (5.6). Then by the proof of Lemma 5.3,

[TABLE]

for some $C_{0},C_{1}>0,l\in\mathbf{N\ }$and $\lambda_{0}$ is the maximal growth rate defined before. Let $U_{\delta}(t,x)$ be the solution to (7.2) with initial value $u_{c}+\delta u_{\delta}(0)$ and $u_{\delta}=U_{\delta}-u_{c}$, then $u_{\delta}$ satisfies

[TABLE]

where

[TABLE]

Define $T_{1}>0$ to be the maximal time such that

[TABLE]

Define $T_{\delta}$ by

[TABLE]

where $\theta>0$ is to be determined. We will show $T_{1}>T_{\delta}$. Suppose otherwise, $T_{1}\leq T_{\delta}$. From (7.6), we have

[TABLE]

Then when $0\leq t\leq T_{1}\leq T_{\delta}$, by using assumption (7.5) we have

[TABLE]

by choosing $\varepsilon>0$ small. By the same arguments as in the proof of Theorem 1.2, this leads to a contradiction with the definition of $T_{1}$. Therefore, $T_{1}\geq T_{\delta}$ and

[TABLE]

when $\theta$ is chosen to be small. This finishes the proof of the Theorem.

Remark 7.1

For multi-periodic perturbations, following the same arguments, we can prove the nonlinear modulational orbital instability of the generalized BBM equation. Moreover, since the generalized BBM equation is an infinite dimensional ODE in $H^{1}$, one can even construct invariant (stable, unstable and center) manifolds by the standard theory.

8 Applications

In this section, we apply our results to some concrete examples.

8.1 Whitham equation

Consider the Whitham equation for surface water waves,

[TABLE]

where $\mathcal{M}$ is the Fourier multiplier given by

[TABLE]

It is clear that $\|\mathcal{M}(\cdot)\|_{H^{1/2}}\sim\|\cdot\|_{L^{2}}$. It is clear that $m(\xi)=\sqrt{\frac{\tanh\xi}{\xi}}$ is real-valued, analytic and even.

The existence of a periodic traveling wave solutions was shown in [17].

Lemma 8.1** ([17])**

For each $\kappa>0$ and each $b$ with $|b|$ sufficient small, there exists a family of periodic traveling wave solutions to (8.1) taking the form

[TABLE]

for $a$ with $|a|$ sufficiently small , where $w$ and $c$ depend analytically upon $\kappa$, $a$, and $b$. Moreover, $w$ is smooth, even, and $2\pi$-periodic in $z$, and $c$ is even in $a$. Furthermore,

[TABLE]

and

[TABLE]

as $|a|$, $|b|\rightarrow 0$, where

[TABLE]

and

[TABLE]

One can check that

[TABLE]

when $|a|$, |b|\are sufficiently small. So the assumption (2.3) is satisfied.

Moreover, the linear modulational instability of u_{c}(a,b,\kappa)\ is shown in [17] for \kappa>0\large enough. Therefore, we can apply Theorem 1.1 to obtain nonlinear modulational instability of u_{c}(a,b,\kappa)\when $|a|$, |b|\are sufficiently small and \kappa>0\is sufficiently large.

8.2 The Nonlinear Schrödinger equation

We consider in this section the focusing NLS equation

[TABLE]

in which $x\in\mathbf{R}$, $t\in\mathbf{R}^{+}$, and $u(x,t)\in\mathbb{C}$. Note that like the generalized BBM equation discussed in Section 7, the NLS equation is also semi-linear, with no loss of derivative in the nonlinear term. From the results in [11], we know that (8.2) possesses a family of small periodic waves of the form $u_{a,b}(x,t)=e^{-it}e^{il_{a,b}x}P_{a,b}(k_{a,b}x)$, where

[TABLE]

[TABLE]

[TABLE]

as $(a,b)\rightarrow 0$.

In [11], $u_{a,b}(x,t)$ were written in the form of

[TABLE]

and solutions of (8.2) of the form $u(x,t)=e^{i(p_{a,b}x-t)}Q(2k_{a,b}x,t)$ were considered, where

[TABLE]

Here $Q_{a,b}(z)$ were claimed to be a member of a two-parameter family of traveling and rotating waves, see Claim 2 in [11]. Moreover, $Q_{a,b}(z)$ were regarded as an equilibrium of a corresponding evolution equation, and the spectrum of a linear operator at $Q_{a,b}(x)$ was studied to obtain the linear modulational instability of the small periodic waves $u_{a,b}(x,t)$. Thus, we can use the same arguments as in Section 7 to prove nonlinear modulational instability of the small periodic waves $u_{a,b}(x,t)$ as a solution of (8.2).

8.3 Fractional KDV-type equation

Consider the KDV-type equation

[TABLE]

where the pseudo differential operator $\Lambda=\sqrt{-\partial_{x}^{2}}$ is defined by its Fourier multiplier as $\widehat{\Lambda u}(\xi)=|\xi|\hat{u}(\xi)$. Here we consider $m>\frac{1}{2}$ and either $p\in\mathbb{N}$ or $p=\frac{q}{n}$ with $q$ and $n$ being even and odd natural numbers, respectively.

It is clear that $\|\mathcal{M}(\cdot)\|_{L^{2}}\sim\|\cdot\|_{H^{m}}$ and $\alpha(\xi)=|\xi|^{m}$ is real-valued and even.

In [21], a family of small periodic traveling waves $u_{a,b}(t,x)$ of (8.3) were constructed for $|a|,|b|<<1$. It was also showed in Theorem 3.4 of [21] that $u_{a,b}(t,x)$ is linearly modulationally unstable if $m\in(\frac{1}{2},1)$ or if $m>1$ and $p>p^{\ast}(m)$, where $p^{\ast}(m)$ is defined by

[TABLE]

Therefore, if $m\in(\frac{1}{2},1)$ and $p\in\mathbb{N\ }$or if $m>1$ and $p>p^{\ast}(m)$ and $|a|,|b|<<1$, then Theorems 1.1 and 1.2 can be applied to obtain nonlinear modulational instability of $u_{a,b}(t,x)$ for both multiple periodic and localized perturbations. When $m=2$, equation (8.3) is reduced to the generalized KDV equation.

8.4 BBM equation

Consider the BBM equation

[TABLE]

In [19], the authors showed that (8.4) admits a family of periodic traveling wave solutions $u_{c}$ in the following form,

[TABLE]

with $|a|\ll 1$. Furthermore, it was showed in [19] that $u_{c}(t,x;m,a)$ is linearly modulationally unstable if $m>\sqrt{3}$. Applying Theorem 7.1, one can obtain the nonlinear modulational instability of $u_{c}(t,x;m,a)$.

Acknowledgement

Zhiwu Lin is supported in part by a NSF grant DMS-1411803. Shasha Liao is partially supported by the China Scholarship Council No. 20150620040.