A neutrino mass model with hidden $U(1)$ gauge symmetry

TL;DR

This paper introduces a neutrino mass model with a hidden $U(1)$ gauge symmetry that incorporates a dark matter candidate, fitting neutrino data and addressing relic density, muon g-2, and collider signatures.

Contribution

It presents a novel inverse seesaw model with hidden $U(1)$ symmetry and explores its implications for dark matter, neutrino oscillations, and collider phenomenology.

Findings

Model explains relic density and muon g-2 discrepancy.

Constraints from direct detection and LHC signatures are analyzed.

Scalar spectra and neutrino data fitting are detailed.

Abstract

We propose a realisation of inverse seesaw model controlled by hidden $U(1)$ gauge symmetry, and discuss the impact of a bosonic dark matter (DM) candidate by imposing a $Z_2$ parity. We present the detail of scalar spectra and apply the Casas-Ibarra parametrisation to fit the neutrino oscillation data. For this model, the allowed region is extracted to explain the observed relic density and the muon $g-2$ discrepancy, satisfying flavor constraints with DM involved. We interpret the DM annihilation into $\bar f f $ including all SM charged fermions and investigate the direct detection to place the bound on DM-Higgs coupling. Finally the LHC DM production is explored in light of charged lepton pair signature plus missing transverse energy.

Tables

Table 1. Table 1: Charge assignments of the our fields under S ​ U ​ ( 3 ) C × S ​ U ​ ( 2 ) L × U ​ ( 1 ) Y × U ​ ( 1 ) H × Z 2 𝑆 𝑈 subscript 3 𝐶 𝑆 𝑈 subscript 2 𝐿 𝑈 subscript 1 𝑌 𝑈 subscript 1 𝐻 subscript 𝑍 2 SU(3)_{C}\times SU(2)_{L}\times U(1)_{Y}\times U(1)_{H}\times Z_{2} , where all the SM fields are zero charges under the U ​ ( 1 ) H 𝑈 subscript 1 𝐻 U(1)_{H} symmetry and even under the Z 2 subscript 𝑍 2 Z_{2} .

	$U_R\left(U_L\right)$	$D_R\left(D_L\right)$	$E_R\left(E_L\right)$	$N_R\left(N_L\right)$	$H^{\prime }$	$\phi$	${\phi }^{\prime }$	$\chi$
$SU{\left(3\right)}_C$	$\mathrm{𝟑}$	$\mathrm{𝟑}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$
$SU{\left(2\right)}_L$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟐}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$	$\mathrm{𝟏}$
$U{\left(1\right)}_Y$	$\frac{2}{3}$	$-\frac{1}{3}$	$-1$	$0$	$\frac{1}{2}$	$0$	$0$	$0$
$U{\left(1\right)}_H$	$4\left(1\right)$	$-4\left(-1\right)$	$-4\left(-1\right)$	$4\left(1\right)$	$4$	$-3$	$-2$	$1$
$Z_2$	$-$	$-$	$-$	$+$	$+$	$+$	$+$	$-$

Table 2. Table 2: Number of events after each step of selection criterion for one generation of VLL with benchmark points M E = 150 , 200 , 250 subscript 𝑀 𝐸 150 200 250 M_{E}=150,200,250 GeV, for an integrated luminosity of ∫ L ​ 𝑑 t = 35.9 ​ fb − 1 𝐿 differential-d 𝑡 35.9 superscript fb 1 \int{Ldt=35.9}~{}\mbox{fb}^{-1} at a s = 13 𝑠 13 \sqrt{s}=13 TeV LHC.

$\begin{array}{c}13\text{TeV},35.9{\text{fb}}^{-1}\\ M_X\text{(GeV)}\end{array}$	$M_E=150\text{(GeV)}$			$M_E=200\text{(GeV)}$			$M_E=250\text{(GeV)}$			SM BG	Observed
	$60$	$80$	$100$	$100$	$120$	$140$	$120$	$140$	$160$
	$56.8$	$62.9$	$70.3$	$15.8$	$19.1$	$23.4$	$5.12$	$5.69$	$7.08$	-	-
$\begin{array}{c}\Sigma M_T>350\text{GeV}\\ E_T^{\text{miss}}>50\text{GeV}\end{array}$	$5.06$	$3.72$	$1.62$	$2.35$	$1.96$	$1.18$	$1.26$	$1.23$	$1.05$	-	-
$\Delta \varphi (l_1,l_2)>1.5$	$4.81$	$3.40$	$1.43$	$2.27$	$1.86$	$1.12$	$1.20$	$1.19$	$0.99$	${4.35}_{-1.53}^{+1.75}$	$5$

Equations

$$-{\mathcal{L}}_Y$$

$$+y_{{E\phi }_{aa}}{\bar{E}}_{R_a}{E}_{L_a}\phi +(y_{u\chi }{)}_{ia}{\bar{u}}_{R_i}{U}_{L_a}{\chi }^{\ast }+(y_{d\chi }{)}_{ia}{\bar{d}}_{R_i}{D}_{L_a}\chi +(y_{e\chi }{)}_{ia}{\bar{e}}_{R_i}{E}_{L_a}\chi +h.c.,$$

$$V$$

$$+\left[{\lambda }_0(H^{†}{H}^{\prime }){\phi }^{\prime 2}-\mu \chi \chi {\phi }^{\prime }+h.c.\right],$$

$$\frac{\partial V}{\partial v_H}=\frac{\partial V}{\partial v_{H^{\prime }}}=\frac{\partial V}{\partial v_{\phi }}=\frac{\partial V}{\partial v_{{\phi }^{\prime }}}=0,$$

$$v_H\simeq \sqrt{\frac{2\left({\lambda }_{H{\phi }^{\prime }}{\mu }_{{\phi }^{\prime }}^2-2{\lambda }_{{\phi }^{\prime }}{\mu }_H^2\right)}{4{\lambda }_H{\lambda }_{{\phi }^{\prime }}-{\lambda }_{H{\phi }^{\prime }}^2}},v_{{\phi }^{\prime }}\simeq \sqrt{\frac{2\left({\lambda }_{H{\phi }^{\prime }}{\mu }_H^2-2{\lambda }_H{\mu }_{{\phi }^{\prime }}^2\right)}{4{\lambda }_H{\lambda }_{{\phi }^{\prime }}-{\lambda }_{H{\phi }^{\prime }}^2}}$$

$$v_{H^{\prime }}\simeq \frac{-{\lambda }_0{v}_H{v}_{{\phi }^{\prime }}^2}{2{\mu }_{H^{\prime }}^2+({\lambda }_{H{H}^{\prime }}+{\lambda }_{H{H}^{\prime }}^{\prime })v_H^2},v_{\phi }\simeq \sqrt{\frac{-{\mu }_{\phi }^2}{{\lambda }_{\phi }}}$$

$$\displaystyle H=\left[\begin{array}[]{c}w^{+}\\ \frac{v_{H}+h+i\eta}{\sqrt{2}}\end{array}\right],\quad H^{\prime}=\left[\begin{array}[]{c}w^{\prime+}\\ \frac{v_{H^{\prime}}+h^{\prime}+i\eta^{\prime}}{\sqrt{2}}\end{array}\right],$$

$$w^{+}\simeq G^{+},\eta \simeq G_Z,w^{\prime +}\simeq H^{+},$$

$$\phi =\frac{v_{\phi }+{\phi }_R}{\sqrt{2}}{e}^{i\frac{\alpha }{v_{\phi }}},{\phi }^{\prime }=\frac{v_{{\phi }^{\prime }}+{\phi }_R^{\prime }}{\sqrt{2}}{e}^{i\frac{{\alpha }^{\prime }}{v_{{\phi }^{\prime }}}}.$$

$${\alpha }_{NG}$$

$$c_X\equiv \cos X$$

$$M^{2}_{\phi}\simeq\left(\begin{array}[]{cccc}2\lambda_{H}v_{H}^{2}&\left(\lambda_{HH^{\prime}}+\lambda^{\prime}_{HH^{\prime}}\right)v_{H}v_{H^{\prime}}&\lambda_{H\varphi}v_{H}v_{\varphi}&\lambda_{H\varphi^{\prime}}v_{H}v_{\varphi^{\prime}}+\lambda_{0}v_{H^{\prime}}v_{\varphi^{\prime}}\\ \left(\lambda_{HH^{\prime}}+\lambda^{\prime}_{HH^{\prime}}\right)v_{H}v_{H^{\prime}}&-\frac{\lambda_{0}}{2}\frac{v_{H}}{v_{H^{\prime}}}v_{\varphi^{\prime}}^{2}&\lambda_{H^{\prime}\varphi}v_{H^{\prime}}v_{\varphi}&\lambda_{H^{\prime}\varphi^{\prime}}v_{H^{\prime}}v_{\varphi^{\prime}}+\lambda_{0}v_{H}v_{\varphi^{\prime}}\\ \lambda_{H\varphi}v_{H}v_{\varphi}&\lambda_{H^{\prime}\varphi}v_{H^{\prime}}v_{\varphi}&2\lambda_{\varphi}v^{2}_{\varphi}&\lambda_{\varphi\varphi^{\prime}}v_{\varphi}v_{\varphi^{\prime}}\\ \lambda_{H\varphi^{\prime}}v_{H}v_{\varphi^{\prime}}+\lambda_{0}v_{H^{\prime}}v_{\varphi^{\prime}}&\lambda_{H^{\prime}\varphi^{\prime}}v_{H^{\prime}}v_{\varphi^{\prime}}+\lambda_{0}v_{H}v_{\varphi^{\prime}}&\lambda_{\varphi\varphi^{\prime}}v_{\varphi}v_{\varphi^{\prime}}&2\lambda_{\varphi^{\prime}}v^{2}_{\varphi^{\prime}}\end{array}\right),$$

$$m_h^2\simeq {\lambda }_H{v}_H^2+{\lambda }_{{\phi }^{\prime }}{v}_{{\phi }^{\prime }}^2-\sqrt{({\lambda }_H{v}_H^2-{\lambda }_{{\phi }^{\prime }}{v}_{{\phi }^{\prime }}^2{)}^2+{\lambda }_{H{\phi }^{\prime }}^2{v}_H^2{v}_{{\phi }^{\prime }}^2},$$

$$m_{H_1}^2\simeq {\lambda }_H{v}_H^2+{\lambda }_{{\phi }^{\prime }}{v}_{{\phi }^{\prime }}^2+\sqrt{({\lambda }_H{v}_H^2-{\lambda }_{{\phi }^{\prime }}{v}_{{\phi }^{\prime }}^2{)}^2+{\lambda }_{H{\phi }^{\prime }}^2{v}_H^2{v}_{{\phi }^{\prime }}^2},$$

$$m_{H_2}^2\simeq -\frac{{\lambda }_0}{2}\frac{v_H}{v_{H^{\prime }}}{v}_{{\phi }^{\prime }}^2,$$

$$m_{H_3}^2\simeq 2{\lambda }_{\phi }{v}_{\phi }^2,$$

$$\displaystyle\left(\begin{array}[]{c}h\\ \varphi^{\prime}_{R}\end{array}\right)$$

$$\tan {\theta }_h$$

$$-\left({\lambda }_{{\phi }^{\prime }}{v}_{{\phi }^{\prime }}^2-{\lambda }_H{v}_H^2\right)].$$

$$\mathcal{L}\supset \frac{1}{2v_{{\phi }^{\prime }}}{\phi }_R^{\prime }{\partial }_{\mu }{\alpha }_G{\partial }^{\mu }{\alpha }_G=\frac{\sin {\theta }_h}{2v_{{\phi }^{\prime }}}{h}_{SM}{\partial }_{\mu }{\alpha }_G{\partial }^{\mu }{\alpha }_G+\frac{\cos {\theta }_h}{2v_{{\phi }^{\prime }}}{H}_1{\partial }_{\mu }{\alpha }_G{\partial }^{\mu }{\alpha }_G,$$

$${\Gamma }_{h_{SM}\to {\alpha }_G{\alpha }_G}=\frac{m_h^3{\sin }^2{\theta }_h}{256\pi v_{{\phi }^{\prime }}^2}.$$

$$BR(h_{SM}\to {\alpha }_G{\alpha }_G)\simeq 0.052{\left(\frac{1000\mathrm{GeV}}{v_{{\phi }^{\prime }}}\right)}^2{\left(\frac{\sin {\theta }_h}{0.3}\right)}^2,$$

$$\mathcal{L}$$

$${\Gamma }_{H_1\to {\alpha }_G{\alpha }_G}=\frac{m_{H_1}^3{\cos }^2{\theta }_h}{256\pi v_{{\phi }^{\prime }}^2}$$

$${\Gamma }_{H_1\to XX}=\frac{{\mu }^2{\cos }^2{\theta }_h}{16\pi m_{H_1}}\sqrt{1-\frac{4m_X^2}{m_{H_1}^2}}.$$

$${\Gamma }_{H_1\to \bar{t}t}=\frac{3G_F{m}_{H_1}{m}_t^2{\sin }^2{\theta }_h}{4\sqrt{2}\pi }{\left(1-\frac{4m_t^2}{m_{H_1}^2}\right)}^{3/2}$$

$${\Gamma }_{H_1\to WW}=\frac{G_F{m}_{H_1}^3{\sin }^2{\theta }_h}{8\sqrt{2}\pi }\sqrt{1-\frac{4m_W^2}{m_{H_1}^2}}\left(\frac{12m_W^4}{m_{H_1}^4}-\frac{4m_W^2}{m_{H_1}^2}+1\right)$$

$${\Gamma }_{H_1\to ZZ}=\frac{G_F{m}_{H_1}^3{\sin }^2{\theta }_h}{16\sqrt{2}\pi }\sqrt{1-\frac{4m_Z^2}{m_{H_1}^2}}\left(\frac{12m_Z^4}{m_{H_1}^4}-\frac{4m_Z^2}{m_{H_1}^2}+1\right)$$

$$m_{{\chi }_R}^2={\mu }_{\chi }^2+\frac{1}{2}({\lambda }_{H\chi }{v}_H^2+{\lambda }_{H^{\prime }\chi }{v}_{H^{\prime }}^2+{\lambda }_{\phi \chi }{v}_{\phi }^2+{\lambda }_{{\phi }^{\prime }\chi }{v}_{{\phi }^{\prime }}^2)-\sqrt{2}\mu v_{{\phi }^{\prime }}$$

$$m_{{\chi }_I}^2={\mu }_{\chi }^2+\frac{1}{2}({\lambda }_{H\chi }{v}_H^2+{\lambda }_{H^{\prime }\chi }{v}_{H^{\prime }}^2+{\lambda }_{\phi \chi }{v}_{\phi }^2+{\lambda }_{{\phi }^{\prime }\chi }{v}_{{\phi }^{\prime }}^2)+\sqrt{2}\mu v_{{\phi }^{\prime }},$$

$$M_N$$

$$m_{\nu }\approx m_D^{\ast }{M}^{-1}{\mu }_L(M^T{)}^{-1}{m}_D^{†},$$

$$m_D=U_{MNS}\sqrt{D_{\nu }}{O}_{mix}{R}_N^{-1}.$$

$$\displaystyle R_{N}^{-1}=\left[\begin{array}[]{ccc}\frac{1}{a}&0&0\\ -\frac{d}{ab}&\frac{1}{b}&0\\ \frac{-be+df}{abc}&\frac{f}{bc}&\frac{1}{c}\\ \end{array}\right]$$

$$a=\sqrt{{\mu }_{M,11}},d=\frac{{\mu }_{M,12}}{a},b=\sqrt{{\mu }_{M,22}-d^2},f=\frac{d{\mu }_{M,13}-a{\mu }_{M,23}}{ab}$$

$$e=\frac{{\mu }_{M,13}}{a}+2\frac{d}{b}f,c=\sqrt{{\mu }_{M,33}-{\left(e-2\frac{d}{b}f\right)}^2-f^2},$$

$$U_{MNS}^{\prime }\equiv \left(1-\frac{1}{2}F{F}^{†}\right)U_{MNS},$$

Full text

I Introduction

The discovery of the Higgs boson at the Large Hadron Collider (LHC) in 2012 announced the success of the Standard Model (SM) and data collected so far have affirmatively validated the high precision of this framework in explaining most of phenomenology. However the deviations from the expected SM prediction within the current experimental uncertainty can still accommodate the possibility of new particles existence motivated by those interesting scenarios of extra dimension, supersymmetry and composite Higgs model, etc. There are also several aspects such as non-zero masses of neutrinos and dark matter (DM) candidate which should involve physics beyond the SM. For neutrinos, several recent experiments observing the neutrino oscillations confirmed that the neutrino has a tiny mass at the order $<0.1$ eV, which is much smaller compared with the SM quarks and leptons. One favourable neutrino model is supposed to account for the important features related to three active neutrinos with mixing angles of $\theta_{12,13,23}$ and the two neutrino mass square differences, $\Delta m_{12}^{2}$ and $|\Delta m_{23}^{2}|$ consistent with the observations  Forero:2014bxa ; Capozzi:2018ubv ; Esteban:2018azc . Furthermore, several issues are very poorly understood, including whether neutrino is a Dirac fermion or Majorana one, in normal or inverted hierarchy pattern for the mass ordering, and the exact value of CP violation phase, and so on. In particular, the presence of Majorana field violating the lepton number in this type of models leads to the neutrinoless double beta decay detectable in experiments, as well as possibility to explain the Baryon Asymmetry of the Universe via ”leptogenesis” Fukugita:1986hr . Thus it is important to explore and analyze viable neutrino mass models in order to reveal the nature and role of the neutrino sector.

The simplest idea to realize a tiny neutrino mass is seesaw mechanism by introducing heavier neutral fermions which can obtain Majorana mass at the GUT scale $\sim 10^{15}$ GeV. There are several types of seesaw models after a long time of evolving, such as type-I seesaw (aka canonical seesaw) or type-III seesaw involving either a $SU(2)_{L}$ singlet or triplet right-handed neutral fermions  Seesaw1 ; Seesaw2 ; Seesaw3 ; Seesaw4 . One alternative mechanism to obtain a small mass is the radiative seesaw provided the neutrino mass can only be generated at the one-loop level, such as the model proposed in the paper of Ma:2006km , where the neutral $Z_{2}$ odd scalar interacting with neutrino could be the DM candidate. While an inverse seesaw is a promising scenario to reproduce neutrino masses and their mixings by introducing both left and right-handed neutral fermions so that the seesaw mechanism is proceeded via a two-step mediation. This type of mechanism is often considered in extended gauge models such as the superstring inspired model or left-right models in unified gauge group Mohapatra:1986bd ; Wyler:1982dd . In this paper, we propose an inverse seesaw model with several extra scalars charged by a hidden U(1) symmetry, which provides rather natural hierarchies among active neutrinos and heavy neutral fermions even at the tree level compared to another similar scenario of linear seesaw Akhmedov:1995ip ; Akhmedov:1995vm . In analogy to the radiative seesaw, an inert scalar is identified as DM candidate in this model, whose interaction with charged SM leptons and quarks will not only produce the observed relic density, but also give rise to rich LHC phenomenology. The typical LHC signature related to DM production is jets or leptons plus large missing energy, which provides complimentary limits for the parameter space. In fact our scenario allows certain advantage for the DM production at the collider, since in order to obtain the LHC bound, it is crucial to tag the accompanied SM particles like charged leptons in our case.

This letter is organised as follows. In Section II, we present our model by showing new particle fields and symmetries, where the inverse seesaw mechanism is implemented in a framework of hidden $U(1)$ gauge symmetry. We add an inert boson that is expected to be a dark matter (DM) candidate, where a $Z_{2}$ symmetry is imposed to assure the stability of DM. The scalar potential is constructed to trigger spontaneous symmetry breaking and generate the required mass hierarchy. In Section III, We review the electroweak bounds from the lepton flavor violation processes related to charged lepton and $Z$ boson decays. In particular, we provide an analytic formula for the annihilation amplitude squared in terms of four momentum of DM and SM fermions, verified with the chiral limit result as an expansion of $v_{\rm rel}$ in the literature. We further interpret the impact from the observed relic density, along with the direct direction bound on a Higgs-portal term. A numerical analysis is carried out to search for the allowed parameter region. In Section IV, we discuss the LHC collider physics in our model by exploring the pair production of vector-like charged leptons, which subsequently decay into the DM plus SM leptons. Finally we devote the Section V to the summary and conclusion of our results.

II The Model

We will start by presenting the particle content in our model. First of all, we introduce three families of right(left)-handed vector-like fermions $U,D,E,N$ which are charged under $U(1)_{H}$ gauge symmetry; note that actually they are chiral under $U(1)_{H}$ and become vector-like fermions after its spontaneous symmetry breaking Ko:2016ala ; Ko:2016wce . To have gauge anomaly-free for $[U(1)_{H}]^{2}[U(1)_{Y}]$ and $[U(1)_{H}][U(1)_{Y}]^{2}$, the number of family has to be the same for each fermion, although $[U(1)_{H}]^{3}$ and $[U(1)_{H}]$ are anomaly free between $U$ and $D$ or $E$ and $N$.111We can show the non-trivial anomaly free conditions for $[U(1)_{H}]^{2}[U(1)_{Y}]$ and $[U(1)_{H}][U(1)_{Y}]^{2}$. For $[U(1)_{H}]^{2}[U(1)_{Y}]$: $n_{f}\left[3\cdot\frac{2}{3}(4^{2}-1)-3\cdot\frac{1}{3}(4^{2}-1)-(4^{2}-1)\right]=0$; For $[U(1)_{H}][U(1)_{Y}]^{2}$: $n_{f}\left[3\cdot\left(\frac{2}{3}\right)^{2}(4-1)+3\cdot\left(-\frac{1}{3}\right)(4-1)+(-4+1)\right]=0$. The $n_{f}$ is required to be the same for $U,D,E$ so that the anomaly cancellation is achieved. In this model, we can set $n_{f}=3$. In scalar sector, we add an isospin doublet boson $H^{\prime}$ with charge 4 under the $U(1)_{H}$ symmetry that plays an role in having Dirac mass terms in the neutrino sector after spontaneous electroweak symmetry breaking. Also we require three isospin singlet bosons $(\varphi,\varphi^{\prime},\chi)$ with charges $(-3,-2,1)$ under the $U(1)_{H}$ symmetry, where $\varphi,\varphi^{\prime}$ have nonzero vacuum expectation values to induce masses for $U,D,E,N$, while $\chi$ is expected to be an inert boson that can be a DM candidate. Here, we denote that all the SM fields are neutral under $U(1)_{H}$ symmetry, and each of vacuum expectation value is symbolized by $\langle H^{(^{\prime})}\rangle\equiv v_{H^{(^{\prime})}}/\sqrt{2}$, and $\langle\varphi^{(^{\prime})}\rangle\equiv v_{\varphi^{(^{\prime})}}/\sqrt{2}$, where $H$ is the SM Higgs field. In addition we introduce $Z_{2}$ symmetry assigning odd parity to $\{\chi,U,D,E\}$ so that the stability of $\chi$ is guaranteed as a dark matter (DM) candidate. The $Z_{2}$ parity forbids additional interaction terms: $\lambda_{0}\chi\varphi^{*}\varphi^{\prime 2}$, $\lambda_{0}\left(H^{\prime\dagger}H\right)\varphi^{*}\chi$ and $\mu_{0}\chi\varphi\varphi^{\prime*}$, which are permitted by the $U(1)_{H}$ symmetry but could lead to the decay of $\chi$ into SM particles. All the new field contents and their charge assignments are summarized in Table 1. The relevant renormalizable Yukawa Lagrangian and Higgs potential under these symmetries are given by

[TABLE]

[TABLE]

where $\tilde{H}=i\sigma_{2}H^{*}$, $\lambda^{(^{\prime})}_{\phi\phi^{\prime}}=\lambda^{(^{\prime})}_{\phi^{\prime}\phi}$, and the upper indices $(a,b,i)=1,2,3$ are the number of families. All the Yukawa couplings in Eq. (1) are assumed to be diagonal except for $y_{N\varphi^{\prime}}$. Thus in this model, the mixing of active neutrinos are induced via $y_{N\varphi^{\prime}}$ and as illustrated by Figure 1, the neutrino mass is generated by the inverse seesaw. With the outline of particle content and Lagrangian, we are going to present the detail for each sector.

II.1 Scalar sector

We will first focus on the scalar spectra by demanding the VEV of $\chi$ to be vanishing. The non-zero VEVs of scalar fields are obtained by the minimum conditions:

[TABLE]

Since the $v_{H^{\prime}}$ generates a mass term of $\bar{L}H^{\prime}N_{R}$, it is natural to be $\mathcal{O}(1)$ GeV. On the other hand the $v_{\varphi}$ gives a Dirac mass to extra fermions $N_{L(R)}$ which is expected to be in TeV scale (refer to Section II.2 for more detail). Thus we can impose a VEV hierarchy of $v_{H^{\prime}}\ll v_{H}<v_{\varphi^{\prime}}\ll v_{\varphi}$ in order to realise the inverse seesaw mechanism. In this limit, we will approximately obtain the expressions:

[TABLE]

assuming $\{\lambda_{H\varphi},\lambda_{H^{\prime}\varphi},\lambda_{H^{\prime}\varphi^{\prime}},\lambda_{\varphi\varphi^{\prime}}\}\ll 1$. Since we prefer a notable mixing between $H$ and $\varphi^{\prime}$ to induce DM-nucleon scattering, the coupling $\lambda_{H\varphi^{\prime}}$ is only slightly less than $\lambda_{H},\lambda_{\varphi^{\prime}}$. And we require $\{\left(4\lambda_{H}\lambda_{\varphi^{\prime}}-\lambda_{H\varphi^{\prime}}^{2}\right),\left(\lambda_{H\varphi^{\prime}}\mu_{\varphi^{\prime}}^{2}-2\lambda_{\varphi^{\prime}}\mu_{H}^{2}\right),\left(\lambda_{H\varphi^{\prime}}\mu_{H}^{2}-2\lambda_{H}\mu_{\varphi^{\prime}}^{2}\right)\}>0$, and $2\mu_{H^{\prime}}^{2}+(\lambda_{HH^{\prime}}+\lambda^{\prime}_{HH^{\prime}})v_{H}^{2}>0$ plus $\{-\mu_{\varphi}^{2},-\lambda_{0}\}>0$ to make all VEVs positive. The smallness of $v_{H^{\prime}}$ can be achieved by requiring $\lambda_{0}$ to be negligible, so that the $v=\sqrt{v_{H}^{2}+v_{H^{\prime}}^{2}}=246$ GeV is mainly determined by the $v_{H}$. The two Higgs doublet fields are parameterized to be:

[TABLE]

where one massless combination of the charged scalars $w^{+},w^{\prime+}$ is absorbed by the SM gauge boson $W^{\pm}$, and one degree of freedom composed by the CP-odd scalars $\eta$ and $\eta^{\prime}$ is eaten by the neutral SM gauge boson $Z$. In the case of $v_{H^{\prime}}\ll v_{H}$ we can approximate

[TABLE]

Here $G^{+}$ and $G_{Z}$ indicate Nambu-Goldstone boson and $H^{+}$ is remaining physical charged Higgs boson, same as the two-Higgs doublet models.

In the symmetry breaking phase, we also have massless Nambu-Goldstone (NG) boson absorbed by $Z^{\prime}$ and physical Goldstone boson from singlet scalar fields $\varphi$ and $\varphi^{\prime}$. To discuss these massless bosons we first write $\varphi$ and $\varphi^{\prime}$ by:

[TABLE]

While the NG boson $\alpha_{NG}$ and physical Goldstone boson $\alpha_{G}$ should be recasted in terms of linear combination of $\alpha$ and $\alpha^{\prime}$, with the mixing angle determined by relative sizes of VEVs. We therefore obtain the expression for the NG boson and physical Goldstone boson:

[TABLE]

and we can simply write $\alpha_{NG}\simeq\alpha$ and $\alpha_{G}\simeq\alpha^{\prime}$ for the sake of $v_{\varphi^{\prime}}\ll v_{\varphi}$.

For neutral CP-even scalar bosons, we obtain the mass matrix in basis of $(h,h^{\prime},\varphi_{R},\varphi^{\prime}_{R})$ as follows:

[TABLE]

To induce DM-nucleon scattering, we can assume only two CP-even scalars $(h,\varphi^{\prime}_{R})$ have sizable mixing. This can be realised by setting the corresponding coupling for other mixing to be tiny: $\{\lambda_{HH^{\prime}},\lambda^{\prime}_{HH^{\prime}},\lambda_{H\varphi}\}\sim\mathcal{O}\left(\frac{v_{H^{\prime}}}{v_{H}}\right)^{2}$ and $\{\lambda_{H^{\prime}\varphi},\lambda^{\prime}_{H\varphi^{\prime}},\lambda_{\varphi\varphi^{\prime}},\lambda_{0}\}\sim\mathcal{O}\left(\frac{v_{H^{\prime}}}{v_{\varphi}}\right)^{2}$. Thus the mass eigenvalues reads:

[TABLE]

and the mixing among $h$ and $\varphi^{\prime}_{R}$ is parameterised as

[TABLE]

Note that if the quartic coupling $\lambda_{H\varphi^{\prime}}$ is tuned to be large enough, this will result in a sizable mixing angle. In such case the Higgs coupling is universally rescaled by a mixing angle and its current bound is $\sin\theta_{h}\lesssim 0.3$ from the analysis of Higgs precision measurements Chpoi:2013wga ; Cheung:2015dta . In addition, the mixing between those CP-even scalars will cause invisible Higgs decays $h\to\chi_{R(I)}\chi_{R(I)}$ depending on the mass spectrum as well as $h\to\alpha_{G}\alpha_{G}$ via the kinematic term. Since we plan to take $m_{\chi_{R(I)}}>m_{h}/2$ in our analysis below, thus only the process $h\to\alpha_{G}\alpha_{G}$ will be considered here. From the kinetic terms of $\varphi^{\prime}$ we obtain

[TABLE]

where we applied the scalar mixing in Eq. (26). The decay width of $h_{SM}\to\alpha_{G}\alpha_{G}$ process is given by

[TABLE]

Then the branching ratio is estimated as

[TABLE]

Thus for $\Gamma_{h}=4.19$ MeV, $v_{\varphi^{\prime}}>500$ GeV and $\sin\theta_{h}\sim 0.1$, it is safe from the current upper bound $BR_{\rm invisible}<0.25$ Aad:2015pla . For phenomenology interest, we should consider the branching ratio of $H_{1}$ decay since it can be produced at the LHC via scalar mixing. We find out that depending on the $\mu$ parameter, the $H_{1}$ mainly decays into $\chi_{R(I)}\chi_{R(I)}$, $\alpha_{G}\alpha_{G}$ plus SM particles. While the $H_{1}$ decay involving either $\varphi_{R}$ or $H^{\prime}$ are subdominant for $\lambda_{\varphi\varphi^{\prime}},\lambda_{H^{\prime}\varphi^{\prime}}\ll 1$, so is its decay into SM Higgs pair. The last point can be illustrated by an explicit calculation. The interactions inducing $H_{1}\to h_{SM}h_{SM}$ are expressed by:

[TABLE]

where the $H_{1}h_{SM}^{2}$ coupling depends on $\lambda_{\varphi^{\prime}},\lambda_{H},\lambda_{H\varphi^{\prime}}$ if we fix $v_{H}\simeq 246$ GeV and $v_{\varphi^{\prime}}=1000$ GeV. Three couplings will be solved by the conditions of $m_{h}=125$ GeV, $m_{H_{1}}=500$ GeV and a specific value of $\sin\theta_{h}$ using Eq.(15 -16) and Eq.(26). This gives $\Gamma_{H_{1}\to h_{SM}h_{SM}}=\frac{C_{Hhh}^{2}}{32\pi m_{H_{1}}}\sqrt{1-\frac{4m_{h}^{2}}{m_{H_{1}}^{2}}}<6\cdot 10^{-6}$ for $\sin\theta_{h}<0.3$. The partial decay widths for the major $H_{1}$ decay channels are written as:

[TABLE]

Note that $\Gamma_{H_{1}\to\chi_{I}\chi_{I}}=\Gamma_{H_{1}\to XX}$ in case of $m_{\chi_{I}}=m_{X}$, which should be summed up into the total width.

In Fig. 2 we present the dependence of $BR(H_{1}\to\rm{anything})$ on a single variable $\mu$ and the contour of $BR(H_{1}\to XX)$ in the plane of $(\mu,\sin\theta_{h})$, with other parameters indicated in the caption. The plots show that in the low $\mu$ region, $H_{1}$ mainly decays into $\alpha_{G}\alpha_{G}$ and SM particles $\bar{t}t$, $WW$ and $ZZ$ regardless of the mixing angle. While near the corner of large $\mu$ and small $\sin\theta_{h}$, the dominant decay of $H_{1}$ is into DM and its partners $XX+\chi_{I}\chi_{I}$. For $\mu\sim 170$ GeV and $\sin\theta\sim 0.05$, we roughly obtain $BR_{H_{1}\to XX}\simeq 0.4$.

The $Z_{2}$ odd scalar $\chi$ is written as $\chi=(\chi_{R}+i\chi_{I})/\sqrt{2}$. The masses for each component are given by

[TABLE]

where the last term in right-hand side provides the mass difference between $\chi_{R}$ and $\chi_{I}$. Depending on the sign of $\mu$ coupling, either the real or the imaginary part of the $\chi$ scalar can be the DM candidate.

II.2 Neutrino sector

After the spontaneous symmetry breaking, one has neutral fermion masses which are defined by $m_{D}\equiv y_{N}v_{H^{\prime}}/\sqrt{2}$, $M\equiv y_{N\varphi}v_{\varphi}/\sqrt{2}$, and $\mu_{L}\equiv y_{N\varphi^{\prime}}v_{\varphi^{\prime}}/\sqrt{2}$. Then, the neutral fermion mass term in the basis of $\left(\nu_{L}^{i},N_{R}^{C,\,a},N_{L}^{a}\right)$, $(i,a)=1,2,3$, is given by

[TABLE]

The active neutrino mass matrix can be approximated as:

[TABLE]

which can be directly calculated from Feynman diagram as well under the seesaw limit of $\mu_{L}\lesssim m_{D}\ll M$ and assuming that $\mu_{L}$($=\mu_{L}^{T}$), $M$ to be real. The neutrino mass (9$\times$9) matrix is diagonalized by a unitary matrix $U_{MNS}$, i.e. $D_{\nu}=U_{MNS}^{T}m_{\nu}U_{MNS}$, with $D_{\nu}={\rm diag}(m_{1},m_{2},m_{3})$. One of the elegant ways to reproduce the current neutrino oscillation data Forero:2014bxa is to apply the Casas-Ibarra parametrization Casas:2001sr . Without loss of generality, we find the following relation:

[TABLE]

where $O_{mix}$ is an arbitrary 3 by 3 orthogonal matrix with complex values, and $R_{N}$ is a lower unit triangular Nomura:2018ktz , which can uniquely be decomposed to be $\mu_{M}\equiv M^{-1}\mu_{L}(M^{T})^{-1}=R_{N}R^{T}_{N}$, since it is symmetric. For clarity, we provide the explicit form of $R_{N}$ in term of the elements of $\mu_{M}=M^{-1}\mu_{L}(M^{T})^{-1}$ 222The $R_{N}$ parametrisation in ref. Nomura:2018ktz is not fully correct.:

[TABLE]

[TABLE]

Note that the absolute value of all components in $m_{D}$ should not exceed $\frac{1}{\sqrt{2}}$ GeV with $v_{H^{\prime}}=1$ GeV, once the perturbative limit for $|y_{N}|=|\sqrt{2}m_{D}/v_{H^{\prime}}|$ is taken to be 1.

Non-unitarity: We should mention the possibility of non-unitarity matrix $U^{\prime}_{MNS}$ due to the mixing related to heavy fermions. This is typically parametrized by the form:

[TABLE]

where $F$ is a hermitian matrix determined by each model, $U_{MNS}$ is the three by three unitarity matrix, while $U^{\prime}_{MNS}$ represents the deviation from the unitarity. Then $F$ is given by Das:2017ski ; Das:2012ze ; Das:2017nvm

[TABLE]

The global constraints are found via several experimental results such as the SM $W$ boson mass $M_{W}$, the effective Weinberg angle $\theta_{W}$, several ratios of $Z$ boson fermionic decays, invisible decay of $Z$, EW universality, measured CKM, and LFVs Fernandez-Martinez:2016lgt . The result can be given by Agostinho:2017wfs

[TABLE]

We can show a benchmark point satisfying the observed neutrino masses, three mixing angles, plus the Dirac CP violation phase deSalas:2017kay , 333We use the best fit value in the case of normal hierarchy, namely $\Delta m_{21}^{2}=7.55\times 10^{-5}~{}{\rm eV}^{2}$, $\Delta m_{31}^{2}=2.50\times 10^{-3}~{}{\rm eV}^{2}$, $\sin^{2}\theta_{12}=0.32$, $\sin^{2}\theta_{23}=0.547$, $\sin^{2}\theta_{13}=0.0216$ and $\delta=1.21\pi$. The other parameters are fixed to be $v_{H^{\prime}}=1$ GeV and $m_{\nu_{1}}=10^{-13}$ GeV. without conflict with the unitarity bound in Eq. (54):

[TABLE]

For comparison, we comment on an alternative non-unitarity parametrisation: $U^{\prime}_{MNS}=\left(1-\alpha\right)\tilde{U}_{MNS}$, where $\alpha$ is a lower triangular matrix. Defining $\eta=\frac{FF^{\dagger}}{2}$, the translation from the previous one gives $\alpha_{\beta\beta}\simeq\eta_{\beta\beta}$, and $\alpha_{\beta\gamma}\simeq 2\eta_{\beta\gamma}$. In fact the latter one imposes a slightly looser bound according to refs.  Blennow:2016jkn ; Escrihuela:2016ube , although being more model-independent. In our inverse seesaw, the light neutrino flavors decompose into mass eigenstates as $\nu^{i}_{L}\simeq(1-\frac{FF^{\dagger}}{2})\nu_{m}^{i}-\frac{F}{\sqrt{2}}N_{1}^{a}-\frac{F}{\sqrt{2}}N_{2}^{a}$, where the unitarity deviation is same as in Type-I seesaw. Thus for $M\sim\mathcal{O}\mbox{(TeV)}\gg m_{D}$, two formalisms are equivalent up to small corrections.

From the Lagrangian in Eq. (1), we can derive the masses for exotic charged fermions $U,D,E$ after scalars gain VEVs, which are denoted as: $M_{U}=y_{U\varphi}v_{\varphi}/\sqrt{2}$, $M_{D}=y_{D\varphi}v_{\varphi}/\sqrt{2}$, and $M_{E}=y_{E\varphi}v_{\varphi}/\sqrt{2}$. These parameters are not correlated to the neutrino oscillation data, but they should be constrained by DM relic density and LHC direct bound.

II.3 Heavy $Z^{\prime}$ boson

Here we briefly discuss the Hidden gauge boson in the model where we assume the gauge kinetic mixing between $U(1)_{H}$ and $U(1)_{Y}$ is negligibly small. In such a way, a massive $Z^{\prime}$ boson will arise after the symmetry is broken, whose mass is given by:

[TABLE]

where $g_{H}$ is the $U(1)_{H}$ gauge coupling. Note that we have $Z$-$Z^{\prime}$ mixing since $H^{\prime}$ is charged both under $SU(2)_{L}\times U(1)_{Y}$ and $U(1)_{H}$ symmetries. However the mixing effect is highly suppressed by a factor of $v_{H^{\prime}}^{2}/m_{Z^{\prime}}^{2}$ if we take $v_{H^{\prime}}\ll v_{\varphi},v_{\varphi^{\prime}}$. Thus the $Z^{\prime}$ interaction with SM particles is very small, which makes its detection potential at the LHC Run-II evadable.

III Flavour and Dark matter Bounds

As we describe in the model part, extra scalars and sterile neutrinos are introduced to realise an inverse seesaw, with their interactions governed by the hidden gauge symmetry $U(1)_{H}$. In particular, the presence of an inert scalar $\chi$ and exotic charged fermions gives rise to the charged LFVs and flavor-changing $Z$ decays. These interactions will induce a shift in the muon magnetic moment in an expected order provided the Yukawa coupling $(y_{e\chi})_{2a}$, $a=1,2,3$ are relatively large. Due to the $Z_{2}$ parity, the real part of $\chi$ is stabilised as a DM candidate for $\mu>0$, so that its impact on relic density and DM-nucleon scattering would impose constraints as well. Another interesting aspect is the DM production at the LHC, which is characterised by a pair of charged leptons plus missing transverse energy in this model. For a better illustration we will first focus on the bounds from flavour and DM physics here and put the discussion of LHC phenomenology in the next section.

III.1 Lepton flavor violations(LFVs)

The charged LFV decay can be generated with the mediation of a neutral scalar at the one-loop level. For an inert neutral scalar with no mixing, exotic charged fermions are necessary to present assuming the invariance under an extra symmetry such as the $U(1)_{H}$. In our model, the LFV decays can arise from the Yukawa term $(y_{e\chi})_{ia}\bar{e}_{R,i}E_{L,a}\chi$ as illustrated in Figure 3(a). The branching ratio ${\rm BR}(\ell_{i}\to\ell_{j}\gamma)$ is given by:

[TABLE]

where $\int[dx]_{3}\equiv\int_{0}^{1}dx\int_{0}^{1-x}dy$, ${\rm G_{F}}\approx 1.17\times 10^{-5}$[GeV]-2 is the Fermi constant, $\alpha_{\rm em}\approx 1/137$ is the fine structure constant, $C_{21}\approx 1$, $C_{31}\approx 0.1784$, and $C_{32}\approx 0.1736$. Experimental upper bounds are respectively given by ${\rm BR}(\mu\to e\gamma)\lesssim 4.2\times 10^{-13}$, ${\rm BR}(\tau\to e\gamma)\lesssim 3.3\times 10^{-8}$, and ${\rm BR}(\tau\to\mu\gamma)\lesssim 4.4\times 10^{-8}$ TheMEG:2016wtm ; Adam:2013mnn ; Aubert:2009ag .

New contribution to the muon anomalous magnetic moment (muon $g-2$: $\Delta a_{\mu}$) arises from the same term as in LFVs, and its analytic formula reads:444For a comprehensive review on new physics models for the muon $g-2$ anomaly as well as lepton flavour violation, please see Ref. Lindner:2016bgg .

[TABLE]

To explain the current 3.3$\sigma$ deviation Hagiwara:2011af

[TABLE]

Note that in case of a large $h-\varphi$ mixing (by tuning $\lambda_{H\varphi}\sim\lambda_{H\varphi^{\prime}}$), we would have the Barr-Zee type diagrams which contribute to the muon $g-2$. However they only give a small contribution due to two-loop suppression and necessity to satisfy $H\to 2\gamma$ constraint Chiang:2017tai .

III.2 Flavor-Changing Leptonic $Z$ Boson Decays

As a complementary constraint, we include the bound from the decays of the $Z$ boson into two charged leptons of different flavors at the one-loop level. 555Although the quark pairs are also induced from the $y_{u\chi}$ and $y_{d\chi}$, we do not consider them because their experimental bounds are not so stringent. Since we are mainly interested in the parameter region that can achieve a sizeable muon $g-2$, the flavor-changing $Z$ decay widths are expected to get non-trivial contribution from an ${\cal O}(1)$ Yukawa coupling $(y_{e\chi})_{22}$. The relevant form factor is obtained through the vertex and wave-function renormalisation depicted in Figure 3(b), with the analytic expression calculated to be Chiang:2017tai ; Fernandez-Martinez:2016lgt :

[TABLE]

where

[TABLE]

with $\Delta\equiv-xym_{Z}^{2}+(x+y)(m_{a}^{2}-m_{b}^{2})+m_{b}^{2}$ and the total $Z$ decay width $\Gamma_{Z}^{\rm tot}=2.4952\pm 0.0023$ GeV. The current upper limit for the lepton flavor-changing $Z$ boson decay branching ratios are published to be Patrignani:2016xqp :

[TABLE]

where the upper bounds are quoted at 95 % CL. After scanning the parameter space, we found that these constraints are less stringent than the charged LFV ones, and this also applies to flavor-conserving processes ${\rm BR}(Z\to\ell^{\pm}\ell^{\mp})$ ($\ell=e,\mu,\tau$).

III.3 Bosonic dark matter candidate

Fixing $X=\chi_{R}$ to be DM, we can first evaluate the relic abundance by assuming the Higgs portal interaction is negligibly small. This hypothesis is quite reasonable since the $hXX$ coupling is strongly constrained by the spin independent DM-nucleon scattering as we will discuss later. The DM annihilations come from $XX\to{\bar{f}}f$ via Yukawa couplings or $Z^{\prime}$ boson mediation, although the $Z^{\prime}$ one is ignorable. Another possible channel is $XX\to\alpha_{G}\alpha_{G}$, where $\alpha_{G}$ is the physical Goldstone bosons. To figure out the dominant one, we can first examine the couplings. The DM Yukawa interaction is directly read from Eq (1):

[TABLE]

While the DM interaction with $\alpha_{G}$ can be derived from the kinetic term of $\chi$ by a phase rescaling $\chi\to\chi e^{-i\frac{\alpha_{G}}{2v_{\varphi^{\prime}}}}$ Weinberg:2013kea ; Baek:2016wml ; Baek:2018wuo , with $\alpha^{\prime}\simeq\alpha_{G}$ applied:

[TABLE]

which is equivalent to an exponential expansion of the term $\chi\chi\varphi^{\prime}$. Thus in the limit of $v_{H}\ll v_{\varphi^{\prime}}$ and $g_{H}\ll 1$, plus $\mathcal{O}(1)$ Yukawa couplings favored by the muon $g-2$ anomaly, the majority portion of required DM abundance is provided by the annihilation induced by exotic fermions.

We explicitly calculate the amplitude squared for the DM annihilation process of $XX\to\bar{f_{i}}f_{j}$ as shown in Figure 4 to be:

[TABLE]

with $p_{1,2}$ and $k_{1,2}$ denoting the four momenta of DM and SM fermions. Thus the velocity weighted cross section crucial for the relic density is determined by:

[TABLE]

where the indices $i,j$ sum over all the SM leptons and quarks. In our case, only the $\sigma(XX\to\bar{t}\,t)$ is corrected by a phase space factor of $k_{ij}=\sqrt{1-4m_{t}^{2}/s}$, for other channels $k_{ij}\simeq 1$ is used in the chiral limit of $m_{f}/M_{X}\to 0$. In powers of the relative velocity $v_{\rm rel}$ , we get an expansion $\sigma v\simeq a_{\rm eff}+b_{\rm eff}v_{\rm rel}^{2}+d_{\rm eff}v_{\rm rel}^{4}$, where $a_{\rm eff}$, $b_{\rm eff}$ and $d_{\rm eff}$ are s-wave, p-wave and d-wave coefficients respectively. Defining $k_{t}=\sqrt{1-m_{t}^{2}/m_{X}^{2}}$, the coefficients read:

[TABLE]

with $a,i,j=1,2,3$ and $k=1,2$ without counting top quark for d-wave, since only for $m_{f}/M_{X}\to 0$, the $|{\cal{\bar{M}}}_{ij}|^{2}$ behaves like $v_{\rm rel}^{4}$. Hence for the top quark, $s$- and $p$-waves are the leading terms, but for those light fermions, $\sigma_{ij}v$ is $d$-wave dominant. We assume that $y_{u\chi}$ and $y_{d\chi}$ to be diagonal to avoid the constraint from quark sector. For the light fermions, we will take into account the contribution of internal Bremsstrahlung Giacchino:2013bta ; Toma:2013bka .

[TABLE]

Due to the fact $a_{\rm VIB}\sim d_{\rm eff}\,v_{\rm ref}^{4}$ for $r\to 1$, the annihilation cross section is enhanced by a $\mathcal{O}(1)$ boost factor. The resulting relic density is found to be:

[TABLE]

with $\theta(M_{X}-m_{t})=1$ for $M_{X}>m_{t}$, otherwise zero. Here $g_{*}(x_{f}\approx 25)\approx 100$ counts the degrees of freedom for relativistic particles, and $M_{\rm PL}\approx 1.22\times 10^{19}$ GeV is the Planck mass. The present relic density is $0.12\pm 0.003$ at the 3$\sigma$ confidence level (CL) Aghanim:2018eyx . In the left plot of Figure 5, as an estimation, we investigate the sole impact of relic density on DM couplings with SM fermions, where the region between the two lines of same color is allowed. We take universal Yukawa couplings $y_{u\chi}=y_{d\chi}$ for exotic quarks with a degenerate mass $M_{U}=M_{D}=1.0$ TeV. While for all exotic leptons, we set their masses to correlate with the DM mass $M_{E}=1.2M_{X}$, such that due to $r\simeq 1.4$ close to $1.0$, the enhancement for $\langle\sigma v_{\rm rel}\rangle$ from the internal Bremsstrahlung effect is not negligible. The plot shows that the annihilation process $XX\to\bar{t}t$ starts to effectuate beyond the threshold of $M_{X}=m_{t}$. For $M_{X}=200$ GeV and $|(y_{u\chi})_{33}(y^{\dagger}_{u\chi})_{33}|^{2}=0.9$, the coupling sum of $\sum_{a,i,j}|(y_{e\chi})_{ia}(y_{e\chi}^{\dagger})_{ai}|^{2}\sim 20$ is required to obtain the correct relic density. But in case of a smaller $|(y_{u\chi})_{33}(y^{\dagger}_{u\chi})_{33}|^{2}=0.5$, $~{}\sum_{a,i,j}|(y_{e\chi})_{ia}(y_{e\chi}^{\dagger})_{ai}|^{2}\sim 27$ is expected for compensating the reduced $s$ and $p$ waves contribution from the top quark. A more comprehensive analysis will be explored in the next section, where the lepton flavour bounds are fully included.

DM Direct detection measures the nucleon recoil energy for the DM-nucleon scattering in underground experiments. Those searches impose bound for $(M_{X},\mu)$ and $\sin\theta_{h}$, so that the DM production via $H_{1}$ decay at the LHC will be discussed afterwards. The DM-Nucleon scattering is induced via $h_{SM}$ and $H_{1}$ exchange where the relevant interactions are

[TABLE]

where $N(=p,n)$ denote nucleon field and $f_{N}=\frac{2}{9}+\frac{7}{9}\sum_{q=u,d,s}f_{q}$ is the effective coupling for the interaction between SM Higgs and nucleon. The spin-independent DM-nucleon scattering cross section for $m_{H_{1}}\gg m_{h}$ is evaluated as Cline:2013gha :

[TABLE]

where $\mu_{nX}=m_{n}M_{X}/(m_{n}+M_{X})$ is the reduced mass, with $f_{N}\simeq 0.287$ for the neutron-DM scattering Belanger:2013oya (proton-DM scattering is almost same). The most stringent constraint comes from XENON1T data Aprile:2017iyp ; Aprile:2018dbl which gives $90\%$ confidence level upper limit on $\sigma_{SI}$, consistent with the looser bound from LUX Akerib:2016vxi or PandaX-II Cui:2017nnn . This bound fixes the ratio of $\frac{\mu\sin\theta_{h}\cos\theta_{h}}{M_{X}}$ and is recasted into the allowed region of $(M_{X},\mu)$ as shown in the right plot of Figure 5. Based on that we can investigate the DM production via $gg\to H_{1}\to XX$ for two limits where $\sin\theta_{h}$ is either small or sizable. Considering a benchmark point of $M_{X}=200$ GeV, the bound $\sigma_{\chi_{R}-n}\lesssim 1.78\times 10^{-46}{\rm cm}^{2}$ leads to $\mu\sin\theta_{h}\cos\theta_{h}\lesssim 3.6$ GeV. For $\sin\theta_{h}\sim 0.04$ and $\mu<90$ GeV, we have $BR(H_{1}\to XX)<0.3$ as indicated by Figure 2, but a very small $\sigma_{H_{1}}^{ggF}$ for $m_{H_{1}}=500{\rm GeV}$ due to almost vanishing mixing. While for a sizable $\sin\theta_{h}=0.1$, we find that at the $\sqrt{s}=13$ TeV LHC $\sigma_{H_{1}}^{ggF}(m_{H_{1}}=500{\rm GeV})\sim\mathcal{O}(10)$ fb, but in such case $BR(H_{1}\to XX)<0.05$ is too small since we require $\mu<36$ GeV. Thus we can conclude that the DM production rate via $H_{1}$ exchange is negligible in this model.

III.4 Numerical analysis

Now we can combine all the bounds and carry out a numerical scan to find out the parameter space which can explain DM relic density and muon $g-2$. In this analysis, we show the correlation between $M_{E}$ (the lightest VLL mass) and $M_{X}$ by recasting the bounds from the observed relic density and various leptonic flavor constraints. We take the upper limit of Yukawa couplings as $\sqrt{4\pi}$, and the regions of $M_{X}$, $m_{\chi_{I}}$ $M_{U,D}$, and $M_{E}$ are scanned in the regions of $(90,500)$ GeV, $(1.2M_{X},550)$ GeV, $(1000,2000)$ GeV, and $(1.2M_{X},1000)$ GeV respectively. Here the lower bound of $m_{\chi_{I}}$ is set to forbid the co-annihilation modes between $X$ and $\chi_{I}$ for simplicity, and the lower limit of vector-like lepton $M_{E}>108$ GeV complies with the LEP experiment, although the relevant LHC limit can be more stringent. The left plot in Figure 6 represents the allowed regions for $(M_{E},M_{X})$, which are consistent with precise observations of $\Delta a_{\mu}=(26.1\pm 8.0)\times 10^{-10}$ and $\Omega h^{2}=0.12\pm 0.003$, as well as satisfy LFV and Z decay bounds. We adopt different colours in the plot to emphasize the experimental constraint from the muon g-2 at the confidence level of $68\%$ (green), $95\%$ (blue), $99.7\%$ (red). The LFV bounds specifically lead to the consequence that the typical value of $(y_{e\chi})_{22}$ should be 2$\sim$3 as verified by the right plot in Figure  6 and the other Yukawa couplings can be less than 1. While the masses of exotic heavy quarks $M_{U,D}$ are not so much restricted in this simplified model. In particular Figure 6 indicates that the upper bounds for DM and vector-like leptons masses are required to be $M_{E}\lesssim 450$ GeV and $M_{X}\lesssim 400$ GeV respectively, while the mass splitting between these two particles tends to be small, roughly in a scale of $\sim 50$ GeV.

IV LHC phenomenology

In this sector, we proceed to provide an analysis for the LHC constraint by scanning over the mass region allowed by the bounds of flavour and relic density. Due to the $Z_{2}$ parity presented in this model, exotic fermions $U$, $D$ and $E$ can be pair produced. In order to interpret the LHC measurement in this hidden $U(1)$ symmetry model, we only consider the Drell-Yan production of vector-like lepton (VLL) pairs, with the subsequent decaying of $E_{a}\to X/\chi_{I}+l_{i}$. For estimation, the mass difference of $(m_{\chi_{I}}-M_{X})$ will be ignored. The final state of $\tau$ lepton pairs plus $\not{\!\!{E}}_{T}$ was recently adopted by the CMS collaboration to extract the upper limit of cross section for $\tau$ slepton pair productions Sirunyan:2018vig . By recasting the CMS analysis into our DM scenario, we obtain a loose bound for $(M_{E},M_{X})$ under the assumption of universal $E_{a}$-$\chi$-lepton couplings, i.e. $y_{E\chi}^{e,a}=y_{E\chi}^{\mu,a}=y_{E\chi}^{\tau,a}$.

In order to simulate the $2\tau_{h}+\not{\!\!{E}}_{T}$ signal in this model, we employ MG5_aMC@NLO  Alwall:2014hca to generate events for the production of $pp\to Z,\gamma$ $\to E^{+}E^{-}$ at the leading order precision, with the VLL decay into $\chi+\{e,\mu,\tau\}$ handled by MadSpin. The events are passed through Pythia 8 Sjostrand:2007gs for parton shower and hadronization, where the $tau$ lepton decays in both leptonic and hadronic modes are sophisticatedly processed. Event reconstruction is finally performed by Madanalysis 5 package Conte:2012fm , so that the jets are clustered using the anti-$k_{T}$ algorithm implemented in FastJet, with $p_{T}>20$ GeV and a distance parameter of $R=0.4$. The CMS discriminant for $\tau_{h}$ reconstruction results in an efficiency $\sim 60\%$, which is also counted in our simulation. The event analysis is conducted first by a baseline selection, demanding two hadronic taus in opposite signs, with a veto for electrons or muons in the final state. Subsequent kinematic cuts are applied afterwards, including the $M_{T2}$ variable, sum of transverse mass $\Sigma M_{T}(\tau_{i})$, missing energy $\not{\!\!{E}}_{T}$ and $\Delta\phi(\tau_{1},\tau_{2})$, in order to optimize the signal and suppress the SM background. The $M_{T2}$ variable is a generalization of transverse mass into the case with two invisible particles  Barr:2003rg ; Cheng:2008hk . In this analysis we use the CMS interpretation by setting the trial mass $\mu_{X}$ of two missing particles to be zero for a direct comparison purpose. We calculate the $M_{T2}$ as the minimum of all possible maximum of $\left(M_{T}(\tau_{1}),M_{T}(\tau_{2})\right)$, with the partition of missing momentum in two DMs added up to be $\not{\!\!{E}}_{T}$ measured in the event:

[TABLE]

where the transverse mass in the case of massless particles is defined as:

[TABLE]

Following the CMS analysis, we employ the event selection criteria in the search region 2 (SR2) for the $\tau_{h}\tau_{h}$ final states, and ignore the selection in the other two isolated regions of SR1 and SR3 due to their insensitivity and a larger number of expected SM background than the LHC observed data. Therefore events should satisfy these requirements: (1) $40~{}\mbox{GeV}<M_{T2}<90~{}\mbox{GeV}$, (2) $\Sigma M_{T}>350~{}\mbox{GeV}$, (3) $\not{\!\!{E}}_{T}>50~{}\mbox{GeV}$, and (4) $\Delta\phi(l_{1},l_{2})>1.5$. The number of events after each cut is reported in the table 2, for the case that only the lightest VLL is effective. The assumption of universal coupling results in equal branching ratios of $Br(E^{-}\to e^{-}+\chi)=Br(E^{-}\to\mu^{-}+\chi)=Br(E^{-}\to\tau^{-}+\chi)=1/3$, which can be consistent with the flavour constraint. The cut table indicates that for a fixed VLL mass $M_{E}$, the event number after those $\Sigma M_{T}$, $\not{\!\!{E}}_{T}$ and $\Delta\phi(l_{1},l_{2})$ cuts decreases for an increasing $M_{X}$. While the event number after the $M_{T2}$ cut will instead be enhanced in that situation. This is a reflection of the $M_{T2}$ quality as a function of the trial mass for missing particle. If the trial mass $\mu_{X}$ equals the true mass $M_{X}$, the end point of $M_{T2}$ gives the exact mass of the parent particle $M_{E}$. However for the large deviation $M_{X}\gg\mu_{X}=0$, the end point $M_{T2}$ will drop below $M_{E}$ because there is less measured missing energy. In the left plot of Figure 7, we present the $M_{T2}$ distribution after the basic cut for $M_{E}=200$ GeV, $M_{X}=100,150$ GeV. For a larger DM mass, the event distribution shifts into the lower mass region due to a false trial mass, leading to an increase for the $M_{T2}$ cut acceptance.

The CMS collaboration provides a simulation for the SM background, which is $4.35^{+1.75}_{-1.53}$ in the SR2 signal region, and the observed event numbrer is $5$ at the 13 TeV LHC. This can be translated into a $68\%$ C.L. exclusion limit for $(M_{E},M_{X})$ presented in the right plot of Figure 7. As we can see, with only one generation of VLL, the LHC constraint is not stringent, excluding a small mass region with $M_{X}\lesssim 110$ GeV for $M_{E}\sim 200$ GeV. While for the three-generation scenario, the exclusion becomes much more relevant. The upper exclusion limit for $M_{X}$ reaches $185$ GeV, which possibly overlaps with the mass region permitted by the relic density and flavor bounds displayed in Figure 6. Note that all three generations of VLLs contribute to this specific LHC signal with a mass hierarchy of $M_{E1}<M_{E2}<M_{E3}$. However since the Yukawa coupling $(y_{e\chi})_{22}$ is preferred to be larger than other ones, the assumption of universal couplings is less observed for the second generation. Thus the realistic LHC exclusion region for $(M_{E},M_{X})$ would most likely lie below the upper boundary of the orange band.

V Summary and Conclusions

We have proposed an inverse seesaw scenario in a framework of hidden $U(1)_{H}$ gauge symmetry where extra scalars and vector-like neutrinos are introduced to assist the mass generation of neutrino, while vector-like quarks and leptons are required in order to cancel the $U(1)_{H}$ gauge anomaly. For the neutrino sector, we apply the Casas-Ibarra parametrisation to fit neutrino oscillation data and bound of non-unitary PMNS matrix. This model features a bosonic dark matter candidate stabilised by a $Z_{2}$ parity. Specifically the DM interaction with exotic charged fermions plays an important role to realize the observed relic density in a viable limit $v_{H^{\prime}}\ll v_{H}<v_{\varphi^{\prime}}\ll v_{\varphi}$. By tuning the scalar potential, a minimal mixing among the SM Higgs and extra scalars is achieved. Under this assumption, we provide the allowed region capable to accommodate the discrepancy in muon $g-2$, while consistent with the relic density and flavour bounds. In case that a notable $h-\varphi^{\prime}$ mixing is invoked, the Higgs portal DM-nucleon scattering fixes the upper limit for $\mu\sin\theta_{h}\cos\theta_{h}/M_{X}$. Our analysis shows that after taking into account the constraint from direct detection, the cross section of DM pair production via a heavy Higgs $H_{1}$ is almost negligible.

Concerning the possibility to extract the DM mass bound at the LHC, we focus on the Drell-Yan pair production of vector-like charged lepton since it provides clear signal of charged leptons plus missing transverse energy involving DM. We recasted the CMS analysis for $2\tau_{h}+\not{\!\!{E}}_{T}$ events at the $\sqrt{s}=13$ TeV LHC based on the $M_{T2}$ selection, which shows that the lower regions in ($M_{E}$, $M_{X}$) are ruled out depending on the mass degeneracy among vector-like charged leptons and their branching ratios into tau leptons. However due to the current insensitivity to $\tau_{h}$, most of the allowed region from relic density and flavor physics would survive for Yukawa couplings in correct orders.

Acknowledgments

This research is supported by the Ministry of Science, ICT & Future Planning of Korea, the Pohang City Government, and the Gyeongsangbuk-do Provincial Government (H. C. and H. O.).