Probing the Decoupled Seesaw Scalar in Rare Higgs Decay

TL;DR

This paper explores how a tiny mixing between the Higgs boson and a decoupled singlet scalar can enable collider detection of right-handed neutrinos through a distinctive same-sign lepton pair signature, offering a new way to probe neutrino mass origins.

Contribution

It introduces a novel collider search strategy for right-handed neutrinos via Higgs-mediated pair production with minimal Higgs-singlet mixing, enhancing detection prospects.

Findings

Higgs-singlet mixing enables right-handed neutrino production at colliders.

The proposed channel is nearly background-free in proton-proton collisions.

Current and future colliders can effectively probe this scenario.

Abstract

The Higgs boson can mix with a singlet scalar that dynamically generates the Majorana mass of the right-handed neutrino $N_R$. We show that even a tiny mixing between the Higgs boson and a `decoupled' singlet scalar allows for Higgs-mediated pair production of $N_R$ without significant mixings between the active neutrinos and $N_R$, and thus testable at colliders via a characteristic signal of two same-sign same-flavor lepton pairs, plus missing energy. We demonstrate that this search channel is mostly background-free in $pp$-collision and can be a highly sensitive probe of the Higgs-singlet mixing at the current and future $pp$ colliders. Such channel provides a clean signal to discover the singlet scalar and explore the origin of neutrino masses.

Tables

Table 1. Table 1: Event rates for the signal from h 1 → N ​ N → e ± ​ e ± ​ μ ∓ ​ μ ∓ ​ 2 ​ ν → subscript ℎ 1 𝑁 𝑁 → superscript 𝑒 plus-or-minus superscript 𝑒 plus-or-minus superscript 𝜇 minus-or-plus superscript 𝜇 minus-or-plus 2 𝜈 h_{1}\to NN\to e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}2\nu with m N = 20 subscript 𝑚 𝑁 20 m_{N}=20 GeV and BF h 1 → e ± ​ e ± ​ μ ∓ ​ μ ∓ ​ 2 ​ ν = 10 − 7 subscript BF → subscript ℎ 1 superscript 𝑒 plus-or-minus superscript 𝑒 plus-or-minus superscript 𝜇 minus-or-plus superscript 𝜇 minus-or-plus 2 𝜈 superscript 10 7 {\rm BF}_{h_{1}\to e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}2\nu}=10^{-7} and for background processes of 4 ​ τ 4 𝜏 4\tau and W ​ W ​ Z 𝑊 𝑊 𝑍 WWZ . The numbers at the HL-LHC (FCC-hh/SppC) correspond to center-of-mass energy s = 14 ​ ( 100 ) 𝑠 14 100 \sqrt{s}=14(100) TeV and 3 (20) ab − 1 superscript ab 1 \rm ab^{-1} integrated luminosity.

		$NN$	$4\tau$	$WWZ$
HL-LHC	initial	16.4	$2.9\times {10}^5$	$1.1\times {10}^3$
	pre-selection	1.8	7.0	1.4
	BDT $>$ 0.267	1.5	$6.6\times {10}^{-2}$	$9.6\times {10}^{-4}$
FCC-hh/SppC	initial	1481	$1.2\times {10}^7$	$1.1\times {10}^5$
	pre-selection	343	1183	346
	BDT $>$ 0.170	298	7.5	$1.5\times {10}^{-1}$

Table 2. Table 2: Selection efficiencies for the signal N ​ N 𝑁 𝑁 NN and background processes of 4 ​ τ 4 𝜏 4\tau and W ​ W ​ Z 𝑊 𝑊 𝑍 WWZ at the HL-LHC (with center-of-mass energy s = 14 𝑠 14 \sqrt{s}=14 TeV) and FCC-hh/SppC (with s = 100 𝑠 100 \sqrt{s}=100 TeV) for m N = subscript 𝑚 𝑁 absent m_{N}= 10, 20, 30, 40, 50, 60 GeV, respectively. Here h 2 subscript ℎ 2 h_{2} is assumed to be heavy and only h 1 subscript ℎ 1 h_{1} decay contributes to the signal N ​ N 𝑁 𝑁 NN .

$m_N$	$\sqrt{s}$	selection	$NN$	$4\tau$	$WWZ$
10 GeV	14 TeV	pre-selection	$1.10\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.428	$9.52\times {10}^{-1}$	$2.86\times {10}^{-3}$	$1.50\times {10}^{-4}$
	100 TeV	pre-selection	$2.26\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$0.311	$8.97\times {10}^{-1}$	$9.46\times {10}^{-4}$	$1.57\times {10}^{-5}$
20 GeV	14 TeV	pre-selection	$1.08\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.267	$8.60\times {10}^{-1}$	$9.43\times {10}^{-3}$	$6.87\times {10}^{-4}$
	100 TeV	pre-selection	$2.32\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$0.170	$8.69\times {10}^{-1}$	$6.30\times {10}^{-3}$	$4.25\times {10}^{-4}$
30 GeV	14 TeV	pre-selection	$1.12\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.177	$6.82\times {10}^{-1}$	$1.54\times {10}^{-2}$	$1.96\times {10}^{-3}$
	100 TeV	pre-selection	$2.56\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$ 0.150	$5.56\times {10}^{-1}$	$5.25\times {10}^{-3}$	$1.22\times {10}^{-3}$
40 GeV	14 TeV	pre-selection	$1.35\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.122	$6.58\times {10}^{-1}$	$3.82\times {10}^{-2}$	$9.69\times {10}^{-3}$
	100 TeV	pre-selection	$3.07\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$0.131	$4.35\times {10}^{-1}$	$5.56\times {10}^{-3}$	$3.59\times {10}^{-3}$
50 GeV	14 TeV	pre-selection	$1.56\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.128	$5.34\times {10}^{-1}$	$2.46\times {10}^{-2}$	$1.13\times {10}^{-2}$
	100 TeV	pre-selection	$3.62\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$0.117	$4.71\times {10}^{-1}$	$9.42\times {10}^{-3}$	$7.20\times {10}^{-3}$
60 GeV	14 TeV	pre-selection	$1.77\times {10}^{-1}$	$2.41\times {10}^{-5}$	$1.33\times {10}^{-3}$
		BDT$>$0.138	$5.88\times {10}^{-1}$	$1.79\times {10}^{-2}$	$1.16\times {10}^{-2}$
	100 TeV	pre-selection	$4.03\times {10}^{-1}$	$1.01\times {10}^{-4}$	$3.25\times {10}^{-3}$
		BDT$>$0.131	$4.95\times {10}^{-1}$	$5.20\times {10}^{-3}$	$5.06\times {10}^{-3}$

Equations

$$\mathcal{L}$$

$$\begin{array}[]{c|cc}&\phi&s\\ \hline\cr\phi&m^{2}_{\phi}&\lambda v_{\phi}v_{s}\\ s&\lambda v_{\phi}v_{s}&m_{s}^{2}\\ \end{array}.$$

$$\left(\begin{array}[]{c}h_{1}\\ h_{2}\end{array}\right)=\left(\begin{array}[]{cc}\cos\alpha&-\sin\alpha\\ \sin\alpha&\cos\alpha\end{array}\right)\left(\begin{array}[]{c}\phi\\ s\end{array}\right).$$

$$\alpha =\frac{\lambda v_{\Phi }{v}_S}{|m_s^2-m_{\varphi }^2|},$$

$${\sin }^2\alpha$$

$$\lambda \cdot \max (v_S^2,v_{\Phi }^2)$$

$${\Gamma }_{h_1\to NN}=\frac{1}{2}{\sin }^2\alpha \cdot \frac{y_N^2{m}_{h_1}}{8\pi }{\left(1-\frac{4m_N^2}{m_{h_1}^2}\right)}^{3/2}$$

$${\mathrm{BR}}_{h_1\to NN}=\frac{{\Gamma }_{h_1\to NN}}{{\Gamma }_h+{\Gamma }_{h_1\to NN}}.$$

$${\mathrm{BF}}_{sig.}$$

$${\sigma }_{sig.}=$$

$${\sigma }_{\mathrm{stat}}=\sqrt{2[(N_s+N_b)\ln (1+\frac{N_s}{N_b})-N_s]},$$

Full text

1 Introduction

The seesaw mechanism has been extensively studied to generate the tiny active neutrino mass. By adding a right-handed neutrino $N_{R}$ and introducing a Dirac neutrino mass $m_{D}\,\nu_{L}N_{R}$, the Standard Model’s (SM’s) neutrino $\nu_{L}$ and the hypothetical $N_{R}$ can mix into the active neutrino $\nu$ and a heavy neutrino $N$, where the active neutrino mass $m_{\nu}\sim m_{D}^{2}/M$ can be brought down to sub-eV scale by having a high-scale Majorana mass $M$ of $N_{R}$.

The heavy $N$ is an important target of new physics searches as the crucial seesaw ingredient. At colliders it can be weakly produced via its active neutrino component or by its $N_{R}$ component’s coupling to SM particles. One major difficulty for the weak production is that it is limited by $N_{R}$’s typically small mixing with the weakly charged $\nu_{L}$. Nowadays Large Hadron Collider (LHC) searches Sirunyan:2018xiv ; Aaboud:2018spl and electroweak precision measurements Abada:2013aba ; Akhmedov:2013hec probe the effective $\nu_{l}-N_{R}$ mixing parameters $|V_{lN}|^{2}$ Atre:2009rg to $\sim 10^{-5}$. Alternatively, a $N_{R}$ uncharged under SM gauge interactions can still have a sizable coupling to massive Beyond the Standard Model (BSM) scalars which mix with the SM Higgs boson. Well-motivated scenarios include the $N_{R}$ coupling to the neutral scalar components in Left-Right symmetric models Wyler:1982dd , gauged $U(1)_{B-L}$ symmetric models Mohapatra:1979ia , and seesaw models with gauge-singlet scalars Shoemaker:2010fg , etc.

While direct massive BSM mediator production is an important aspect of future high energy searches, their contribution to the rare decay of the SM bosons must be carefully examined for a potentially tiny $N_{R}$ related mixing. Well known instances include the rare Higgs decay channel $h\rightarrow l^{\pm}l^{\pm}4j$ that is enhanced by the Higgs-BSM scalar mixing Shoemaker:2010fg ; Maiezza:2015lza ; Nemevsek:2016enw , which have been studied as a good test of lepton number violation (LNV) in case of a Majorana heavy neutrino. $N_{R}$ characteristic and collider-testable rare Higgs decay channels are very important precision searches, especially as so far collider searches show no robust sign of BSM mediators.

Generally, a singlet scalar that dynamically generates the Majorana mass of the $N_{R}$ can mix with the Higgs boson, which allows the Higgs boson’s decay mode of $h\to NN$ without requiring sizeable left-right neutrino mixings. Such a singlet extension of the SM is often motivated for the dynamical $N_{R}$ mass generation, and it potentially helps understanding several important phenomena: the baryon number asymmetry via a strong first order electroweak phase transition Pietroni:1992in ; Davies:1996qn , the abundance of dark matter Silveira:1985rk ; McDonald:1993ex and the issue of vacuum stability Clark:2009dc ; Lerner:2009xg ; Gonderinger:2009jp , etc. In this study we advocate that such new singlet scalar could be discovered at $pp$ colliders by searching for the fully leptonic rare Higgs decay channel $h\to NN\to 4l+\not{E}_{T}$.

For minimal parameter dependence we start with a small-mixing scenario of the BSM scalar sector in a Type I seesaw setup where the L-R mixing is typically small,

[TABLE]

$\mathit{\Phi}$ denotes the scalar field(s) that yield the 125 GeV Higgs boson, and $S$ for a singlet scalar field that couples to the right-hand singlet neutrino $N_{R}$. Here we leave out the specific form of the potentials and only assume ‘small-mixing’, which is a very subleading mixing term $\frac{1}{2}\lambda|\mathit{\Phi}|^{2}S^{2}$, so that $V(\mathit{\Phi})$ and $V(S)$ minimize independently. $\mathit{\Phi}$ and $S$ develop their own vacuum expectation values (vev): $\mathit{\Phi}=(v_{\mathit{\Phi}}+\phi)/\sqrt{2}$ to give rise to electroweak symmetry breaking and generate a Higgs-like boson $\phi$, and $S=v_{S}+s$ to yield a necessary Majorana mass $m_{N_{R}}=2y_{N}v_{S}$. The small but non-zero mixing term also portals $N_{R}$ into $\phi$ decay, yielding our search signal. For simplicity here we neglected the $|H|^{2}S$ term which has a dimensional coupling, by assuming its coupling is subdominant to $\lambda v_{\Phi}$.

The $\lambda$ term develops a $\phi-s$ mass mixing term $\lambda v_{\mathit{\Phi}}v_{S}\cdot\phi s$ after potential minimization, where the scalar mass-square matrix can be parametrized as,

[TABLE]

With ‘small mixing’ such that $\lambda v_{\mathit{\Phi}}v_{S}\ll m_{\phi}^{2},m_{s}^{2}$, this matrix is dominated by the diagonal terms and the mass eigenstate {$h_{1,2}$} is a rotation by a small angle $\alpha$ from the gauge eigenstates {$\phi$,$s$},

[TABLE]

Without loss of generality $h_{1}$ identifies with the SM Higgs boson and contains an $s$-component of the magnitude

[TABLE]

which leads to a decay channel $h_{1}\to NN$ in additional to the SM decays, if kinematically permitted. Similarly $h_{2}$ is mostly $s$ and has only a small $\phi$-component. Since $s$ does not couple to the SM fermions, $h_{1}$’s $s$-component leaves the relative branching ratios unchanged from their SM values. Such a mixing can be sizable Farina:2013fsa ; Lopez-Val:2014jva if $S$ is completely decoupled ($y_{N}\rightarrow 0$). For small mixings we would take ‘decoupling’ $\mathit{\Phi},S$ conditions:

[TABLE]

The latter simplifies to $\lambda v_{S}^{2}\leq{\cal O}(10^{-2})m_{h_{1}}^{2}$ if we consider an $h_{2}$ comparable to or heavier than the Higgs boson.

A finite $y_{N}$ leads to the $h_{1}$ decay partial width

[TABLE]

for each $N$ species, where $m_{h_{1}}$=125 GeV, $m_{N}=2y_{N}v_{S}$ and the factor $1/2$ is due to $N$ being Majorana. Given the rather small SM Higgs boson’s decay width $\Gamma_{h}\sim$ 4 MeV and relatively light heavy neutrino mass $m_{N}\lesssim m_{h_{1}}/2$, the branching ratio of $h_{1}$ decay to $NN$ is

[TABLE]

This leads to ${\rm BR}_{h_{1}\to NN}\approx\Gamma_{h_{1}\to NN}/\Gamma_{h}$ for $\sin^{2}\alpha\ll 1$. In both Majorana and Dirac $N_{R}$ cases, the subsequent $NN$ decay can lead to a characteristic two same-sign, same-flavor lepton pair, plus missing transverse energy signal $e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}+\not{E}_{T}$.

2 Signal at $pp$ colliders

The Higgs-like $h_{1}$ can be copiously produced via gluon fusion during $pp$ collisions with a cross-section $\sigma_{h_{1}}=\sigma_{pp\to h}$. $NN$ decay then yields a characteristic and remarkably clean signal of two same-sign, same-flavor (SSSF) lepton pairs as illustrated in Fig. 1. The two lepton pairs should have different lepton flavors and opposite charges (i.e. $e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}$ and no opposite-sign, same-flavor lepton pair exists). This channel differs from the LNV $NN\to l^{\pm}l^{\pm}4j$ case that it does not necessarily violate lepton number conservation (LNC) if $N$ couples to more than one lepton flavor, as the final state $\nu$ can be either neutrinos or anti-neutrinos. Therefore, this final state can be present either $N$ is Majorana or Dirac.

With a Majorana $N$, the $NN$ system has a higher decay branching to this final state as one $N$ can also decay as its own antiparticle. In such case, for each species of $N$ the branching factor of the signal final state from $NN$ is obtained by summing up the LNV and LNC processes,

[TABLE]

where $\Sigma^{2}\equiv|V_{eN}|^{2}+|V_{\mu N}|^{2}+|V_{\tau N}|^{2}$, and ${i,j}=\{e,\mu\}$ for lepton flavors. ${\rm BF}^{l}_{i}$ is the $W^{(*)}\rightarrow l_{i}\nu$ branching fraction in $N$ decay. Note the $N\to ll^{\prime}\nu$ requires charged weak gauge boson mediation, which makes ${\rm BF}^{l}_{i}$ depend on both $m_{N}$ and whether BSM mediators are present in $N$ decay. The maximal BFsig. is reached if $N$ only couples to one lepton flavor, and ${\rm BF}_{\rm sig.}\leq({\rm BF}^{l})^{2}/2$.

Similarly, the $h_{2}$ scalar can also be produced via its $\phi$-component at a cross-section $\sigma_{h_{2}}=\sigma_{pp\to h_{2}}=\left.\sin^{2}\alpha\cdot\sigma_{pp\to h}\right|_{m^{\rm kin.}_{h}=m_{h_{2}}}$, where the latter is the SM Higgs-like boson production cross-section evaluated for a different kinematical mass $m_{h_{2}}$. Since both the $h_{1}\rightarrow NN$ branching and the $h_{2}$’s production cross-section scale as $\sin^{2}\alpha$, assuming a near-unity branching of $h_{2}\rightarrow NN$, their contributions are the same order in terms of $h-s$ mixing.

We note that $h_{2}$ decay modes depend on the relative size between $m_{h_{1}},m_{h_{2}}$ and $m_{N}$. When $m_{h_{2}}>2m_{h_{1}}$, the $h_{2}\to h_{1}h_{1}$ channel opens up, but it is suppressed at small $\lambda$. $h_{2}$ may also decay to the SM fermion/gauge boson pairs, with an $\alpha^{2}$ suppression. Therefore, in case of a kinematically accessible $m_{h_{2}}>2m_{N}$ and a small $\lambda$, $h_{2}\to NN$ dominates $h_{2}$ decays and acquire a near-unity branching. In case $m_{h_{2}}\ll m_{h_{1}}$ and $m_{h_{2}}<2m_{N}$, however, the relative branching ratios become less transparent as $h_{2}$ decays via the $\phi$- and $s$- components compete between the $\alpha^{2}$-dependent two-body channels and the $|V_{lN}|^{2}$, $m_{N}$-dependent multi-body channels.

The total signal cross-section is then

[TABLE]

where $A_{\rm eff}^{\rm(1,2)}$ is the detector selection efficiency for $h_{1},h_{2}$ mediated event samples, which can be separately studied via Monte Carlo simulations. $h_{2}$’s event rate may quickly become subdominant with a heavy $h_{2}$ mass, and with $m_{h_{2}}\sim{\cal O}(m_{h_{1}})$ it can still be the main contributor due to its near-unity decay branching into $NN$. Note this $h_{1,2}$ mediated signal rate bypasses the vertex suppression from $\nu_{L}-N_{R}$ mixing, and serves as a probe for low $\nu_{L}-N_{R}$ mixing scenarios, e.g. Type-I seesaw models.

For convenience in phenomenology study we consider the simplified case where only the mass of one heavy neutrino $N_{1}$ lies below $\frac{1}{2}m_{h_{1}}$ and $N_{1}$ mixes with only one active neutrino flavor $\nu_{\mu}$. In this case only the LNV diagram in Fig. 1 contributes, and ${\rm BF}_{\rm sig.}$ reaches its maximum ${\rm BF}_{\rm sig.}=0.16\%$. Complication rises when both $m_{N}$ and $|V_{lN}|^{2}$ are small the $N$ lifetime can be quite long, which leads to displaced decay at colliders. Our study relies on reconstructing decay kinematics so we require $N$ decays inside the detector. Note that this decay-length issue can be circumvented in BSM models: $|V_{lN}|^{2}$ can be enhanced by inverse-seesaw Mohapatra:1986aw after introducing mixture into an extra singlet fermion; also if $N_{R}$ is charged under BSM interactions, e.g. in extra-$U(1)$ and Left-Right symmetric models, $N$ gains extra partial width that helps make its decay visible but it also suppresses BFl at the same time.

The major kinematic difference from a similar large invariant-mass search for doubly-charged Higgs Du:2018eaw ; Aaboud:2017qph is that our four leptons are much softer in energy, only adding up to a fraction of the Higgs boson mass in the optimal mass range $m_{N}<\frac{1}{2}m_{h_{1}}$. In addition, the same-sign lepton pair must rise from different parent particles in the $NN$ decay, leading to different kinematic topologies. Moreover, our final state contain two neutrinos which act as missing energy.

3 Background and multivariate analysis

The $e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}+\not{E}_{T}$ signal has very few SM background events at the $pp$ colliders due to the presence of two pairs of SSSF leptons. The irreducible background includes contribution from $pp\to 4\tau,WWZ(Z\to 2\tau)$ channels in which the flavor combinatorics among the four $e/\mu$ leptons after $\tau,W$ decays can produce the same final state. Because $4\tau$ channel is mainly $ZZ\to 4\tau$ after all selection cuts, it is related to the di-boson production, and is the leading background at lower invariant masses. The $pp\to 4W$ channel is much subdominant due to its higher number of weak interaction vertices. Another potential background may arise from events with jets, usually from decays of hadrons into leptons, misidentified as leptons. However, a reliable fake-lepton background is usually obtained by the “data-driven” approach, which relies on experimental data. A very relevant experiment study Aaboud:2017qph which has a similar signal final state shows that the background from doubly faked leptons can be subdominant to di-boson production. Besides, our signal requires two muons while faked leptons are mostly electrons. Thus, it might indicate that the fake-lepton contribution is controllable in our case. We include $4\tau$ and $WWZ$ channels as the main background in our $pp$ collision event simulations, and leave the fake-lepton process for further study when data become available.

We quote the measured Higgs boson’s single production cross-section via gluon fusion process $\sigma_{pp\to h}=54.7$ pb at 14-TeV  gghCrs14TeV and 740 pb at 100-TeV gghCrs100TeV . As $h_{2}$ is also mostly produced by gluon fusion, we adopt the N3LO scalar mass dependence Anastasiou:2016hlm to scale the ${h_{2}}$ production rate in $\sigma_{h_{2}}$. For data simulations we use MadGraph5_aMC@NLO version 2.6.1 Alwall:2014hca for parton level event generation, Pythia6 Sjostrand:2006za for the parton shower and hadronization, and Delphes 3.4.1 deFavereau:2013fsa for detector simulation.

At the HL-LHC (FCC-hh/SppC), we generate about 150 (165) million $4\tau$, 10 (15) million $WWZ$, and 0.1 million signal events for kinematic analyses. Simulated events are analyzed to reject the SM background and maximize the signal significance. First the following pre-selection is applied: (a) exactly four leptons with $p_{\rm T}(l_{1,2})>$ 10 GeV and $p_{\rm T}(l_{3,4})>$ 5 GeV; (b) one same-sign $e^{\pm}e^{\pm}$ pair and one $\mu^{\mp}\mu^{\mp}$ pair; (c) no tagged $\tau$ leptons or $b$-jets.

After pre-selection, in order to construct useful kinematical observables, the stransverse mass $M_{\rm T2}$Cheng:2008hk ; Lester:1999tx ; Barr:2003rg is exploited to identify the electron and muon from the same heavy neutrino decay. Since the final state has four leptons (e.g. $\mu_{1}^{+}\mu_{2}^{+}e_{1}^{-}e_{2}^{-}$) and the two leptons from the same $N$ decay must have opposite sign and different flavor (i.e. $\mu^{+}e^{-}$), we calculate the $M_{\rm T2}$ values corresponding to two different combinations of the two-lepton system for $\mu_{1}$ (i.e. $\mu_{1}^{+}e_{1}^{-}$ and $\mu_{1}^{+}e_{2}^{-}$). The combination with smaller $M_{\rm T2}$ value (e.g. $\mu_{1}^{+}e_{2}^{-}$) is considered from the same $N$ decay, while the remaining lepton pair (i.e. $\mu_{2}^{+}e_{1}^{-}$) is considered to be from the other $N$. We then input a comprehensive collection of 53 kinematic observables to the TMVA package Hocker:2007ht to perform the MVA.

1. I.

global observables:

1. I.1)

the missing transverse energy $\not{E}_{T}$; 2. I.2)

$M_{\rm T2}(\mu_{1}+e,\mu_{2}+e)$. 2. II.

observables for the system of lepton(s):

1. II.1)

$p_{\rm T}$ and the pseudorapidity $\eta$ of lepton $p_{\rm T}(\mu_{1})$, $p_{\rm T}(\mu_{2})$, $p_{\rm T}(e_{1})$, $p_{\rm T}(e_{2})$, $\eta(\mu_{1})$, $\eta(\mu_{2})$, $\eta(e_{1})$, $\eta(e_{2})$; 2. II.2)

$p_{\rm T}$, $\eta$ and the invariant mass $M$ of leptons $p_{\rm T}(\mu_{1}+\mu_{2})$, $\eta(\mu_{1}+\mu_{2})$, $M(\mu_{1}+\mu_{2})$, $p_{\rm T}(e_{1}+e_{2})$, $\eta(e_{1}+e_{2})$, $M(e_{1}+e_{2})$, $p_{\rm T}(\mu_{1}+e)$, $\eta(\mu_{1}+e)$, $M(\mu_{1}+e)$, $p_{\rm T}(\mu_{2}+e)$, $\eta(\mu_{2}+e)$, $M(\mu_{2}+e)$, $p_{\rm T}(\mu_{1}+\mu_{2}+e_{1}+e_{2})$, $\eta(\mu_{1}+\mu_{2}+e_{1}+e_{2})$, $M(\mu_{1}+\mu_{2}+e_{1}+e_{2})$; 3. II.3)

the pseudorapidity difference $\Delta\eta$ and the azimuthal angle difference $\Delta\phi$ of the system of leptons $\Delta\eta(\mu_{1},\mu_{2})$, $\Delta\phi(\mu_{1},\mu_{2})$, $\Delta\eta(e_{1},e_{2})$, $\Delta\phi(e_{1},e_{2})$, $\Delta\eta(\mu_{1},e)$, $\Delta\phi(\mu_{1},e)$, $\Delta\eta(\mu_{2},e)$, $\Delta\phi(\mu_{2},e)$, $\Delta\eta(\mu_{1}+e,\mu_{2}+e)$, $\Delta\phi(\mu_{1}+e,\mu_{2}+e)$. 3. III.

observables between the missing energy and lepton(s):

1. III.1)

$\Delta\phi$ between the missing transverse momentum and lepton(s) $\Delta\phi(\not{E}_{T},\mu_{1})$, $\Delta\phi(\not{E}_{T},\mu_{2})$, $\Delta\phi(\not{E}_{T},e_{1})$, $\Delta\phi(\not{E}_{T},e_{2})$, $\Delta\phi(\not{E}_{T},\mu_{1}+\mu_{2})$, $\Delta\phi(\not{E}_{T},e_{1}+e_{2})$, $\Delta\phi(\not{E}_{T},\mu_{1}+e)$, $\Delta\phi(\not{E}_{T},\mu_{2}+e)$, $\Delta\phi(\not{E}_{T},\mu_{1}+\mu_{2}+e_{1}+e_{2})$; 2. III.2)

the transverse mass $M_{\rm T}$ of the system formed by the missing momentum plus lepton(s) $M_{\rm T}(\not{E}_{T},\mu_{1})$, $M_{\rm T}(\not{E}_{T},\mu_{2})$, $M_{\rm T}(\not{E}_{T},e_{1})$, $M_{\rm T}(\not{E}_{T},e_{2})$, $M_{\rm T}(\not{E}_{T},\mu_{1}+\mu_{2})$, $M_{\rm T}(\not{E}_{T},e_{1}+e_{2})$, $M_{\rm T}(\not{E}_{T},\mu_{1}+e)$, $M_{\rm T}(\not{E}_{T},\mu_{2}+e)$, $M_{\rm T}(\not{E}_{T},\mu_{1}+\mu_{2}+e_{1}+e_{2})$.

Here $\mu_{1}$, $\mu_{2}$, $e_{1}$, $e_{2}$ denote muons or electrons with the first or second leading transverse momentum, while for ($\mu_{1}+e$) or $(\mu_{1},e)$ the muon and electron are two leptons from the same N decay. The variables can be ranked according to their importance in the TMVA analysis. For the benchmark case with signal from $h_{1}\to NN$ and $m_{N}=20$ GeV, at the HL-LHC, the most useful observables are found to be $M_{\rm T2}(\mu_{1}+e,\mu_{2}+e)$, $\Delta\phi(\mu_{2},e)$, $M(\mu_{1}+e)$, $\Delta\phi(\mu_{1},e)$, $M(\mu_{2}+e)$, $\Delta\eta(\mu_{2},e)$, $\Delta\eta(\mu_{1},e)$, $M_{\rm T}(\not{E}_{T},\mu_{1}+\mu_{2}+e_{1}+e_{2})$, $\Delta\phi(\not{E}_{T},\mu_{1}+\mu_{2})$, $\Delta\phi(\not{E}_{T},e_{1}+e_{2})$, $\Delta\phi(\mu_{1},\mu_{2})$, $M(\mu_{1}+\mu_{2}+e_{1}+e_{2})$. We note that since the kinematics vary with $m_{N}$, the rank may also change for different heavy neutrino masses.

Fig. 5 in the Appendix A illustrates the distributions of a few input observables for a $m_{N}$=20 GeV signal sample (black, filled), and those for the SM background including $4\tau$ (red) and $WWZ$ (blue) processes after the pre-selection cuts. Many observables show clear distinctions between the signal and background. It is worth noting that the signal’s transverse mass $M_{\rm T2}$ distribution develops a sharp end-point around $m_{N}$, which means that it can be used to determine the mass of the heavy neutrino as well if this new physics scenario is discovered.

The resulting Boosted Decision Trees (BDT) distributions from the MVA demonstrate excellent separation between signal and background, as shown in Fig. 2, which can reduce the background to a negligible level. Since the kinematics vary with $m_{N}$, here we show a representative result of $h_{1}$ decay with $m_{N}=20$ GeV. Table 1 lists the corresponding background event rate estimated after pre-selection and the MVA, where the BDT cut is optimized for the signal significance, given by

[TABLE]

where $N_{s}$ ($N_{b}$) is the number of signal (total background) events after all selection cuts with 3 $\rm ab^{-1}$ integrated luminosity at the HL-LHC or 20 $\rm ab^{-1}$ integrated luminosity at the FCC-hh/SppC. Note that Table 1 does not include $h_{2}$ contribution; $h_{2}$ increases the signal but the BDT cut needs to be re-optimized according to its fraction in the signal.

4 Collider sensitivites

Since the final state kinematics vary with the heavy neutrino mass, We simulate the signal sample and perform the MVA for each $M_{N}$ case from 10 to 60 GeV, at increment of 10 GeV. The corresponding BDT distributions at the HL-LHC with $\sqrt{s}=14$ TeV are shown in Fig. 6, and distributions at the FCC-hh/SppC with $\sqrt{s}=100$ TeV are shown in Fig. 7, see Appendix B.

As $m_{N}$ increases, the kinematics of the $4\tau$ background becomes similar to that of the signal process $NN$, which renders its BDT distribution largely overlapping with that of the signal. At each $m_{N}$, the BDT criterion is optimized for maximal signal significance from its BDT distribution. The selection efficiencies for both $NN$ signal and background processes at the HL-LHC and FCC-hh/SppC for different heavy neutrino masses are presented in Table 2.

Fig. 3 shows the 95% confidence level (C.L.) limits on the combined branching fraction of Higgs boson into our final state when $h_{2}$ is heavy and only $h_{1}$ is relevant to the $NN$ signal. Here, the 95% C.L. limits correspond to 2-$\sigma$ signal significance given by the Eq. (11). The number of events after all selections, $N_{s}(N_{b})$, is the product of the production cross-section, luminosity and total selection efficiency for the signal (total background). Note the signal production cross-section is proportional to the Higgs decay branching fraction, cf. Eq. (10). The $m_{N}$ dependence of the limit is the result of two competing reasons: the four-lepton pre-selection efficiencies increase at larger $m_{N}$, while the BDT cuts reject background more efficiently at lower $m_{N}$. When $m_{N}=20$ GeV, the combined branching fraction can be probed to as tiny as $4.4\times 10^{-7}$, $6.7\times 10^{-8}$, $1.9\times 10^{-9}$ at the 13, 14, 100 TeV $pp$ collision runs, respectively. Since the kinematics are quite similar between 13-TeV and 14-TeV, the 13-TeV limits are derived by adopting the same detector selection efficiencies as those for 14-TeV. $\sigma_{pp\to h}=48.6$ pb at 13-TeV [23] is used for signal production cross-section. For background processes, their production cross-sections at 13 TeV are calculated by MadGraph5.

Now we interpret these limits in the content of our ‘small mixing’ scenario. By Eq. (4) the mixing angle is most sensitive to the mass-square difference between $\phi$ and $s$, which can be approximated the mass-square difference between $h_{1}$ and $h_{2}$, off by a correction $\Delta m^{2}_{s,\phi}=\Delta m^{2}_{h_{2},h_{1}}\cdot[1+{\cal O}(\sin^{2}\alpha)]$. Generally, $m_{s}$ can be lighter than $v_{S}$ and it is possible to have $h_{2}$ at the same (or lower) mass scale as the SM Higgs, so that $|\Delta m^{2}_{s,\phi}|\sim{m_{h}^{2}}$ even if $v_{S}$ is more massive, at the TeV scale or above. So we adopt a 150 GeV $h_{2}$ as a showcase scenario, with the consideration that an $m_{h_{2}}$ below the $2m_{W}$ threshold can avoid a dramatic increment in ${\rm BF}_{h_{2}\to WW,ZZ}$ which suppresses the $NN$ branching and complicate our physics picture. In this scenario, both $h_{2}$ and $h_{1}$ decays contribute significantly. Fig. 4 illustrates the 95% C.L. contour limits on $\lambda,v_{S}$ from various $pp$ collision runs. The width of each filled band is due to varying $m_{N}$ from 10 to 60 GeV, which affects the cut efficiencies. $R$ is the fraction of $h_{1}$ contribution in the total $NN$ signal events. $h_{1}$ contributes mostly to the signal at low $v_{S}$, while $h_{2}$ can also contribute due to its relatively light mass, and becomes dominant towards $v_{S}\sim$ TeV. The limits indicate that the 13-TeV LHC with 280 fb*-1* data can access the weak-scale $v_{S}$ part of the decoupling region, while the future HL-LHC and FCC-hh/SppC can be sensitive to even lower $\mathit{\Phi}-S$ mixing region, where the singlet components in $h_{1}$ can be as low as $10^{-4}$ to $10^{-6}$.

For heavier $h_{2}$, at the TeV scale, $h_{2}$’s contribution to the signal diminishes and the mixing $\sin^{2}\alpha$ is suppressed by the TeV $m_{s}$. However, since $y_{N}=m_{N}/(2v_{S})$, an $m_{N}<\frac{1}{2}m_{h1}$ also significantly suppresses the $hNN$ coupling, leading to $m_{s}^{-4}$ suppression in the signal rate. Therefore, a large $m_{s}$ would require a sizable $\lambda$ to yield a detectable signal rate, which could violate the decoupling condition as $v_{S}$ is generally expected to be comparable to or higher than $m_{s}$. In such cases, the $h-s$ mixing needs to be parametrized under specific models where $\mathit{\Phi},S$ sectors are no longer independent. Note in $SU(2)_{L}$ doublet(s)+singlet scalar models, the joint scalar potential minimization removes $\lambda$ dependence from the diagonal scalar mass-square terms at the tree-level, indicating Eq. (6) to be a strong requirement if $\lambda\ll 1$, which suppresses the singlet’s loop contribution to the Higgs mass.

5 Conclusion

We show that the fully leptonic Higgs decay channel $e^{\pm}e^{\pm}\mu^{\mp}\mu^{\mp}+\not{E}_{T}$ is almost SM background-free and is accessible in both the lepton number violating and conserving heavy neutrino $N$ decays. This rare channel can be a high-precision probe for the mixing between the Higgs boson and a singlet scalar boson which dynamically gives the $N$ mass in the $|V_{lN}|$-suppressed simple seesaw scheme. We consider a generic ‘decoupled’ scenario where the Higgs and singlet scalar bosons only share a small mixing, and demonstrate that the signal channel is capable of testing such decoupled scenario if the singlet and the Higgs bosons are in the comparable mass range. Comprehensive signal and SM background analyses with machine-learning MVA yield stringent limits on the signal Higgs boson decay branching fraction at the current and future $pp$ colliders. For a benchmark case with the singlet-dominated $h_{2}$ at 150 GeV, the 13-TeV LHC, future HL-LHC and FCC-hh/SppC are sensitive to the Higgs-singlet mixing to $10^{-3}$, $10^{-4}$ and $10^{-6}$ with 280 $\rm fb^{-1}$, 3 $\rm ab^{-1}$, and 20 $\rm ab^{-1}$ $pp$ collision luminosities, respectively. In addition, this test does not require a sizable left-right neutrino mixing and it is complimentary to the existing $|V_{lN}|^{2}$-based searches.

Appendix A Distributions of representative input observables

In Fig. 5, we show the distributions of some observables for the signal (black, filled), and those for the SM background including $4\tau$ (red) and $WWZ$ (blue) processes after the pre-selection cuts. These plots illustrate a $h_{1}$ decay signal by gluon fusion at the HL-LHC, with $m_{N}$=20 GeV.

Appendix B Distributions of BDT responses

In Fig. 6, we show the distributions of BDT responses of the signal and the SM background processes at the HL-LHC with $\sqrt{s}=14$ TeV for different $m_{N}$ assumptions. In these plots, the signals are from $h_{1}$ decay only. The distributions at the FCC-hh/SppC with $\sqrt{s}=100$ TeV are presented in Fig. 7.

Appendix C Selection efficiency table

In Table 2, we show selection efficiencies for both signal $NN$ and background processes at the HL-LHC and FCC-hh/SppC for different heavy neutrino masses. The total selection efficiency after both pre-selection and BDT criteria is the product of two efficiencies in the same column. The numbers of events for background after all selections can be calculated by multiplying the initial numbers in the Table 1 by the total cut efficiency, while the number of events for signal can be calculated from Eq. (10) by substituting the total cut efficiency for $NN$ as $A_{\rm eff}^{(1)}$.

Acknowledgements.

We thank Monica D’Onofrio and Yusheng Wu for helpful discussions. Y.G. is supported by the IHEP, CAS under the CEPC theory grant (2019-2020) and partially under grant no. Y7515560U1. M.J. by the National Natural Science Foundation of China under grant no. 11475191. K.W. by the Excellent Young Talents Program of the Wuhan University of Technology, the National Natural Science Foundation of China under grant no. 11905162, and the CEPC theory grant (2019-2020) of IHEP, CAS.