Resolving the degeneracy in top quark Yukawa coupling with Higgs pair production

TL;DR

This paper investigates how Higgs pair production at high-energy colliders can disentangle the effects of different dimension-six operators modifying the top quark Yukawa coupling, providing clearer insights into new physics beyond the Standard Model.

Contribution

It demonstrates that combining multiple Higgs production channels can resolve degeneracies in Wilson coefficients affecting the top Yukawa coupling at future colliders.

Findings

Higgs pair production constrains $c_y$ at 100 TeV colliders.

Combining single and double Higgs production separates contributions of $c_H$ and $c_y$.

Different models' predictions for $c_y$ and $c_t$ can be distinguished.

Abstract

The top quark Yukawa coupling ($y_t$) can be modified by two dimension-six operators $\mathcal{O}_H$ and $\mathcal{O}_y$ with the corresponding Wilson coefficients $c_H$ and $c_y$, whose individual contribution cannot be distinguished by measuring $y_t$ alone. However, such a degeneracy can be resolved with Higgs boson pair production. In this work we explore the potential of resolving the degeneracy of the unknown Wilson coefficients $c_H$ and $c_y$ at the 14~TeV LHC and the 100~TeV hadron colliders. Combining the information of the single Higgs production, $t\bar{t}h$ associated production and Higgs pair production, the individual contribution of $c_H$ and $c_y$ to $y_t$ can be separated. Regardless of the value of $c_H$, the Higgs pair production can give a strong constraint on $c_y$ at the 100 TeV hadron collider. We further show that it is possible to differentiate various $c_y$ and $c_t$ values predicted in several benchmark models.

Tables

Table 1. Table 1: The Wilson coefficients c H subscript 𝑐 𝐻 c_{H} and c y subscript 𝑐 𝑦 c_{y} predicted by various NP models  del Aguila et al. ( 2000 ); Low et al. ( 2010 ); Corbett et al. ( 2018 ) .

A) singlet scalar Low et al. (2010); Corbett et al. (2018)	B) 2HDM Corbett et al. (2018) or VLQs del Aguila et al. (2000)
$c_H>0,c_y=0$	$c_H=0,c_y>0$
C) real triplet scalar Low et al. (2010)	D) complex triplet scalar Low et al. (2010)
E) vectors Low et al. (2010)	F) 2HDM Corbett et al. (2018) or VLQs  del Aguila et al. (2000)
$c_H=3c_y>0$

Equations

$$\mathcal{L}={\mathcal{L}}_{\text{SM}}+c_H{\mathcal{O}}_H+(c_y{\mathcal{O}}_y+\text{h.c.})+\cdots$$

$${\mathcal{O}}_H$$

$${\mathcal{O}}_y$$

$${\mathcal{L}}_h$$

$$\displaystyle\overline{|{\cal M}|^{2}}=\frac{\alpha_{s}^{2}\hat{s}^{2}}{256\pi^{2}v^{4}}\bigg{[}$$

$$h\to \frac{h}{\sqrt{1+c_H}}-\frac{c_H{h}^2}{2(1+c_H{)}^{2}v},m_t\to \frac{m_t}{1+\frac{1}{2}{c}_y}.$$

$$c_t=\frac{2+3c_y}{\sqrt{1+c_H}(2+c_y)},c_{2t}=\frac{c_H(3c_y-2)+6c_y}{2(c_H+1{)}^2(c_y+2)}.$$

$$c_g^{\pm }=\frac{3}{2}\left(-c_t{F}_{△}\pm \sqrt{R_h}|F_{△}|\right),$$

$$R_{hh}$$

$$(a)c_g=0,c_3=1;$$

$$(c)R_h=0.9,c_3=1,c_g^{+};$$

$$(e)R_h=1,c_3=1,c_g^{+};$$

$$(g)R_h=1.1,c_3=1,c_g^{+};$$

$${\sigma }_{\text{cut}}=\int d{m}_{hh}\frac{d\sigma }{d{m}_{hh}}\mathcal{A}(m_{hh})$$

$$14\mathrm{TeV}$$

$$100\mathrm{TeV}$$

$$\mathcal{Z}$$

$$t=-2\ln \frac{L(c_H,c_y)}{L(0,0)}$$

$$L(c_H,c_y)=P(\text{data}|n_b+n_s(c_H,c_y)).$$

Full text

††preprint:

MSUHEP-19-009

**Introduction. **Top quark Yukawa coupling ($y_{t}$) is the only coupling with the magnitude of order one in the Standard Model (SM). As the largest Yukawa coupling, it is important for vacuum stability and cosmology Degrassi et al. (2012); Bezrukov and Shaposhnikov (2015). Besides, in many new physics (NP) scenarios Martin (1997); Contino (2011); Bellazzini et al. (2014); Panico and Wulzer (2016); Csaki and Tanedo (2015), top quark plays an important role in triggering the electroweak symmetry breaking (EWSB) and is directly connected to new physics beyond the SM. Therefore, it is highly motivated to understand the top quark Yukawa sector better, both theoretically and experimentally. The parameter $y_{t}$ can be measured directly by the $t\bar{t}h$ associated production. Recently, this process is confirmed by both the ATLAS and CMS collaborations with signal strengths $\mu_{t\bar{t}h}=1.32_{-0.26}^{+0.28}$ and $1.26^{+0.31}_{-0.26}$ Aaboud et al. (2018); Sirunyan et al. (2018a), respectively, at the Large Hadron Collider (LHC) with $\sqrt{s}=13~{}\text{TeV}$. Besides, $y_{t}$ can also be measured in loop-induced single Higgs boson production collaboration (2018); Collaboration (2018), $t(\bar{t})hj$ associated production Sirunyan et al. (2018b) and multi-top production processes Cao et al. (2017a, 2019a). With higher luminosity being accumulated, one expects the accuracy on $y_{t}$ can be further improved. It is thus timely to study what kind of NP can modify $y_{t}$.

In general, we can parameterize NP effects on $y_{t}$ by several higher dimensional operators in a model independent way. Out of the complete set of dimension-6 operators listed in Ref. Pomarol and Riva (2014), we consider in this work the two operators which can modify $y_{t}$ at tree level:

[TABLE]

in the so called Strongly-Interacting Light Higgs (SILH) basis Giudice et al. (2007); Contino et al. (2013), where the dimension-six operators

[TABLE]

Here, $c_{H}$ and $c_{y}$ are the corresponding Wilson coefficients with $c_{y}$ being assumed to be real, $Q_{L}$ is the left-handed third-family quark doublet, $v=246~{}{\rm GeV}$ is the vacuum expectation value, $y_{t}^{\rm SM}=m_{t}/v$ is the SM top Yukawa coupling, and $m_{t}$ is the top quark mass. As $\mathcal{O}_{H}$ modifies the Higgs boson wave function, it can universally shift all the single Higgs couplings, hence, affects $y_{t}$.

Theoretically, the operators $\mathcal{O}_{H}$ and $\mathcal{O}_{y}$ can be induced by several different NP scenarios. For example, scalar singlets interacting with the Higgs doublet can induce the universal $\mathcal{O}_{H}$ operator Craig et al. (2013, 2015); Dawson and Murphy (2017); Corbett et al. (2018); Cao et al. (2018), while additional vector-like fermions can induce the operator $\mathcal{O}_{y}$ via mixing with the top quark del Aguila et al. (2000); Chen et al. (2017). As both the operators in Eq. (2) can induce deviations in $y_{t}$, one cannot differentiate their individual contributions if we only measure the top Yukawa coupling. Even if $y_{t}$ is measured to be consistent with the SM prediction, one cannot exclude the possibility of having cancellation among different NP operators. Or, if the deviation in $y_{t}$ is established, we still need to separate the effects of $\mathcal{O}_{H}$ and $\mathcal{O}_{y}$ for better understanding the origin of NP. Since the effect of the $\mathcal{O}_{H}$ operator is to simply rescale any amplitude involving a single Higgs boson by a factor $1/\sqrt{1+c_{H}}$ after renormalizing the Higgs boson field, its effect is universal and can be measured from studying the $hVV~{}(V=W^{\pm},Z)$ couplings. However as shown in Ref. Cao et al. (2019b), in case that Higgs boson is a pseudo Nambu-Goldstone boson, other operator (at the order of $\mathcal{O}(p^{2})$) can mimic the effect induced by $\mathcal{O}_{H}$ on $hVV$ couplings. Hence, it requires novel method to separately measure the coefficients of those two types of operators Cao et al. (2019b). Likewise, in this work, we explore the possibility of separately measuring the coefficients of $\mathcal{O}_{H}$ and $\mathcal{O}_{y}$, both contributing to top Yukawa coupling $y_{t}$, through Higgs boson pair production $gg\to hh$. In addition to modifying the single Higgs effective coupling of $ht\bar{t}$, both $\mathcal{O}_{H}$ and $\mathcal{O}_{y}$ can also contribute to the effective coupling $hht\bar{t}$, but with different combinations, which can be measured via Higgs boson pair production. Namely, studying Higgs boson pair production can be utilized not only for measuring Higgs self interactions, but also for discriminating new physics scenarios in the top sector. Only after the individual contribution of each effective operator is extracted can we further solidify the SM or otherwise establish NP models.

**Higgs boson pair production. **The gluon-initiated Higgs pair production can be used to measure the trilinear Higgs self-coupling Glover and van der Bij (1988); Baur et al. (2003, 2004); Dolan et al. (2012); Baglio et al. (2013); Papaefstathiou et al. (2013); Shao et al. (2013); Goertz et al. (2013); Barger et al. (2014); Barr et al. (2014); Ferreira de Lima et al. (2014); Li et al. (2015); Behr et al. (2016); Alves et al. (2017); Adhikary et al. (2018); Kim et al. (2018); Goncalves et al. (2018); Chang et al. (2018); Borowka et al. (2018); Homiller and Meade (2019); Kim et al. (2019a, b). It is also sensitive to various NP models Agashe et al. (2005); Contino et al. (2007); Arkani-Hamed et al. (2001, 2002); Dawson and Murphy (2017); Corbett et al. (2018); Huang et al. (2018); Basler et al. (2018); Babu and Jana (2018); Li et al. (2019). After the EWSB, the effective Lagrangian related to the non-resonant Higgs pair production is Cao et al. (2016, 2017b); Chen and Low (2014); Goertz et al. (2015); Azatov et al. (2015); Dawson et al. (2015)

[TABLE]

where $a$ is the color index, $\alpha_{s}=g_{s}^{2}/4\pi$ with $g_{s}$ being the strong coupling strength, and $m_{h}$ is the Higgs boson mass. The SM, at tree level, corresponds to $c_{3}=c_{t}=1$ and $c_{2t}=c_{g}=0$. Then the squared amplitude of $gg\to hh$, after averaging over the gluon polarizations and colors, is Dawson et al. (2015)

[TABLE]

where $F_{\bigtriangleup}\equiv F_{\bigtriangleup}({\hat{s}},{\hat{t}},m_{h}^{2},m_{t}^{2})$, $F_{\Box}\equiv F_{\Box}({\hat{s}},{\hat{t}},m_{h}^{2},m_{t}^{2})$ and $G_{\Box}\equiv G_{\Box}({\hat{s}},{\hat{t}},m_{h}^{2},m_{t}^{2})$ are the form factors Plehn et al. (1996) with $\hat{s}$ and $\hat{t}$ being the canonical Mandelstam variables. The first term inside the bracket contributes to $s$-wave and the $G_{\Box}$ term to $d$-wave component whose contribution to total cross section is numerically negligible Dawson et al. (2013).

To avoid any momentum dependent contributions to the Higgs self couplings induced by $\mathcal{O}_{H}$, we adopt the generalized canonical normalization of the Higgs filed and perform the field redefinition Plehn (2012); Goertz et al. (2015):

[TABLE]

Applying this shift of the Higgs field throughout the Standard Model Lagrangian $\mathcal{L}_{\text{SM}}$ leads to multiple Higgs couplings to any pair of massive gauge bosons or massive fermions. The effective couplings $c_{t}$ and $c_{2t}$ are derived as

[TABLE]

We note that $c_{t}$ and $c_{2t}$ have different dependence on the Wilson coefficients $c_{H}$ and $c_{y}$. Also, $c_{t}=1-c_{H}/2+c_{y}$ and $c_{2t}=3c_{y}/2-c_{H}/2$, when $c_{y},c_{H}\ll 1$ Azatov et al. (2015); Goertz et al. (2015).

In Fig. 1 the dependence of $c_{t}$ and $c_{2t}$ on $c_{H}$ and $c_{y}$ is shown. It is clear that the slope of $c_{2t}$ (dashed lines) is different from $c_{t}$ (solid lines), especially when $c_{2t}<0$ and $c_{t}<1$. Precise study on the effective coupling $c_{2t}$ therefore offers the possibility to discriminate the effects of $c_{H}$ and $c_{y}$.

As shown in Eqs. (3) and (4), $c_{3}$ and $c_{g}$ also contribute to the Higgs pair production cross section. However, $c_{g}$ is already constrained to be within $c_{g}^{-}$ and $c_{g}^{+}$ by the signal strength measurements of single Higgs production $gg\to h$ collaboration (2018); Collaboration (2018)

[TABLE]

where $R_{h}\equiv\sigma(gg\to h)/\sigma^{\rm SM}(gg\to h)$, the sign ”$\pm$” refers to the cases $c_{t}F_{\bigtriangleup}>-2/3c_{g}$ and $c_{t}F_{\bigtriangleup}<-2/3c_{g}$, respectively. The combined fit to the single Higgs production, which depends on both $c_{t}$ and $c_{g}$ Cao et al. (2016), and decay using 13 TeV LHC data gives rise to $R_{h}=1.07\pm 0.09$ (ATLAS collaboration (2018)) and $R_{h}=1.23\pm 0.13$ (CMS Collaboration (2018)). By fixing $R_{h}$, we can replace $c_{g}$ with a function of $c_{t}$. Furthermore, the coupling $c_{3}$ in Eq. (3) is approximately equal to $1-{3\over 2}c_{H}+c_{6}$, as shown in Refs. Azatov et al. (2015); Goertz et al. (2015), and is weakly constrained, $-5.0<c_{3}<12.1$, by present data collaboration (2018); Sirunyan et al. (2018c). Here, $c_{6}$ is the coefficient of the dimension-6 operator ${\cal O}_{6}=\lambda/v^{2}|H|^{6}$. As the Higgs pair production cross section is more sensitive to the sign of $c_{3}$, rather than its magnitude Cao et al. (2017c), we will choose $c_{3}=\pm 1$ as our benchmark value in the following numerical analysis.

To compare $\sigma(gg\to hh)$ with the SM prediction, we define a ratio $R_{hh}$ as

[TABLE]

In Fig. 2, we show the contours of $R_{hh}$ at the 14 TeV LHC in the plane of $c_{t}$ and $c_{2t}$, with other parameters chosen as $c_{g}=0$ and $c_{3}=1$. $R_{hh}$ can be enhanced largely for both the positive and negative $c_{2t}$, but $R_{hh}$ is more sensitive to negative $c_{2t}$ when $c_{t}$ is of order one, which can be understood with Eq. (4). In the large top quark mass limit, $F_{\bigtriangleup}\to 2/3$ and $F_{\Box}\to-2/3$ Plehn et al. (1996). Besides, the $c_{t}^{2}F_{\Box}$ term dominates over the $c_{t}F_{\bigtriangleup}$ term for $c_{t}\gtrsim 1$. Therefore, a negative $c_{2t}$ can enhance $R_{hh}$ more easily than a positive $c_{2t}$, for this choice of $c_{g}$ and $c_{3}$ Cao et al. (2017b).

With the information from Figs. 1 and  2, we conclude that it is hopeful to discriminate the effects of $c_{H}$ and $c_{y}$ through Higgs pair production, especially for the negative $c_{2t}$ region. Furthermore, we could translate the above results in the plane of $(c_{H},c_{y})$. In Fig. 3, we show the contours of $R_{hh}$ with respect to the Wilson coefficients $c_{H}$ and $c_{y}$, where various choices of the effective couplings $c_{g}$ and $c_{3}$ are considered. They correspond to

[TABLE]

For the cases $(a)$ and $(b)$, the $hgg$ effective coupling $c_{g}$ is assumed to vanish while the trilinear Higgs self-coupling $c_{3}$ is assumed to be $+1$ and $-1$, respectively. In these cases, we include the constraints on the parameter space of $c_{H}$ and $c_{y}$ from the single Higgs production collaboration (2018); Collaboration (2018); cf. the gray band of Fig. 3 (a, b). It amounts to $c_{t}=1.03\pm 0.04$ (ATLAS) and $c_{t}=1.11\pm 0.06$ (CMS). For the cases from $(c)$ to $(h)$, $c_{g}$ is derived from a given $R_{h}$ value, cf. Eq. (7), while $c_{3}$ is fixed to be identical to the SM value. For comparison, we also show the parameter space constrained by the $t\bar{t}h$ measurements at the 13 TeV LHC, with an integrated luminosity of $79.8~{}{\rm fb}^{-1}$ from ATLAS Aaboud et al. (2018) and of $35.9~{}{\rm fb}^{-1}$ from CMS Sirunyan et al. (2018a). Here, we take the combined best fit $t\bar{t}h$ signal strength normalized to the SM prediction, such that $c_{t}=1.15^{+0.12}_{-0.11}$ (ATLAS) and $c_{t}=1.12^{+0.14}_{-0.12}$ (CMS).

With the result depicted in Fig. 3, several comments are in order:

• •

$R_{hh}$ is enhanced in some parameter space of $c_{H}$ and $c_{y}$, the magnitude of the enhancement strongly depends on the choice of $c_{3}$ and $c_{g}$.

• •

In case $(b)$, the cancellation between the triangle diagram and box diagram does not happen, because of the negative $c_{3}$ which further enhances $R_{hh}$ Baglio et al. (2013).

• •

In case of $c_{g}^{+}$ or $c_{g}^{-}$, the value of $c_{g}$ is extracted from the signal strength of single Higgs production process $R_{h}$. We find that the variation of $R_{h}=0.9,\ 1,$ or $1.1$ can only slightly change the value of $R_{hh}$.

• •

$R_{hh}$ is more sensitive to $c_{g}^{-}$ than $c_{g}^{+}$. According to Eq. (7), $c_{g}^{+}$ is close to zero for a positive $c_{t}$, and accordingly the contours of $R_{hh}$ are similar to the contours in case $(a)$. On the other hand, negative $c_{g}^{-}$ can significantly deviate from zero and $R_{hh}$ is largely enhanced (cf. Eq. (4)).

• •

In cases $(a)$, $(c)$, $(e)$ and $(g)$, the possible enhancement of $R_{hh}$ can only come from the deviations of $c_{t}$ and $c_{2t}$. With the information of Figs. 1 and 2, $R_{hh}$ can be largely enhanced when $c_{2t}<0$, which corresponds to the region $c_{y}<0$ in Fig. 3.

**Sensitivity at the 14 TeV LHC and the 100 TeV hadron collider. **Now we discuss the potential of discriminating the Wilson coefficients $c_{H}$ and $c_{y}$ at the 14 TeV LHC and the 100 TeV proton-proton hadron collider. As a concrete example, we examine the $b\bar{b}\gamma\gamma$ channel, which has been studied by the ATLAS collaboration at the High-Luminosity LHC (HL-LHC), operating at the center-of-mass energy of $14~{}{\rm TeV}$ with an integrated luminosity of $3~{}{\rm ab}^{-1}$ ATL (2014a). As being discussed in Refs. Cao et al. (2016, 2017b), an analytical function can be used to describe the fraction of signal events passing through the kinematic cuts. Since the Higgs boson is a scalar particle and the $gg\to hh$ process is dominated by the $s$-wave contribution, the acceptance of the kinematic cuts, in the inclusive Higgs pair production, will mainly depend on the invariant mass of the Higgs boson pair ($m_{hh}$). The cross section of $gg\to hh$, after imposing the kinemati cuts, can be written as Cao et al. (2016)

[TABLE]

where the efficiency function $\mathcal{A}(m_{hh})$ has been given in Refs. Cao et al. (2016, 2017b), both for the 14 TeV LHC and the 100 TeV hadron collider. In Ref. Cao et al. (2017b), it is demonstrated that the results obtained from the analytic cut efficiency functions agree very well with other results with more detailed simulations Azatov et al. (2015).

The SM backgrounds for the process of $gg\to hh$ production include $b\bar{b}\gamma\gamma$, $c\bar{c}\gamma\gamma$, $b\bar{b}\gamma j$, $jj\gamma\gamma$, $b\bar{b}jj$, $t\bar{t}(\geqslant 1\ell^{\pm})$, $t\bar{t}\gamma$, $Z(\to b\bar{b})h(\to\gamma\gamma)$, $t\bar{t}h(\to\gamma\gamma)$ and $b\bar{b}h(\to\gamma\gamma)$, etc. At the 14 TeV LHC, with the integrated luminosity of $\mathcal{L}=3~{}{\rm ab}^{-1}$ ATL (2014a), and the 100 TeV hadron collider, with $\mathcal{L}=30~{}{\rm ab}^{-1}$ Contino et al. (2017), the signal ($n_{s}$) and background ($n_{b}$) events in the SM are, respectively,

[TABLE]

With the event numbers listed above, the discovery potential for the signal process can be evaluated by using Cowan et al. (2011)

[TABLE]

In Figs. 4 and 5, we show the contours of discovery potential for $gg\to hh\to b\bar{b}\gamma\gamma$ at the 14 TeV LHC and the 100 TeV hadron collider, with various integrated luminosities. The $2\sigma$ and $5\sigma$ discovery potentials correspond to $\mathcal{Z}=2$ and $\mathcal{Z}=5$, respectively. As mentioned earlier, the signal strength $R_{h}$ is measured with an accuracy of about $10\%$, and $R_{hh}$ is not sensitive to the value of $R_{h}$ (for $0.9\leq R_{h}\leq 1.1$), we therefore take $R_{h}=1$ as the benchmark to show the discovery potential in those figures.

Several comments are in order regarding the discovery potential of $gg\to hh$ with respect to the Wilson coefficients $c_{H}$ and $c_{y}$. In Fig. 4, only the parameter space on the left of the curve, or below the curve, labeled by a specified integrated luminosity at the HL-LHC, can be reached at the $2\sigma$ or $5\sigma$ C.L.. For case (a), the SM point cannot be probed by measuring only the $gg\to hh$ production. A larger parameter space is reachable at $2\sigma$ C.L. for case $(b)$, as compared to cases (a) and (c), due to the large enhancement of $R_{hh}$ with negative $c_{3}$. For case $(d)$, with negative $c_{g}$, the Higgs pair production cross section can be enhanced by a factor of about 10, as compared to the SM value (cf. Fig. 3$(f)$). Consequently, with an integrated luminosity of $1~{}{\rm ab}^{-1}$ at the HL-LHC, most of the considered parameter space (for $|c_{H}|<1$ and $|c_{y}|<1$) can already be reached at the $5\sigma$ C.L.. For comparison, in the same figure, we also show the constraints imposed by the projected $2\sigma$ errors in the single Higgs production and $t\bar{t}h$ measurement at the HL-LHC, with an integrated luminosity of $3~{}{\rm ab}^{-1}$. It amounts to $c_{t}=1\pm 0.02$ (single Higgs with $c_{g}=0$) and $c_{t}=1\pm 0.05$ ($t\bar{t}h$ production) ATL (2014b) which have included the statistical and experimental systematic uncertainties.

In Fig. 5, we see that the discovery potential for the $gg\to hh$ measurement is much improved at the 100 TeV hadron collider, for all the benchmark cases. The $5\sigma$ discovery significance can be easily reached. For cases $(a)$ and $(c)$, large region of $c_{y},c_{H}<0.2$ can be discovered with the integrated luminosity of $0.5~{}\text{ab}^{-1}$. For case $(b)$, with negative $c_{3}$, the region of $c_{y},c_{H}<0.5$ can be almost discovered with only $0.1~{}\text{ab}^{-1}$. For case $(d)$, with negative $c_{g}$, the currently allowed region can be explored with $2~{}\text{fb}^{-1}$. For comparison, in the same figure, we also show the constraints imposed by the projected $2\sigma$ errors in the single Higgs production and $t\bar{t}h$ measurement at the 100 TeV hadron collider, with an integrated luminosity of $3~{}{\rm ab}^{-1}$. It amounts to $c_{t}=1\pm 0.0072$ (single Higgs production, with $c_{g}=0$) and $c_{t}=1\pm 0.013$ ($t\bar{t}h$ production) Mangano et al. (2016, 2018). Note that both the statistical and systematic uncertainties are included in the single Higgs analysis, while only statistical error is discussed in $t\bar{t}h$ production. Here, we have scaled down the error in $t\bar{t}h$ measurement by the inverse of the square root of integrated luminosity.

To estimate the expected accuracy for measuring $(c_{H},c_{y})$ with the Higgs pair production at the 14 TeV LHC and the 100 TeV hadron collider, respectively, we perform a log likelihood ratio test Cowan et al. (2011) for the hypothesis with non-zero $c_{H},c_{y}$ against the hypothesis with $c_{H}=c_{y}=0$. The test ratio is defined as Cowan et al. (2011)

[TABLE]

where the likelihood function $L(c_{H},c_{y})$ is

[TABLE]

Here $P(k|\lambda)$ is the usual Poisson distribution function, $P(k|\lambda)=\lambda^{k}e^{-\lambda}/k!$. We assume the observed data is generated under the hypothesis with $c_{H}=c_{y}=0$ Cowan et al. (2011); Kumar and Martin (2015) and calculate the two-sided $p$-value. For convenience, we convert the $p$-value into the equivalent significance $Z=\Phi^{-1}(1-1/2p)=\sqrt{2}~{}\text{Erf}^{-1}(1-p)$ Tanabashi et al. (2018); Cowan et al. (2011), where $\Phi$ the cumulative distribution of the standard Gaussian and Erf is the error function. In our case, Z=\sqrt{2\big{[}n_{0}\ln\dfrac{n_{0}}{n_{1}}+(n_{1}-n_{0})\big{]}} with $n_{0}=n_{b}+n_{s}(0,0)$ and $n_{1}=n_{b}+n_{s}(c_{H},c_{y})$. The discrimination between the hypothesis with arbitrary $(c_{H},c_{y})$ and the hypothesis with $c_{H}=c_{y}=0$ is shown in Figs. 6 and 7. The hypothesis with $(c_{H},c_{y})$ outside of the blue bands is rejected at $1\sigma$ level for the HL-LHC and the 100 TeV hadron collider, respectively. After combining the measurements of single Higgs production (gray band) and $t\bar{t}h$ production (yellow band) at the HL-LHC, it is possible to differentiate $c_{H}$ and $c_{y}$ at the HL-LHC for cases $(b)$ and $(d)$, but it becomes challenging for cases $(a)$ and $(c)$, cf. Fig. 6. The situation will be much improved at the 100 TeV hadron collider, cf. Fig. 7. It is obvious that the Higgs pair production is more sensitive to $c_{y}$ than to $c_{H}$. We find that at the $1\sigma$ C.L., the combined constraint from single Higgs production, $t\bar{t}h$ production and the Higgs pair production measurements, at the 100 TeV hadron collider with the integrated luminosity of $3~{}\text{ab}^{-1}$, yields $-0.04<c_{y}<0$ and $-0.08<c_{H}<0.08$. To further constrain $c_{H}$, it is necessary to improve the measurement of $t\bar{t}h$ associated production. However, if the Higgs boson is a SM-like particle, $c_{H}$ could be constrained by the $hVV$ coupling measurement to $1\%\sim 2\%$ level Mangano et al. (2018).

Given the good sensitivity of differentiating $c_{H}$ with $c_{y}$ at the 100 TeV hadron collider, it is worthwhile clarifying the specific values of $(c_{H},c_{y})$, as induced by several generic classes of NP models Corbett et al. (2018); del Aguila et al. (2000); Low et al. (2010). Table 1 lists the Wilson coefficients $c_{H}$ and $c_{y}$ predicted by various NP models del Aguila et al. (2000); Low et al. (2010); Corbett et al. (2018). Both heavy scalars and vectors could contribute to $c_{H}$ and $c_{y}$, while the additional heavy vector-like quarks (VLQs) could contribute to $c_{y}$ and $c_{g}$. The sign of $c_{y}$ is arbitrary in two Higgs doublet (2HDM) Corbett et al. (2018) and VLQ models del Aguila et al. (2000). Those NP models can be easily discriminated if they modify $c_{H}$ or $c_{y}$ by a sizable amount, cf. Fig. 7.

Conclusions. Both the Wilson coefficients $c_{H}$ and $c_{y}$ of dimension-six operators can contribute to the top quark Yukawa coupling simultaneously, thus their individual contributions cannot be separated with the measurement of $ht\bar{t}$ coupling alone. In this work, we demonstrate that $c_{H}$ and $c_{y}$ also contribute to the $t\bar{t}hh$ effective coupling $c_{2t}$, whose information can be well extracted out from the Higgs pair production. Thus this process can be used to distinguish the effects of $c_{H}$ and $c_{y}$ at the 14 TeV LHC and the 100 TeV hadron collider. Regarding the discovery potential for the process $gg\to hh$, it shows that the $2\sigma$ confidence level is reachable for some parameter space at the 14 TeV LHC in general, and the sensitivity can be much improved at the 100 TeV hadron collider. Regrading the sensitivity of measuring $c_{t}$ and $c_{H}$, we find it is challenging to differentiate $c_{H}$ and $c_{y}$ at the HL-LHC, except for some special scenarios. The situation will be much improved at the 100 TeV hadron collider. After combing the single Higgs production, $t\bar{t}h$ production at the 100 TeV hadron collder, the Higgs pair production can give a strong constraint on $c_{y}$ regardless of the value $c_{H}$. The precise measurement of both $c_{H}$ and $c_{y}$ enable us to discriminate various new physics models.

Acknowledgements. We thank Kirtimaan A. Mohan for helpful discussions. The work of G. Li is supported by the MOST (Grant No. MOST 106-2112-M-002-003-MY3). L.-X. Xu is supported in part by the National Science Foundation of China under Grants No. 11635001, 11875072. B. Yan and C.-P. Yuan are supported by the U.S. National Science Foundation under Grant No. PHY-1719914. C.-P. Yuan is also grateful for the support from the Wu-Ki Tung endowed chair in particle physics.