Search for top-quark decays $t \rightarrow Hq$ with 36 fb$^{-1}$ of $pp$ collision data at $\sqrt{s}=13$ TeV with the ATLAS detector

TL;DR

This paper searches for rare top-quark decays into a Higgs boson and an up-type quark using 36 fb$^{-1}$ of 13 TeV collision data, setting upper limits on the branching ratios and couplings with no significant signal found.

Contribution

First combined search for $t ightarrow Hq$ decays in multiple Higgs decay channels using ATLAS data at 13 TeV, improving constraints on these rare processes.

Findings

No significant excess observed above background.

95% CL upper limits on $t ightarrow Hc$ and $t ightarrow Hu$ branching ratios are around 1.1-1.2 x 10^{-3}.

Upper limits on the couplings $| ext{lambda}_{tcH}|$ and $| ext{lambda}_{tuH}|$ are approximately 0.055-0.066.

Abstract

A search for flavour-changing neutral current decays of a top quark into an up-type quark ($q=u, c$) and the Standard Model Higgs boson, $t\rightarrow Hq$, is presented. The search is based on a dataset of $pp$ collisions at $\sqrt{s}=13$ TeV recorded in 2015 and 2016 with the ATLAS detector at the CERN Large Hadron Collider and corresponds to an integrated luminosity of 36.1 fb$^{-1}$. Two complementary analyses are performed that search for top-quark pair events in which one top quark decays into $Wb$ and the other top quark decays into $Hq$, and target the $H\rightarrow b\bar{b}$ and $H\rightarrow\tau^+\tau^-$ decay modes, respectively. The high multiplicity of $b$-quark jets, or the presence of hadronically decaying $\tau$-leptons, are exploited in the two analyses respectively. Multivariate techniques are used to separate the signal from the background, which is dominated by top-quark pair production. No significant excess of events above the background expectation is found, and 95% CL upper limits on the $t\rightarrow Hq$ branching ratios are derived. The combination of these searches with ATLAS searches in diphoton and multilepton final states yields observed (expected) 95% CL upper limits on the $t\rightarrow Hc$ and $t\rightarrow Hu$ branching ratios of $1.1 \times 10^{-3}$ ($8.3 \times 10^{-4}$) and $1.2 \times 10^{-3}$ ($8.3 \times 10^{-4}$), respectively. The corresponding combined observed (expected) upper limits on the $|\lambda_{tcH}|$ and $|\lambda_{tuH}|$ couplings are 0.064 (0.055) and 0.066 (0.055), respectively.

Tables

Table 1. Table 1: Summary of preselection requirements for the t ​ q ​ H ​ ( b ​ b ¯ ) 𝑡 𝑞 𝐻 𝑏 ¯ 𝑏 tqH(b\bar{b}) and t ​ q ​ H ​ ( τ ​ τ ) 𝑡 𝑞 𝐻 𝜏 𝜏 tqH(\tau\tau) searches. The leading and trailing τ had subscript 𝜏 had \tau_{\mathrm{had}} candidates are denoted by τ had , 1 subscript 𝜏 had 1 \tau_{\mathrm{had,1}} and τ had , 2 subscript 𝜏 had 2 \tau_{\mathrm{had,2}} respectively.

Preselection requirements
Requirement	$tqH\left(b\overline{b}\right)$ search	$tqH\left(\tau \tau \right)$ search
		${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$ channel	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$ channel
Trigger	single-lepton trigger	single-lepton trigger	di-$\tau$ trigger
Leptons	=1 isolated $e$ or $\mu$	=1 isolated $e$ or $\mu$	no isolated $e$ or $\mu$
	–	$\ge$1 ${\tau }_{\mathrm{had}}$	$\ge$2 ${\tau }_{\mathrm{had}}$
Electric charge ($q$)	–
Jets	$\ge$4 jets	$\ge$3 jets	$\ge$3 jets
$b$-tagging	$\ge$2 $b$-tagged jets	=1 $b$-tagged jets	=1 $b$-tagged jets

Table 2. Table 2: t ​ q ​ H ​ ( τ ​ τ ) 𝑡 𝑞 𝐻 𝜏 𝜏 tqH(\tau\tau) search: Discriminating variables used in the training of the BDT for each search region (denoted by × \times ). The description of each variable is provided in the text.

	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$		${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$
Variable	3j	$\ge$4j	3j	$\ge$4j
$m_{\tau \tau }^{\text{fit}}$	$\times$	$\times$	$\times$	$\times$
$m_{Hq}$	$\times$	$\times$	$\times$	$\times$
$m_{\text{T,lep}}$	$\times$	$\times$
$p_{\text{T,1}}$	$\times$	$\times$	$\times$	$\times$
$p_{\text{T,2}}$	$\times$	$\times$	$\times$	$\times$
$E_{\text{T}}^{\text{miss}}$ $\varphi$ centrality	$\times$	$\times$	$\times$	$\times$
$E_{\text{T},\parallel }^{\text{miss}}$	$\times$	$\times$	$\times$	$\times$
$E_{\text{T},⟂}^{\text{miss}}$	$\times$	$\times$
$m_{b{j}_1}$	$\times$	$\times$	$\times$	$\times$
$m_{\text{lep}j}$	$\times$	$\times$
$m_{\tau j}$	$\times$	$\times$
$x_1^{\text{fit}}$	$\times$	$\times$	$\times$	$\times$
$x_2^{\text{fit}}$	$\times$	$\times$	$\times$	$\times$
$m_{b{j}_1{j}_2}$		$\times$		$\times$

Table 3. Table 3: Summary of 95% CL upper limits on ℬ ​ ( t → H ​ c ) ℬ → 𝑡 𝐻 𝑐 \mathscr{B}(t\to Hc) and ℬ ​ ( t → H ​ u ) ℬ → 𝑡 𝐻 𝑢 \mathscr{B}(t\to Hu) , in each case neglecting the other decay mode. Signatures with two same-charge (three) leptons and no τ had subscript 𝜏 had \tau_{\mathrm{had}} candidates are denoted by 2 ​ ℓ 2 ℓ 2\ell SS ( 3 ​ ℓ 3 ℓ 3\ell ).

	95% CL upper limits	95% CL upper limits
	on $ℬ\left(t\to Hc\right)$	on $ℬ\left(t\to Hu\right)$
	Observed (Expected)	Observed (Expected)
$H\to b\overline{b}$	$4.2\times {10}^{-3}$ ($4.0\times {10}^{-3}$)	$5.2\times {10}^{-3}$ ($4.9\times {10}^{-3}$)
$H\to \tau \tau$ (${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, ${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$)	$1.9\times {10}^{-3}$ ($2.1\times {10}^{-3}$)	$1.7\times {10}^{-3}$ ($2.0\times {10}^{-3}$)
$H\to W{W}^{\ast },\tau \tau ,Z{Z}^{\ast }$ ($2\ell$SS, $3\ell$) [30]	$1.6\times {10}^{-3}$ ($1.5\times {10}^{-3}$)	$1.9\times {10}^{-3}$ ($1.5\times {10}^{-3}$)
$H\to \gamma \gamma$ [29]	$2.2\times {10}^{-3}$ ($1.6\times {10}^{-3}$)	$2.4\times {10}^{-3}$ ($1.7\times {10}^{-3}$)
Combination	$1.1\times {10}^{-3}$ ($8.3\times {10}^{-4}$)	$1.2\times {10}^{-3}$ ($8.3\times {10}^{-4}$)

Table 4. Table 4: t ​ q ​ H ​ ( b ​ b ¯ ) 𝑡 𝑞 𝐻 𝑏 ¯ 𝑏 tqH(b\bar{b}) search: Predicted and observed yields in each of the analysis regions considered. The prediction is shown before the fit to data. Also shown are the signal expectations for t ​ t ¯ → W ​ b ​ H ​ c → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑐 t\bar{t}\to WbHc and t ​ t ¯ → W ​ b ​ H ​ u → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑢 t\bar{t}\to WbHu assuming ℬ ​ ( t → H ​ c ) = 1 % ℬ → 𝑡 𝐻 𝑐 percent 1 \mathscr{B}(t\to Hc)=1\% and ℬ ​ ( t → H ​ u ) = 1 % ℬ → 𝑡 𝐻 𝑢 percent 1 \mathscr{B}(t\to Hu)=1\% respectively. The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, excluding the normalisation uncertainty of the t ​ t ¯ + ≥ 1 ​ b limit-from 𝑡 ¯ 𝑡 1 𝑏 t\bar{t}+\geq 1b background, which is determined via a likelihood fit to data.

	4j, 2b	4j, 3b	4j, 4b
$t\overline{t}\to WbHc$	$1990\pm 190$	$1260\pm 190$	$24.8\pm 9.5$
$t\overline{t}\to WbHu$	$1950\pm 190$	$1110\pm 170$	$19\pm 16$
$t\overline{t}$+light-jets	$87000\pm 11000$	$4300\pm 1200$	$10.2\pm 9.6$
$t\overline{t}+\ge 1c$	$8300\pm 4300$	$1050\pm 640$	$3.2\pm 3.3$
$t\overline{t}+\ge 1b$	$3620\pm 440$	$2900\pm 580$	$95\pm 33$
$t\overline{t}V$	$176\pm 31$	$34.8\pm 6.9$	$2.84\pm 0.74$
$t\overline{t}H$	$61.7\pm 9.2$	$48.7\pm 8.3$	$5.1\pm 1.0$
$W$+jets	$5400\pm 2400$	$280\pm 130$	$3.3\pm 1.8$
$Z$+jets	$2120\pm 960$	$115\pm 55$	$2.4\pm 1.4$
Single top	$7100\pm 1300$	$400\pm 120$	$7.8\pm 6.0$
Diboson	$267\pm 97$	$17.2\pm 6.5$	$0.58\pm 0.27$
Multijet	$7800\pm 3400$	$930\pm 360$	$31\pm 17$
Total background	$120000\pm 15000$	$10000\pm 2000$	$162\pm 44$
Data	120572	11275	176

Table 5. Table 5: t ​ q ​ H ​ ( b ​ b ¯ ) 𝑡 𝑞 𝐻 𝑏 ¯ 𝑏 tqH(b\bar{b}) search: Predicted and observed yields in each of the analysis regions considered. The background prediction is shown after the fit to data under the signal-plus-background hypothesis (assuming t ​ t ¯ → W ​ b ​ H ​ c → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑐 t\bar{t}\to WbHc as signal). The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, computed taking into account correlations among nuisance parameters and among processes.

	4j, 2b	4j, 3b	4j, 4b
$t\overline{t}\to WbHc$	$-30\pm 470$	$-20\pm 300$	$-0.4\pm 5.9$
$t\overline{t}$+light-jets	$82900\pm 4200$	$4900\pm 500$	$16\pm 12$
$t\overline{t}+\ge 1c$	$11400\pm 4800$	$1360\pm 550$	$5.9\pm 4.2$
$t\overline{t}+\ge 1b$	$4270\pm 590$	$3400\pm 350$	$110\pm 17$
$t\overline{t}V$	$174\pm 28$	$35.0\pm 5.9$	$2.69\pm 0.55$
$t\overline{t}H$	$62.6\pm 7.8$	$47.3\pm 6.3$	$4.68\pm 0.69$
$W$+jets	$4800\pm 1800$	$260\pm 100$	$2.9\pm 1.3$
$Z$+jets	$1870\pm 730$	$102\pm 41$	$1.9\pm 1.0$
Single-top	$6360\pm 980$	$393\pm 96$	$7.6\pm 5.2$
Diboson	$242\pm 84$	$16.3\pm 5.7$	$0.50\pm 0.22$
Multijet	$9000\pm 3500$	$820\pm 240$	$29\pm 16$
Total	$121100\pm 2200$	$11290\pm 280$	$181\pm 23$
Data	120572	11275	176

Table 6. Table 6: t ​ q ​ H ​ ( b ​ b ¯ ) 𝑡 𝑞 𝐻 𝑏 ¯ 𝑏 tqH(b\bar{b}) search: Predicted and observed yields in each of the analysis regions considered. The background prediction is shown after the fit to data under the signal-plus-background hypothesis (assuming t ​ t ¯ → W ​ b ​ H ​ u → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑢 t\bar{t}\to WbHu as signal). The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, computed taking into account correlations among nuisance parameters and among processes.

	4j, 2b	4j, 3b	4j, 4b
$t\overline{t}\to WbHu$	$40\pm 550$	$20\pm 320$	$0.4\pm 5.3$
$t\overline{t}$+light-jets	$82700\pm 4400$	$4860\pm 530$	$15\pm 12$
$t\overline{t}+\ge 1c$	$11500\pm 5100$	$1400\pm 580$	$5.8\pm 4.2$
$t\overline{t}+\ge 1b$	$4260\pm 590$	$3400\pm 350$	$110\pm 17$
$t\overline{t}V$	$173\pm 28$	$34.8\pm 5.8$	$2.68\pm 0.54$
$t\overline{t}H$	$62.4\pm 7.7$	$47.1\pm 6.2$	$4.66\pm 0.68$
$W$+jets	$4800\pm 1900$	$260\pm 100$	$2.9\pm 1.4$
$Z$+jets	$1880\pm 740$	$103\pm 42$	$1.9\pm 1.0$
Single-top	$6380\pm 990$	$392\pm 96$	$7.5\pm 5.2$
Diboson	$243\pm 85$	$16.3\pm 5.7$	$0.50\pm 0.22$
Multijet	$9000\pm 3500$	$810\pm 240$	$29\pm 16$
Total	$121000\pm 2300$	$11290\pm 290$	$181\pm 23$
Data	120572	11275	176

Table 7. Table 7: t ​ q ​ H ​ ( τ ​ τ ) 𝑡 𝑞 𝐻 𝜏 𝜏 tqH(\tau\tau) search: Predicted and observed yields in each of the analysis regions considered. The prediction is shown before the fit to data. Also shown are the signal expectations for t ​ t ¯ → W ​ b ​ H ​ c → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑐 t\bar{t}\to WbHc and t ​ t ¯ → W ​ b ​ H ​ u → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑢 t\bar{t}\to WbHu assuming ℬ ​ ( t → H ​ c ) = 1 % ℬ → 𝑡 𝐻 𝑐 percent 1 \mathscr{B}(t\to Hc)=1\% and ℬ ​ ( t → H ​ u ) = 1 % ℬ → 𝑡 𝐻 𝑢 percent 1 \mathscr{B}(t\to Hu)=1\% respectively. The contributions with real τ had subscript 𝜏 had \tau_{\mathrm{had}} candidates from t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} , t ​ t ¯ ​ V 𝑡 ¯ 𝑡 𝑉 t\bar{t}V , t ​ t ¯ ​ H 𝑡 ¯ 𝑡 𝐻 t\bar{t}H , and single-top-quark backgrounds are combined into a single background source referred to as “Top (real τ had subscript 𝜏 had \tau_{\mathrm{had}} )”, whereas the small contributions from Z → ℓ + ​ ℓ − → 𝑍 superscript ℓ superscript ℓ Z\to\ell^{+}\ell^{-} ( ℓ = e , μ ℓ 𝑒 𝜇 \ell=e,\mu ) and diboson backgrounds are combined into “Other”. The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, excluding the normalisation uncertainty of the fake τ had subscript 𝜏 had \tau_{\mathrm{had}} background, which is determined via a likelihood fit to data.

	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, $\ge$4j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, $\ge$4j
$t\overline{t}\to WbHc$	$89\pm 14$	$226\pm 43$	$46\pm 14$	$122\pm 32$
$t\overline{t}\to WbHu$	$100\pm 17$	$237\pm 47$	$32\pm 10$	$114\pm 28$
Fake ${\tau }_{\mathrm{had}}$	$2828\pm 78$	$3200\pm 100$	$710\pm 110$	$500\pm 62$
Top (real ${\tau }_{\mathrm{had}}$)	$3840\pm 720$	$3160\pm 890$	$113\pm 72$	$117\pm 35$
$Z\to \tau \tau$	$420\pm 140$	$320\pm 120$	$283\pm 99$	$267\pm 96$
Other	$168\pm 56$	$103\pm 33$	$8.9\pm 2.5$	$11.2\pm 2.5$
Total background	$7260\pm 730$	$6770\pm 880$	$1120\pm 120$	$900\pm 120$
Data	7259	6768	1119	894

Table 8. Table 8: t ​ q ​ H ​ ( τ ​ τ ) 𝑡 𝑞 𝐻 𝜏 𝜏 tqH(\tau\tau) search: Predicted and observed yields in each of the analysis regions considered. The background prediction is shown after the fit to data under the signal-plus-background hypothesis (assuming t ​ t ¯ → W ​ b ​ H ​ c → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑐 t\bar{t}\to WbHc as signal). The contributions with real τ had subscript 𝜏 had \tau_{\mathrm{had}} candidates from t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} , t ​ t ¯ ​ V 𝑡 ¯ 𝑡 𝑉 t\bar{t}V , t ​ t ¯ ​ H 𝑡 ¯ 𝑡 𝐻 t\bar{t}H , and single-top-quark backgrounds are combined into a single background source referred to as “Top (real τ had subscript 𝜏 had \tau_{\mathrm{had}} )”, whereas the small contributions from Z → ℓ + ​ ℓ − → 𝑍 superscript ℓ superscript ℓ Z\to\ell^{+}\ell^{-} ( ℓ = e , μ ℓ 𝑒 𝜇 \ell=e,\mu ) and diboson backgrounds are combined into “Other”. The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, computed taking into account correlations among nuisance parameters and among processes.

	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, $\ge$4j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, $\ge$4j
$t\overline{t}\to WbHc$	$-4.2\pm 8.2$	$-11\pm 21$	$-2.4\pm 4.3$	$-10\pm 11$
Fake ${\tau }_{\mathrm{had}}$	$2290\pm 680$	$2640\pm 880$	$640\pm 110$	$440\pm 100$
Top (real ${\tau }_{\mathrm{had}}$)	$4300\pm 670$	$3660\pm 860$	$147\pm 84$	$139\pm 35$
$Z\to \tau \tau$	$500\pm 100$	$359\pm 90$	$320\pm 79$	$306\pm 76$
Other	$178\pm 45$	$112\pm 28$	$9.6\pm 2.6$	$12.5\pm 2.6$
Total	$7230\pm 160$	$6760\pm 170$	$1117\pm 65$	$893\pm 45$
Data	7259	6768	1119	894

Table 9. Table 9: t ​ q ​ H ​ ( τ ​ τ ) 𝑡 𝑞 𝐻 𝜏 𝜏 tqH(\tau\tau) search: Predicted and observed yields in each of the analysis regions considered. The background prediction is shown after the fit to data under the signal-plus-background hypothesis (assuming t ​ t ¯ → W ​ b ​ H ​ u → 𝑡 ¯ 𝑡 𝑊 𝑏 𝐻 𝑢 t\bar{t}\to WbHu as signal). The contributions with real τ had subscript 𝜏 had \tau_{\mathrm{had}} candidates from t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} , t ​ t ¯ ​ V 𝑡 ¯ 𝑡 𝑉 t\bar{t}V , t ​ t ¯ ​ H 𝑡 ¯ 𝑡 𝐻 t\bar{t}H , and single-top-quark backgrounds are combined into a single background source referred to as “Top (real τ had subscript 𝜏 had \tau_{\mathrm{had}} )”, whereas the small contributions from Z → ℓ + ​ ℓ − → 𝑍 superscript ℓ superscript ℓ Z\to\ell^{+}\ell^{-} ( ℓ = e , μ ℓ 𝑒 𝜇 \ell=e,\mu ) and diboson backgrounds are combined into “Other”. The quoted uncertainties are the sum in quadrature of statistical and systematic uncertainties of the yields, computed taking into account correlations among nuisance parameters and among processes.

	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{lep}}{\tau }_{\mathrm{had}}$, $\ge$4j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, 3j	${\tau }_{\mathrm{had}}{\tau }_{\mathrm{had}}$, $\ge$4j
$t\overline{t}\to WbHu$	$-5.7\pm 8.6$	$-14\pm 21$	$-2,0\pm 2.8$	$-7.1\pm 9.8$
Fake ${\tau }_{\mathrm{had}}$	$2270\pm 680$	$2620\pm 880$	$640\pm 110$	$440\pm 100$
Top (real ${\tau }_{\mathrm{had}}$)	$4320\pm 660$	$3680\pm 860$	$148\pm 84$	$140\pm 35$
$Z\to \tau \tau$	$470\pm 100$	$359\pm 89$	$321\pm 79$	$308\pm 77$
Other	$177\pm 44$	$111\pm 27$	$9.7\pm 2.6$	$12.5\pm 2.6$
Total	$7230\pm 160$	$6760\pm 160$	$1118\pm 66$	$892\pm 45$
Data	7259	6768	1119	894

Equations

$$L(\mathbf{x})=\frac{P^{\text{sig}}(\mathbf{x})}{P^{\text{sig}}(\mathbf{x})+P^{\text{bkg}}(\mathbf{x})},$$

$$E_{\text{T}}^{\text{miss}}\varphi \mathrm{centrality}=\frac{\sin ({\varphi }_{\mathrm{miss}}-{\varphi }_1)+\sin ({\varphi }_{\mathrm{miss}}-{\varphi }_2)}{\sqrt{{\sin }^2({\varphi }_{\mathrm{miss}}-{\varphi }_1)+{\sin }^2({\varphi }_{\mathrm{miss}}-{\varphi }_2)}}$$

$${\mathcal{L}}_{\mathrm{FCNC}}=-{\lambda }_{t_{\mathrm{L}}{q}_{\mathrm{R}}}{\bar{t}}_{\mathrm{L}}{q}_{\mathrm{R}}H-{\lambda }_{q_{\mathrm{L}}{t}_{\mathrm{R}}}{\bar{q}}_{\mathrm{L}}{t}_{\mathrm{R}}H+h.c.$$

Full text

\AtlasTitle

Search for top-quark decays $t\to Hq$ with 36 fb*-1* of $pp$ collision data at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ with the ATLAS detector \AtlasAbstract A search for flavour-changing neutral current decays of a top quark into an up-type quark ($q=u,c$) and the Standard Model Higgs boson, $t\to Hq$, is presented. The search is based on a dataset of $pp$ collisions at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ recorded in 2015 and 2016 with the ATLAS detector at the CERN Large Hadron Collider and corresponding to an integrated luminosity of 36.1 fb*-1*. Two complementary analyses are performed to search for top-quark pair events in which one top quark decays into $Wb$ and the other top quark decays into $Hq$, and target the $H\to b\bar{b}$ and $H\to\tau^{+}\tau^{-}$ decay modes, respectively. The high multiplicity of $b$-quark jets, or the presence of hadronically decaying $\tau$-leptons, is exploited in the two analyses respectively. Multivariate techniques are used to separate the signal from the background, which is dominated by top-quark pair production. No significant excess of events above the background expectation is found, and 95% CL upper limits on the $t\to Hq$ branching ratios are derived. The combination of these searches with ATLAS searches in diphoton and multilepton final states yields observed (expected) 95% CL upper limits on the $t\to Hc$ and $t\to Hu$ branching ratios of $1.1\times 10^{-3}$ ($8.3\times 10^{-4}$) and $1.2\times 10^{-3}$ ($8.3\times 10^{-4}$), respectively. The corresponding combined observed (expected) upper limits on the $|\lambda_{tcH}|$ and $|\lambda_{tuH}|$ couplings are 0.064 (0.055) and 0.066 (0.055), respectively.

\AtlasRefCodeTOPQ-2017-07 \AtlasJournalJHEP \AtlasJournalRefJHEP 05 (2019) 123 \AtlasDOI10.1007/JHEP05(2019)123 \PreprintIdNumberCERN-EP-2018-295 \AtlasCoverSupportingNoteSearch for $t\bar{t}\to WbHq$, $H\to b\bar{b}$https://cds.cern.ch/record/2257631 \AtlasCoverSupportingNoteSearch for $t\bar{t}\to WbHq$, $H\to\tau^{+}\tau^{-}$https://cds.cern.ch/record/2273683 \AtlasCoverSupportingNoteCombination of $t\bar{t}\to WbHq$ searcheshttps://cds.cern.ch/record/2312520/ \AtlasCoverCommentsDeadline4 November 2018 \AtlasCoverAnalysisTeam $H\to b\bar{b}$: Trisha Farooque, Davide Gerbaudo, Aurelio Juste, Nicola Orlando,

Cheng Peng, Laura Pereira, Yulia Rodina, Yanjun Tu, Loïc Valéry,

Tal Van Daalen, Jia-Shian Wang, Daiki Yamaguchi;

$H\to\tau^{+}\tau^{-}$: Xin Chen, Antonio De Maria, Boyang Li, Ligang Xia, Gang Zhang;

Combination: Peter Onyisi, Harish Potti \AtlasCoverEdBoardMemberKetevi Assamagan (chair), Garabed Halladjian, Fabrice Hubaut, Michele Weber (chair) \AtlasCoverEgroupEditorsatlas-TOPQ-2017-07-editors@cern.ch \AtlasCoverEgroupEdBoardatlas-TOPQ-2017-07-editorial-board@cern.ch

1 Introduction

Following the observation of the Higgs boson by the ATLAS and CMS experiments [1, 2] at the Large Hadron Collider (LHC), a comprehensive programme of measurements of its properties is underway. An interesting possibility is the presence of flavour-changing neutral-current (FCNC) interactions between the Higgs boson, the top quark, and a $u$- or $c$-quark, $tqH$ ($q=u,c$). Since the Higgs boson is lighter than the top quark [3], such interactions would manifest themselves as FCNC top-quark decays [4], $t\to Hq$. In the Standard Model (SM), such decays are suppressed relative to the dominant $t\to Wb$ decay mode, since $tqH$ interactions are forbidden at the tree level and suppressed even at higher orders in the perturbative expansion due to the Glashow–Iliopoulos–Maiani (GIM) mechanism [5]. As a result, the SM predictions for the $t\to Hq$ branching ratios ($\mathscr{B}$) are exceedingly small, $\mathscr{B}(t\to Hu)\sim 10^{-17}$ and $\mathscr{B}(t\to Hc)\sim 10^{-15}$ [6, 7, 8, 9], making them undetectable in the foreseeable future. In contrast, large enhancements of these branching ratios are possible in some scenarios beyond the SM. Examples include quark-singlet models [10], two-Higgs-doublet models (2HDM) of type I, with explicit flavour conservation, and of type II, such as the minimal supersymmetric SM (MSSM) [11, 12, 13, 14], supersymmetric models with R-parity violation [15], composite Higgs models with partial compositeness [16], or warped extra dimensions models with SM fermions in the bulk [17]. In these scenarios, branching ratios can be as high as $\mathscr{B}(t\to Hq)\sim 10^{-5}$. An even larger branching ratio of $\mathscr{B}(t\to Hc)\sim 10^{-3}$ can be reached in 2HDM without explicit flavour conservation (type III), since a tree-level FCNC coupling is not forbidden by any symmetry [18, 19, 20, 21, 22, 23, 24, 25]. While other FCNC top couplings ($tq\gamma$, $tqZ$, $tqg$) are also enhanced in these scenarios beyond the SM, the largest enhancements are typically found for the $tqH$ couplings, and in particular the $tcH$ coupling [4].

Searches for $t\to Hq$ decays have been performed by the ATLAS and CMS collaborations, taking advantage of the large samples of top-quark pair ($t\bar{t}$) events collected in proton-proton ($pp$) collisions at centre-of-mass energies of $\sqrt{s}=7\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ and $8\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ [26, 27, 28] during Run 1 of the LHC, as well as at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ [29, 30, 31] using early Run 2 data. In these searches, one of the top quarks is required to decay into $Wb$, while the other top quark decays into $Hq$, yielding $t\bar{t}\to WbHq$.111 In the following, $WbHq$ is used to denote both $W^{+}bH\bar{q}$ and its charge conjugate, $HqW^{-}\bar{b}$. Similarly, $WbWb$ is used to denote $W^{+}bW^{-}\bar{b}$. The Higgs boson is assumed to have a mass of $m_{H}=125\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and to decay as predicted by the SM. The simplifying assumption of SM-like Higgs boson branching ratios is motivated by the fact that measurements of the flavour-diagonal Higgs boson couplings by the ATLAS and CMS collaborations are in agreement with the SM prediction within about 10% [32, 33]. Furthermore, typical beyond-the-SM scenarios that predict significant enhancements to $\mathscr{B}(t\to Hq)$, also predict modifications to the Higgs boson branching ratios at the few percent level or below, well beyond the current experimental precision. Some of the most sensitive single-channel searches have been performed in the $H\to\gamma\gamma$ decay mode, which has a small branching ratio of $\mathscr{B}(H\to\gamma\gamma)\simeq 0.2\%$, but benefits from having a very small background contamination and excellent diphoton mass resolution. Searches targeting signatures with two same-charge leptons or three leptons (electrons or muons), generically referred to as multileptons, are able to exploit a branching ratio that is significantly larger for the $H\rightarrow WW^{*},\tau\tau$ decay modes than for the $H\rightarrow\gamma\gamma$ decay mode, and are also characterised by relatively small backgrounds. Finally, searches have also been performed exploiting the dominant Higgs boson decay mode, $H\to b\bar{b}$, which has a branching ratio of $\mathscr{B}(H\to b\bar{b})\simeq 58\%$. Compared with Run 1, the Run 2 searches benefit from the increased $t\bar{t}$ cross section at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$, as well as the larger integrated luminosity. Using 36.1 fb*-1* of data at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$, the ATLAS Collaboration has derived upper limits at 95% confidence level (CL) of $\mathscr{B}(t\to Hc)<0.22\%$ using $H\to\gamma\gamma$ decays [29], and of $\mathscr{B}(t\to Hc)<0.16\%$ based on multilepton signatures resulting from $H\to WW^{*}$, $H\to\tau^{+}\tau^{-}$ in which both $\tau$-leptons decay leptonically, or $H\to ZZ^{*}$ [30]. These upper limits are derived assuming that $\mathscr{B}(t\to Hu)=0$. Similar upper limits are obtained for $\mathscr{B}(t\to Hu)$ if $\mathscr{B}(t\to Hc)=0$. The CMS Collaboration has performed a search using $H\to b\bar{b}$ decays [31] with 35.9 fb*-1* of data at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$, resulting in upper limits of $\mathscr{B}(t\to Hc)<0.47\%$ and $\mathscr{B}(t\to Hu)<0.47\%$, in each case neglecting the other decay mode. Compared with previous searches, the search in Ref. [31] considers in addition the contribution to the signal from $pp\to tH$ production [34].

The searches presented in this paper are focussed on fermionic decay modes of the Higgs boson. Therefore, they help to complete the ATLAS experiment’s programme of searches for $t\to Hq$ decays based on $pp$ collision data at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ recorded in 2015 and 2016. The corresponding integrated luminosity is 36.1 fb*-1*. Two analyses are performed, searching for $t\bar{t}\to WbHq$ production (ignoring $pp\to tH$ production) and targeting the $H\to b\bar{b}$ and $H\to\tau^{+}\tau^{-}$ decay modes, which this paper refers to as “$tqH(b\bar{b})$ search” and “$tqH(\tau\tau)$ search”, respectively. The $tqH(b\bar{b})$ search selects events with one isolated electron or muon from the $W\to\ell\nu$ decay, and multiple jets, several of which are identified with high purity as originating from the hadronisation of $b$-quarks. The $tqH(\tau\tau)$ search selects events with two $\tau$-lepton candidates, at least one of which decays hadronically, as well as multiple jets. The latter requirement aims to select events with a hadronically decaying $W$ boson, since this allows an improved reconstruction of the event kinematics.

Both searches employ multivariate techniques to discriminate between the signal and the background on the basis of their different kinematics. These two searches are combined with previous ATLAS searches in the diphoton and multilepton final states using the same dataset [29, 30], and bounds are set on $\mathscr{B}(t\to Hc)$ and $\mathscr{B}(t\to Hu)$, as well as on the corresponding non-flavour-diagonal Yukawa couplings. The combination is performed after verifying the overall consistency of the results obtained by the different searches, which exploit very different experimental signatures and thus are affected by different backgrounds and related systematic uncertainties. By combining all searches, the expected sensitivity is improved by about a factor of two relative to the most sensitive individual results.

2 ATLAS detector

The ATLAS detector [35] at the LHC covers almost the entire solid angle around the collision point,222ATLAS uses a right-handed coordinate system with its origin at the nominal interaction point (IP) in the centre of the detector. The $x$-axis points from the IP to the centre of the LHC ring, the $y$-axis points upward, and the $z$-axis coincides with the axis of the beam pipe. Cylindrical coordinates ($r$,$\phi$) are used in the transverse plane, $\phi$ being the azimuthal angle around the beam pipe. The pseudorapidity is defined in terms of the polar angle $\theta$ as $\eta=-\ln\tan(\theta/2)$. Angular distance is measured in units of $\Delta R\equiv\sqrt{(\Delta\eta)^{2}+(\Delta\phi)^{2}}$. and consists of an inner tracking detector surrounded by a thin superconducting solenoid producing a 2 T axial magnetic field, electromagnetic and hadronic calorimeters, and a muon spectrometer incorporating three large toroid magnet assemblies with eight coils each. The inner detector contains a high-granularity silicon pixel detector, including the insertable B-layer [36, 37, 38], installed in 2014, and a silicon microstrip tracker, together providing a precise reconstruction of tracks of charged particles in the pseudorapidity range $|\eta|<2.5$. The inner detector also includes a transition radiation tracker that provides tracking and electron identification for $|\eta|<2.0$. The calorimeter system covers the pseudorapidity range $|\eta|<4.9$. Within the region $|\eta|<3.2$, electromagnetic (EM) calorimetry is provided by barrel and endcap high-granularity lead/liquid-argon (LAr) sampling calorimeters, with an additional thin LAr presampler covering $|\eta|<1.8$, to correct for energy loss in material upstream of the calorimeters. Hadronic calorimetry is provided by a steel/scintillator-tile calorimeter, segmented into three barrel structures within $|\eta|<1.7$, and two copper/LAr hadronic endcap calorimeters. The solid angle coverage is completed with forward copper/LAr and tungsten/LAr calorimeter modules optimised for electromagnetic and hadronic measurements, respectively. The calorimeters are surrounded by a muon spectrometer within a magnetic field provided by air-core toroid magnets with a bending integral of about 2.5 Tm in the barrel and up to 6 Tm in the endcaps. The muon spectrometer measures the trajectories of muons with $|\eta|<2.7$ using multiple layers of high-precision tracking chambers, and is instrumented with separate trigger chambers covering $|\eta|<2.4$. A two-level trigger system [39], consisting of a hardware-based level-1 trigger followed by a software-based high-level trigger, is used to reduce the event rate to a maximum of around 1 kHz for offline storage.

3 Event reconstruction

The event reconstruction is affected by multiple $pp$ collisions in a single bunch crossing and by collisions in neighbouring bunch crossings, referred to as pile-up. Interaction vertices from the $pp$ collisions are reconstructed from at least two tracks with transverse momentum ($p_{\text{T}}$) larger than $400\leavevmode\nobreak\ \text{Me\kern-1.00006ptV}$ that are consistent with originating from the beam collision region in the $x$–$y$ plane. If more than one primary vertex candidate is found, the candidate whose associated tracks form the largest sum of squared $p_{\text{T}}$ [40] is selected as the hard-scatter primary vertex.

Electron candidates [41, 42] are reconstructed from energy clusters in the EM calorimeter that are matched to reconstructed tracks in the inner detector; electron candidates in the transition region between the EM barrel and endcap calorimeters ($1.37<|\eta_{\textrm{cluster}}|<1.52$) are excluded. In the $tqH(b\bar{b})$ ($tqH(\tau\tau)$) search, electron candidates are required to have $p_{\text{T}}>30\leavevmode\nobreak\ (15)\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta_{\textrm{cluster}}|<2.47$, and to satisfy tight (medium) likelihood-based identification criteria [41] based on calorimeter, tracking and combined variables that provide separation between electrons and jets.

Muon candidates [43] are reconstructed by matching track segments in different layers of the muon spectrometer to tracks found in the inner detector; the resulting muon candidates are re-fitted using the complete track information from both detector systems. In the $tqH(b\bar{b})$ ($tqH(\tau\tau)$) search, muon candidates are required to have $p_{\text{T}}>30\leavevmode\nobreak\ (10)\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<2.5$ and to satisfy medium identification criteria [43].

Electron (muon) candidates are matched to the primary vertex by requiring that the significance of their transverse impact parameter, $d_{0}$, satisfies $|d_{0}/\sigma(d_{0})|<5\,(3)$, where $\sigma(d_{0})$ is the measured uncertainty in $d_{0}$, and by requiring that their longitudinal impact parameter, $z_{0}$, satisfies $|z_{0}\sin\theta|<0.5$ mm. To further reduce the background from non-prompt leptons, photon conversions and hadrons, lepton candidates are also required to be isolated in the tracker and in the calorimeter. A track-based lepton isolation criterion is defined by calculating the quantity $I_{R}=\sum p_{\text{T}}^{\textrm{trk}}$, where the scalar sum includes all tracks (excluding the lepton candidate itself) within the cone defined by $\Delta R<R_{\textrm{cut}}$ around the direction of the lepton. The value of $R_{\textrm{cut}}$ is the smaller of $r_{\textrm{min}}$ and $10\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}/p_{\text{T}}^{\ell}$, where $r_{\textrm{min}}$ is set to 0.2 (0.3) for electron (muon) candidates, and $p_{\text{T}}^{\ell}$ is the lepton $p_{\text{T}}$. The $tqH(b\bar{b})$ search requires lepton candidates to satisfy $I_{R}/p_{\text{T}}^{\ell}<0.06$, while the $tqH(\tau\tau)$ search makes $p_{\text{T}}$-dependent requirements on $I_{R}/p_{\text{T}}^{\ell}$. Additionally, the $tqH(\tau\tau)$ search requires leptons to satisfy a calorimeter-based isolation criterion: the sum of the transverse energy within a cone of size $\Delta R<0.2$ around the lepton, after subtracting the contributions from pile-up and the energy deposit of the lepton itself, is required to be less than a $p_{\text{T}}$-dependent fraction of the lepton energy.

Candidate jets are reconstructed with the anti-$k_{t}$ algorithm [44, 45] with a radius parameter $R=0.4$, as implemented in the FastJet package [46]. Jet reconstruction in the calorimeter starts from topological clustering [47] of individual calorimeter cells calibrated to the electromagnetic energy scale. The reconstructed jets are then calibrated to the particle level by the application of a jet energy scale derived from simulation and in situ corrections based on $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ data [48]. The calibrated jets used in the $tqH(b\bar{b})$ search are required to have $p_{\text{T}}>25\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<2.5$, while the $tqH(\tau\tau)$ search uses jets with $p_{\text{T}}>30\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<4.5$. Jet four-momenta are corrected for pile-up effects using the jet-area method [49].

Quality criteria are imposed to reject events that contain any jets arising from non-collision sources or detector noise [50]. To reduce the contamination due to jets originating from pile-up interactions, additional requirements are imposed on the jet vertex tagger (JVT) [51] output for jets with $p_{\text{T}}<60\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<2.4$, or on the forward JVT [52] output for jets with $p_{\text{T}}<50\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|>2.5$.

Jets containing $b$-hadrons are identified ($b$-tagged) via an algorithm [53, 54] that uses multivariate techniques to combine information about the impact parameters of displaced tracks and the topological properties of secondary and tertiary decay vertices reconstructed within the jet. For each jet, a value for the multivariate $b$-tagging discriminant is calculated. In the $tqH(\tau\tau)$ search, a jet is considered $b$-tagged if this value is above the threshold corresponding to an average 70% efficiency to tag a $b$-quark jet, with a light-jet333Light-jet refers to a jet originating from the hadronisation of a light quark ($u$, $d$, $s$) or a gluon. rejection factor of about 380 and a charm-jet rejection factor of about 12, as determined for jets with $p_{\text{T}}>20\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<2.5$ in simulated $t\bar{t}$ events. In contrast, the $tqH(b\bar{b})$ search employs a tighter $b$-tagging requirement, corresponding to an average efficiency of 60% to tag a $b$-quark jet, and light-jet and charm-jet rejection factors of about 1500 and 34, respectively.

Hadronically decaying $\tau$-lepton ($\tau_{\mathrm{had}}$) candidates are reconstructed from energy clusters in the calorimeters and associated inner-detector tracks [55]. Candidates are required to have either one or three associated tracks, with a total charge of $\pm 1$. Candidates are required to have $p_{\text{T}}>25\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $|\eta|<2.5$, excluding the EM calorimeter’s transition region. A boosted decision tree (BDT) discriminant [56, 57, 58] using calorimeter- and tracking-based variables is used to identify $\tau_{\mathrm{had}}$ candidates and reject jet backgrounds. Three working points labelled loose, medium and tight are defined, and correspond to different $\tau_{\mathrm{had}}$ identification efficiency values, with the efficiency designed to be independent of $p_{\text{T}}$. The $tqH(\tau\tau)$ search uses the medium working point for the nominal selection, while the loose working point is used for background estimation. The medium working point has a combined reconstruction and identification efficiency of 55% (40%) for one-prong (three-prong) $\tau_{\mathrm{had}}$ decays [59], and an expected rejection factor against light-jets of 100 [55]. Electrons that are reconstructed as one-prong $\tau_{\mathrm{had}}$ candidates are removed via a BDT trained to reject electrons. Any $\tau_{\mathrm{had}}$ candidate that is also $b$-tagged is rejected.

Overlaps between reconstructed objects are removed sequentially. In the $tqH(b\bar{b})$ search, firstly, electron candidates that lie within $\Delta R=0.01$ of a muon candidate are removed to suppress contributions from muon bremsstrahlung. Overlaps between electron and jet candidates are resolved next, and finally, overlaps between remaining jet candidates and muon candidates are removed. Energy clusters from identified electrons are not excluded during jet reconstruction. In order to avoid double-counting of electrons as jets, the closest jet whose axis is within ${\Delta}R=0.2$ of an electron is discarded. If the electron is within ${\Delta}R=0.4$ of the axis of any jet after this initial removal, the jet is retained and the electron is removed. The overlap removal procedure between the remaining jet candidates and muon candidates is designed to remove those muons that are likely to have arisen in the decay of hadrons and to retain the overlapping jet instead. Jets and muons may also appear in close proximity when the jet results from high-$p_{\text{T}}$ muon bremsstrahlung, and in such cases the jet should be removed and the muon retained. Such jets are characterised by having very few matching inner-detector tracks. Selected muons that satisfy $\Delta R(\mu,{\textrm{jet}})<0.04+10\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}/p_{\text{T}}^{\mu}$ are rejected if the jet has at least three tracks originating from the primary vertex; otherwise the jet is removed and the muon is kept. The overlap removal procedure in the $tqH(\tau\tau)$ search is similar to that of the $tqH(b\bar{b})$ search, except that the first step is the removal of $\tau_{\mathrm{had}}$ candidates within $\Delta R=0.2$ of electrons or muons, and the last step is the removal of jets whose axis lies within $\Delta R=0.2$ of the leading (highest-$p_{\text{T}}$) $\tau_{\mathrm{had}}$ candidate or the two leading $\tau_{\mathrm{had}}$ candidates (depending on the search channel). In addition, the muon–jet overlap removal is slightly different: if a muon lies within $\Delta R=0.2$ of the axis of a jet, the jet is removed if either it has fewer than three tracks originating from the primary vertex or it has a small $p_{\text{T}}$ compared with that of the muon (the $p_{\text{T}}$ of the jet is less than 50% of the $p_{\text{T}}$ of the muon, or the scalar sum of the $p_{\text{T}}$ of the tracks associated with the jet is less than 70% of the $p_{\text{T}}$ of the muon).

The missing transverse momentum $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ (with magnitude $E_{\text{T}}^{\text{miss}}$) is defined as the negative vector sum of the $p_{\text{T}}$ of all selected and calibrated objects in the event, including a term to account for momentum from soft particles in the event which are not associated with any of the selected objects. This soft term is calculated from inner-detector tracks matched to the selected primary vertex to make it more resilient to contamination from pile-up interactions [60].

4 Data sample and event preselection

Both searches are based on a dataset of $pp$ collisions at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ with 25 ns bunch spacing collected in 2015 and 2016, corresponding to an integrated luminosity of 36.1\leavevmode\nobreak\ \mbox{fb{}^{-1}}. Only events recorded with a single-electron trigger, a single-muon trigger, or a di-$\tau$ trigger under stable beam conditions and for which all detector subsystems were operational are considered. The number of $pp$ interactions per bunch crossing in this dataset ranges from about 8 to 45, with an average of 24.

Single-electron and single-muon triggers with low $p_{\text{T}}$ thresholds and lepton isolation requirements are combined in a logical OR with higher-threshold triggers but with a looser identification criterion and without any isolation requirement. The lowest $p_{\text{T}}$ threshold used for muons is 20 (26) GeV in 2015 (2016), while for electrons the threshold is 24 (26) GeV. For di-$\tau$ triggers, the $p_{\text{T}}$ threshold of the leading (trailing) $\tau_{\mathrm{had}}$ candidate is 35 (25) GeV. In both searches, events satisfying the trigger selection are required to have at least one primary vertex candidate.

Events selected by the $tqH(b\bar{b})$ search are recorded with a single-electron or single-muon trigger and are required to have exactly one electron or muon that matches, with $\Delta R<0.15$, the lepton reconstructed by the trigger. Furthermore, at least four jets are required, of which at least two must be $b$-tagged.

In the $tqH(\tau\tau)$ search, events are classified into $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ and $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channels depending on the multiplicity of selected leptons. Events in the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel are recorded with a single-electron or single-muon trigger and are required to have exactly one selected electron or muon and at least one $\tau_{\mathrm{had}}$ candidate. The selected electron or muon is required to match, with $\Delta R<0.15$, the lepton reconstructed by the trigger and to have a $p_{\text{T}}$ exceeding the trigger $p_{\text{T}}$ threshold by 1 GeV or 2 GeV (depending on the lepton trigger and data-taking conditions). In addition, its electric charge is required to be of opposite sign to that of the leading $\tau_{\mathrm{had}}$ candidate. Events in the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel are recorded with a di-$\tau$ trigger, and are required to have at least two $\tau_{\mathrm{had}}$ candidates and no selected electrons or muons. The two leading $\tau_{\mathrm{had}}$ candidates are required to have charges of opposite sign. In addition, in both $tqH(\tau\tau)$ search channels, trigger matching for $\tau_{\mathrm{had}}$ candidates, at least three jets and exactly one $b$-tagged jet are required.

The above requirements apply to the reconstructed objects defined in Section 3. These requirements, which ensure a negligible overlap between the $tqH(b\bar{b})$ and $tqH(\tau\tau)$ searches, are referred to as the preselection and are summarised in Table 1.

5 Signal and background modelling

Signal and most background processes are modelled using Monte Carlo (MC) simulation. After the event preselection, the main background is $t\bar{t}$ production, often in association with jets, denoted by $t\bar{t}$+jets in the following. Small contributions arise from single-top-quark, $W/Z$+jets, multijet and diboson ($WW,WZ,ZZ$) production, as well as from the associated production of a vector boson $V$ ($V=W,Z$) or a Higgs boson and a $t\bar{t}$ pair ($t\bar{t}V$ and $t\bar{t}H$). All backgrounds with prompt leptons, i.e. those originating from the decay of a $W$ boson, a $Z$ boson, or a $\tau$-lepton, are estimated using samples of simulated events and initially normalised to their theoretical cross sections. In the simulation, the top-quark and SM Higgs boson masses are set to $172.5\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $125\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$, respectively, and the Higgs boson is allowed to decay into all SM particles with branching ratios calculated using Hdecay [61]. Backgrounds with non-prompt electrons or muons, with photons or jets misidentified as electrons, or with jets misidentified as $\tau_{\mathrm{had}}$ candidates, generically referred to as fake leptons, are estimated using data-driven methods. The background prediction is further improved during the statistical analysis by performing a likelihood fit to data using several signal-depleted analysis regions, as discussed in Sections 6 and 7.

5.1 Simulated signal and background processes

Samples of simulated $t\bar{t}\to WbHq$ events were generated with the next-to-leading-order (NLO) generator444In the following, the order of a generator should be understood as referring to the order in the strong coupling constant at which the matrix-element calculation is performed. Madgraph5_aMC@NLO 2.4.3 [62] (referred to in the following as MG5_aMC) with the NNPDF3.0 NLO [63] parton distribution function (PDF) set and interfaced to Pythia 8.212 [64] with the NNPDF2.3 LO [65] PDF set for the modelling of parton showering, hadronisation, and the underlying event. The A14 [66] set of tuned parameters in Pythia controlling the description of multiparton interactions and initial- and final-state radiation, referred to as the tune, was used. The signal sample is normalised to the same total cross section as used for the inclusive $t\bar{t}\to WbWb$ sample (see discussion below) and assuming an arbitrary branching ratio of $\mathscr{B}_{\mathrm{ref}}(t\to Hq)=1\%$. The case of both top quarks decaying into $Hq$ is neglected in the analysis given the existing upper limits on $\mathscr{B}(t\to Hq)$ (Section 1).

The nominal sample used to model the $t\bar{t}$ background was generated with the NLO generator Powheg-Box v2 [67, 68, 69, 70] using the NNPDF3.0 NLO PDF set. The Powheg-Box model parameter $h_{\textrm{damp}}$, which controls matrix element to parton shower matching and effectively regulates the high-$p_{\text{T}}$ radiation, was set to 1.5 times the top-quark mass. The parton showers, hadronisation, and underlying event were modelled by Pythia 8.210 with the NNPDF2.3 LO PDF set in combination with the A14 tune. Alternative $t\bar{t}$ simulation samples used to derive systematic uncertainties are described in Section 8.3. The generated $t\bar{t}$ samples are normalised to a theoretical cross section of $832^{+46}_{-51}$ pb, computed using Top++ v2.0 [71] at next-to-next-to-leading order (NNLO), including resummation of next-to-next-to-leading logarithmic (NNLL) soft gluon terms [72, 73, 74, 75, 76].

The $t\bar{t}$ background selected by the $tqH(b\bar{b})$ search is enriched in $t\bar{t}$+heavy-flavour production, and thus requires a more sophisticated treatment than provided by the nominal $t\bar{t}$ sample; this treatment is briefly outlined below. A detailed discussion can be found in Ref. [77]. The simulated $t\bar{t}$ events are categorised depending on the flavour content of additional particle jets not originating from the decay of the $t\bar{t}$ system. Events labelled as either $t\bar{t}$+$\geq$1$b$ or $t\bar{t}$+$\geq$1$c$ are generically referred to in the following as $t\bar{t}$+HF events, where HF stands for heavy flavour. The remaining events are labelled as $t\bar{t}$+light-jets events, including those with no additional jets. A finer categorisation of $t\bar{t}$+$\geq$1$b$ events is considered for the purpose of applying further corrections and assigning systematic uncertainties associated with the modelling of heavy-flavour production in different event topologies [77]. In particular, the $t\bar{t}$+$\geq$1$b$ events are reweighted to an NLO prediction in the four-flavour (4F) scheme of $t\bar{t}$+$\geq$1$b$ production including parton showering [78], based on Sherpa+OpenLoops [79, 80] (referred to as SherpaOL in the following) using the CT10 4F PDF set. This reweighting is performed in such a way that the inter-normalisations of the $t\bar{t}$+$\geq$1$b$ categories are at NLO accuracy, while preserving the $t\bar{t}$+$\geq$1$b$ cross section of the nominal $t\bar{t}$ sample. This reweighting is also applied to the alternative $t\bar{t}$ samples that are used to study systematic uncertainties.

Samples of single-top-quark events corresponding to the $t$-channel production mechanism were generated with the Powheg-Box v1 [81] generator, using the 4F scheme for the NLO matrix-element calculations and the fixed 4F CT10f4 [82] PDF set. Samples corresponding to the $tW$- and $s$-channel production mechanisms were generated with Powheg-Box v1 using the CT10 PDF set. Overlaps between the $t\bar{t}$ and $tW$ final states were avoided by using the diagram removal scheme [83]. The parton showers, hadronisation and the underlying event were modelled using Pythia 6.428 [84] with the CTEQ6L1 [85, 86] PDF set in combination with the Perugia 2012 tune [87]. The single-top-quark samples are normalised to the approximate NNLO theoretical cross sections [88, 89, 90].

Samples of $W/Z$+jets events were generated with the Sherpa 2.2.1 [79] generator. The matrix element was calculated for up to two partons at NLO and up to four partons at LO using Comix [91] and OpenLoops [80]. The matrix-element calculation is merged with the Sherpa parton shower [92] using the ME+PS@NLO prescription [93]. The PDF set used for the matrix-element calculation is NNPDF3.0 NNLO [63] with a dedicated parton shower tuning developed for Sherpa. Separate samples were generated for different $W/Z$+jets categories using filters for a $b$-jet ($W/Z$+$\geq$1$b$+jets), a $c$-jet and no $b$-jet ($W/Z$+$\geq$1$c$+jets), and with a veto on $b$- and $c$-jets ($W/Z$+light-jets), which are combined into the inclusive $W/Z$+jets samples. Both the $W$+jets and $Z$+jets samples are normalised to their respective inclusive NNLO theoretical cross sections calculated with FEWZ [94].

Samples of $WW/WZ/ZZ$+jets events were generated with Sherpa 2.2.1 using the CT10 PDF set and include processes containing up to four electroweak vertices. In the case of $WW/WZ$+jets ($ZZ$+jets) the matrix element was calculated for zero (up to one) additional partons at NLO and up to three partons at LO using the same procedure as for the $W/Z$+jets samples. The final states simulated require one of the bosons to decay leptonically and the other hadronically. All diboson samples are normalised to their NLO theoretical cross sections provided by Sherpa.

Samples of $t\bar{t}V$ and $t\bar{t}H$ events were generated with MG5_aMC 2.2.1, using NLO matrix elements and the NNPDF3.0 NLO PDF set, and interfaced to Pythia 8.210 with the NNPDF2.3 LO PDF set and the A14 tune. Instead, the $t\bar{t}V$ samples used in the $tqH(b\bar{b})$ search are based on LO matrix elements computed for up to two additional partons using the NNPDF3.0 NLO PDF set, and merged using the CKKW-L approach [95]. The $t\bar{t}V$ samples are normalised to the NLO cross section computed with MG5_aMC, while the $t\bar{t}H$ sample is normalised using the NLO cross section recommended in Ref. [96].

All generated samples, except those produced with the Sherpa [79] event generator, utilise EvtGen 1.2.0 [97] to model the decays of heavy-flavour hadrons. To model the effects of pile-up, events from minimum-bias interactions were generated using Pythia 8.186 [64] in combination with the A2 tune [98], and overlaid onto the simulated hard-scatter events according to the luminosity profile of the recorded data. The generated events were processed through a simulation [99] of the ATLAS detector geometry and response using Geant4 [100]. A faster simulation, where the full Geant4 simulation of the calorimeter response is replaced by a detailed parameterisation of the shower shapes [101], was adopted for some of the samples used to estimate systematic uncertainties in background modelling. Simulated events were processed through the same reconstruction software as the data, and corrections were applied so that the object identification efficiencies, energy scales and energy resolutions match those determined from data control samples.

5.2 Backgrounds with fake leptons

5.2.1 Fake electrons and muons

In the $tqH(b\bar{b})$ search, the background from multijet production (multijet background in the following) contributes to the selected data sample via several production and misreconstruction mechanisms. In the electron channel, it consists of non-prompt electrons (from semileptonic $b$- or $c$-hadron decays) as well as misidentified photons (from a conversion of a photon into an $e^{+}e^{-}$ pair) or jets with a high fraction of their energy deposited in the EM calorimeter. In the muon channel, the multijet background originates mainly from non-prompt muons. The multijet background normalisation and shape are estimated directly from data by using the matrix method technique [102, 103], which exploits differences in lepton identification and isolation properties between prompt leptons and leptons that are either non-prompt or result from the misidentification of photons or jets.

5.2.2 Fake $\tau$-lepton candidates

In the $tqH(\tau\tau)$ search, the background with one or more fake $\tau_{\mathrm{had}}$ candidates mainly arises from $t\bar{t}$ or multijet production, depending on the search channel, with $W$+jets production contributing to a lesser extent. Studies based on the simulation show that, for all the above processes, fake $\tau_{\mathrm{had}}$ candidates primarily result from the misidentification of light-quark jets, with the contribution from $b$-quarks and gluon jets playing a subdominant role. It is also found that the fake rate decreases for all jet flavours as the $\tau_{\mathrm{had}}$ candidate $p_{\text{T}}$ increases.

This background is estimated directly from data by defining control regions (CR) enriched in fake $\tau_{\mathrm{had}}$ candidates via loosened $\tau_{\mathrm{had}}$ requirements or flipped charge. These CRs do not overlap with the main search regions (SRs), discussed in Section 7. The CR selection requirements are analogous to those used to define the different SRs, except that the leading (trailing) $\tau_{\mathrm{had}}$ candidate in the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$) channel is required to fail the medium $\tau_{\mathrm{had}}$ identification but pass the loose identification, or the two $\tau_{\mathrm{had}}$ candidates have the same charge.

The fake $\tau_{\mathrm{had}}$ background prediction in a given SR is modelled by the distribution (referred to as the fake $\tau_{\mathrm{had}}$ template) derived from data in the corresponding CR. The fake $\tau_{\mathrm{had}}$ template is defined as the data distribution from which the contributions from the simulated backgrounds with real $\tau_{\mathrm{had}}$ candidates, originating primarily from $W(\to\tau\nu)$+jets and $Z(\to\tau\tau)$+jets, are subtracted. In the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel, simulation studies indicate that the fake $\tau_{\mathrm{had}}$ background composition is consistent between the SR and the CR, and dominated by $t\bar{t}$ production. In the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel, the fake $\tau_{\mathrm{had}}$ background is expected to be dominated by multijet production. However, simulation studies indicate that the contribution of $t\bar{t}$ events to the fake $\tau_{\mathrm{had}}$ background is higher in the SR than in the CR. Therefore, an appropriate number of simulated $t\bar{t}$ events with fake $\tau_{\mathrm{had}}$ candidates in the CR is added to the fake $\tau_{\mathrm{had}}$ template to match the fake $\tau_{\mathrm{had}}$ background composition in the SR. In both the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ and $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channels, the fake $\tau_{\mathrm{had}}$ template in each SR is initially normalised to the estimated fake $\tau_{\mathrm{had}}$ background yield, defined as the data yield minus the contributions from the simulated backgrounds with real $\tau_{\mathrm{had}}$ candidates (assuming no signal contribution). During the statistical analysis, the normalisation of the fake $\tau_{\mathrm{had}}$ background in each SR is allowed to vary freely in the fit to data, as discussed in Section 10.2.

6 Strategy for the $tqH(b\bar{b})$ search

This section presents an overview of the analysis strategy adopted in the $tqH(b\bar{b})$ search, which closely follows that of the previous search performed on the Run 1 dataset [27].

6.1 Event categorisation

Given that the $W\to\ell\nu$ and $H\to b\bar{b}$ decay modes are chosen, the $t\bar{t}\to WbHq$ signal is expected to have four jets in the final state, three of them originating from $b$-quarks, which can be effectively exploited to suppress the background. Additional jets can also be present because of initial- or final-state radiation. However, the use of the 60% $b$-tagging efficiency operating point, characterised by a low mistag rate for $c$- and light-jets, results in both the $t\bar{t}\to WbHc$ and $t\bar{t}\to WbHu$ signals having a similar $b$-tag multiplicity distribution, with a very small fraction of events having four or more $b$-tagged jets.

In order to optimise the sensitivity of the search, the selected events are categorised into different analysis regions depending on the number of jets (4, 5 and $\geq$6) and on the number of $b$-tagged jets (2, 3 and $\geq$4). Therefore, a total of nine analysis regions are considered: (4j, 2b), (4j, 3b), (4j, 4b), (5j, 2b), (5j, 3b), (5j, $\geq$4b), ($\geq$6j, 2b), ($\geq$6j, 3b), and ($\geq$6j, $\geq$4b), where ($n$j, $m$b) indicates $n$ selected jets and $m$ $b$-tagged jets.

The overall rate and composition of the $t\bar{t}$+jets background strongly depends on the jet and $b$-tag multiplicities, as illustrated in Figure 1. Regions with exactly two $b$-tagged jets are dominated by $t\bar{t}$+light-jets, while regions with at least four $b$-tagged jets are dominated by $t\bar{t}$+$\geq$1$b$. Intermediate compositions are found in regions with exactly three $b$-tagged jets. Most of the $t\bar{t}$+light-jets background events in these regions have a $b$-tagged charm jet from the hadronic $W$ boson decay, in addition to the two $b$-jets from the top-quark decays.

In the regions with four or five jets and exactly three $b$-tagged jets, which dominate the sensitivity of this search, the selected signal events have a $H\to b\bar{b}$ decay in more than 97% of the events. The other regions have significantly lower signal-to-background ratios, but they are used to improve the $t\bar{t}$+jets background prediction and constraining the related systematic uncertainties through a likelihood fit to data. Because of a somewhat larger fraction of $t\bar{t}\to WbHc$ signal in the regions with exactly three $b$-tagged jets, resulting from the higher mistag rate for $c$-jets than for light-jets, this analysis is expected to have slightly better sensitivity to a $t\bar{t}\to WbHc$ signal than to a $t\bar{t}\to WbHu$ signal.

6.2 Likelihood discriminant

After event categorisation, the signal-to-background ratio is insufficient even in the best cases to achieve sensitivity, and a suitable discriminating variable between signal and background needs to be constructed in order to improve the sensitivity of the search. Since both signal and background result from the $t\bar{t}$ decay, their discrimination is a challenge and it is based on a few measured quantities. The most prominent features are the different resonances present in the decay (the Higgs boson in the case of the $t\bar{t}\to WbHq$ signal and a hadronically decaying $W$ boson in the case of the $t\bar{t}\to WbWb$ background), and the different flavours of the jets forming those resonances. However, the large number of jets in the final state causes ambiguities in the calculation of these kinematic variables to discriminate signal events from background events.

This search uses a likelihood (LH) discriminant similar to that developed in Ref. [27]. The LH variable for a given event is defined as:

[TABLE]

where $P^{\textrm{sig}}(\mathbf{x})$ and $P^{\textrm{bkg}}(\mathbf{x})$ represent the probability density functions (pdf) of a given event under the signal hypothesis ($t\bar{t}\to WbHq$) and under the background hypothesis ($t\bar{t}\to WbWb$), respectively. Both $P^{\textrm{sig}}$ and $P^{\textrm{bkg}}$ are functions of $\mathbf{x}$, representing the four-momentum vectors of all final-state particles at the reconstruction level: the lepton, the missing transverse momentum, and the selected jets in a given analysis region. The value of the multivariate $b$-tagging discriminant for each jet is also included in $\mathbf{x}$. As in Ref. [27], $P^{\textrm{sig}}$ and $P^{\textrm{bkg}}$ are approximated as a product of one-dimensional pdfs over the set of two-body and three-body invariant masses that correspond to the expected resonances in the event (the leptonically decaying $W$ boson, the Higgs boson or the hadronically decaying $W$ boson, and the corresponding parent top quarks) and averaged over all possible parton–jet matching combinations. Combinations are weighted using the per-jet multivariate $b$-tagging discriminant value to suppress the impact from parton–jet assignments that are inconsistent with the correct flavour of the parton candidates. The invariant masses are computed from the reconstructed lepton, missing transverse momentum, and jets. After a suitable transformation of the three-body invariant masses (see Ref. [27]), all considered invariant mass variables are largely uncorrelated, thus making possible the factorisation of $P^{\textrm{sig}}$ and $P^{\textrm{bkg}}$ as discussed above.

Two background hypotheses are considered, corresponding to the dominant backgrounds in the analysis: $t\bar{t}$+light-jets and $t\bar{t}$+$\geq$1$b$. Thus, $P^{\textrm{bkg}}$ is computed as the average of the pdfs for the two hypotheses, weighted by their relative fractions found in simulated $t\bar{t}$+jets events, which depend on the analysis region considered. Furthermore, in a significant fraction of $t\bar{t}\to WbHq$ simulated events (about 40–50% in regions with exactly three $b$-tagged jets), the light-quark jet from the hadronic top-quark decay is not among the selected jets. Similarly, in about 30–40% (50–90%) of simulated $t\bar{t}$+light-jets ($t\bar{t}+\geq 1b$) background events in regions with exactly three $b$-tagged jets, the light-quark jet originating from the $W$ boson decay is also not selected. Thus, the calculation of $P^{\textrm{sig}}$ and $P^{\textrm{bkg}}$ also includes an additional hypothesis to account for this topology, again weighted by the corresponding fractions. In this case, the invariant masses involving the missing jet are computed using the highest-$p_{\text{T}}$ jet not matched to a decay product from the $t\bar{t}$ system.

Figure 2 shows a comparison between data and prediction in the most sensitive analysis region, (4j, 3b), for several kinematic variables associated with the reconstructed lepton, jets, and missing transverse momentum. The distributions shown correspond to the lepton $p_{\text{T}}$, the $E_{\text{T}}^{\text{miss}}$, the scalar sum of the transverse momenta of the jets, and the invariant mass distribution of the two $b$-tagged jets with lowest $\Delta R$ separation. The variables displayed do not correspond directly to those used internally in the evaluation the LH discriminant, as to build them it is necessary to select a particular signal or background hypothesis and a jet permutation. Instead, these distributions are shown to demonstrate that a good description of the data by the background prediction is observed in several kinematic variables related to the information used in the LH discriminant construction.

Figure 3 compares the shape of the LH discriminant distribution between the $t\bar{t}\to WbHc$ and $t\bar{t}\to WbHu$ signals and the $t\bar{t}\to WbWb$ background in each of the analysis regions considered. Since this analysis has higher expected sensitivity to a $t\bar{t}\to WbHc$ signal than to a $t\bar{t}\to WbHu$ signal, in order to allow probing of the $\mathscr{B}(t\to Hu)$ versus $\mathscr{B}(t\to Hc)$ plane, the LH discriminant optimised for $t\bar{t}\to WbHc$ is used for both decay modes. It was verified that using the $t\bar{t}\to WbHc$ discriminant for the $t\bar{t}\to WbHu$ search does not result in a significant sensitivity loss.

7 Strategy for the $tqH(\tau\tau)$ search

The analysis strategy adopted in the $tqH(\tau\tau)$ search closely follows that developed in Ref. [104] and is summarised in this section.

7.1 Event categorisation and kinematic reconstruction

In the $tqH(\tau\tau)$ search, the $t\bar{t}\to WbHq$ signal being probed is characterised by the presence of $\tau$-leptons from the decay of the Higgs boson and at least four jets, only one of which originates from a $b$-quark. If one of the $\tau$-leptons decays leptonically, an isolated electron or muon and significant $E_{\text{T}}^{\text{miss}}$ is also expected. However, in a significant fraction of the events the lowest-$p_{\text{T}}$ jet from the $W$ boson decay fails the minimum $p_{\text{T}}$ requirement of $30\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$, resulting in signal events with only three jets reconstructed. In order to optimise the sensitivity of the search, the selected events are categorised into four SRs depending on the number of $\tau_{\text{lep}}$ and $\tau_{\mathrm{had}}$ candidates, and on the number of jets: ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, 3j), ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, $\geq$4j), ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, 3j), and ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, $\geq$4j).

This event categorisation is primarily motivated by the different quality of the event kinematic reconstruction, depending on the amount of $E_{\text{T}}^{\text{miss}}$ in the event (larger in $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ events compared with $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ events), and whether a jet from the hadronic top-quark decay is missing or not (events with exactly three jets or at least four jets). The event kinematic reconstruction is based on the strategy used in Ref. [104], and is summarised below.

Events with exactly three jets that are compatible with having a fully reconstructed hadronically decaying top quark ($t\to Wb\to qqb$) are rejected, as the $t\to Hq$ decay cannot be reconstructed due to the missing light-quark jet. This compatibility is assessed via a likelihood function that depends on the reconstructed mass of the three-jet system and the two non-$b$-tagged jets. For the remaining events, the selected jets are assigned to the different top-quark decay products via a criterion based on minimising a sum of angular distances between objects. Finally, the four-momenta of the invisible decay products for each $\tau$-lepton decay are estimated by minimising a $\chi^{2}$ function based on the probability density functions for the angular distance of the visible and invisible products of the $\tau$-lepton decay, and including Gaussian constraints on the $\tau$-lepton mass, the Higgs boson mass and the measured $E_{\text{T}}^{\text{miss}}$ within their expected resolutions. The resolution on the $\tau$-lepton mass and the Higgs boson mass are taken to be $1.8\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ and $20\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$, respectively, while the resolution on the measured $E_{\text{T}}^{\text{miss}}$ is parameterised as a linear function of $\sqrt{\sum E_{T}}$, with $\sum E_{T}$ denoting the scalar sum of the $p_{\text{T}}$ of all physics objects contributing to the $E_{\text{T}}^{\text{miss}}$ reconstruction [60]. After the $\chi^{2}$ minimisation, the Higgs boson four-momentum, and hence its invariant mass, as well as the four-momentum of the parent top quark, are determined with better resolution. Following the event kinematic reconstruction, several kinematic variables that discriminate between signal and background are defined. These variables are used in the multivariate analysis discussed in the next section.

7.2 Multivariate discriminant

Boosted decision trees are used in each SR to improve the separation between signal and background. In the training, only $t\bar{t}\to W(qq)bH(\tau\tau)q$ signal events are used against the total SM background (including both real and fake $\tau_{\mathrm{had}}$ contributions), whereas to obtain the result the contributions from $t\bar{t}\to W(\ell\nu)bHq$ signal events are also taken into account.

A large set of potential variables were investigated in each SR separately, and only those variables that led to better discrimination by the BDT were kept. The discrimination of a given variable was quantified by the “separation" and “importance" measures provided by the TMVA package [105]. The BDT input variables in each SR are listed in Table 2 and defined in the following:

• •

$m_{\tau\tau}^{\text{fit}}$: the invariant mass of the two $\tau$-lepton candidates after the reconstruction of the neutrinos, indicating the reconstructed Higgs boson mass.

• •

$m_{Hq}$: the invariant mass of the reconstructed Higgs boson and the associated light-quark jet in the $t\to Hq$ decay, corresponding to the reconstructed mass of the parent top quark.

• •

$m_{\text{T,lep}}$: the transverse mass calculated from the lepton and $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ in the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel.

• •

$p_{\text{T,1}}$ and $p_{\text{T,2}}$: the transverse momenta of the lepton and $\tau_{\mathrm{had}}$ candidate (referred to as particles 1 and 2 respectively) in the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel, or the transverse momenta of the leading and trailing $\tau_{\mathrm{had}}$ candidates (referred to as particles 1 and 2 respectively) in the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel.

• •

$E_{\text{T}}^{\text{miss}}$ $\phi$ centrality: a variable that quantifies the angular position of $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ relative to the visible $\tau$-lepton decay products in the transverse plane. It is defined as:

[TABLE]

where $\phi_{\mathrm{miss}}$ denotes the azimuthal angle of $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$, and $\phi_{1}$ and $\phi_{2}$ denote the azimuthal angles the two $\tau$-lepton candidates (the lepton and $\tau_{\mathrm{had}}$ candidate in the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel, or the leading and trailing $\tau_{\mathrm{had}}$ candidates in the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel), referred to as particles 1 and 2 respectively.

• •

$E_{\text{T},\parallel}^{\text{miss}}$: the magnitude of the projection of the original $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ vector parallel to the fitted $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ vector, minus the magnitude of the fitted $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ vector.

• •

$E_{\text{T},\perp}^{\text{miss}}$: the magnitude of the projection of the original $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ vector perpendicular to the fitted $\vec{p}_{\textrm{T}}^{\textrm{\;miss}}$ vector.

• •

$m_{bj_{1}}$: the invariant mass of the $b$-jet and the leading jet candidate from the hadronically decaying $W$ boson.

• •

$m_{\text{lep}j}$: the invariant mass of the lepton and the jet that has the smallest angular distance to the $\tau_{\mathrm{lep}}$ candidate.

• •

$m_{\tau j}$: the invariant mass of the $\tau_{\mathrm{had}}$ candidate and the jet that has the smallest angular distance to the $\tau_{\mathrm{had}}$ candidate.

• •

$x_{1}^{\text{fit}}$ and $x_{2}^{\text{fit}}$: the momentum fractions carried by the visible decay products from the two $\tau$-lepton candidates (whether $\tau_{\text{lep}}$ or $\tau_{\mathrm{had}}$) per event. It is based on the best-fit four-momentum of the neutrino(s) according to the event reconstruction procedure outlined in Section 7.1.

• •

$m_{bj_{1}j_{2}}$: the invariant mass of the $b$-jet and the two jets originating from the $W$ boson in the $t\to Wb\to j_{1}j_{2}b$ decay, corresponding to the reconstructed mass of the parent top quark. This variable is only defined for events with at least four jets.

Among these variables, the most discriminating are $m_{\tau\tau}^{\text{fit}}$, $p_{\text{T},2}$, $x_{1}^{\text{fit}}$ and $x_{2}^{\text{fit}}$. A comparison between data and the predicted background for some of these variables in each of the SRs considered is shown in Figures 4 and 5. A good description of the data by the background model is observed in all cases. The level of discrimination between signal and background achieved by the BDTs is illustrated in Figure 6.

8 Systematic uncertainties

Several sources of systematic uncertainty that can affect the normalisation of signal and background and/or the shape of their corresponding discriminant distributions are considered. Each source is considered to be uncorrelated with the other sources. Correlations of a given systematic uncertainty are maintained across processes and channels as appropriate. The following sections describe the systematic uncertainties considered.

8.1 Luminosity

The uncertainty in the integrated luminosity is 2.1%, affecting the overall normalisation of all processes estimated from the simulation. It is derived, following a methodology similar to that detailed in Ref. [106], and using the LUCID-2 detector for the baseline luminosity measurements [107], from a calibration of the luminosity scale using $x$–$y$ beam-separation scans.

8.2 Reconstructed objects

Uncertainties associated with electrons, muons, and $\tau_{\mathrm{had}}$ candidates arise from the trigger, reconstruction, identification and isolation (in the case of electrons and muons) efficiencies, as well as the momentum scale and resolution. These are measured using $Z\to\ell^{+}\ell^{-}$ and $J/\psi\to\ell^{+}\ell^{-}$ events ($\ell=e,\mu$) [41, 43] in the case of electrons and muons, and using $Z\to\tau^{+}\tau^{-}$ events in the case of $\tau_{\mathrm{had}}$ candidates [59].

Uncertainties associated with jets arise from the jet energy scale and resolution, and the efficiency to pass the JVT requirements. The largest contribution results from the jet energy scale, whose uncertainty dependence on jet $p_{\text{T}}$ and $\eta$, jet flavour, and pile-up treatment, is split into 21 uncorrelated components that are treated independently [48].

Uncertainties associated with energy scales and resolutions of leptons and jets are propagated to $E_{\text{T}}^{\text{miss}}$. Additional uncertainties originating from the modelling of the underlying event, in particular its impact on the $p_{\text{T}}$ scale and resolution of unclustered energy, are negligible.

Efficiencies to tag $b$-jets and $c$-jets in the simulation are corrected to match the efficiencies in data by $p_{\text{T}}$-dependent factors, whereas the light-jet efficiency is scaled by $p_{\text{T}}$- and $\eta$-dependent factors. The $b$-jet efficiency is measured in a data sample enriched in $t\bar{t}$ events [108], while the $c$-jet efficiency is measured using $t\bar{t}$ events [109] or $W$+$c$-jet events [53]. The light-jet efficiency is measured in a multijet data sample enriched in light-flavour jets [110]. Since the $t\bar{t}$ sample used to measure the $c$-jet tagging efficiency overlaps with the analysis sample, the $tqH(b\bar{b})$ search uses instead the $W$+$c$-jet scale factors. In the case of the $tqH(b\bar{b})$ ($tqH(\tau\tau)$) search, the uncertainties in these scale factors include a total of 6 independent sources affecting $b$-jets, 1 (2) source(s) affecting $c$-jets, and 17 sources affecting light-jets. These systematic uncertainties are taken as uncorrelated between $b$-jets, $c$-jets, and light-jets. An additional uncertainty is included due to the extrapolation of these corrections to jets with $p_{\text{T}}$ beyond the kinematic reach of the data calibration samples used ($p_{\text{T}}>300\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ for $b$- and $c$-jets, and $p_{\text{T}}>750\leavevmode\nobreak\ \text{Ge\kern-1.00006ptV}$ for light-jets); it is taken to be correlated among the three jet flavours. Since the fraction of signal and background in this kinematic regime is very small, these uncertainties have a negligible impact in the analyses. Finally, an uncertainty related to the application of $c$-jet scale factors to $\tau$-jets is considered, which also has a negligible impact.

8.3 Background modelling

A number of sources of systematic uncertainty affecting the modelling of $t\bar{t}$+jets are considered. An uncertainty of 6% is assigned to the inclusive $t\bar{t}$ production cross section [71], including contributions from varying the factorisation and renormalisation scales, as well as from the top-quark mass, the PDF and $\alpha_{\textrm{S}}$. The latter two represent the largest contribution to the overall theoretical uncertainty in the cross section and were calculated using the PDF4LHC prescription [111] with the MSTW 2008 68% CL NNLO, CT10 NNLO [82, 112] and NNPDF2.3 5F FFN [65] PDF sets. The uncertainty associated with the choice of NLO generator is derived by comparing the nominal prediction from Powheg-Box+Pythia 8 with a prediction from Sherpa 2.2.1. For the latter, the matrix-element calculation is performed for up to two partons at NLO and up to four partons at LO using Comix and OpenLoops, and merged with the Sherpa parton shower using the ME+PS@NLO prescription. The uncertainty due to the choice of parton shower and hadronisation (PS & Had) model is derived by comparing the predictions from Powheg-Box interfaced either to Pythia 8 or Herwig 7. The latter uses the MMHT2014 LO [113] PDF set in combination with the H7UE tune [114]. The uncertainty in the modelling of additional radiation is assessed with two alternative Powheg-Box+Pythia 8 samples: a sample with increased radiation (referred to as radHi) is obtained by decreasing the renormalisation and factorisation scales by a factor of two, doubling the $h_{\textrm{damp}}$ parameter, and using the Var3c upward variation of the A14 parameter set; a sample with decreased radiation (referred to as radLow) is obtained by increasing the scales by a factor of two and using the Var3c downward variation of the A14 set [115].

In the case of the $tqH(b\bar{b})$ search, where the $t\bar{t}$+HF background plays a prominent role (see Fig. 1), a more detailed treatment of its associated systematic uncertainties is used. In particular, since several analysis regions have a sufficiently large number of $t\bar{t}$+$\geq$1$b$ background events, its normalisation is determined in the fit to data. In the case of the $t\bar{t}$+$\geq$1$c$ normalisation, an uncertainty of 50% is assumed, as the fit to the data is unable to precisely determine it, and the analysis has very limited sensitivity to this uncertainty. Since the diagrams that contribute to $t\bar{t}$+light-jets, $t\bar{t}$+$\geq$1$c$, and $t\bar{t}$+$\geq$1$b$ production are different, all above uncertainties in $t\bar{t}$+jets background modelling (NLO generator, PS & Had, and radHi/radLow), except the uncertainty of the inclusive cross section, are considered to be uncorrelated among these processes. Additional uncertainties of the $t\bar{t}$+$\geq$1$b$ background are considered associated with the NLO prediction from SherpaOL, which is used for reweighting the nominal Powheg-Box+Pythia 8 prediction. These include three different scale variations, a different shower-recoil model scheme, and two alternative PDF sets (MSTW 2008 NLO and NNPDF2.3 NLO). Additional uncertainties are assessed for the contributions to the $t\bar{t}$+$\geq$1$b$ background originating from multiple parton interactions. Finally, an additional uncertainty is assigned to the $t\bar{t}$+$\geq$1$b$ background by comparing the predictions from Powheg-Box+Pythia 8 and SherpaOL 4F (5F vs 4F). In the derivation of the above uncertainties, the overall normalisations of the $t\bar{t}$+$\geq$1$c$ and $t\bar{t}$+$\geq$1$b$ backgrounds at the particle level are fixed to the nominal prediction. In order to maintain the inclusive $t\bar{t}$ cross section, the normalisation of the $t\bar{t}$+light-jets background at the particle level is adjusted accordingly.

Uncertainties affecting the normalisation of the $V$+jets background are estimated for the sum of $W$+jets and $Z$+jets, and separately for $V$+light-jets, $V$+$\geq$1$c$+jets, and $V$+$\geq$1$b$+jets subprocesses. The total normalisation uncertainty of $V$+jets processes is estimated by comparing the data and total background prediction in the different analysis regions considered, but requiring exactly zero $b$-tagged jets. Agreement between data and predicted background in these modified regions, which are dominated by $V$+light-jets, is found to be within approximately 30%. This bound is taken to be the normalisation uncertainty, correlated across all $V$+jets subprocesses. Since Sherpa 2.2 has been found to underestimate $V$+heavy-flavour production by about a factor of 1.3 [116], additional 30% normalisation uncertainties are assumed for $V$+$\geq$1$c$+jets and $V$+$\geq$1$b$+jets subprocesses, considered uncorrelated between them.

Uncertainties affecting the modelling of the single-top-quark background include a $+5\%$/$-4\%$ uncertainty of the total cross section estimated as a weighted average of the theoretical uncertainties in $t$-, $tW$- and $s$-channel production [88, 89, 90]. Additional uncertainties associated with the modelling of additional radiation are assessed by comparing the nominal samples with alternative samples where generator parameters are varied. For the $t$- and $tW$-channel processes, an uncertainty due to the choice of parton shower and hadronisation model is derived by comparing events produced by Powheg-Box interfaced to Pythia 6 or Herwig++. These uncertainties are treated as fully correlated among single-top-quark production processes, but uncorrelated with the corresponding uncertainty of the $t\bar{t}$+jets background. An additional systematic uncertainty in $tW$-channel production concerning the separation between $t\bar{t}$ and $tW$ at NLO is assessed by comparing the nominal sample, which uses the diagram removal scheme [117], with an alternative sample using the diagram subtraction scheme [117].

Uncertainties of the diboson background normalisation include 5% from the NLO theory cross sections [118, 119], as well as an additional 24% normalisation uncertainty added in quadrature for each additional inclusive jet-multiplicity bin, based on a comparison among different algorithms for merging LO matrix elements and parton showers [120] (it is assumed that two jets originate from the $W/Z$ decay, as in $WW/WZ\to\ell\nu jj$). Therefore, the total normalisation uncertainty is $5\%\oplus\sqrt{N-2}\times 24\%$, where $N$ is the selected jet multiplicity, resulting in 34%, 42%, and 48%, for events with exactly 4 jets, exactly 5 jets, and $\geq$6 jets, respectively. Recent comparisons between data and Sherpa 2.1.1 for $WZ(\to\ell\nu\ell\ell)+\geq$4 jets show agreement within the experimental uncertainty of approximately 40% [121], which further justifies the above uncertainty. Given the very small contribution of this background to the total prediction, the final result is not affected by the assumed modelling uncertainties.

Uncertainties of the $t\bar{t}V$ and $t\bar{t}H$ cross sections are 15% and $+10\%$/$-13\%$, respectively, from the uncertainties of their respective NLO theoretical cross sections [122, 123, 96].

Uncertainties of the data-driven multijet background in the $tqH(b\bar{b})$ search include contributions from the limited size of the data sample, particularly at high jet and $b$-tag multiplicities, as well as from the uncertainty in the rate of fake leptons, estimated in different control regions (e.g. selected with an upper requirement on either $E_{\text{T}}^{\text{miss}}$ or $m_{\mathrm{T}}^{W}$). A combined normalisation uncertainty of 50% due to all these effects is assigned, which is taken as correlated across jet and $b$-tag multiplicity bins, but uncorrelated between electron and muon channels. No explicit shape uncertainty is assigned since the large statistical uncertainties associated with the multijet background prediction, which are uncorrelated between bins in the final discriminant distribution, effectively cover all possible shape uncertainties.

Uncertainties of the data-driven fake $\tau_{\mathrm{had}}$ background in the $tqH(\tau\tau)$ search are obtained by using additional signal-depleted regions. The construction is similar to that of the SRs and corresponding CRs discussed in Section 5.2, but employing further loosened $\tau_{\mathrm{had}}$ identification criteria, and thus referred to as “loose SR” and “loose CR”. In each loose SR, after subtracting the small simulation-predicted contribution from real $\tau_{\mathrm{had}}$ candidates, the relative difference in the shape of the distribution between the remaining data and the fake $\tau_{\mathrm{had}}$ background estimate based on its associated loose CR is assigned as an uncertainty of the prediction in the nominal SR. In addition, a 30% uncertainty is applied to the fraction of $t\bar{t}$ events with a fake $\tau_{\mathrm{had}}$ candidate from the simulation that are added to the fake $\tau_{\mathrm{had}}$ template in the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel as part of the fake $\tau_{\mathrm{had}}$ background estimation procedure. This uncertainty, associated with the modelling of the fake $\tau_{\mathrm{had}}$ rate by the simulation, is estimated by comparing data and simulation in a sample enriched in $t\bar{t}$ dilepton events plus a fake $\tau_{\mathrm{had}}$ candidate. The same uncertainty is assigned to the selected signal events with fake $\tau_{\mathrm{had}}$ candidates. In addition, a systematic uncertainty is assigned to account for the different fractional composition of particles (various types of leptons and partons) producing the fake $\tau_{\mathrm{had}}$ candidates between each SR and its corresponding CR in the $t\bar{t}$ simulation. Finally, the normalisation of the fake $\tau_{\mathrm{had}}$ background in each SR is determined in the fit to data.

8.4 Signal modelling

Several normalisation and shape uncertainties are taken into account for the $t\bar{t}\to WbHq$ signal. The uncertainty of the $t\bar{t}$ cross section also applies to the $t\bar{t}\to WbHq$ signal and is taken to be the same as, and fully correlated with, the uncertainty assigned to the $t\bar{t}\to WbWb$ background. Uncertainties of the Higgs boson branching ratios are taken into account following the recommendation in Ref. [96]. Additional uncertainties associated with the modelling of additional radiation, with the choice of NLO generator, and with the choice of parton shower and hadronisation model, are estimated from the comparison of the nominal and alternative $t\bar{t}\to WbWb$ background samples (discussed in Section 8.3) and applied to $t\bar{t}\to WbHq$ signal. These modelling uncertainties are taken to be uncorrelated with those affecting the $t\bar{t}\to WbWb$ background.

9 Statistical analysis

For each search, the final discriminant distributions across all analysis regions considered are jointly analysed to test for the presence of a signal. The statistical analysis uses a binned likelihood function ${\cal L}(\mu,\theta)$ constructed as a product of Poisson probability terms over all bins considered in the search. This function depends on the signal-strength parameter $\mu$, defined as a factor multiplying the expected yield of $t\bar{t}\to WbHq$ signal events normalised to a reference branching ratio $\mathscr{B}_{\mathrm{ref}}(t\to Hq)=1\%$, and $\theta$, a set of nuisance parameters that encode the effect of systematic uncertainties on the signal and background expectations. Therefore, the expected total number of events in a given bin depends on $\mu$ and $\theta$. All nuisance parameters are subject to Gaussian or log-normal constraints in the likelihood, with the exception of a few parameters that control the normalisation of some background components (e.g. the $t\bar{t}$+$\geq$1$b$ background in the case of the $tqH(b\bar{b})$ search), which are treated as free parameters in the fit.

For a given value of $\mu$, the nuisance parameters $\theta$ allow variations of the expectations for signal and background according to the corresponding systematic uncertainties, and their fitted values result in the deviations from the nominal expectations that globally provide the best fit to the data. This procedure allows a reduction of the impact of systematic uncertainties on the search sensitivity by taking advantage of the highly populated background-dominated bins included in the likelihood fit. Statistical uncertainties in each bin of the predicted final discriminant distributions are taken into account by dedicated parameters in the fit. The best-fit $\mathscr{B}(t\to Hq)$ is obtained by performing a binned likelihood fit to the data under the signal-plus-background hypothesis, maximising the likelihood function ${\cal L}(\mu,\theta)$ over $\mu$ and $\theta$.

The fitting procedure was initially validated through extensive studies using mock data, defined as the sum of all predicted backgrounds plus an injected signal of variable strength, as well as by performing fits to real data where bins of the final discriminant variable with a signal contamination above 5% are excluded (referred to as blinding requirements). In both cases, the robustness of the model for systematic uncertainties is established by verifying the stability of the fitted background when varying assumptions about some of the leading sources of uncertainty. After this, the blinding requirements are removed in the data and a fit under the signal-plus-background hypothesis is performed. Further checks involve the comparison of the fitted nuisance parameters before and after removal of the blinding requirements, and their values are found to be consistent. In addition, it is verified that the fit is able to correctly determine the strength of a simulated signal injected into the real data.

The test statistic $q_{\mu}$ is defined as the profile likelihood ratio, $q_{\mu}=-2\ln({\cal L}(\mu,{\hat{\theta}}_{\mu})/{\cal L}(\hat{\mu},\hat{\theta}))$, where $\hat{\mu}$ and $\hat{\theta}$ are the values of the parameters that maximise the likelihood function (subject to the constraint $0\leq\hat{\mu}\leq\mu$), and ${\hat{\theta}}_{\mu}$ are the values of the nuisance parameters that maximise the likelihood function for a given value of $\mu$. The test statistic $q_{\mu}$ is evaluated with the RooFit package [124, 125]. A related statistic is used to determine whether the observed data is compatible with the background-only hypothesis (the so-called discovery test) by setting $\mu=0$ in the profile likelihood ratio and leaving $\hat{\mu}$ unconstrained: $q_{0}=-2\ln({\cal L}(0,{\hat{\theta}}_{0})/{\cal L}(\hat{\mu},\hat{\theta}))$. The $p$-value (referred to as $p_{0}$), representing the level of agreement between the data and the background-only hypothesis, is estimated by integrating the distribution of $q_{0}$ based on the asymptotic formulae in Ref. [126], above the observed value of $q_{0}$ in the data. Upper limits on $\mu$, and thus on $\mathscr{B}(t\to Hq)$, are derived by using $q_{\mu}$ in the CL${}_{\textrm{s}}$ method [127, 128]. For a given signal scenario, values of the $\mathscr{B}(t\to Hq)$ yielding CL${}_{\textrm{s}}<0.05$, where CL${}_{\textrm{s}}$ is computed using the asymptotic approximation [126], are excluded at $\geq 95\%$ CL.

10 Results

This section presents the results obtained from the individual searches for $t\bar{t}\to WbHq$, as well as their combination, following the statistical analysis discussed in Section 9.

10.1 $tqH(b\bar{b})$ search

A binned likelihood fit under the signal-plus-background hypothesis is performed on the LH discriminant distributions in the nine analysis regions considered. In the regions with exactly three $b$-tagged jets, which have the highest sensitivity, the full LH distribution is used with ten equal-width bins. In contrast, in the regions with at least four $b$-tagged jets, which have a limited number of data events and a small signal fraction, only two equal-width bins are used. Finally, in the regions with exactly two $b$-tagged jets the total event yield after requiring the LH discriminant to be above 0.6, is used. The unconstrained parameters of the fit are the signal strength and a global normalisation factor applied to the $t\bar{t}$+$\geq$1$b$ background common to all analysis regions. Figures 7 and 8 show a comparison of the LH discriminant for data and prediction in the regions with exactly three and at least four $b$-tagged jets, both before and after performing the fit to data, in the case of the $t\bar{t}\to WbHc$ search. Tables summarising the pre-fit and post-fit yields can be found in Appendix A.

The best-fit branching ratio obtained is $\mathscr{B}(t\to Hc)=[-0.2^{+0.7}_{-0.7}\,(\mathrm{stat})^{+2.2}_{-2.3}\,(\mathrm{syst})]\times 10^{-3}$, assuming $\mathscr{B}(t\to Hu)=0$. A similar fit is performed for the $t\bar{t}\to WbHu$ search, yielding $\mathscr{B}(t\to Hu)=[0.2^{+0.8}_{-0.7}\,(\mathrm{stat})^{+2.5}_{-2.9}\,(\mathrm{syst})]\times 10^{-3}$, assuming $\mathscr{B}(t\to Hc)=0$. The total uncertainties of the measured branching ratios are dominated by systematic uncertainties.

The large number of events in the analysis regions considered, together with their different background compositions, allows the fit to place constraints on the combined effect of several sources of systematic uncertainty. As a result, an improved background prediction is obtained with a significantly reduced uncertainty, not only in the signal-depleted regions, but also in the most sensitive analysis regions for this search, (4j, 3b) and (5j, 3b). The regions with two $b$-tagged jets are used to constrain the leading uncertainties affecting the $t\bar{t}$+light-jets background prediction, while the channels with at least four $b$-tagged jets are sensitive to the uncertainties affecting the $t\bar{t}$+HF background prediction. In particular, one of the main corrections applied by the fit is an increase of the $t\bar{t}+\geq 1b$ normalisation by a factor of $1.17\pm 0.15$ relative to the nominal prediction by adjusting the corresponding nuisance parameter. The $t\bar{t}+\geq 1c$ normalisation is also increased, by a factor of $1.34\pm 0.40$. These corrections are in agreement with those found in Ref. [77]. Additionally, a few nuisance parameters are adjusted by the fit, with the largest effects corresponding to the leading nuisance parameters related to the $b$-tagging and $c$-tagging calibrations (by about 0.8 standard deviations), and those related to $t\bar{t}+\geq 1b$ and $t\bar{t}+\geq 1c$ modelling, which are based on a comparison with alternative generators (by 0.5 standard deviations or less). The leading uncertainties affecting the signal extraction by the fit are related to the $c$-tagging calibration ($\Delta\mathscr{B}$ $\sim$ $1.5\times 10^{-3}$), followed by the $t\bar{t}$+light-jets PS & Had uncertainty ($\Delta\mathscr{B}$ $\sim$ $1.2\times 10^{-3}$). Smaller contributions ($\Delta\mathscr{B}$ $\sim$ 0.5–$1.0\times 10^{-3}$ each) result from the uncertainties associated with the $t\bar{t}+\geq 1b$ 5F vs 4F comparison, the dependence of jet energy scale on the jet flavour, the uncertainty of the $t\bar{t}+\geq 1c$ normalisation, and the limited size of the simulated samples in some of the bins with the highest signal-to-background ratio. The uncertainty most strongly constrained by the fit is that related to the $c$-tagging calibration. It is reduced by about a factor of two of its value as originally determined in $W$+$c$-jet events [53]. This is possible because the fit exploits the large number of $t\bar{t}$ events with two and three $b$-tagged jets to effectively perform a $c$-tagging calibration, whose results are found to be consistent with those of Ref. [109]. Beyond the constraints on a few individual uncertainties, the significant reduction of the total background uncertainty achieved by the fit primarily derives from the anti-correlations found among systematic uncertainties from different sources.

In the absence of a significant excess of data events above the background expectation, 95% CL limits are set on $\mathscr{B}(t\to Hc)$ and $\mathscr{B}(t\to Hu)$. The observed (expected) 95% CL upper limits on the branching ratios are $\mathscr{B}(t\to Hc)<4.2\times 10^{-3}\,(4.0\times 10^{-3})$ and $\mathscr{B}(t\to Hu)<5.2\times 10^{-3}\,(4.9\times 10^{-3})$.

10.2 $tqH(\tau\tau)$ search

A binned likelihood fit under the signal-plus-background hypothesis is performed on the BDT discriminant distributions in the four analysis regions considered. The unconstrained parameters of the fit are the signal strength, and four independent parameters associated with the normalisation of the fake $\tau_{\mathrm{had}}$ background in each of the analysis regions. No significant pulls or constraints are obtained for the fitted nuisance parameters, resulting in a post-fit background prediction in each analysis region that is very close to the pre-fit prediction, albeit with reduced uncertainties due to the anti-correlations among sources of systematic uncertainty resulting from the fit. Figure 9 shows a comparison of the data and prediction for the BDT discriminant distribution in the ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, 3j) and ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, $\geq$4j) regions, both pre- and post-fit to data, in the case of the $t\bar{t}\to WbHc$ search. A similar comparison for the ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, 3j) and ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, $\geq$4j) regions is shown in Figure 10. Tables summarising the pre-fit and post-fit yields can be found in Appendix B.

The best-fit branching ratio obtained is $\mathscr{B}(t\to Hc)=[-4.4^{+7.7}_{-7.0}\,(\mathrm{stat})^{+6.2}_{-4.9}\,(\mathrm{syst})]\times 10^{-4}$, assuming $\mathscr{B}(t\to Hu)=0$. The best-fit normalisation factors for the fake $\tau_{\mathrm{had}}$ background are: $0.82\pm 0.23$ in the ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, 3j) region, $0.84^{+0.25}_{-0.28}$ in the ($\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$, $\geq$4j) region, $0.94^{+0.18}_{-0.17}$ in the ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, 3j) region, and $0.90\pm 0.26$ in the ($\tau_{\mathrm{had}}\tau_{\mathrm{had}}$, $\geq$4j) region. A similar fit is performed for the $t\bar{t}\to WbHu$ search, yielding $\mathscr{B}(t\to Hu)=[-5.3^{+7.3}_{-6.5}\,(\mathrm{stat})^{+5.3}_{-4.2}\,(\mathrm{syst})]\times 10^{-4}$, assuming $\mathscr{B}(t\to Hc)=0$. The obtained normalisation factors for the fake $\tau_{\mathrm{had}}$ background agree within 1% with those obtained by the $t\bar{t}\to WbHc$ search. In both cases, the uncertainty of the measured branching ratio is dominated by the statistical uncertainty. The main contributions to the total systematic uncertainty arise from the fake $\tau_{\mathrm{had}}$ background estimation and the uncertainty associated with the different responses to quark-initiated and gluon-initiated jets. No significant excess of data events above the background expectation is found, and observed (expected) 95% CL limits are set on $\mathscr{B}(t\to Hc)$ and $\mathscr{B}(t\to Hu)$: $\mathscr{B}(t\to Hc)<1.9\times 10^{-3}\,(2.1\times 10^{-3})$ and $\mathscr{B}(t\to Hu)<1.7\times 10^{-3}\,(2.0\times 10^{-3})$. These results are dominated by the $\tau_{\mathrm{had}}\tau_{\mathrm{had}}$ channel, which has a sensitivity a factor of two better than that of the $\tau_{\mathrm{lep}}\tau_{\mathrm{had}}$ channel.

10.3 Combination of ATLAS searches

The $tqH(b\bar{b})$ and $tqH(\tau\tau)$ searches are combined with the ATLAS searches in diphoton [29] and multilepton [30] final states of events in the same data set, referred to as “$tqH(\gamma\gamma)$ search” and “$tqH(\mathrm{ML})$ search”, respectively. Since all searches, with the exception of the $tqH(b\bar{b})$ search, are dominated by the data statistical uncertainty, and in each search the dominant systematic uncertainties are different, the combined result is insensitive to the assumed correlations of systematic uncertainties across searches. Therefore, the only systematic uncertainties taken to be fully correlated among the four searches are those affecting the integrated luminosity, the $t\bar{t}$ cross section, signal modelling, a subset of the uncertainties on the Higgs boson branching ratios (those associated with uncertainties in $\alpha_{\mathrm{S}}$ and $m_{b}$), and a subset of jet-related uncertainties (jet energy resolution and JVT requirement). The rest of the jet-related uncertainties (jet energy scale and $b$-tagging) are taken as fully correlated among the $tqH(b\bar{b})$, $tqH(\tau\tau)$, and $tqH(\mathrm{ML})$ searches, but uncorrelated with the $tqH(\gamma\gamma)$ search. The rest of the uncertainties, e.g. those related to leptons and to background modelling, are taken as uncorrelated among the four searches.

The first set of combined results is obtained for each branching ratio separately, setting the other branching ratio to zero. The best-fit combined branching ratios are $\mathscr{B}(t\to Hc)=[3.0^{+3.0}_{-2.7}\,(\mathrm{stat})^{+2.6}_{-2.1}\,(\mathrm{syst})]\times 10^{-4}$ and $\mathscr{B}(t\to Hu)=[4.2^{+3.2}_{-2.9}\,(\mathrm{stat})^{+2.6}_{-2.1}\,(\mathrm{syst})]\times 10^{-4}$. A comparison of the best-fit branching ratios for the individual searches and their combination is shown in Figure 11 for $\mathscr{B}(t\to Hc)$ and Figure 12 for $\mathscr{B}(t\to Hu)$. The observed (expected) 95% CL combined upper limits on the branching ratios are $\mathscr{B}(t\to Hc)<1.1\times 10^{-3}\,(8.3\times 10^{-4})$ and $\mathscr{B}(t\to Hu)<1.2\times 10^{-3}\,(8.3\times 10^{-4})$. A summary of the upper limits on the branching ratios obtained by the individual searches, as well as their combination, is given in Table 3 and in Figures 13 and 14.

Upper limits on the branching ratios $\mathscr{B}(t\to Hq)$ ($q=u,c$) can be translated into upper limits on the non-flavour-diagonal Yukawa couplings $\lambda_{tqH}$ appearing in the Lagrangian [129]:

[TABLE]

The branching ratio $\mathscr{B}(t\to Hq)$ is estimated as the ratio of its partial width [9] to the SM $t\to Wb$ partial width [130], which is assumed to be dominant. Both predicted partial widths include next-to-leading-order QCD corrections. Using the expression derived in Ref. [26], the coupling $|\lambda_{tqH}|$ can be extracted as $|\lambda_{tqH}|=(1.92\pm 0.02)\sqrt{\mathscr{B}(t\to Hq)}$. The $\lambda_{tqH}$ coupling corresponds to the sum in quadrature of the couplings relative to the two possible chirality combinations of the quark fields, $\lambda_{tqH}\equiv\sqrt{|\lambda_{t_{\mathrm{L}}q_{\mathrm{R}}}|^{2}+|\lambda_{q_{\mathrm{L}}t_{\mathrm{R}}}|^{2}}$ [129]. The observed (expected) upper limits on the couplings from the combination of the searches are $|\lambda_{tcH}|<0.064\,(0.055)$ and $|\lambda_{tuH}|<0.066\,(0.055)$.

A similar set of results can be obtained by simultaneously varying both branching ratios in the likelihood function. Figure 15(a) shows the 95% CL upper limits on the branching ratios in the $\mathscr{B}(t\to Hu)$ versus $\mathscr{B}(t\to Hc)$ plane. The small differences between the limiting values (on the $x$- and $y$-axes) of the branching ratio limits obtained in the two-dimensional scan and those reported in Table 3, result from slightly different choices in the $tqH(\mathrm{ML})$ search regarding the final discriminant, which in the two-dimensional case should be common to both signals, and its binning. The corresponding upper limits on the couplings in the $|\lambda_{tuH}|$ versus $|\lambda_{tcH}|$ plane are shown in Figure 15(b).

11 Conclusion

A search for flavour-changing neutral-current decays of a top quark into an up-type quark ($q=u,c$) and the Standard Model Higgs boson, $t\to Hq$, is presented. The search is based on a dataset of $pp$ collisions at $\sqrt{s}=13\leavevmode\nobreak\ \text{Te\kern-1.00006ptV}$ recorded in 2015 and 2016 with the ATLAS detector at the CERN Large Hadron Collider and corresponding to an integrated luminosity of 36.1 fb*-1*. Two complementary analyses are performed to search for top-quark pair events in which one top quark decays into $Wb$ and the other top quark decays into $Hq$, and target the $H\to b\bar{b}$ and $H\to\tau^{+}\tau^{-}$ decay modes, respectively. The $tqH(b\bar{b})$ search selects events with one isolated electron or muon from the $W\to\ell\nu$ decay, and multiple jets, with several of them being identified with high purity as originating from the hadronisation of $b$-quarks. The $tqH(\tau\tau)$ search selects events with either one or two hadronically decaying $\tau$-lepton candidates, as well as multiple jets. Both searches employ multivariate techniques to discriminate between the signal and the background on the basis of their different kinematics. No significant excess of events above the background expectation is found, and 95% CL upper limits on the $t\to Hq$ branching ratios are derived. In the case of the $tqH(b\bar{b})$ search, the observed (expected) 95% CL upper limits on the $t\to Hc$ and $t\to Hu$ branching ratios are $4.2\times 10^{-3}\,(4.0\times 10^{-3})$ and $5.2\times 10^{-3}\,(4.9\times 10^{-3})$, respectively. In the case of the $tqH(\tau\tau)$ search, the observed (expected) 95% CL upper limits on the $t\to Hc$ and $t\to Hu$ branching ratios are $1.9\times 10^{-3}\,(2.1\times 10^{-3})$ and $1.7\times 10^{-3}\,(2.0\times 10^{-3})$, respectively. The combination of these searches with ATLAS searches in diphoton and multilepton final states yields observed (expected) 95% CL upper limits on the $t\to Hc$ and $t\to Hu$ branching ratios of $1.1\times 10^{-3}$ ($8.3\times 10^{-4}$) and $1.2\times 10^{-3}$ ($8.3\times 10^{-4}$), assuming $\mathscr{B}(t\to Hu)=0$ and $\mathscr{B}(t\to Hc)=0$ respectively. The corresponding combined observed (expected) upper limits on the $|\lambda_{tcH}|$ and $|\lambda_{tuH}|$ couplings are 0.064 (0.055) and 0.066 (0.055), respectively.

Acknowledgements

We thank CERN for the very successful operation of the LHC, as well as the support staff from our institutions without whom ATLAS could not be operated efficiently.

We acknowledge the support of ANPCyT, Argentina; YerPhI, Armenia; ARC, Australia; BMWFW and FWF, Austria; ANAS, Azerbaijan; SSTC, Belarus; CNPq and FAPESP, Brazil; NSERC, NRC and CFI, Canada; CERN; CONICYT, Chile; CAS, MOST and NSFC, China; COLCIENCIAS, Colombia; MSMT CR, MPO CR and VSC CR, Czech Republic; DNRF and DNSRC, Denmark; IN2P3-CNRS, CEA-DRF/IRFU, France; SRNSFG, Georgia; BMBF, HGF, and MPG, Germany; GSRT, Greece; RGC, Hong Kong SAR, China; ISF and Benoziyo Center, Israel; INFN, Italy; MEXT and JSPS, Japan; CNRST, Morocco; NWO, Netherlands; RCN, Norway; MNiSW and NCN, Poland; FCT, Portugal; MNE/IFA, Romania; MES of Russia and NRC KI, Russian Federation; JINR; MESTD, Serbia; MSSR, Slovakia; ARRS and MIZŠ, Slovenia; DST/NRF, South Africa; MINECO, Spain; SRC and Wallenberg Foundation, Sweden; SERI, SNSF and Cantons of Bern and Geneva, Switzerland; MOST, Taiwan; TAEK, Turkey; STFC, United Kingdom; DOE and NSF, United States of America. In addition, individual groups and members have received support from BCKDF, CANARIE, CRC and Compute Canada, Canada; COST, ERC, ERDF, Horizon 2020, and Marie Skłodowska-Curie Actions, European Union; Investissements d’ Avenir Labex and Idex, ANR, France; DFG and AvH Foundation, Germany; Herakleitos, Thales and Aristeia programmes co-financed by EU-ESF and the Greek NSRF, Greece; BSF-NSF and GIF, Israel; CERCA Programme Generalitat de Catalunya, Spain; The Royal Society and Leverhulme Trust, United Kingdom.

The crucial computing support from all WLCG partners is acknowledged gratefully, in particular from CERN, the ATLAS Tier-1 facilities at TRIUMF (Canada), NDGF (Denmark, Norway, Sweden), CC-IN2P3 (France), KIT/GridKA (Germany), INFN-CNAF (Italy), NL-T1 (Netherlands), PIC (Spain), ASGC (Taiwan), RAL (UK) and BNL (USA), the Tier-2 facilities worldwide and large non-WLCG resource providers. Major contributors of computing resources are listed in Ref. [131].

Appendix

Appendix A Pre-fit and post-fit event yields in the $tqH(b\bar{b})$ search

Table 4 presents the observed and predicted yields in each of the analysis regions for the $tqH(b\bar{b})$ search before the fit to data. Tables 5 and 6 present the observed and predicted yields in each of the analysis regions after the fit to the data under the signal-plus-background hypothesis, assuming $t\bar{t}\to WbHc$ and $t\bar{t}\to WbHu$ as signal, respectively.

Appendix B Pre-fit and post-fit event yields in the $tqH(\tau\tau)$ search

Table 7 presents the observed and predicted yields in each of the analysis regions for the $tqH(\tau\tau)$ search before the fit to data. Tables 8 and 9 present the observed and predicted yields in each of the analysis regions after the fit to the data under the signal-plus-background hypothesis, assuming $t\bar{t}\to WbHc$ and $t\bar{t}\to WbHu$ as signal, respectively.