Search for heavy particles decaying into a top-quark pair in the fully hadronic final state in $pp$ collisions at $\sqrt{s} =13$ TeV with the ATLAS detector

TL;DR

This paper reports a search for new heavy particles decaying into top-quark pairs in fully hadronic final states using 13 TeV proton-proton collision data from ATLAS, setting limits on various hypothetical particles' masses.

Contribution

It introduces two optimized analysis techniques for reconstructing top-quark pairs across different kinematic regimes, enhancing the sensitivity of the search for new particles.

Findings

No significant deviation from the Standard Model observed.

Excludes Z' bosons up to 3.6 TeV and Kaluza-Klein gluons up to 3.4 TeV.

Limits on dark-matter mediators in specific mass ranges.

Abstract

A search for new particles decaying into a pair of top quarks is performed using proton-proton collision data recorded with the ATLAS detector at the Large Hadron Collider at a center-of-mass energy of $\sqrt{s} = $13 TeV corresponding to an integrated luminosity of 36.1 fb$^{-1}$. Events consistent with top-quark pair production and the fully hadronic decay mode of the top quarks are selected by requiring multiple high transverse momentum jets including those containing $b$-hadrons. Two analysis techniques, exploiting dedicated top-quark pair reconstruction in different kinematic regimes, are used to optimize the search sensitivity to new hypothetical particles over a wide mass range. The invariant mass distribution of the two reconstructed top-quark candidates is examined for resonant production of new particles with various spins and decay widths. No significant deviation from the Standard Model prediction is observed and limits are set on the production cross-section times branching fraction for new hypothetical $Z'$ bosons, dark-matter mediators, Kaluza-Klein gravitons and Kaluza-Klein gluons. By comparing with the predicted production cross-sections, the $Z'$ boson in the topcolor-assisted-technicolor model is excluded for masses up to 3.1$-$3.6 TeV, the dark-matter mediators in a simplified framework are excluded in the mass ranges from 0.8 TeV to 0.9 TeV and from 2.0 TeV to 2.2 TeV, and the Kaluza-Klein gluon is excluded for masses up to 3.4 TeV, depending on the decay widths of the particles.

Tables

Table 1. Table 1: Performance of the resolved t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t}  reconstruction with the “buckets of tops” algorithm estimated using simulated SM t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} and Z TC2 ′ subscript superscript 𝑍 ′ TC2 Z^{\prime}_{\text{TC2}} (850  GeV ) events in the fully hadronic final state. The fraction of events in each of the five possible top bucket categories is shown for all events satisfying the selection criteria described in Section  5.1 . For each event category the relative fraction of events that have correctly matched top-quark pairs is presented. The measure of accuracy is based on a geometrical matching in the η 𝜂 \eta – ϕ italic-ϕ \phi plane. Specifically the matched top buckets are required to be within Δ ​ R = 0.3 Δ 𝑅 0.3 \Delta R=0.3 of a simulated top quark. The momenta of the simulated top quarks are evaluated immediately before the decay. The errors indicate the statistical uncertainty only.

Top buckets category	Fraction of events [$\%$]			Matched top-quark pairs [$\%$]
	SM $t\overline{t}$	$Z_{\text{TC2}}^{\prime }$ (850 GeV)		SM $t\overline{t}$	$Z_{\text{TC2}}^{\prime }$ (850 GeV)
$(t_0,t_0)$	$16.5\pm 0.3$	$12.6\pm 0.7$		$57.1\pm 1.0$	$63.6\pm 2.7$
$(t_{-},t_{-})$	$17.5\pm 0.3$	$15.0\pm 0.9$		$66.7\pm 0.9$	$74.2\pm 2.6$
$(t_{-},t_W)$	$7.8\pm 0.2$	$7.9\pm 0.8$		$72.2\pm 1.3$	$80.0\pm 3.9$
$(t_W,t_{-})$	$30.2\pm 0.4$	$30.9\pm 1.2$		$78.9\pm 0.6$	$82.6\pm 1.5$
$(t_W,t_W)$	$28.0\pm 0.4$	$33.6\pm 1.3$		$88.7\pm 0.5$	$90.7\pm 1.1$

Table 2. Table 2: Performance of the boosted t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} reconstruction in the boosted analysis estimated using simulated SM t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} and Z TC2 ′ subscript superscript 𝑍 ′ TC2 Z^{\prime}_{\text{TC2}} (3  TeV ) events in the fully hadronic final state. The fraction of events in each of the eight possible boosted signal regions is shown for all events satisfying the selection criteria described in Section  5.1 , together with the relative fraction of events that have correctly matched top-quark pairs. The measure of accuracy is based on a geometrical matching in the η 𝜂 \eta – ϕ italic-ϕ \phi plane. Specifically the matched large- R 𝑅 R jets are required to be within Δ ​ R = 0.4 Δ 𝑅 0.4 \Delta R=0.4 of a simulated top quark. The notation used to define each signal region is described in Section  5.3 . The momenta of the simulated top quarks are evaluated immediately before the decay. The errors indicate the statistical uncertainties only.

Signal region category	Fraction of events [$\%$]			Matched top-quark pairs [$\%$]
	SM $t\overline{t}$	$Z_{\text{TC2}}^{\prime }$ (3 TeV)		SM $t\overline{t}$	$Z_{\text{TC2}}^{\prime }$ (3 TeV)
Medium R1 1b	$1.80\pm 0.07$	$2.41\pm 0.08$		$89.8\pm 4.4$	$86.7\pm 4.1$
Medium R1 2b	$5.24\pm 0.11$	$4.39\pm 0.10$		$94.0\pm 2.7$	$84.3\pm 2.8$
Tight R1 1b	$2.55\pm 0.08$	$2.07\pm 0.10$		$93.8\pm 4.0$	$83.5\pm 4.2$
Tight R1 2b	$7.75\pm 0.14$	$4.18\pm 0.10$		$97.2\pm 2.3$	$83.5\pm 2.8$
Medium R2 1b	$1.20\pm 0.06$	$1.99\pm 0.07$		$83.8\pm 5.3$	$86.4\pm 4.4$
Medium R2 2b	$3.13\pm 0.09$	$3.08\pm 0.08$		$91.4\pm 3.3$	$86.3\pm 3.3$
Tight R2 1b	$0.89\pm 0.05$	$1.54\pm 0.06$		$90.0\pm 6.6$	$89.8\pm 5.2$
Tight R2 2b	$2.25\pm 0.07$	$2.59\pm 0.07$		$93.9\pm 4.1$	$86.5\pm 3.6$

Table 3. Table 3: Event categorization in the resolved analysis. The multijet-enriched regions A–C and the main signal region D, as well as the additional validation regions A 0 subscript A 0 \text{A}_{0} , A − subscript A \text{A}_{-} , C 0 subscript C 0 \text{C}_{0} , C − subscript C \text{C}_{-} selected with looser requirements on the top-quark pair candidates are shown. The events are also classified according to the two b 𝑏 b -tagging criteria, i.e, satisfying the tight or satisfying the loose but failing to satisfy the tight efficiency working points for both b 𝑏 b -tagged jets. The expected fraction of t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} events to the total background events in each region, as estimated from the simulation, is given in parentheses. The error indicates the statistical uncertainty only.

Top buckets category	$(t_0,t_0)$	$(t_{-},t_{-})$	$(t_W,t_{-})$	$(t_W,t_W)$
Loose $b$-tag	${\text{A}}_0$ $\left(2.1\pm 0.0\right)\%$	${\text{A}}_{-}$   $\left(4.2\pm 0.1\right)\%$	A $\left(12.3\pm 0.2\right)\%$	B $\left(38.9\pm 0.9\right)\%$
Tight $b$-tag	${\text{C}}_0$ $\left(8.0\pm 0.1\right)\%$	${\text{C}}_{-}$ $\left(16.9\pm 0.2\right)\%$	C $\left(44.9\pm 0.5\right)\%$	D $\left(79.6\pm 1.3\right)\%$

Table 4. Table 4: List of the event categories considered in the boosted analysis. The index i 𝑖 i is the region number defined in Figure  LABEL:fig:categorization_boosted_b . The indices j 𝑗 j and k 𝑘 k correspond to the L τ 32 subscript 𝐿 subscript 𝜏 32 L_{\tau_{32}} and n b subscript 𝑛 𝑏 n_{b} categories, respectively, defined in Figure  LABEL:fig:categorization_boosted_a . The TR i ​ ( Inclusive , 0 ​ b ) 𝑖 Inclusive 0 𝑏 i(\text{Inclusive},0b) is used to estimate the multijet background shape in the SR i ​ ( j , k ) 𝑖 𝑗 𝑘 i(j,k) and the CR i ​ ( Inclusive , k ) 𝑖 Inclusive 𝑘 i(\text{Inclusive},k) are used to estimate the shape correction.

	Category	Mass region	$i$	$j$	$k$
Signal Region (SR)	$\text{SR}i(j,k)$	$\text{R}i$	1, 2	Medium, Tight	$1b$, $2b$
Validation Region (VR)	$\text{VR}i(j,k)$	$\text{R}i$	1,2	Loose	$1b$, $2b$
Control Region (CR)	$\text{CR}i(j,k)$	$\text{CR}i$	1, …, 4	Loose, Medium,	$0b$, $1b$, $2b$
				Tight, Inclusive
Template Region (TR)	$\text{TR}i(j,k)$	$\text{R}i$	1, 2	Loose, Medium,	$0b$
				Tight, Inclusive

Table 5. Table 5: Expected and observed yields in the main signal region D and multijet-enriched regions A–C for the resolved analysis. The yields and their uncertainties are evaluated after the fit to data under the background-only hypothesis. The expected Z TC2 ′ subscript superscript 𝑍 ′ TC2 Z^{\prime}_{\text{TC2}} signal yields with masses of 0.75 and 1  TeV  are calculated using the μ = 1 𝜇 1 \mu=1 hypothesis. The multijet contribution also contains all other small non- t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} backgrounds.

Type	Region A		Region B		Region C		Region D
$t\overline{t}$	$4299$$\pm$	$277$	$2740$$\pm$	$188$	$9816$$\pm$	$458$	$8986$$\pm$	$248$
Multijet (template)	$\mathrm{31\hspace{0.17em}418}$$\pm$	$771$	$4440$$\pm$	$364$	$\mathrm{12\hspace{0.17em}842}$$\pm$	$531$	$1816$$\pm$	$251$
Total background	$\mathrm{35\hspace{0.17em}717}$$\pm$	$772$	$7180$$\pm$	$367$	$\mathrm{22\hspace{0.17em}658}$$\pm$	$347$	$\mathrm{10\hspace{0.17em}801}$$\pm$	$186$
Data	$\mathrm{35\hspace{0.17em}722}$		$7186$		$\mathrm{22\hspace{0.17em}665}$		$\mathrm{10\hspace{0.17em}821}$
$Z_{\text{TC2}}^{\prime }\left(0.75\text{TeV}\right)$	$470$$\pm$	$68$	$367$$\pm$	$91$	$1196$$\pm$	$141$	$1203$$\pm$	$182$
$Z_{\text{TC2}}^{\prime }\left(1\text{TeV}\right)$	$460$$\pm$	$65$	$296$$\pm$	$37$	$1018$$\pm$	$127$	$1014$$\pm$	$148$

Table 6. Table 6: Expected and observed yields in the signal regions for the boosted analysis. The yields and their uncertainties are evaluated after the background-only fit to the data. The expected Z TC2 ′ subscript superscript 𝑍 ′ TC2 Z^{\prime}_{\text{TC2}} signal yields with masses of 1.5 and 3  TeV  are calculated using the μ = 1 𝜇 1 \mu=1 hypothesis. The multijet contribution also contains all other small non- t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} backgrounds.

Type	$\text{SR}1(\text{Medium},1b)$		$\text{SR}1(\text{Medium},2b)$		$\text{SR}1(\text{Tight},1b)$		$\text{SR}1(\text{Tight},2b)$
$t\overline{t}$	$320$$\pm$	$50$	$930$$\pm$	$50$	$440$$\pm$	$70$	$1350$$\pm$	$70$
Multijet (template)	$1360$$\pm$	$60$	$810$$\pm$	$50$	$510$$\pm$	$70$	$330$$\pm$	$50$
Total background	$1680$$\pm$	$40$	$1740$$\pm$	$40$	$950$$\pm$	$30$	$1680$$\pm$	$50$
Data	$1689$		$1730$		$952$		$1676$
$Z_{\text{TC2}}^{\prime }\left(1.5\text{TeV}\right)$	$100$$\pm$	$20$	$280$$\pm$	$20$	$150$$\pm$	$20$	$460$$\pm$	$30$
$Z_{\text{TC2}}^{\prime }\left(3\text{TeV}\right)$	$4$$\pm$	$1$	$8$$\pm$	$1$	$4$$\pm$	$1$	$8$$\pm$	$1$
Type	$\text{SR}2(\text{Medium},1b)$		$\text{SR}2(\text{Medium},2b)$		$\text{SR}2(\text{Tight},1b)$		$\text{SR}2(\text{Tight},2b)$
$t\overline{t}$	$250$$\pm$	$40$	$690$$\pm$	$60$	$190$$\pm$	$30$	$510$$\pm$	$40$
Multijet (template)	$2760$$\pm$	$60$	$1640$$\pm$	$70$	$820$$\pm$	$50$	$510$$\pm$	$50$
Total background	$3010$$\pm$	$50$	$2330$$\pm$	$50$	$1010$$\pm$	$30$	$1020$$\pm$	$30$
Data	$3006$		$2322$		$989$		$1021$
$Z_{\text{TC2}}^{\prime }\left(1.5\text{TeV}\right)$	$80$$\pm$	$10$	$210$$\pm$	$20$	$70$$\pm$	$10$	$190$$\pm$	$20$
$Z_{\text{TC2}}^{\prime }\left(3\text{TeV}\right)$	$4$$\pm$	$1$	$6$$\pm$	$1$	$3$$\pm$	$1$	$5$$\pm$	$1$

Table 7. Table 7: The relative impact of the post-fit uncertainties on the signal strength parameter μ 𝜇 \mu using the Z TC2 ′ subscript superscript 𝑍 ′ TC2 Z^{\prime}_{\text{TC2}} benchmark model with m = 0.75 𝑚 0.75 m=0.75 (3)  TeV  in the resolved (boosted) analysis. The eight systematic uncertainties with the highest impact on the signal strength parameter in the resolved and boosted analyses, respectively, are shown. The uncertainty on the extrapolation using an exponential function at high m t ​ t ¯ reco superscript subscript 𝑚 𝑡 ¯ 𝑡 reco m_{t\bar{t}}^{\text{reco}} above 1.5  TeV  applies to the boosted analysis only. To estimate the impact from a given source of systematic uncertainty, the fit is performed with the nuisance parameter for the test fixed to the ± 1 ​ σ plus-or-minus 1 𝜎 \pm 1\sigma value after the nominal fit and the other nuisance parameters floated. The differences between the best-fit μ 𝜇 \mu values in the tests and the nominal fit are divided by total post-fit uncertainty in μ 𝜇 \mu are shown in this table. The total systematic uncertainty is different from the sum in quadrature of the different components due to correlations between nuisance parameters built by the fit. The statistical uncertainty in the data is evaluated by fixing all the nuisance parameters in the fit to the best-fit values except for the free-floating normalization factors.

	Resolved ($Z_{\text{TC2}}^{\prime }$ $m=0.75\text{TeV}$)	Boosted ($Z_{\text{TC2}}^{\prime }$ $m=3\text{TeV}$)
Source of uncertainty	Relative impact on $\mu$	Relative impact on $\mu$
Luminosity		$+0.03$/$-0.03$
$b$-tagging efficiency	$+0.05$/$-0.04$	$+0.07$/$-0.07$
Small- and large-$R$ JES and JER	$+0.20$/$-0.24$	$+0.21$/$-0.09$
$t\overline{t}$ modeling	$+0.34$/$-0.33$	$+0.10$/$-0.09$
Multijet estimation	$+0.25$/$-0.27$	$+0.16$/$-0.13$
Extrapolation	–	$+0.34$/$-0.33$
PDF	$+0.07$/$-0.08$	$+0.10$/$-0.10$
Pileup reweighting	$+0.07$/$-0.05$
Simulation statistical uncertainty	$\pm 0.41$	–
Total systematic uncertainty	$\pm 0.92$	$\pm 0.67$
Data statistical uncertainty	$\pm 0.39$	$\pm 0.74$

Table 8. Table 8: Summary of expected and observed excluded mass ranges at 95% CL for the benchmark models studied.

Signal		Expected excluded mass [TeV]	Observed excluded mass [TeV]
$Z_{\text{TC2}}^{\prime }$	($\Gamma =1\%$)	$[0.57,2.8]$	$[0.58,3.1]$
	($\Gamma =3\%$)	$[0.51,3.6]$	$[0.53,3.6]$
$Z_{\text{med}}^{\prime }$	(vector)	$[0.75,1.07]\cup [2.0,2.1]$	$[0.74,0.97]\cup [2.0,2.2]$
	(axial-vector)	$[1.99,2.04]$	$[0.80,0.92]\cup [2.0,2.2]$
$g_{\text{KK}}$	($\Gamma =10\%$)
	($\Gamma =20\%$)
	($\Gamma =30\%$)
	($\Gamma =40\%$)

Equations

$${\Delta }^2=\omega {\Delta }_{B_1}^2+{\Delta }_{B_2}^2,{\Delta }_{B_{1(2)}}=|m_{B_{1(2)}}-m_{\text{top}}|,\omega =100,$$

$$155\text{GeV}<m_{B_{1,2}}<200\text{GeV}$$

$$\left|\frac{m_{kl}}{m_{B_i}}-\frac{m_W}{m_{\text{top}}}\right|<0.15,$$

$${\Delta }_B^{bj}=|m_B-145\text{GeV}|$$

$$75\text{GeV}<m_{B_i}^{bj}<155\text{GeV}.$$

$${\tau }_n=\frac{1}{d_0}\sum _i{p}_{\text{T}}^i\times \min (\Delta R_{1,i},\Delta R_{2,i},\cdots ,\Delta R_{n,i}),$$

$$N_{\text{SR}i(j,k)}^{\text{MJ}}=F_{\text{N}}(j,k)\times N_{\text{TR}i(j,0b)}^{\text{MJ}},$$

$$F_{\text{N}}(j,k)=\frac{{\sum }_i{N}_{\text{CR}i(j,k)}^{\text{MJ}}}{{\sum }_i{N}_{\text{CR}i(j,0b)}^{\text{MJ}}},$$

Full text

\AtlasTitle

Search for heavy particles decaying into a top-quark pair in the fully hadronic final state in $pp$ collisions at $\sqrt{s}$ =13 TeV with the ATLAS detector \AtlasAbstractA search for new particles decaying into a pair of top quarks is performed using proton–proton collision data recorded with the ATLAS detector at the Large Hadron Collider at a center-of-mass energy of $\sqrt{s}=13$ TeV corresponding to an integrated luminosity of 36.1 fb*-1*. Events consistent with top-quark pair production and the fully hadronic decay mode of the top quarks are selected by requiring multiple high transverse momentum jets including those containing $b$-hadrons. Two analysis techniques, exploiting dedicated top-quark pair reconstruction in different kinematic regimes, are used to optimize the search sensitivity to new hypothetical particles over a wide mass range. The invariant mass distribution of the two reconstructed top-quark candidates is examined for resonant production of new particles with various spins and decay widths. No significant deviation from the Standard Model prediction is observed and limits are set on the production cross-section times branching fraction for new hypothetical $Z^{\prime}$ bosons, dark-matter mediators, Kaluza–Klein gravitons and Kaluza–Klein gluons. By comparing with the predicted production cross sections, the $Z^{\prime}$ boson in the topcolor-assisted-technicolor model is excluded for masses up to 3.1–3.6 TeV, the dark-matter mediators in a simplified framework are excluded in the mass ranges from 0.8 to 0.9 TeV and from 2.0 to 2.2 TeV, and the Kaluza–Klein gluon is excluded for masses up to 3.4 TeV, depending on the decay widths of the particles.

\AtlasRefCodeEXOT-2016-24 \PreprintIdNumberCERN-EP-2018-350 \AtlasJournalPhys. Rev. D \AtlasJournalRefPhys. Rev. D 99 (2019) 092004 \AtlasDOI10.1103/PhysRevD.99.092004

\size@chapter\sectfont

Contents

@afterheading@starttoc

toc

1 Introduction

The Large Hadron Collider (LHC), currently operating at a center-of-mass energy of $\sqrt{s}$ = 13 TeV, has the potential to discover phenomena beyond the Standard Model (SM) at the TeV scale. The heaviest elementary particle known in the SM, the top quark, is produced abundantly at the LHC. It is often predicted to be a probe for new physics phenomena at the TeV scale, in models such as the two-Higgs-doublet model (2HDM) [1], topcolor-assisted-technicolor [2, 3, 4] and Randall–Sundrum (RS) models of warped extra dimensions [5, 6]. Resonant production of a pair of top and antitop quarks ($t\bar{t}$) is particularly interesting as it provides a clear signature indicating the existence of new heavy particles decaying into $t\bar{t}$. Such new particles could manifest themselves as a localized deviation from the SM prediction in the high invariant mass distribution of the $t\bar{t}$ system ($m_{t\bar{t}}$). In this paper, a search for new particles in events containing $t\bar{t}$ pairs, where both the top and antitop quarks decay hadronically ($t\bar{t}\to W^{+}bW^{-}\bar{b}$ with $W\to q\bar{q^{\prime}}$), is presented. The analysis is based on $36.1\leavevmode\nobreak\ \text{fb}^{-1}$ of proton–proton collision data at a center-of-mass energy of $\sqrt{s}$ =13 TeV recorded with the ATLAS detector at the LHC in 2015 and 2016.

The fully hadronic final state is characterized by the presence of multiple hadronic jets, two of which contain $b$-hadrons, and the absence of reconstructed leptons. This all-jets topology benefits from the largest top-quark decay branching fraction (45.7% of $t\bar{t}$ decays), but suffers from large backgrounds due to QCD multijet production. Dedicated top-quark reconstruction and identification techniques are used to enhance selection of $t\bar{t}$ over multijet events to maximize the sensitivity to the benchmark signals considered. Two different search strategies are employed, each targeting a different mass range of the hypothetical resonance. In the mass range below approximately 1.2 TeV, where the decay products of the top quarks can be resolved as separate small-radius jets, the “buckets of tops” algorithm [7] is used to optimize the reconstruction of top-quark-pair candidates. At higher masses, top-quark decay products often merge into a single large-radius jet due to the high transverse momentum ($p_{\text{T}}$) of the top quarks, hence a second strategy with a jet-substructure-based top-quark identification technique [8, 9] is exploited. In the intermediate mass range of about 1.1 to 1.6 TeV, signals are searched for using both strategies separately. The two results are compared at each mass point and the one with the better expected sensitivity is selected.

The ATLAS and CMS collaborations performed searches for heavy particles decaying into $t\bar{t}$ using $pp$ collision data recorded at $\sqrt{s}$ = 7 TeV [10, 11, 12, 13, 14], 8 TeV [15, 16, 17, 18] and 13 TeV [19, 20, 21] and set lower limits on the masses for several benchmark signal models. The ATLAS search at 13 TeV [19], using data equivalent to 36.1 fb*-1*, exploits the lepton-plus-jets topology, where a high-$p_{\text{T}}$ electron or muon and large missing transverse momentum are required, and excludes masses below 3.0 (3.8) TeV for the new $Z^{\prime}_{\text{TC2}}$ boson with an intrinsic decay width111In the rest of this paper, the decay width of a resonance divided by the resonance mass is referred to as the width. of $\Gamma=1\%$ (3%) in the topcolor-assisted-technicolor model [2, 3] (described in Section 2). The CMS search with the lepton-plus-jets, all-jets, and dilepton topologies at 13 TeV [21] excludes the $Z^{\prime}_{\text{TC2}}$ boson with $\Gamma=1\%$ up to 3.8 TeV using 35.9 fb*-1*. The Kaluza–Klein (KK) excitation of the graviton $G_{\text{KK}}$ predicted in the specific “bulk” RS model [22, 23] decaying into $t\bar{t}$ (see details in Section 2) was also searched for by the ATLAS Collaboration and the mass range from 0.45 to 0.65 TeV is excluded assuming $k/\overline{M}_{\mathrm{Pl}}=1$, where $k$ is the curvature of the warped extra dimension and $\overline{M}_{\mathrm{Pl}}=M_{\mathrm{Pl}}/\sqrt{8\pi}$ is the reduced Planck mass. The KK excitation of the gluon, $g_{\text{KK}}$, predicted in an RS model with a single warped extra dimension [6] with $\Gamma=15\%$ (30%) is excluded by the ATLAS search up to 3.8 (3.7) TeV. The CMS search [21] considered a slightly different model [24], including a KK gluon with $\Gamma=20\%$ and larger production cross section, and set a lower limit of 4.55 TeV on the mass.

The paper is organized as follows. The signal models considered are discussed in Section 2. After a brief description of the ATLAS detector in Section 3, the data and simulation samples are summarized in Section 4. The analysis strategy including event selection, reconstruction and categorization is presented in Section 5. The background estimation is described in Section 6 and the systematic uncertainties in the background and signal predictions in Section 7. After describing the signal search and the statistical procedure in Section 8, the results are presented in Section 9 with the conclusions given in Section 10.

2 Signal models

Several benchmark signal models are considered in this analysis, in which new spin-1 or spin-2 color-singlet and color-octet bosons with masses ranging from 0.5 to 5 TeV are introduced. The width of these bosons can vary from $\Gamma=1\%$ to 30% to cover resonances narrower or wider than the typical detector resolution of about 10%.

As the first benchmark, a topcolor-assisted-technicolor (TC2) model [2, 3] is considered, which predicts a spin-1 color-singlet boson. This leptophobic $Z^{\prime}$ boson (denoted by $Z^{\prime}_{\text{TC2}}$), referred to as Model IV in Ref. [4], couples only to first- and third-generation quarks and is mainly produced by $q\bar{q}$ annihilation. The model parameters are chosen to maximize the branching fraction for the $Z^{\prime}_{\text{TC2}}\rightarrow t\bar{t}$ decay, which reaches 33%, and the width is set to $\Gamma=1\%$ or 3%.

A framework of simplified models for dark matter (DM) interactions is considered as the second benchmark. An axial-vector mediator $Z^{\prime}_{\text{med,ax}}$ and a vector mediator $Z^{\prime}_{\text{med,vec}}$ are used, following the recommendation of the LHC Dark Matter Working Group in Ref. [25]. In the simplified model there are five parameters relevant for $pp\to Z^{\prime}_{\text{med}}\to t\bar{t}$ processes ($Z^{\prime}_{\text{med}}$ is either $Z^{\prime}_{\text{med,ax}}$ or $Z^{\prime}_{\text{med,vec}}$): the mediator mass $m_{\text{med}}$, the dark-matter mass $m_{\text{DM}}$, and the mediator couplings to quarks $g_{q}$, to leptons $g_{\ell}$, and to dark matter $g_{\text{DM}}$. This search considers the coupling parameters defined in the A1 (V1) scenario of Ref. [25] for the axial-vector (vector) mediator. The branching fraction of the mediators into $t\bar{t}$ is 8.8% and the width is approximately constant at $\Gamma=5.6\%$ over the search range considered. The DM mass $m_{\text{DM}}$ is fixed to 10 GeV.

An RS model with the SM fields propagating in the bulk of a single warped extra dimension [6] is used as the third benchmark, which predicts a spin-1 color-octet boson, the first KK excitation of the gluon, $g_{\text{KK}}$. The $g_{\text{KK}}$ is primarily produced in $q\bar{q}$ annihilation and decays predominantly into $t\bar{t}$ with a branching fraction of approximately 92.5% as predicted in Ref. [6]. In this analysis, the coupling of the KK gluon to quarks is set to $g_{q}=-0.2g_{\mathrm{s}}$, where $g_{\mathrm{s}}$ is the strong coupling constant in the SM. The left-handed coupling to the top quark is fixed to $g_{\mathrm{s}}$ while the right-handed coupling is varied to change the intrinsic width.

The “bulk” RS model [22, 23] with the SM fields propagating in the bulk, inherited from the original RS model, is used as the fourth benchmark to predict a spin-2 color-singlet boson. The first KK excitation of the graviton, $G_{\text{KK}}$, in this model is mainly produced in gluon–gluon fusion, and the production rate and width are controlled by a dimensionless coupling constant $k/\overline{M}_{\mathrm{Pl}}$. In this analysis $k/\overline{M}_{\mathrm{Pl}}$ is chosen to be 1, resulting in the $G_{\text{KK}}$ width varying from $\Gamma=3\%$ to 6% in the mass range between 0.5 and 3 TeV. The branching fraction of the $G_{\text{KK}}$ into $t\bar{t}$ increases from 18% to 50% between 400 and 600 GeV and stays approximately constant at 68% for masses larger than 1 TeV. In addition, the $G_{\text{KK}}$ can decay into a pair of $W$, $Z$ or Higgs bosons and, with negligible branching fraction, into light fermions or photons.

Representative leading-order (LO) Feynman diagrams of the benchmark signals are presented in Figure 1.

3 ATLAS detector

The ATLAS detector at the LHC is a multipurpose, forward–backward symmetric detector 222ATLAS uses a right-handed coordinate system with its origin at the nominal interaction point (IP) in the center of the detector and the $z$-axis along the beam pipe. The $x$-axis points from the IP to the center of the LHC ring, and the $y$-axis points upwards. Cylindrical coordinates $(r,\phi)$ are used in the transverse plane, $\phi$ being the azimuthal angle around the $z$-axis. The pseudorapidity is defined in terms of the polar angle $\theta$ as $\eta=-\ln\tan(\theta/2)$. Angular distance is measured in units of $\Delta R\equiv\sqrt{(\Delta\eta)^{2}+(\Delta\phi)^{2}}$. with nearly full solid angle coverage, as described in Refs. [26, 27, 28]. It consists of an inner tracking detector (ID) surrounded by a thin superconducting solenoid, a calorimeter system composed of electromagnetic (EM) and hadronic calorimeters, and a muon spectrometer.

The ID consists of a silicon pixel detector, a silicon microstrip tracker and a transition radiation tracker, all immersed in a $2\text{\,}\mathrm{T}$ axial magnetic field, and provides charged-particle tracking in the range $|\eta|<2.5$. The EM calorimeter is a lead/liquid-argon (LAr) sampling calorimeter with accordion geometry. It is divided into a barrel section covering $|\eta|<1.475$ and two endcap sections covering $1.375<|\eta|<3.2$. For $|\eta|<2.5$ it is divided into three layers in depth, which are finely segmented in $\eta$ and $\phi$. In the region $|\eta|<1.8$, an additional thin LAr presampler layer is used to correct for energy losses in the material upstream of the calorimeters. The hadronic calorimeter is a sampling calorimeter composed of steel/scintillator tiles in the central region ($|\eta|<1.7$), while copper/LAr modules are used in the endcap ($1.5<|\eta|<3.2$) regions. The forward region ($3.1<|\eta|<4.9$) is instrumented with copper/LAr and tungsten/LAr calorimeter modules optimized for electromagnetic and hadronic measurements, respectively. Surrounding the calorimeters is a muon spectrometer that includes three air-core superconducting toroidal magnets and multiple types of tracking chambers, providing precision tracking for muons with $|\eta|<2.7$ and trigger capability in the range $|\eta|<2.4$.

A two-level trigger system is used to select events for offline analysis [29]. Events are first selected by the level-1 trigger implemented in custom electronics, which uses a subset of the detector information to reduce the event rate to $100\text{\,}\mathrm{kHz}$. This is followed by a software-based trigger that reduces the accepted event rate to $1\text{\,}\mathrm{kHz}$ on average by refining the first-level trigger selection.

4 Data and simulation

This analysis is based on $36.1\leavevmode\nobreak\ \text{fb}^{-1}$ of $pp$ collisions recorded by the ATLAS experiment at the LHC at a center-of-mass energy of 13 TeV in 2015 and 2016. A number of quality criteria were imposed to ensure that the data were collected during stable beam conditions with the relevant detectors operational. Simulated signal and background events are used to optimize the event selection, to estimate the background contribution and to perform the hypothesis test of the benchmark signal models considered.

The main backgrounds after applying criteria to enhance potential signals originate from SM $t\bar{t}$ and multijet production. The $t\bar{t}$ contribution and the related modeling uncertainties are evaluated using Monte Carlo (MC) simulated events, while the multijet contribution is estimated directly from data. However, simulated events of multijet processes are used to optimize selection criteria and derive residual corrections to the multijet distributions.

For the generation of SM $t\bar{t}$ events, the next-to-leading-order (NLO) generator Powheg-Box v2 [30, 31, 32] was used with the CT10 [33, 34] parton distribution function (PDF) set in the matrix element calculations. The $t\bar{t}$ production cross section in $pp$ collisions at $\sqrt{s}=13$ TeV is $\sigma_{t\bar{t}}=832^{+46}_{-52}$ pb for a top-quark mass of $172.5$ GeV. It was calculated at next-to-next-to leading order (NNLO) in QCD including resummation of next-to-next-to-leading logarithmic soft gluon terms with Top++2.0 [35, 36, 37, 38, 39, 40, 41]. Parton showering, hadronization and the underlying event were simulated using Pythia v6.428 [42] with the CTEQ6L1 [43] PDF set and the corresponding Perugia 2012 set of tuned parameters [44]. The $h_{\mathrm{damp}}$ parameter, which controls the transverse momentum of the first additional parton emission beyond the Born configuration, was set equal to the top-quark mass. The top-quark kinematics in $t\bar{t}$ events were corrected to account for electroweak higher-order effects [45]. The generated events were weighted by this correction factor as a function of the flavor and center-of-mass energy of the initial partons, and of the decay angle of the top quarks in the center-of-mass frame of the initial partons. The value of the correction factor decreases with increasing $m_{t\bar{t}}$ from 0.98 at $m_{t\bar{t}}=0.4$ TeV to 0.87 at $m_{t\bar{t}}=3.5$ TeV. Multijet processes were simulated with the Pythia v8.186 [46] generator using the LO NNPDF2.3 [47] PDF set.

Simulated signal samples of spin-1 color-singlet $Z^{\prime}_{\text{TC2}}$ bosons decaying into $t\bar{t}$ were generated with Pythia v8.165 [46] with the LO NNPDF2.3 PDF set and the A14 set [48] of tuned parameters. To account for higher-order contributions, the LO calculation of the cross section was multiplied by a factor 1.3 obtained at NLO in QCD [49] using the PDF4LHC2015 PDF set [50]. For the spin-1 mediators $Z^{\prime}_{\text{med}}$ in the DM simplified model, the same samples are used after being reweighted to have the approximate mediator width and cross section as simulated by MadGraph5_aMC@NLO [51]. The production cross sections were calculated at LO accuracy using the LO NNPDF2.3 PDF set. The production of a spin-2 bulk RS graviton $G_{\text{KK}}$ was performed using MadGraph5_aMC@NLO with the LO NNPDF2.3 PDF set, interfaced to Pythia v8.165 with the A14 set of tuned parameters for parton shower and hadronization. Simulated samples of spin-1 color-octet KK gluons $g_{\text{KK}}$ with $\Gamma=30\%$ were generated with Pythia v8.165 with the same PDF and tuned parameters as those used for the $Z^{\prime}_{\text{TC2}}$ samples. Samples of $g_{\text{KK}}$ with different widths (from 10% to 40%) were derived by reweighting the shapes of corresponding samples with $\Gamma=30\%$ and adjusting their normalization according to the appropriate prediction. The $Z^{\prime}_{\text{TC2}}$ and $g_{\text{KK}}$ samples were generated for the mass range between 0.5 and 5 TeV. Signal masses were sampled at intervals of 100–150 GeV below 1 TeV, 250 GeV between 1 and 3 TeV and 500 GeV above 3 TeV for the $Z^{\prime}_{\text{TC2}}$. The $g_{\text{KK}}$ samples were produced at fixed intervals of 500 GeV in all mass ranges. The $G_{\text{KK}}$ samples were generated between 0.5 and 3 TeV in steps of 250 GeV (1 TeV) below (above) 1 TeV. The simulated samples are also used to evaluate the acceptance and selection efficiencies for the signals considered in the search.

The EvtGen v1.2.0 program [52] was used in all simulated samples to model the properties of heavy-flavor hadron decays. All simulated samples include the effects of multiple $pp$ interactions in the same and neighboring bunch crossings (pileup) and are processed through the ATLAS detector simulation  [53] based on Geant4 [54]. Pileup effects were emulated by overlaying simulated minimum-bias events generated with Pythia v8.186, using the MSTW2008LO PDF set [55] and the A2 set of tuned parameters [56]. The number of overlaid minimum-bias events was adjusted to match the luminosity profile of the recorded data. Simulated events were processed through the same reconstruction software as the data, and corrections are applied so that the object identification efficiencies, energy scales and energy resolutions match those determined from control samples of data.

5 Event reconstruction, selection and categorization

The production of a pair of hadronically decaying top quarks is characterized by the presence of multiple hadronic jets. When the top quarks have moderate transverse momentum, $p_{\text{T}}$, of less than approximately 500 GeV, the decay products can be reconstructed as separate jets, which is referred to as the “resolved” event topology. At higher transverse momentum, the decay products of each of the two top or antitop quarks are merged into a single large-radius jet, referred to as the “boosted” event topology. For both topologies the identification and reconstruction of the jets originating from the top quarks is crucial for reconstructing the top-quark pair, resulting in a better separation of signal from background. The resolved and boosted event analyses are employed in parallel in the analysis.

5.1 Object reconstruction and event preselection

Events are required to have at least one $pp$ interaction vertex associated with two or more tracks with $p_{\text{T}}>400$ MeV. If more than one vertex is found in an event, the one with the largest $\sum p_{\text{T}}^{2}$ of associated tracks is chosen as the primary interaction vertex. Depending on the kinematic regime of the top quarks, resolved or boosted, different jet reconstruction techniques are applied. Events containing leptons (electrons or muons) are included in the complementary search targeting the lepton-plus-jets topology [19] but are rejected in the analyses presented here.

Small-$R$ jets are built from three-dimensional topological clusters of energy deposits in the calorimeter [57], calibrated at the electromagnetic energy scale, using the anti-$k_{t}$ algorithm [58] with a radius parameter $R=0.4$. These jets are calibrated to the hadronic energy scale by applying $p_{\text{T}}$- and $\eta$-dependent corrections derived from MC simulations and in situ measurements obtained from $Z/\gamma$+jets and multijet events at $\sqrt{s}=13$ TeV [59]. Jets from pileup interactions are suppressed by applying the jet vertex tagger [60], which uses information from tracks associated with the hard-scatter and pileup vertices, to jets with $p_{\text{T}}<60$ GeV and $|\eta|<2.4$. Events containing jets from calorimeter noise or non-collision backgrounds are removed by discarding events containing at least one jet failing to satisfy the loose quality criteria defined in Ref. [61]. Jets that satisfy all the selection requirements and have $p_{\text{T}}>25$ GeV and $|\eta|<2.5$ are considered in the resolved analysis. Small-$R$ jets containing $b$-hadrons are identified using an algorithm [62] based on multivariate techniques to combine information from the impact parameters of displaced tracks as well as topological properties of secondary and tertiary decay vertices reconstructed within the jet. Two working points with 70% (tight) and 85% (loose) efficiencies for $b$-quark-induced jets are chosen, where the efficiencies are averaged values derived from simulated SM $t\bar{t}$ events. The corresponding misidentification rates of the tight (loose) working point are 0.26% (3%) and 8% (32%) for jets containing hadrons composed of light-flavor quarks and $c$-quarks, respectively. Efficiencies to tag jets from $b$- and $c$-quarks in the simulation are corrected to match the efficiencies in data using $p_{\text{T}}$-dependent factors, whereas the light-jet efficiency is scaled by $p_{\text{T}}$- and $\eta$-dependent factors [62].

Large-$R$ jets are built from three-dimensional topological clusters of energy deposits in the calorimeter calibrated with the local cluster weighting (LCW) procedure [57] using the anti-$k_{t}$ algorithm with a radius parameter $R=1.0$. The non-compensating response of the calorimeter and the energy loss in dead material and due to out-of-cluster leakage from charged and neutral particles are corrected in the LCW procedure before jet reconstruction. The reconstructed jets are “trimmed” [63] to mitigate contributions from pileup and soft radiation. In the trimming procedure, the jet constituents are reclustered into subjets using the $k_{t}$ algorithm [64, 65, 66] with a radius parameter $R=0.2$ and subjets with $p_{\text{T}}$ less than 5% of the $p_{\text{T}}$ of the parent jet are removed [67]. Finally, the large-$R$ jets are formed from the momentum vectors of the remaining subjets and selected by requiring $p_{\text{T}}>200$ GeV and $|\eta|<2.0$ in the boosted analysis. For highly boosted top quarks, the mass resolution of a large-$R$ jet containing the top-quark decay products deteriorates with increasing top $p_{\text{T}}$ due to the limited angular granularity of the calorimeter. To overcome this the mass of the large-$R$ jet, $m_{\text{J}}$, is calculated by combining the calorimeter energy measurement with the track information from the ID, as described in Ref. [68]. The two jets with the highest $p_{\text{T}}$ in the event are required to have 50 GeV$<m_{\text{J}}<350$ GeV.

Track-jets are built from charged-particle tracks using the anti-$k_{t}$ algorithm with a radius parameter $R=0.2$. Tracks used in the reconstruction are selected by requiring that they are associated with the primary vertex, and have $p_{\text{T}}>400$ MeV and $|\eta|<2.5$. Track-jets composed of at least two constituent tracks and having $p_{\text{T}}>10$ GeV and $|\eta|<2.5$ are used to identify jets containing $b$-hadrons in the boosted analysis. In the dense environment characteristic of the boosted topology, the $b$-tagging is more efficient if performed on track-jets than on calorimeter jets [69]. The same $b$-tagging algorithm as used for small-$R$ jets with 77% (tight) and 85% (loose) efficiency working points from $b$-quark-induced jets is employed. The training of the multivariate algorithm and the evaluation of systematic uncertainties associated with the track-jet $b$-tagging efficiency are performed separately from those for the small-$R$ calorimeter jets. The corresponding misidentification rates at the tight (loose) working point are 1.7% (5.3%) and 23.8% (40.5%) for light-flavor quarks and $c$-quarks, respectively.

Electrons are reconstructed from clusters of EM calorimeter energy deposits matched to an ID track with $|\eta|<2.47$, excluding the barrel and endcap transition region of $1.37<|\eta|<1.52$. The electron candidates are required to have $E_{\text{T}}>25$ GeV and to satisfy the “tight” identification criteria defined in Ref. [70]. To suppress contamination from misidentified hadrons, the electron candidates are further required to be isolated from other hadronic activity in the event. This is achieved by requiring the scalar sum of track $p_{\text{T}}$ within a cone around the electron direction, excluding the track associated with the electron, to be less than 6% of the electron transverse momentum $p_{\text{T}}^{e}$. The cone size is given by the minimum of $\Delta R=10\,\text{GeV}/p_{\text{T}}^{e}$ and $\Delta R=0.2$.

Muons are reconstructed by combining tracks separately reconstructed in the ID and the muon spectrometer. The muon candidates are required to have $p_{\text{T}}>25$ GeV and $|\eta|<2.5$, and satisfy the “medium” quality requirements defined in Ref. [71]. The muons are also required to be isolated by using the same track-based isolation conditions as for electrons, except that the value of $\Delta R=0.2$ is replaced with $\Delta R=0.3$.

Electron and muon candidate tracks are required to be associated with the primary vertex using criteria based on the longitudinal and transverse impact parameters. To avoid the misidentification of jets as electrons and electrons from heavy-flavor decays, the closest small-$R$ jet within $\Delta R_{y}=\sqrt{(\Delta y)^{2}+(\Delta\phi)^{2}}=0.2$ around a reconstructed electron is removed.333The rapidity is defined as $y=\frac{1}{2}\ln\frac{E+p_{z}}{E-p_{z}}$ where $E$ is the energy and $p_{z}$ is the longitudinal component of the momentum along the beam direction. If an electron is then found within $\Delta R_{y}=0.4$ of a jet, the electron is removed. If a muon is found within \Delta R_{y}=0.04+10\,\text{\text{Ge\kern-1.00006ptV}}/p_{\text{T}}^{\mu} of a jet (where $p_{\text{T}}^{\mu}$ is the muon transverse momentum), the muon is removed if the jet contains at least three tracks, otherwise the jet is removed.

In the resolved analysis, the event selection is based on multijet triggers requiring the presence of at least five small-$R$ jets with $p_{\text{T}}>60$–65 GeV depending on the data-taking periods. Events are further required to have at least six jets with $p_{\text{T}}>25$ GeV and $|\eta|<2.5$, out of which the five highest-$p_{\text{T}}$ jets must have $p_{\text{T}}>75$ GeV and $|\eta|<2.4$. Among those six jets at least two of them are required to be $b$-tagged with $|\eta|<1.6$ using the loose efficiency working point. The trigger efficiency for the events satisfying the offline selection criteria is estimated using a lower-threshold multijet trigger. The trigger efficiency is above 99% and consistent between data and the simulated events.

In the boosted analysis, events are selected using triggers that require at least one large-$R$ jet with $p_{\text{T}}>360$–420 GeV depending on the data-taking periods. Events are required to have at least two large-$R$ jets with $p_{\text{T}}>400\,\text{Ge\kern-1.00006ptV}$ to ensure that the jets can fully contain the top-quark decay products. The large-$R$ jets with the highest and the second-highest $p_{\text{T}}$ in the event are referred to as the leading and sub-leading jets, respectively. The leading jet has to satisfy $p_{\text{T}}>500$ GeV to ensure a nearly full trigger efficiency. The trigger efficiency is measured using a control sample in data and found to be approximately 100% in this $p_{\text{T}}$ range. The invariant mass $m_{\text{JJ}}$ of the two leading large-$R$ jets is required to be $m_{\text{JJ}}>1$ TeV to avoid a kinematic bias caused by the jet $p_{\text{T}}$ requirements. The two leading jets are required to have an azimuthal angle difference larger than 1.6. In addition, each jet is required to have at least one track-jet within $\Delta R=1.0$ satisfying the loose $b$-tagging efficiency working point. The fraction of events with more than two $b$-tagged jets is negligibly small, and those events are rejected to simplify the data-driven multijet background estimation.

5.2 Top-quark pair reconstruction

In the resolved analysis, the top-quark pair reconstruction is achieved by exploiting the “buckets of tops” algorithm [7] using small-$R$ jets. In this algorithm, all jets in the event are assumed to originate from $t\bar{t}$ events, including those from initial- or final-state radiation, and are assigned to one of three groups, referred to as “buckets”. The first two buckets correspond to reconstructed candidates of the two top quarks in $t\bar{t}$ events and the third bucket contains all jets from extra radiation. The assignment of small-$R$ jets to buckets is performed by taking all jet combinations and minimizing a metric based on the difference between the invariant mass of jets falling into one of the first two buckets and the top-quark mass. In this analysis the metric $\Delta^{2}$ is defined as

[TABLE]

where $m_{B_{1(2)}}$ is the invariant mass of the jets falling into bucket 1(2), denoted by $B_{1(2)}$, and $m_{\text{top}}=173.5$ GeV is the top-quark mass. The difference from $m_{\text{top}}$ used in the simulation (172.5 GeV) does not affect the performance of the $t\bar{t}$ reconstruction. The $\omega$ factor is introduced to ensure that $B_{1}$ has a mass closer to $m_{\text{top}}$ than $B_{2}$, i.e. $\Delta_{B_{1}}<\Delta_{B_{2}}$, as described in Ref. [7]. No restriction is imposed on the multiplicity of jets falling into the buckets except that $B_{1}$ and $B_{2}$ are required to contain exactly one $b$-tagged jet each. Furthermore, the mass window requirements of

[TABLE]

are applied to increase the fraction of $t\bar{t}$ events. The preferred two “top buckets” $B_{1,2}$ are further classified according to the hadronic $W$-boson decay. If the following condition is satisfied for at least one combination of two non-$b$-tagged jets ($k$,$l$), the bucket is considered to contain a $W$-boson candidate and labeled $t_{W}$, otherwise it is labeled $t_{-}$:

[TABLE]

where $m_{kl}$ is the invariant mass of the ($k$, $l$) jet combination inside $B_{i}$, and $m_{W}=80.4$ GeV is the $W$-boson mass. To retain $t\bar{t}$ events where one of the jets originating from the top-quark decay, presumably the softer quark from $W\to q\bar{q^{\prime}}$, falls outside the top buckets, two-jet top buckets are formed. The metric used to form the bucket is adjusted to be

[TABLE]

if the bucket mass $m_{B}$ is smaller than 155 GeV, otherwise $\Delta_{B}^{bj}$ is set to an arbitrary large number. The mass criteria are based on the top-decay kinematics in which only the $b$-quark and the harder quark from $W\to q\bar{q^{\prime}}$ fall inside the bucket. When the two top buckets are classified as ($t_{W}$, $t_{W}$) the event is kept. If the buckets are classified as ($t_{W}$, $t_{-}$) or ($t_{-}$, $t_{W}$) with the notation that the first bucket in the parentheses is always chosen to be $B_{1}$, the $t_{-}$ bucket is recalculated using the new metric $\Delta_{B}^{bj}$ from all jets excluding those belonging to any $t_{W}$ bucket in the event. Hereafter these two categories are collectively referred to as ($t_{W}$, $t_{-}$). If the two top buckets are ($t_{-}$, $t_{-}$), the new buckets are formed from all jets in the event by minimizing the sum of a new metric $\Delta_{B_{1}}^{bj}+\Delta_{B_{2}}^{bj}$. The new two-jet bucket is finally required to satisfy the mass window requirement of

[TABLE]

If an event has no buckets satisfying the mass window requirements, the event is classified as ($t_{0}$, $t_{0}$). Finally, the top-quark candidate, reconstructed as the sum of the momentum vectors of the jets in the $t_{W}$, $t_{-}$ or $t_{0}$ bucket, is required to have $p_{\text{T}}>200$ GeV to suppress multijet backgrounds. The performance of the resolved $t\bar{t}$ reconstruction is summarized in Table 1. The resolution of the reconstructed $t\bar{t}$ mass for the resolved analysis is typically 6%.

For the boosted analysis, a top-quark pair is reconstructed using the top-quark tagging requirements based on the jet mass and a jet substructure variable called $n$-subjettiness, $\tau_{n}$ [8, 9]. For each large-$R$ jet, $\tau_{n}$ is calculated by reconstructing exactly $n$ subjets with the “winner-take-all” recombination scheme [72] from the large-$R$ jet constituents using the $k_{t}$ algorithm [64, 65, 66] with a radius parameter of $R=0.2$:

[TABLE]

where $p_{\text{T}}^{i}$ is the transverse momentum of the $i$-th large-$R$ jet constituent and $\Delta R_{j,i}$ is the $y$–$\phi$ distance between the subjet $j$ and the $i$-th constituent. The $\tau_{n}$ variable is scaled by $d_{0}^{-1}=(\sum_{i}p_{\text{T}}^{i}\times R)^{-1}$ with $R=1.0$, the radius parameter of the large-$R$ jet. To distinguish fully contained top quarks with a three-prong structure from other backgrounds dominated by a single-prong or two-prong structure, the $\tau_{32}$ variable defined as $\tau_{32}=\tau_{3}/\tau_{2}$ is used as a discriminant. Since there are two top quarks in signal events, the $\tau_{32}$ variables from the two leading large-$R$ jets are used to construct a single likelihood ratio $L_{\tau_{32}}$, which is then used to suppress the multijet background. The likelihood ratio is computed as $L_{\tau_{32}}=P_{\text{s}}/(P_{\text{s}}+P_{\text{b}})$ where $P_{\text{s}}$ and $P_{\text{b}}$ are the probability density functions for the signal and background, respectively, obtained from MC simulations (see Section 4). The performance of the $t\bar{t}$ reconstruction in the boosted analysis is summarized in Table 2, where signal regions as defined in Section 5.3 are used for illustration. The resolution of the reconstructed $t\bar{t}$ mass for the boosted analysis is typically 10%.

5.3 Event categorization

For both the resolved and boosted analyses, the reconstructed events are categorized into several subsamples used for the signal search and background estimation.

In the resolved analysis, events satisfying the preselection criteria in Section 5.1 are classified according to the reconstructed top buckets and number of $b$-tagged jets in the events. The combination of four possible pairs of top buckets, ($t_{W}$, $t_{W}$), ($t_{W}$, $t_{-}$), ($t_{-}$, $t_{-}$) and ($t_{0}$, $t_{0}$), and the two $b$-tagging criteria, i.e., (1) satisfying the tight or (2) satisfying the loose but failing to satisfy the tight efficiency working points for both $b$-tagged jets, are used to classify events into eight different regions A–D, $\text{A}_{0}$, $\text{A}_{-}$, $\text{C}_{0}$ and $\text{C}_{-}$ defined in Table 3. By construction those regions have no overlapping events. Region D, which contains events with ($t_{W}$, $t_{W}$) buckets and tight $b$-tagged jets, is the most sensitive to the benchmark signals and hence chosen to be the main signal region (SR) for the resolved analysis. Regions A–C are used in a joint likelihood fit with the SR to extract the multijet background in the SR as detailed in Section 6. The regions with the ($t_{-}$, $t_{-}$) and ($t_{0}$, $t_{0}$) buckets ($\text{A}_{0}$, $\text{A}_{-}$, $\text{C}_{0}$ and $\text{C}_{-}$) are used to estimate systematic uncertainties associated with the multijet background modeling (see Section 7).

In the boosted analysis, preselected events are first categorized by the number of tight $b$-tagged track-jets ($n_{b}$) and the $\tau_{32}$-likelihood ratio ($L_{\tau_{32}}$) as shown in Figure LABEL:fig:categorization_boosted_a. Most signal events have $n_{b}=1$ or 2, which define the $1b$ and $2b$ regions. The events with $n_{b}=0$ ($0b$ region) are used to model the multijet background.

For the $L_{\tau_{32}}$ variable, the three criteria $0.35\leq L_{\tau_{32}}<0.6$, $0.6\leq L_{\tau_{32}}<0.8$ and $0.8\leq L_{\tau_{32}}\leq 1.0$ define Loose, Medium and Tight regions, respectively, while $0.35\leq L_{\tau_{32}}<1.0$ is referred to as Inclusive. The lower boundaries of the Tight and Medium regions are determined by optimizing the signal sensitivity while the lower boundary of the Loose region is used to ensure that events have kinematic properties similar to those in the Tight and Medium regions. The Loose region is used for validation of the background estimation across the $L_{\tau_{32}}$ regions (see Section 6 for details). The possible contamination from $Z^{\prime}_{\text{TC2}}$ signal events in the Loose region is a few percent as estimated for a signal with a cross section that has already been excluded by previous analyses. It is hence negligible for the signals with higher masses, which have lower predicted cross sections, and also for other benchmark signals with kinematic properties similar to the $Z^{\prime}_{\text{TC2}}$. In each category, events are further classified into different regions using the masses $m_{\text{J}_{1}}$ and $m_{\text{J}_{2}}$ of large-$R$ jets with the leading and sub-leading $p_{\text{T}}$ as shown in Figure LABEL:fig:categorization_boosted_b. Representative distributions of the jet masses are shown in Figure 3 for events satisfying the $(\text{Tight},1b)$ or $(\text{Tight},2b)$ requirements. The jet mass distributions are shown for the data and background predictions obtained after the fit to data (“Post-Fit”), as detailed in Section 8.

Signal regions are defined in the ranges $140<m_{\text{J}_{1,2}}<190$ GeV (denoted by R1) or $140<m_{\text{J}_{1}}<190$ GeV and $50<m_{\text{J}_{2}}<140$ GeV (denoted by R2). About 38% (34%) of the $Z^{\prime}_{\text{TC2}}$ signal events with $m_{Z^{\prime}_{\text{TC2}}}=1.5$ TeV(3 TeV) fall into the region R1. In some cases, not all partons from the top-quark decay ($q\bar{q}^{\prime}b$) are fully contained within the large-$R$ jet, in particular at low $p_{\text{T}}$. In the higher-$p_{\text{T}}$ region above 1.2 TeV, the large-$R$ jets contain all the decay products of the top quark more than 90% of the time, but the mass resolution deteriorates and the number of jets lost due to final state radiation increases as a function of $p_{\text{T}}$. Consequently, a significant fraction of signal events (28% and 27% at $m_{Z^{\prime}_{\text{TC2}}}=1.5$ TeV and 3 TeV, respectively) have a lower mass for the sub-leading large-$R$ jets, falling into the region R2 of $50<m_{\text{J}_{2}}<140$ GeV. Therefore, eight SRs are considered in the boosted analysis, namely the R1 and R2 mass regions for each combination of the Tight or Medium $L_{\tau_{32}}$ requirement, and one or two tight $b$-tagged jets, as illustrated in Figure 2 and Table 4. The same categories but with the Loose $L_{\tau_{32}}$ requirement are collectively called the validation region (VR). The regions labelled as control regions CR1–4 in Figure LABEL:fig:categorization_boosted_b are used to determine the normalization of multijet backgrounds separately for the SR and VR. The mass regions R1 and R2 in the $0b$ region are used to extract the shape of the multijet backgrounds in the SR and VR and are collectively called the template region (TR). The details of the multijet background estimation are discussed in Section 6.

The normalized reconstructed $m_{t\bar{t}}$ distributions, $m_{t\bar{t}}^{\text{reco}}$, in the resolved main SR (region D) and one of the most sensitive boosted SRs ($\text{R}1(\textrm{Tight},2b)$) are shown in Figure 4 for different masses of the hypothesized particle in each of the benchmark signal scenarios considered. The acceptance times efficiency as a function of the top-quark pair invariant mass, $m_{t\bar{t}}$, at the generator level for SR selections are shown in Figure 5. Due to the spin nature of the resonance, the two top quarks from the spin-2 graviton $G_{\text{KK}}$ (spin-1 $Z^{\prime}_{\text{TC2}}$) are likely to be produced in the barrel (endcap) region. Hence the acceptance for the $G_{\text{KK}}$ signal is higher than that of the $Z^{\prime}_{\text{TC2}}$ or $g_{\text{KK}}$ signals.

6 Background estimation

The main SM backgrounds in both the resolved and boosted analyses are from SM production of $t\bar{t}$ pairs and multijet processes. The $t\bar{t}$ events are predicted from simulation as described in Section 4. The multijet backgrounds are estimated using multijet-enriched regions A–C. The data-driven estimation methods are validated in dedicated validation regions. Contributions from the production of single top quarks, $W$/$Z$ bosons in association with jets, and dibosons ($WW$, $WZ$ and $ZZ$) are negligibly small and are accounted for in the multijet background estimate.

The resolved analysis exploits a double-sideband likelihood method to estimate the multijet background contribution in each of the regions A–D, defined in Table 3. The $m_{t\bar{t}}^{\text{reco}}$ templates extracted from the regions A and B, by subtracting the simulated SM $t\bar{t}$ contribution, are used to model the multijet background shape in the region C and the main signal region D, respectively. It is confirmed that the simulated SM $t\bar{t}$ sample can model the data well by comparing the kinematic distributions observed in the $t\bar{t}$-enriched data and the $t\bar{t}$ simulation sample. The multijet yields in the main signal region D are first estimated by multiplying the yield in B by the ratio of the yields in C and A, assuming no contamination from signal in the regions A–C and no correlation between top- and $b$-tagging requirements. This first estimation is used to get the input values of the unconstrained normalization parameters in the following likelihood fit. The presence of a possible contamination from signal in the multijet-enriched regions A–C, the correlation between the top- and $b$-tagging variables and the subtraction of the SM $t\bar{t}$ background in the multijet background estimate are then taken into account by performing a likelihood fit to the data $m_{t\bar{t}}^{\text{reco}}$ distributions in all the regions A–D. This simultaneous likelihood fit allows the multijet background from the three multijet-enriched regions A–C to be estimated and the probability of compatibility of expected backgrounds with observed data in the main signal region D to be quantified at the same time, as described in Section 8. Systematic uncertainties associated with the data-driven method discussed in Section 7.2 are considered in the fit as nuisance parameters.

For the boosted analysis, the multijet yield in a SR is estimated by multiplying the multijet yield in the corresponding TR by the normalization factor ($F_{\text{N}}$) obtained by comparing the data yields in the CR between $1b$ or $2b$ and $0b$ regions. For a SR$i(j,k)$ with the jet mass requirement $i$, $L_{\tau_{32}}$ requirement $j$ and $n_{b}$ requirement $k$ (defined in Table 4), the multijet yield $N^{\text{MJ}}_{\text{SR}i(j,k)}$ is obtained by

[TABLE]

where the $N^{\text{MJ}}_{\text{TR}i(j,0b)}$ is the event yield in the TR$i(j,0b)$. The normalization factor for the SR with the selection $(j,k)$, $F_{\text{N}}(j,k)$, is defined as

[TABLE]

where the $N^{\text{MJ}}_{\text{CR}i(j,k)}$ is the multijet yield in the CR$i(j,k)$. The $N^{\text{MJ}}_{\text{CR}i(j,k)}$ is obtained from data by subtracting the simulated SM $t\bar{t}$ background. The normalization factors obtained separately from the four CRs (CR$i$; $i=1,...,4$) with the selection $(j,k)$ are found to be comparable within the statistical uncertainty; therefore they are averaged into a single $F_{\text{N}}(j,k)$ value for improved statistical accuracy. The obtained $F_{\text{N}}(j,k)$ is about 2.4 (1.4) with a relative uncertainty of about 2% for $k=1b$ ($2b$), and is the same for both $j=$ Medium and Tight within the statistical uncertainty. Contributions from the SM $t\bar{t}$ background in the TR are about 3% and 1% for mass regions R1 and R2, respectively. The contamination from the SM $t\bar{t}$ in the CR is less than 1% for the $0b$ region and a few percent for the $1b$ and $2b$ regions, and at most 9% in the CR$(\text{Tight},2b)$ category.

For the multijet background shape, the inclusive $L_{\tau_{32}}$ range $[0.35,1.0]$ is used in the TR ($\text{TR}i(\text{Inclusive},k)$) to improve the statistical accuracy after checking the compatibility of the $m_{t\bar{t}}^{\text{reco}}$ shapes in the three $L_{\tau_{32}}$ regions. However, the templates are extracted separately for R1 and R2 as they have non-negligible differences. The estimated multijet shapes are further corrected to account for the $p_{\text{T}}$-dependence of the $b$-tagging efficiency as observed in the simulation. This is performed by using the scalar sum of the $p_{\text{T}}$ of the two leading large-$R$ jets, $p_{\text{T}}^{\text{sum}}$, and comparing the $p_{\text{T}}^{\text{sum}}$ distributions of the CR events in the $1b$ and $2b$ regions with the ones in the $0b$ region in the simulated multijet events. The inclusive $L_{\tau_{32}}$ range and the sum of the four CRs (CR1–4) are used for this study. The shape correction is then extracted separately for the $1b$ and $2b$ regions by performing a fit to the ratio of the distributions. Finally, in order to reduce the statistical fluctuation of the predicted multijet contribution at high mass, the estimated $m_{t\bar{t}}^{\text{reco}}$ distribution in the SR is fit in the range from 1.2 TeV to 4 TeV using an exponential function and the prediction replaced with the fit result above 1.5 TeV. The same procedure is applied to the simulated SM $t\bar{t}$ events to improve the statistical accuracy. The method used to estimate the multijet background is validated in the VR$i(\text{Loose},k)$, where good agreement is seen between the observed data and the prediction from the TR$i(\text{Inclusive},0b)$ for $i=1$ and 2 and $k=1b$ and $2b$.

7 Systematic uncertainties

There are two categories of systematic uncertainties considered in the analysis: experimental uncertainties associated with the detector response and reconstruction algorithms, and uncertainties in the background modeling.

Each source of systematic uncertainty is considered to be uncorrelated with other sources, while it is treated as being fully correlated across event categories and between processes, whenever appropriate. In addition, statistical uncertainties in the signal and background predictions due to the limited amount of simulated data are taken into account.

7.1 Experimental uncertainties in simulated samples

The SM $t\bar{t}$ and signal predictions are subject to experimental systematic uncertainties because they are estimated using simulated events. Dominant sources of the experimental systematic uncertainty are associated with the small-$R$ and large-$R$ jet energy scales (JES), jet energy resolutions (JER) and $b$-tagging.

The small-$R$ JES uncertainty is derived using a combination of simulation, test-beam data, and in situ measurements [59]. Additional contributions from jet flavor composition, punch-through, single-particle response, calorimeter response to different jet flavors and pileup are taken into account, resulting in a total of 21 systematic uncertainty components. The total JES uncertainty is typically 4% at $p_{\text{T}}=25$ GeV and varies from 1% to 3% at $p_{\text{T}}>75$ GeV. The small-$R$ JER uncertainty (typically 2%–3% at $p_{\text{T}}=50$ GeV) obtained from an in situ measurement of jet response using dijet events [59] is also included. The uncertainty in the efficiency of the jet vertex tagger (Section 5.1) is also considered following Ref. [60]. The impact on the total background yield (for a 850 GeV $Z^{\prime}_{\text{TC2}}$ signal) in the resolved analysis is about 9% (11%) for the JES uncertainty and 3% (11%) for the JER uncertainty.

The large-$R$ JES uncertainties are estimated with the $R_{\mathrm{trk}}$ method using dijet data control samples [73, 68]. The method assumes that the track-related uncertainties are uncorrelated with the calorimeter cluster-related uncertainties. The procedure works by measuring the ratio $r_{\mathrm{trk}}$ of an observable (which can be the $p_{\text{T}}$, $m_{\text{J}}$ or $\tau_{32}$ variables) using calorimeter jets to that using track-jets reconstructed within the same detector region. The deviation of the average data-to-simulation ratio $\left<R_{\mathrm{trk}}\right>=\left<r_{\mathrm{trk}}^{\textrm{data}}\right>/\left<r_{\mathrm{trk}}^{\textrm{MC}}\right>$ from unity is taken as the uncertainty, together with the uncertainties associated with the track measurement, charged particle multiplicity modeling in simulation and the statistical uncertainty of the dijet sample. The impact on the total background yield (for a 3 TeV$Z^{\prime}_{\text{TC2}}$ signal) in the boosted analysis is about 3% (4%) for the large-$R$ JES uncertainty and 3% (2%) for the large-$R$ JER uncertainty.

Correction factors to the simulated event samples are applied, separately for small-$R$ jets and track-jets, to compensate for differences observed between data and simulation in the $b$-tagging efficiency of $b$-, $c$- and light-quark and gluon-induced jets [62]. The correction factor for $b$-jets is derived from $t\bar{t}$ events with final states containing two leptons, and is consistent with unity within uncertainties at the level of a few percent over most of the jet $p_{\text{T}}$ range. Uncertainties in the correction factors for the $b$-tagging identification efficiency result in a variation of the total background yield of about 5% (4%) for the resolved (boosted) analysis. Uncertainties due to possible correlations between the correction factors in the signal and control regions are checked to have a negligible impact on the final results. An additional term is included to extrapolate the measured uncertainties to the high-$p_{\text{T}}$ region of interest. This term is calculated from simulated events by considering variations of the quantities affecting the $b$-tagging performance such as the impact parameter resolution, percentage of poorly measured tracks, description of the detector material, and track multiplicity per jet. The impact on the 3 TeV $Z^{\prime}_{\text{TC2}}$ signal yield due to such high-$p_{\text{T}}$ extrapolation uncertainty is about 3%.

In addition, smaller uncertainties associated with the luminosity measurement and the trigger efficiency are considered. The uncertainties associated with electron and muon reconstruction and identification are found to be negligible.

The uncertainty in the combined 2015+2016 integrated luminosity is 2.1%. It is derived, following a methodology similar to that detailed in Ref. [74], and using the LUCID-2 detector for the baseline luminosity measurements [75], from calibration of the luminosity scale using $x$–$y$ beam-separation scans. The pileup modeling uncertainty is considered by varying the average number of $pp$ collisions in simulated events.

In the resolved analysis the trigger efficiency is corrected around the jet $p_{\text{T}}$ threshold at the trigger level. The uncertainty in the correction factor, estimated to be below 1%, is dominated by the statistical uncertainty of the lower-threshold trigger data. In the boosted analysis the uncertainty in the trigger efficiency is found to be negligible.

7.2 Background modeling uncertainties

In this section, uncertainties associated with the data-driven estimates of multijet background and theory uncertainties in the SM $t\bar{t}$ prediction are discussed.

As discussed in Section 6, in both the resolved and boosted analyses the multijet background in the SRs is estimated by extrapolating the $m_{t\bar{t}}^{\text{reco}}$ shape obtained from the regions where the $b$-tagging criterion is loosened compared with that in the SRs. Uncertainties in the $m_{t\bar{t}}^{\text{reco}}$ shape and the yield of the multijet background are estimated separately as follows.

The different $b$-tagging criteria between the signal and control regions could produce a bias in the predicted $m_{t\bar{t}}^{\text{reco}}$ distributions. In the resolved analysis this effect is estimated by comparing the $m_{t\bar{t}}^{\text{reco}}$ distributions in the validation regions $\text{A}_{0}$ and $\text{C}_{0}$ (see Table 3) and the difference observed is assigned as a systematic uncertainty in the multijet background shape. The assumption that the potential bias is caused by the $b$-tagging instead of top-quark tagging is verified by repeating the same procedure using the validation regions $\text{A}_{-}$ and $\text{C}_{-}$, which gives a result comparable to the one from the validation regions $\text{A}_{0}$ and $\text{C}_{0}$. For the boosted analysis, the variations of the correction factor applied to the $p_{\text{T}}^{\text{sum}}$ distribution (see Section 6) are considered as an uncertainty in the multijet background shape. These include the statistical uncertainty of the multijet simulation samples and a small residual difference observed in the $m_{t\bar{t}}^{\text{reco}}$ distributions after the shape correction. A possible bias arising from using the inclusive $L_{\tau_{32}}$ range $\left[0.35,1.0\right]$ for the multijet template extraction from $\text{TR}i(\text{Inclusive},0b)$ is also taken into account as a source of systematic uncertainty. The multijet $m_{t\bar{t}}^{\text{reco}}$ distribution obtained from $\text{TR}i(\text{Inclusive},0b)$ is compared with those obtained from the individual $L_{\tau_{32}}$ regions ($\text{TR}i(j,0b);j=$ Medium and Tight) and the maximum difference in shape is considered.

The impact on the multijet yield due to correlation between the top- and $b$-quark tagging variables in the resolved analysis is evaluated by using the ($t_{0}$, $t_{0}$) or ($t_{-}$, $t_{-}$) categories instead of the ($t_{W}$, $t_{-}$) category. As a result, an uncertainty of 20% is added to the normalization of the multijet background, resulting in a 3% uncertainty in the total background yield. In the boosted analysis, the uncertainty in the multijet background normalization is estimated by taking the maximum deviation of the expected yields in the four CRs from the average. This leads to a 3% uncertainty in the overall background yield.

There are several sources of theoretical uncertainties affecting the modeling of SM $t\bar{t}$ background processes in all regions including signal, control and validation regions. The cross-section uncertainty given in Section 4 accounts for the choice of PDF and strong coupling constant calculated using the PDF4LHC prescription [76] with the MSTW2008 68% CL NNLO [55, 77], CT10 NNLO [33, 34] and NNPDF2.3 5f FFN [47] PDF sets, as well as the renormalization and factorization scale uncertainties. In addition to this pure normalization uncertainty, the following modeling uncertainties affecting both the acceptance and shape of the $t\bar{t}$ kinematic distributions are considered. The impact from the modeling of extra QCD radiation is evaluated using Powheg+Pythia samples in which the renormalization and factorization scales and the $h_{\text{damp}}$ parameter are varied within the ranges consistent with the measurements of $t\bar{t}$ production in association with jets [78, 79, 80]. Additionally, the uncertainty in the $t\bar{t}$ event kinematics due to higher-order QCD effects is considered by adding an uncertainty covering the difference between NLO and NNLO QCD calculations of $t\bar{t}$ production. The recent QCD calculations in Ref. [81] are used to derive the difference, which is applied as a function of top-quark $p_{\text{T}}$ and the transverse momentum of the $t\bar{t}$ system at the particle level taking into account the final-state radiation, to estimate this uncertainty. The variation of the event yield at the reconstruction level is less than 4% at $m_{t\bar{t}}^{\text{reco}}$ below 500 GeV, but approaches 11% at $m_{t\bar{t}}^{\text{reco}}$ of 1.2 TeV in the resolved analysis and 20% above 3 TeV in the boosted analysis. The electroweak corrections to top-quark kinematics in $t\bar{t}$ events have an associated uncertainty of about 10%, which varies as a function of $m_{t\bar{t}}^{\text{reco}}$ [45]. The uncertainty associated with the choice of event generator is evaluated by taking the difference between the predictions from the $t\bar{t}$ samples generated with Powheg-Box and aMC@NLO both interfaced to Herwig++ 2.7.1 [82]. The uncertainty in the parton shower modeling is evaluated by comparing the $t\bar{t}$ events simulated with the default Powheg+Pythia with those with the same version of Powheg-Box but interfaced to Herwig 7 [82, 83]. The uncertainty arising from the choice of PDF set is estimated by taking into account the variations from the PDF4LHC15 PDF set, which includes 30 separate uncertainty eigenvectors [50], and the difference between the nominal PDF4LHC15 and CT10 PDF sets. For the boosted analysis, an additional uncertainty is considered in the $m_{t\bar{t}}^{\text{reco}}$ shape due to the extrapolation procedure using an exponential function at high $m_{t\bar{t}}^{\text{reco}}$ above 1.5 TeV (Section 6). This includes the statistical uncertainty in the exponential fit and the stability of the fit results estimated by varying the fit range. The overall impact on the SM $t\bar{t}$ event yields from these uncertainties is estimated to be 29% in the resolved analysis and 24% in the boosted analysis.

8 Statistical analysis

A binned maximum-likelihood fit to the $m_{t\bar{t}}^{\text{reco}}$ distributions is performed to estimate the signal and background yields, separately in the resolved and boosted analyses. The likelihood is defined as a product of the Poisson probabilities to observe $n_{i}$ events when $\lambda_{i}$ events are expected in bin $i$. The $\lambda_{i}$ is expressed as $\lambda_{i}=\mu s_{i}({\bm{\theta}})+b_{i}({\bm{\theta}})$ where $\mu$ is the signal strength, defined as a signal cross section in units of the theoretical prediction, to be determined by the fit, and $s_{i}({\bm{\theta}})$ and $b_{i}({\bm{\theta}})$ are the expected numbers of signal and background events, respectively. The fit includes two background components; $t\bar{t}$ and multijet processes, which are estimated by the simulated samples and the data-driven methods, respectively, as described in Section 6. The systematic uncertainties are taken into account as nuisance parameters, ${\bm{\theta}}$, constrained by Gaussian or log-normal penalty terms in the likelihood. Nuisance parameters are also determined by the fit, varying the normalization and shape of the $m_{t\bar{t}}^{\text{reco}}$ distribution for each component of the signal and background.

In the resolved analysis, the likelihood fit is performed simultaneously in the three multijet-enriched regions A–C and the main signal region D. In each region, the $m_{t\bar{t}}^{\text{reco}}$ distribution is divided into 19 bins spanning the range 0 to 2 TeV. The shape of the multijet background is determined by bin-by-bin unconstrained normalization factors. Assuming that the $m_{t\bar{t}}^{\text{reco}}$ shape does not depend on the $b$-tagging requirement, the bin-by-bin multijet normalization factors for regions A and C as well as for regions B and D are treated as fully correlated. In order to consider the normalization component not depending on the top-tagging requirement but depending on the $b$-tagging requirement, a common free-floating normalization factor is additionally applied to regions C and D. Thus, the correlation between the ($t_{W}$, $t_{-}$) and ($t_{W}$, $t_{W}$) categories is introduced in the background parameterization.

The SRs in the boosted analysis cover the $m_{t\bar{t}}^{\text{reco}}$ range between 1 and 6 TeV, which is divided into 19 bins. The fit is performed simultaneously in the eight SRs defined in Section 5.3. The $m_{t\bar{t}}^{\text{reco}}$ shape and normalization of the multijet background are constrained by the variations due to systematic uncertainties estimated in Section 7 by using them as nuisance parameters in the fit.

A test statistic based on the profile likelihood ratio [84] is used to extract information about $\mu$ from a likelihood fit to data under the signal-plus-background hypothesis, separately for each model considered. The distributions of the test statistic under the signal-plus-background and the background-only hypotheses are obtained from pseudo experiments. The probability that the observed data is compatible with the SM prediction is estimated by computing the local $p_{0}$-value, defined as the probability to observe an excess at least as large as the one observed in data, under the background-only hypothesis. The global $p_{0}$-value is computed by considering the look-elsewhere effect [85, 86] associated with the multiple testing to scan the signal mass points. If no significant excess is observed over the background, expected and observed upper limits on the signal strength are set at 95% confidence level (CL) using the CL${}_{\text{s}}$ prescription [87]. The results of the resolved and boosted analyses are compared in the $m_{t\bar{t}}$ region covered by both analyses and the one providing the better expected limit is selected. The upper limits on $\mu$ are converted into limits on the cross-section times branching fraction of new particles decaying into $t\bar{t}$.

9 Results

The observed $m_{t\bar{t}}^{\text{reco}}$ distributions in the regions A–D for the resolved analysis and in the signal regions SR1 and SR2 for the boosted analysis after the fit (Post-Fit) with the background-only hypothesis are shown in Figures 6, 7 and 8, respectively. The expected signal and background yields as well as the observed number of data events are summarized in Tables 5 and 6 for the resolved and boosted analyses, respectively. The systematic uncertainties with the largest post-fit impact on the signal strength parameter $\mu$ in the resolved and boosted analyses are presented in Table 7. The observed data agree well with the estimated SM background and no significant excess is observed. Assuming a narrow-width resonance modeled by the $Z^{\prime}_{\text{TC2}}$ signal, the minimum local $p_{0}$-value is observed in the boosted analysis to be 0.02 ($2.1\sigma$) at $m=1.75$ TeV. The observed excess corresponds to a global significance of less than $1\sigma$. While the excess is mostly driven by $\text{SR}1(\text{Tight},2b)$ region, it is worth noting that the other regions contribute significantly to the overall sensitivity, e.g. adding the $\text{SR}2$ regions can improve the sensitivity by up to 20% (for a 3 TeV signal) and adding the $1b$ regions to the $2b$ ones adds about 10% more sensitivity. The data and expected background spectra are also compared using BumpHunter [88], which performs a hypothesis test to look for local excesses or deficits in data relative to the background, taking the look-elsewhere effect into account as well. No significant deviation from the background is found.

In the absence of a significant excess above the background prediction, 95% CL upper limits on the cross-section times branching fraction of new particles decaying into $t\bar{t}$ are calculated at each mass value for the different benchmark signal models considered. The expected and observed upper limits on the cross-section times branching fraction of $Z^{\prime}_{\text{TC2}}\to t\bar{t}$ are presented in Figure 9. Due to the strength of the expected limits, results from the resolved analysis are shown at $m_{Z^{\prime}_{\text{TC2}}}$ below 1.2 TeV, whereas the results of the boosted analysis are shown above that value. The NLO theory cross-section predictions for the $Z^{\prime}_{\text{TC2}}$ with $\Gamma=1\%$ and 3%, as well as those at LO with $\Gamma=1.2\%$ are overlaid. The observed (expected) 95% CL exclusion range is set for the $Z^{\prime}_{\text{TC2}}$ masses between 0.58 and 3.1 TeV (0.57 and 2.8 TeV) and 0.53 and 3.6 TeV (0.51 and 3.6 TeV) for $\Gamma=1\%$ and 3%, respectively. Limits are also set on the cross-section times branching fraction of the vector and axial-vector mediators $Z^{\prime}_{\text{med}}$ in the simplified DM model, as shown in Figure 10. The vector (axial-vector) mediator $Z^{\prime}_{\text{med}}$ is excluded in the mass ranges of $0.74\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<0.97\,\text{Te\kern-1.00006ptV}$ and $2.0\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<2.2\,\text{Te\kern-1.00006ptV}$ ($0.80\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,ax}}}<0.92\,\text{Te\kern-1.00006ptV}$ and $2.0\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,ax}}}<2.2\,\text{Te\kern-1.00006ptV}$) at 95% CL by the data with the corresponding expected mass ranges of $0.75\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<1.07\,\text{Te\kern-1.00006ptV}$ and $2.0\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<2.1\,\text{Te\kern-1.00006ptV}$ ($1.99\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,ax}}}<2.04\,\text{Te\kern-1.00006ptV}$). The upper limit on the cross-section times branching fraction of the $G_{\text{KK}}$ in the bulk RS model is shown in Figure 11. The cross-section times branching fraction for $G_{\text{KK}}$ production with the model parameters described in Section 2 is too low to be excluded with the sensitivity of this measurement, hence the limit is presented only up to 3 TeV. Figure 12 shows the upper limit on the cross-section times branching fraction of the $g_{\text{KK}}$ with $\Gamma=30\%$ in the RS model with a single warped extra dimension. The observed and expected lower limits on the $g_{\text{KK}}$ mass are 3.4 and 3.3 TeV, respectively. The exclusion limit is also extracted for the $g_{\text{KK}}$ as a function of the width at representative mass values. Figures 13(a), 13(b), 13(c), 13(d) and 13(e) show the results for $m_{g_{\text{KK}}}=0.5$, 1.0, 1.5, 2.0 and 5.0 TeV, respectively. For $m_{g_{\text{KK}}}>0.5$ TeV, the limits on the cross-section times branching fraction deteriorate with increasing $g_{\text{KK}}$ width as the signal peak of the reconstructed $m_{t\bar{t}}^{\text{reco}}$ distribution becomes broad. The limit at $m_{g_{\text{KK}}}=0.5$ TeV does not depend on the signal width since the events with reconstructed $m_{t\bar{t}}^{\text{reco}}<0.5$ TeV are covered by one bin, as shown in Figure 6.

The extracted lower limits on the masses for various signal hypotheses where the sensitivity of the analysis allows for it are summarized in Table 8.

10 Conclusion

A search for resonant production of $t\bar{t}$ decaying into the fully hadronic final state is performed using 36.1 fb*-1* of $pp$ collision data recorded at $\sqrt{s}=13$ TeV with the ATLAS detector at the LHC. Depending on the mass of new hypothetical particles, the search exploits two analysis techniques optimized for the reconstruction of a top-quark pair and background suppression. No significant deviation from the Standard Model expectation is observed over the search range considered. Upper limits are set on the production cross-section times branching fraction for several benchmark signals, such as $Z^{\prime}_{\text{TC2}}$ boson predicted in the topcolor-assisted-technicolor model, vector and axial-vector mediators $Z^{\prime}_{\text{med}}$ in the dark-matter simplified model, and the Kaluza–Klein excitations of the graviton $G_{\text{KK}}$ and gluon $g_{\text{KK}}$ in the specific models based on the Randall–Sundrum scenario of warped extra dimensions. The $Z^{\prime}_{\text{TC2}}$ boson is excluded in the mass range of 0.58 and 3.1 TeV (0.53 and 3.6 TeV) for the decay width of 1% (3%). The vector (axial-vector) mediator $Z^{\prime}_{\text{med}}$ is excluded in the mass ranges of $0.74\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<0.97\,\text{Te\kern-1.00006ptV}$ and $2.0\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,vec}}}<2.2\,\text{Te\kern-1.00006ptV}$ ($0.80\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,ax}}}<0.92\,\text{Te\kern-1.00006ptV}$ and $2.0\,\text{Te\kern-1.00006ptV}<m_{Z^{\prime}_{\text{med,ax}}}<2.2\,\text{Te\kern-1.00006ptV}$). The lower limit on the $g_{\text{KK}}$ mass is set at 3.4 TeV for the decay width of 30%. The cross-section limits for the $Z^{\prime}_{\text{TC2}}$ boson are comparable at a $Z^{\prime}_{\text{TC2}}$ mass above $\sim 1$ TeV to those from the previous ATLAS lepton-plus-jets analysis performed at 13 TeV [19].

Acknowledgments

We thank CERN for the very successful operation of the LHC, as well as the support staff from our institutions without whom ATLAS could not be operated efficiently.

We acknowledge the support of ANPCyT, Argentina; YerPhI, Armenia; ARC, Australia; BMWFW and FWF, Austria; ANAS, Azerbaijan; SSTC, Belarus; CNPq and FAPESP, Brazil; NSERC, NRC and CFI, Canada; CERN; CONICYT, Chile; CAS, MOST and NSFC, China; COLCIENCIAS, Colombia; MSMT CR, MPO CR and VSC CR, Czech Republic; DNRF and DNSRC, Denmark; IN2P3-CNRS, CEA-DRF/IRFU, France; SRNSFG, Georgia; BMBF, HGF, and MPG, Germany; GSRT, Greece; RGC, Hong Kong SAR, China; ISF and Benoziyo Center, Israel; INFN, Italy; MEXT and JSPS, Japan; CNRST, Morocco; NWO, Netherlands; RCN, Norway; MNiSW and NCN, Poland; FCT, Portugal; MNE/IFA, Romania; MES of Russia and NRC KI, Russian Federation; JINR; MESTD, Serbia; MSSR, Slovakia; ARRS and MIZŠ, Slovenia; DST/NRF, South Africa; MINECO, Spain; SRC and Wallenberg Foundation, Sweden; SERI, SNSF and Cantons of Bern and Geneva, Switzerland; MOST, Taiwan; TAEK, Turkey; STFC, United Kingdom; DOE and NSF, United States of America. In addition, individual groups and members have received support from BCKDF, CANARIE, CRC and Compute Canada, Canada; COST, ERC, ERDF, Horizon 2020, and Marie Skłodowska-Curie Actions, European Union; Investissements d’ Avenir Labex and Idex, ANR, France; DFG and AvH Foundation, Germany; Herakleitos, Thales and Aristeia programmes co-financed by EU-ESF and the Greek NSRF, Greece; BSF-NSF and GIF, Israel; CERCA Programme Generalitat de Catalunya, Spain; The Royal Society and Leverhulme Trust, United Kingdom.

The crucial computing support from all WLCG partners is acknowledged gratefully, in particular from CERN, the ATLAS Tier-1 facilities at TRIUMF (Canada), NDGF (Denmark, Norway, Sweden), CC-IN2P3 (France), KIT/GridKA (Germany), INFN-CNAF (Italy), NL-T1 (Netherlands), PIC (Spain), ASGC (Taiwan), RAL (UK) and BNL (USA), the Tier-2 facilities worldwide and large non-WLCG resource providers. Major contributors of computing resources are listed in Ref. [89].