Collider constraints and new tests of color octet vectors

TL;DR

This paper investigates collider detection strategies for color octet vector resonances, emphasizing the importance of mixed decay modes and proposing new analysis methods that outperform existing searches.

Contribution

It introduces novel search strategies for mixed decay modes of colored resonances, enhancing detection sensitivity beyond current methods.

Findings

Mixed decay mode searches are more sensitive when decay widths are comparable.

Collective lower mass limits of 1.5 TeV can be set with current data.

Existing searches miss potential signals in mixed decay channels.

Abstract

We analyze the collider sensitivity for new colored resonances in $t\bar{t}$, $b \bar{b}$, and $jj$ final states. While searches in the single production channel are model-dependent, the pair production rate is model independent and the existing $(JJ)(JJ)$ and $4t$ searches impose strong constraints on the relevant branching fractions, where $J = j$ or $b$. We point out the missing, complementary searches in the mixed decay modes, $t\bar{t}(jj)$, $t\bar{t}(b\bar{b})$, and $(b\bar{b})(jj)$. We propose analysis strategies for the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ decays and find their sensivity surpasses that of existing searches when the decay widths to tops and light jets are comparable. If no other decays are present, collective lower limits on the resonance mass can be set at 1.5~TeV using 37~fb$^{-1}$ of 13~TeV data.

Tables

Table 1. Table 1: Cut flow for different resonance masses m X subscript 𝑚 𝑋 m_{X} and dominant background t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} +jets for the semi-leptonic search. All branching ratios are applied to signal and background when quoting cross sections. We normalize the signal assuming Br ​ ( X → t ​ t ¯ ) =  Br ​ ( X → J ​ J ) = 50 % Br → 𝑋 𝑡 ¯ 𝑡  Br → 𝑋 𝐽 𝐽 percent 50 \text{Br}(X\to t\bar{t})=\text{ Br}(X\to JJ)=50\% .

Semi-Leptonic Search	Signal			Background $t\overline{t}$+jets
Signal mass $m_X$	1.3 TeV	1.5 TeV	1.7 TeV	1.3 TeV	1.5 TeV	1.7 TeV
Event selection [fb] ($N_J\ge 5$ with $N_j\ge 1$, $N_b\ge 2$;
$N_{\ell }=1$; $p_{T,j}^{\text{leading}},p_{T,b}^{\text{leading}}>250$ GeV; $p_{T,\ell }>20$ GeV)	1.52	0.45	0.13	5.8	5.8	5.8
${\mathrm{E̸}}_T>80$ GeV	84%	87%	89%	64%	64%	64%
$H_T\ge \left(4/3\right)m_X$	83%	79%	76%	32%	19%	10%
Remaining cross section [fb]	1.06	0.31	0.09	1.23	0.71	0.39

Table 2. Table 2: Cut flow for a different resonance masses m X subscript 𝑚 𝑋 m_{X} and dominant background t ​ t ¯ 𝑡 ¯ 𝑡 t\bar{t} +jets for the fully leptonic search. All branching ratios are applied to signal and background when quoting cross sections. We normalize the signal assuming Br ​ ( X → t ​ t ¯ ) =  Br ​ ( X → J ​ J ) = 50 % Br → 𝑋 𝑡 ¯ 𝑡  Br → 𝑋 𝐽 𝐽 percent 50 \text{Br}(X\to t\bar{t})=\text{ Br}(X\to JJ)=50\% .

Fully Leptonic Search	Signal			Background $t\overline{t}$+jets
Signal mass $m_X$	1.3 TeV	1.5 TeV	1.7 TeV	1.3 TeV	1.5 TeV	1.7 TeV
Event selection [fb] ($N_J\ge 4$ with $N_b\ge 2$;
$N_{\ell }=2$; $p_{T,j}^{\text{leading}},p_{T,b}^{\text{leading}}>250$ GeV; $p_{T,\ell }>20$ GeV)	0.148	0.045	0.009	0.75	0.75	0.75
${\mathrm{E̸}}_T>50$ GeV	96%	96%	97%	91%	91%	91%
$m_{\ell \ell }\ge 115$ GeV	66%	68%	70%	37%	37 %	37%
$H_T\ge \left(4/3\right)m_X$	63%	57%	48%	21%	11%	2.7%
Remaining cross section [fb]	0.059	0.017	0.003	0.053	0.027	0.007

Equations

$$\mathcal{L}\supset g_s\left({\bar{Q}}_L{\gamma }_{\mu }\lambda T^a{X}^{\mu a}{Q}_L+{\bar{u}}_R{\gamma }_{\mu }\kappa T^a{X}^{\mu a}{u}_R+{\bar{d}}_R{\gamma }_{\mu }{T}^a\eta X^{\mu a}{d}_R\right),$$

$$\frac{1}{g_s^2}=\frac{1}{h_1^2}+\frac{1}{h_2^2},$$

$$D^{\mu }\Sigma =({\partial }^{\mu }-i{h}_1{G}_1^{\mu a}{T}^a+i{h}_2{G}_2^{\mu a}{T}^a)\Sigma ,$$

$$g^{\mu }=\cos \theta G_1^{\mu }+\sin \theta G_2^{\mu },$$

$$X^{\mu }=\sin \theta G_1^{\mu }-\cos \theta G_2^{\mu },$$

$$g_s\tan \theta \bar{q}{\gamma }^{\mu }{T}^a{X}_{\mu }^{a}q.$$

$$g_s\bar{q}{\gamma }^{\mu }{T}^a{X}_{\mu }^a(\tan \theta P_L-\cot \theta P_R)q,$$

$$g_s\cot \theta (\bar{t}{\gamma }^{\mu }{T}^a{X}_{\mu }t+{\bar{b}}_L{\gamma }^{\mu }{T}^a{X}_{\mu }{b}_L)+g_s\tan \theta ({\bar{b}}_R{\gamma }^{\mu }{T}^a{X}_{\mu }{b}_R+\sum _{i=1\dots 4}{\bar{q}}_i{\gamma }^{\mu }{T}^a{X}_{\mu }{q}_i),$$

$$g_s{T}^a{\gamma }^{\mu }\frac{1}{\sqrt{2}}\frac{1}{16{\pi }^2}\log {\left(\frac{\Lambda }{\mu }\right)}^2\left[P_L(\frac{1}{8}{g}_1^2+\frac{27}{8}{g}_2^2-\frac{11}{2}{g}_s^2)+P_R(Y_{u,d}{g}_1^2-\frac{11}{2}{g}_s^2)\right],$$

$${\lambda }^{ij}$$

$$\mathcal{L}$$

$$+{\bar{u}}_R^m{g}_s{t}^a/X^a{U}_u\kappa U_u^{†}{u}_R^m+{\bar{d}}_R^m{g}_s{t}^a/X^a{U}_d\eta U_d^{†}{d}_R^m.$$

$$\tilde{\lambda }=V_u\lambda V_u^{†},\tilde{\kappa }=U_u\kappa U_u^{†},\tilde{\eta }=U_d\eta U_d^{†}.$$

$${\tilde{\lambda }}_D\equiv V_d{V}_u^{†}{V}_u\lambda V_u^{†}{V}_u{V}_d^{†}=V_{CKM}^{†}\tilde{\lambda }{V}_{CKM},$$

$$O_K^1=({\bar{d}}_{\alpha }{\gamma }_{\mu }{P}_L{s}_{\alpha })({\bar{d}}_{\beta }{\gamma }^{\mu }{P}_L{s}_{\beta }),$$

$$C_K^1=\frac{-1}{6}\frac{{\tilde{\lambda }}_{D,12}^2}{M_X^2}.$$

$$\text{Re}{C}_K^1:$$

$$\text{Im}{C}_K^1:$$

$$|C_{B_d}^1|:$$

$$|C_{B_s}^1|:$$

$$\text{Im}{C}_D^1:$$

$$\Gamma (X\to \bar{q}q)=\frac{{\alpha }_s{m}_X}{12}\left((g_L^2+g_R^2)(1-\frac{m_q^2}{m_X^2})+6\frac{m_q^2}{m_X^2}{g}_L{g}_R\right){\left(1-4\frac{m_q^2}{m_X^2}\right)}^{1/2},$$

$$\Gamma (X\to {\bar{q}}_i{q}_j,{\bar{q}}_j{q}_i)$$

$$\times {\left((1-\frac{m_i^2}{m_X^2}-\frac{m_j^2}{m_X^2}{)}^2-4\frac{m_i^2{m}_j^2}{m_X^4}\right)}^{1/2},$$

$$\Gamma (X\to \bar{q}q)=\frac{{\alpha }_s(g_L^2+g_R^2)m_X}{12}.$$

$$\text{Br}(X\to t\bar{t})=1-\text{ Br}(X\to JJ).$$

$${{\sigma }_{\text{excl}}(JJ)(JJ)|}_{m_X}={\sigma (pp\to X{X}^{\ast })|}_{m_X}\times {\left({\text{Br}}_{JJ}\right)}^2,$$

$${{\sigma }_{\text{excl}}(pp\to (JJ{)}_{\text{res}})|}_{m_X}$$

$${{\sigma }_{\text{excl}}(pp\to (t\bar{t}{)}_{\text{res}})|}_{m_X}$$

Full text

††preprint: MITP/17-004

1. I Introduction

2. II Theory background

3. II.1 Color octet vectors from extended color gauge groups

4. II.2 Color octet vectors from extra dimensions

5. II.3 Non-universal couplings and flavor constraints

6. II.4 Collider phenomenology of color octet vectors

7. III Collider analyses of the mixed channels, $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$

8. III.1 Event selection strategy

9. III.2 Comparison with current searches

10. IV Conclusion

I Introduction

Searches for new particles and new forces beyond the Standard Model (BSM) are a critical endeavor for the ATLAS and CMS experiments at the Large Hadron Collider (LHC), but such results have thus far come up null. The largest rates come from colored particle pair production, where leading order cross sections can be calculated knowing only the color charge and mass Han et al. (2010). Even so, the colored particle decay patterns and corresponding collider signatures are highly model dependent.

In models with an extended color gauge symmetry Hill (1991); Chivukula et al. (1996); Frampton and Glashow (1987a); Martynov and Smirnov (2009); Frampton et al. (2010); Simmons et al. (2011), massive color octet vectors, such as colorons Bai and Dobrescu (2011), are heavy cousins of the familiar gluon and can decay universally to Standard Model (SM) quark–anti-quark pairs. Axigluons Hall and Nelson (1985); Frampton and Glashow (1987a); Bagger et al. (1988) are similarly motivated by ascribing chiral projection operators to the parent gauge groups, which necessitates new fermions transforming non-trivially under the extended gauge group to cancel anomalies. Similar phenomenology occurs in models of universal extra dimensions, where the massive color octet vectors arise as Kaluza-Klein (KK) excitations of the gluon Appelquist et al. (2001); Rizzo (2001); Macesanu et al. (2002). In Randall-Sundrum models Randall and Sundrum (1999a, b) with SM fields propagating in the bulk Davoudiasl et al. (2000); Grossman and Neubert (2000); Pomarol (2000); Chang et al. (2000); Huber and Shafi (2001), the KK gluons exhibit flavor-dependent couplings to quark pairs, and preferentially decay to the heavy flavor quarks as a result of the localization of the bulk fermion wavefunction profile Dicus et al. (2002); Davoudiasl et al. (2001); Lillie et al. (2007); Casagrande et al. (2008); Agashe et al. (2013).

At the LHC, CMS and ATLAS have searched for color octet vector resonances in the paired dijet channel Chatrchyan et al. (2013a); Khachatryan et al. (2015); Collaboration (2016); Aad et al. (2013a); collaboration (2016a, 2017), most recently constraining colorons with a 100% branching fraction to light jets to be heavier than 1.5 TeV collaboration (2017). Searches for four tops Collaboration (2013); Sirunyan et al. (2017); ATL (2012); Aad et al. (2015a); collaboration (2016b, c, d) also constrain the possible $t\bar{t}$ decays of pair-produced resonances. Searches for TeV-scale dijet resonances Collaboration (2017); Aaboud et al. (2017) and $t\bar{t}$ resonances Chatrchyan et al. (2012, 2013b); Aad et al. (2012, 2013b, 2013c, 2015b); collaboration (2016e) offer complementary probes compared to the pair-production searches, since such searches scale with the individual production coupling Dobrescu and Yu (2013).

Given the possibility that the color octet vector has flavor dependent branching fractions to quark pairs, and because color octet vectors have a model independent pair production rate, the mixed decay signature of a paired ditop and dijet resonance is strongly motivated. This channel is complementary to the existing $(JJ)(JJ)$ and $4t$ searches and even offers superior sensitivity when the branching fractions of the resonance to dijets and ditops are comparable.

In Sec. II, we review the theory motivation and collider phenomenology of massive color octet vectors. In Sec. III, we analyze the prospects for discovering color octet vector resonances in the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ mixed decay modes at the LHC and compare with the current constraints. We conclude in Sec. IV.

II Theory background

Massive color octet vector resonances arise in various different beyond the Standard Model extensions, such as models with extended color gauge groups or models with extra dimensions. In the unbroken phase of electroweak symmetry, the general interaction Lagrangian between quarks and a massive color octet vector $X_{\mu}$ is

[TABLE]

where $g_{S}$ is the strong coupling constant, $T^{a}$ are the $SU(3)$ generators, and $\lambda$, $\kappa$, and $\eta$ are the flavor-dependent couplings in the quark gauge basis. While these matrices must be symmetric by $CPT$, nonzero off-diagonal entries are possible in principle and correspond to tree-level flavor violation. Since these processes are strongly constrained by low-energy precision flavor measurements, such as meson mixing measurements and $b\to X_{s}\gamma$ transitions Bona et al. (2008); Chivukula et al. (2013), the simplest ansatz is to adopt a flavor-universal coupling structure, evading the most stringest flavor bounds. In such a scenario, however, the branching ratio for $X$ to light quarks vs. tops is fixed, and there is no model freedom in the complementarity between searching in dijet vs. ditop resonances. Hence, we will focus on the scenario where the $\lambda$, $\kappa$, and $\eta$ matrices are diagonal but not universal in the quark gauge basis. Moving to the quark mass basis will then induce off-diagonal entries proportional to Cabibbo-Kobayashi-Maskawa (CKM) mixing. We will first briefly review models of massive color octet vectors and then discuss the collider and flavor physics phenomenology of the general Lagrangian, Eq. (1).

II.1 Color octet vectors from extended color gauge groups

In models with an extended color symmetry, such as a $SU(3)_{1}\times SU(3)_{2}$ gauge group, the two parent gauge groups are typically broken to the diagonal subgroup by the vacuum expectation value (vev) of a bifundamental, complex scalar field $\Sigma$ Chivukula et al. (1996); Bai and Dobrescu (2011). The diagonal subgroup is then identified with the SM $SU(3)_{c}$ gauge group, which imposes the requirement

[TABLE]

where $h_{1}$ and $h_{2}$ are the gauge couplings of $SU(3)_{1}$ and $SU(3)_{2}$, respectively. The Goldstone modes of the complex scalar field become the longitudinal degrees of freedom for the heavy color octet vector, leaving one real and one pseudoreal color singlet scalar and one real color octet scalar as the dynamical scalar fields below $\langle\Sigma\rangle$. Explicitly, the covariant derivative for $\Sigma$ is

[TABLE]

and after Higgsing $SU(3)_{1}\times SU(3)_{2}\to SU(3)_{c}$, we get the SM gluon and coloron fields,

[TABLE]

respectively, where $\theta=\tan^{-1}(h_{1}/h_{2})$ is the mixing angle.

In this setup, different possibilities for the couplings in Eq. (1) originate by considering various charge assignments of the quarks in the parent $SU(3)_{1}\times SU(3)_{2}$ gauge symmetry. The main feature for universally coupled models is that all flavor representations are assigned identically to the same gauge representation, which ensures that the gauge symmetry commutes with the global SM quark flavor symmetry.

In the coloron model Chivukula et al. (1996); Bai and Dobrescu (2011), all SM quarks transform as $\square$ under one $SU(3)$ gauge group and singlets under the other. Hence, the coloron has flavor universal, purely vector couplings to the SM quarks as

[TABLE]

An alternative prescription is the chiral color model Hall and Nelson (1985); Frampton and Glashow (1987a, b), where left-handed (LH) and right-handed (RH) quark fields are charged under different $SU(3)$ gauge groups. This construction generally requires new fermions to cancel anomalies, notably $SU(3)_{1}^{2}\times U(1)_{Y}$ and $SU(3)_{2}^{2}\times U(1)_{Y}$, but this new matter content can be massive and unobservable at colliders. As a result, if the LH quarks transform as $(\square,\mathbf{1})$ and the RH quarks transform as $(\mathbf{1},\square)$ under $SU(3)_{1}\times SU(3)_{2}$, for example, the resulting massive color vector interaction with quarks is

[TABLE]

where $X_{\mu}^{a}$ is commonly referred as an axigluon in the literature.

To motivate a non-universal yet diagonal coupling structure in Eq. (1), we can straightforwardly assign different quark flavors to different gauge representations under $SU(3)_{1}\times SU(3)_{2}$. For example, the topcolor model charges the third generation quarks differently than the first two generations, with $Q_{L}^{1,2}\sim(\square,\mathbf{1})$, $Q_{L}^{3}\sim(\mathbf{1},\square)$, $u_{R}^{1,2}\sim(\square,\mathbf{1})$, $u_{R}^{3}\sim(\mathbf{1},\square)$, $d_{R}^{1,2,3}\sim(\square,\mathbf{1})$ Hill (1991). This assignment also requires additional matter to cancel anomalies, for which a minimal solution involves two electroweak singlet quarks transforming as $(\mathbf{1},\square)$ and $(\square,\mathbf{1})$, each with hypercharge $-2/3$. The corresponding massive color octet vector does not have flavor-changing couplings in the quark gauge basis, but instead features distinct couplings to light and heavy generation quarks:

[TABLE]

with $\tan\theta=h_{1}/h_{2}$ as before. Although topcolor models are generally motivated by composite Higgs scenarios triggered by top quark condensation Hill (1991), we will only focus on the motivated possibility that $X_{\mu}$ has non-universal couplings.

II.2 Color octet vectors from extra dimensions

Models with extra spacetime dimensions provide an alternative framework for realizing massive color octet vector resonances Appelquist et al. (2001); Rizzo (2001); Macesanu et al. (2002). In such models, the SM fields are the lowest-lying states of a Kaluza-Klein tower, whose masses and dynamics result from solving the five-dimensional equations of motion Davoudiasl et al. (2000); Grossman and Neubert (2000); Pomarol (2000); Chang et al. (2000); Huber and Shafi (2001). In minimal universal extra dimensions Datta et al. (2010), the level-2 KK gluon obtains a coupling to SM quarks from one-loop diagrams with level-1 KK particles running in the loop. These couplings are generated from boundary conditions on the KK gluon and the bulk masses, which provide the only source of translational invariance breaking, and read

[TABLE]

where $Y_{u}=2$ for up-type quarks, $Y_{d}=1/2$ for down-type quarks, $\Lambda$ is an ultraviolet scale larger than the inverse size of the extra dimension, and $\mu$ is the renormalization scale to evaluate the coupling.

In Randall-Sundrum warped scenarios Randall and Sundrum (1999a, b), the fermion mass hierarchy can be explained by allowing fermions to propagate in the bulk, where the observed charged fermion mass hierarchy originates as $\mathcal{O}(1)$ differences in bulk mass parameters. Typically, the KK mass scale must then be $\mathcal{O}(5-10~{}\text{TeV})$ to satisfy low-energy flavor violation probes, especially $\bar{K}-K$ mixing Casagrande et al. (2008), but this scale can be lowered in the case that the bulk fermions obey a flavor symmetry Chen et al. (2010a, b). Since KK parity is absent, the first KK gluon decays to SM zero-mode quarks, and this coupling is given by calculating the wavefunction overlap between the zero modes and the KK gluon in the extra dimension. Using the general coupling structure in Eq. (1) and identifying $X_{\mu}$ with the first KK gluon,

[TABLE]

where RH up-type couplings are obtained by replacing $\lambda\rightarrow\kappa$ and $Q\rightarrow u$, RH down-type couplings are obtained by replacing $\lambda\rightarrow\eta$ and $Q\rightarrow d$, $m_{X}\approx 2.4M_{\text{KK}}$ is fixed by the boundary conditions of the 5D gluon, $L$ is the length of the extra dimension, and $c_{Q}$, $c_{u}$, and $c_{d}$ are the bulk mass parameters for LH and RH quark fields Agashe et al. (2013). In order to reproduce the known SM quark masses, these bulk mass parameters must be chosen to maximize the top quark wavefunction overlap with the TeV-scale infrared brane while the wavefunctions for the light quarks are skewed towards the ultraviolet brane. As a result, the first KK gluon preferentially decays to top quarks, with branching fractions that can exceed 80% Agashe et al. (2013).

II.3 Non-universal couplings and flavor constraints

As we have emphasized, massive color octet vectors arise in numerous models of beyond the Standard Model physics. Their collider and flavor physics phenomenology depends crucially on the particular $\lambda$, $\kappa$, and $\eta$ structure defined in Eq. (1) that is realized in a given model. Given the stringent constraints on tree-level flavor changing neutral currents Bona et al. (2008); Chivukula et al. (2013), we adopt flavor-diagonal couplings for $\lambda$, $\kappa$, and $\eta$ in the gauge basis. Nevertheless, flavor violating effects are still induced in interactions of LH down quarks by the rotation to the quark mass basis. From Eq. (1), we rotate the quark fields to the mass basis by $V_{u}u_{L}=u_{L}^{m}$, $V_{d}d_{L}=d_{L}^{m}$, $U_{u}u_{R}=u_{R}^{m}$, and $U_{d}d_{R}=d_{R}^{m}$, giving

[TABLE]

We see that the $\lambda$, $\kappa$, and $\eta$ matrices can be chosen such that the effective interaction matrices $\tilde{\lambda}$, $\tilde{\kappa}$, and $\tilde{\eta}$ are diagonal in the quark mass basis,

[TABLE]

The corresponding LH down quark interactions have small, off-diagonal entries induced by $V_{CKM}\equiv V_{u}V_{d}^{\dagger}$ Gedalia and Perez (2011); Patrignani et al. (2016),

[TABLE]

and hence $X_{\mu}$ mediates tree-level flavor-changing neutral currents (FCNCs). Since the strongest FCNC constraints come from $\bar{K}_{0}-K_{0}$ mixing Bona et al. (2008), we minimize the impact of these constraints by assuming $\tilde{\lambda}_{11}=\tilde{\lambda}_{22}\neq\tilde{\lambda}_{33}$, which leads to $\tilde{\lambda}_{D,12}\approx\tilde{\lambda}_{D,21}\approx-(3.3-1.3i)\times 10^{-4}(\tilde{\lambda}_{33}-\tilde{\lambda}_{11})$, using global fit values for the CKM elements Patrignani et al. (2016). A tree-level exchange of $X_{\mu}$ can be matched to the four-fermion operator Bona et al. (2008); Casagrande et al. (2008); Bai et al. (2011); Haisch and Westhoff (2011); Altmannshofer et al. (2012); Chivukula et al. (2013)

[TABLE]

with the Wilson coefficient

[TABLE]

Using the requirement $\Lambda>1.1\times 10^{3}$ TeV from Re $C_{K}^{1}$ and $\Lambda>2.2\times 10^{4}$ TeV from Im $C_{K}^{1}$ Bona et al. (2008), we can constrain the maximum size of $\tilde{\lambda}_{11}$, $\tilde{\lambda}_{33}$, assuming the other is vanishing:

[TABLE]

We see that these constraints are easily satisfied for $\mathcal{O}(\text{TeV})$ scale color octets as long as $\tilde{\lambda}\sim\mathcal{O}(0.1)$. Moreover, with this structure, $\tilde{\lambda}_{11}=\tilde{\lambda}_{22}$, the stronger constraints come from $\bar{B}_{d}^{0}-B_{d}^{0}$ and $\bar{B}_{s}^{0}-B_{s}^{0}$ measurements, where $|C_{B_{d}}^{1}|$ requires $\Lambda>1.0\times 10^{3}$ TeV and $|C_{B_{s}}^{1}|$ requires $\Lambda>240$ TeV. Using the tree-level, CKM-induced off-diagonal elements of $\tilde{\lambda}_{D}$, we obtain

[TABLE]

which are again weakened to the sub-TeV scale for $\tilde{\lambda}\sim\mathcal{O}(0.1)$. As a result, we can realistically expect $\mathcal{O}(\text{TeV})$ color octet vectors with dominant branching fractions to either tops, $\tilde{\lambda}_{33}\gg\tilde{\lambda}_{11}$ or jets, $\tilde{\lambda}_{11}\gg\tilde{\lambda}_{33}$, entirely consistent with flavor bounds. We remark that an alternative assumption of the flavor structure can relax these bounds further. In particular, if instead $V_{CKM}^{\dagger}\tilde{\lambda}V_{CKM}$ is diagonal, then the tree-level flavor violating couplings are shifted to the LH up-quark couplings Bai et al. (2011); Haisch and Westhoff (2011). We can define $\tilde{\lambda}^{\prime}=V_{CKM}^{\dagger}\tilde{\lambda}V_{CKM}$ as diagonal, and then the LH up-quark couples via $V_{CKM}\tilde{\lambda}^{\prime}V_{CKM}^{\dagger}$, which is constrained by $D_{0}-\bar{D}_{0}$ mixing, where $\Lambda>700$ TeV Bona et al. (2008). The corresponding bound is

[TABLE]

The $\Delta F=1$ constraints, including $b\to X_{s}\gamma$ and $b\to X_{s}g$ penguin amplitudes, are also immediately satisfied if the $X^{\mu}$ couplings are aligned in the down-quark sector Chivukula et al. (2013); Haisch and Westhoff (2011).

II.4 Collider phenomenology of color octet vectors

Color octet vectors can be singly and doubly produced at colliders. The single-production rate scales with the corresponding partial width into $q\bar{q}$, $q=d,u,s,c$, since single production via gluons is forbidden at tree-level. In our scenario, the induced flavor-violating couplings $\tilde{\lambda}_{D}$ are at most $5\%\times\tilde{\lambda}_{11},\tilde{\lambda}_{33}$, which we will neglect and hence approximate $\tilde{\lambda}_{D}\approx\tilde{\lambda}$. The partial width of $X$ to a pair of quarks $q$ is

[TABLE]

where $g_{L}$ and $g_{R}$ are the corresponding diagonal entries in $\tilde{\lambda}$, $\tilde{\kappa}$, and $\tilde{\eta}$111For reference, the generalized flavor violating partial width is

$\displaystyle\Gamma(X\to\bar{q}_{i}q_{j},\bar{q}_{j}q_{i})$ $\displaystyle=\dfrac{\alpha_{s}m_{X}}{6}\left(((g_{L}^{ij})^{2}+(g_{R}^{ij})^{2})(1-\frac{1}{2}(\frac{m_{i}^{2}}{m_{X}^{2}}+\frac{m_{j}^{2}}{m_{X}^{2}})-\frac{1}{2}(\frac{m_{i}^{2}}{m_{X}^{2}}-\frac{m_{j}^{2}}{m_{X}^{2}})^{2})+6\frac{m_{i}m_{j}}{m_{X}^{2}}g_{L}^{ij}g_{R}^{ij}\right)$

$\displaystyle\times\left((1-\frac{m_{i}^{2}}{m_{X}^{2}}-\frac{m_{j}^{2}}{m_{X}^{2}})^{2}-4\frac{m_{i}^{2}m_{j}^{2}}{m_{X}^{4}}\right)^{1/2}\ ,$

(21)

where $g_{L}^{ij}=g_{L}^{ji}$ and $g_{R}^{ij}=g_{R}^{ji}$.. If the masses of the quarks can be neglected compared to the $X$ mass, the partial width simplifies to

[TABLE]

The branching fractions for $jj$, $b\bar{b}$, and $t\bar{t}$ final states are then simply ratios of the corresponding sum of squared couplings. We see that large branching fractions to top quarks can easily be realized by increasing $\tilde{\kappa}_{33}$, while large branching fractions to bottom quarks corresponds to increasing $\tilde{\eta}_{33}$. This is the effective description of the warped extra dimension scenario with $t_{R}$ and $b_{R}$ wavefunction profiles peaked close to the infrared brane Agashe et al. (2013), and small hierarchies in these third generation couplings are consistent with electroweak precision tests Bai et al. (2011); Haisch and Westhoff (2011).

As previously highlighted, the single dijet resonance constraints scale with the overal partial width into light flavor quarks. Pair-production rates, however, are robustly calculable knowing only $m_{X}$ and its color octet representation. Hence, all searches in pair-production modes provide important complementary reach compared to single resonance searches. The current LHC analyses focus on the simplest topologies, with $XX^{*}\to(JJ)(JJ)$, with $J=j$ or $b$ inclusively Chatrchyan et al. (2013a); Khachatryan et al. (2015); Collaboration (2016); Aad et al. (2013a); collaboration (2016a, 2017) and $XX^{*}\to 4t$ Collaboration (2013); Sirunyan et al. (2017); ATL (2012); Aad et al. (2015a); collaboration (2016b, c, d). In the case where the $X\to b\bar{b}$ decay width is preferred, additional signal discrimination is easily gained by requiring $b$-tags. The orthogonal $(b\bar{b})(b\bar{b})$, $(b\bar{b})(jj)$, and $(jj)(jj)$ signal regions would then have enhanced and complementary sensitivity compared to the current $(JJ)(JJ)$ searches. If the coupling to tops is preferred, then the $4t$ search and our proposed searches in the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ mixed decay channels are critical. In particular, the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ mixed decay searches offer substantial improvements in covering the sensitivity gap when the $X$ decay widths to tops and light quarks are comparable. In the next section, we describe our collider analyses optimized for the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ final states.

III Collider analyses of the mixed channels, $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$

We analyze pair-produced resonances in a new mixed decay mode, $pp\to XX^{*}\to t\bar{t}(jj)$ and $pp\to XX^{*}\to t\bar{t}(b\bar{b})$. Although there are numerous possibilities for the top decays, we mainly focus on the semi-leptonic and fully leptonic final states, which provide clean handles for tagging the reconstructible $t\bar{t}$ and dijet systems. Of course, the main SM background is the irreducible $t\bar{t}+$ jets background, where other single boson and diboson + jets backgrounds are subleading after requiring multiple $b$-jets.

We construct a FeynRules v2.3.26 Alloul et al. (2014) model using Universal FeynRules Output Degrande et al. (2012) to perform leading order Monte Carlo event simulation with MadGraph 5 v2.4.3 Alwall et al. (2014), interfaced with Pythia v8.2 Sjostrand et al. (2008); Sjöstrand et al. (2015) for showering and hadronizations. We remark that the leading order calculation allows a direct comparison to existing $(JJ)(JJ)$ and $4t$ search limits Chatrchyan et al. (2013a); Khachatryan et al. (2015); Collaboration (2016); Aad et al. (2013a); collaboration (2016a, 2017); Collaboration (2013); Sirunyan et al. (2017); ATL (2012); Aad et al. (2015a); collaboration (2016b, c, d). Since the signal will rely on tagging two top candidates, we simulate the SM background, $t\bar{t}j$ with up to two additional jets at leading order, using Sherpa v.2.1.0 Gleisberg et al. (2009). We rescale the background by a flat $K$-factor of 1.5, adopted from Sherpa+BlackHat Gleisberg et al. (2009); Berger et al. (2008). The inclusive SM $t\bar{t}$ + jets production cross section is known at next-to-next-to-leading order and next-to-next-to-leading log (NNLL) precision Czakon et al. (2013), whereas differential cross sections can be obtained at next-to-leading order and NNLL precision Guzzi et al. (2015). Since our $X$ resonance should appear as a resonant peak over a smooth continuum background, a $p_{T}$-dependent $K$-factor should not significantly affect our projected results.

III.1 Event selection strategy

We first discuss the semi-leptonic top decay channel, where our signal process is $XX^{*}\to b\bar{b}\ell^{\pm}\nu jj(JJ)$, with $J=j$ or $b$. Jets are clustered using the anti-$k_{T}$ algorithm Cacciari et al. (2008) with $R=0.5$, and we require jets to have $p_{T}>50$ GeV and $|\eta|<4.9$. We identify $b$-jets using a Delphes v3.1.2 de Favereau et al. (2014) detector simulation, with a $p_{T}$- and $\eta$-dependent tagging efficiency of about $70\%$ for displaced tracks from $B$-mesons within $\Delta R<0.3$ from the main jet axis; the charm misidentification rate is roughly $15\%$ and the light flavor mistag rate is $0.1\%$.

Events must have at least 5 jets, at least two of which must be $b$-tagged and one must be untagged. The leading $b$-tagged and untagged jets must have $p_{T}\geq 250$ GeV, and the subleading untagged jet, if present, must have $p_{T}\geq 80$ GeV. Events must also have exactly one isolated lepton with $p_{T}>20$ GeV and $|\eta|<2.5$, with isolation using Delphes default parameters. We furthermore cut on missing transverse energy (MET), requiring $\not{E}_{T}>80$ GeV, as well as signal mass-optimized $H_{T}=\Sigma_{j}|p_{T,j}|\geq(4/3)\,m_{X}$.

Naively, the $X\to JJ$ signal resonance is extracted from forming the invariant mass of the two leading jets. The jet combinatorics, however, present a major hurdle against using the dijet and ditop masses to discriminate the signal from the irreducible background. Thus, the main goal of our collider strategy is to solve the combinatorial ambiguity, taking advantage of the resonant high mass dijet signal, the large $p_{T}$ of the resonant dijet system, and the relevant angular spread between the $b$-jets, lepton, and jets. Difficulties in resolving multijet combinatorial ambiguities are discussed in, e.g., Rajaraman and Yu (2011).

For this purpose, we define two signal regions. We target the $b\bar{b}$ mode by selecting events with more than two $b$-tagged jets, while the $jj$ mode is optimized by requiring exactly two $b$-tagged jets. For the $b\bar{b}$ decay, we assume that the leading $b$-tagged jet comes from the $X\to JJ$ decay directly, not from $t\bar{t}$. We then reconstruct the $b\bar{b}$ system by adding the remaining leading jet, whether tagged or untagged. We find that this reconstruction best reflects the dijet resonance in spite of the underlying combinatorial ambiguity.

For the $jj$ mode, we start with the leading light jet $j_{1}$ and add to it the hardest light jet $j_{2}$ which satisfies $\Delta R_{j_{1}j_{2}}\leq\pi$. We also add in the next hardest light jet $j_{3}$ with $\Delta R_{j_{1}j_{3}}\leq\pi$ if $j_{1}$ and $j_{2}$ are not balanced in $p_{T}$, when $p_{T,j_{2}}/p_{T,j_{1}}\leq 0.15$. This follows the hemisphere intuition, where the $X\to jj$ decays should be untagged and relatively hard. Adding the third hardest light jet accounts for the wide-angle final state radiation of our signal quarks.

For both $b\bar{b}$ and $jj$ decay modes, we construct the dijet invariant mass as our final kinematic discriminant. The signal and background cut efficiencies for the semileptonic analysis are shown in Table 1, while the most salient kinematic distributions for the background stacked with different signal hypotheses are shown in Fig. 1. The upper left panel in Fig. 1 shows the hardening of the $H_{T}$ distribution coming from the signal jets. The dijet $p_{T,JJ}$ distribution, however, only offers an overal rate shift from the additional signal events and no strong correlation with the signal $X$ mass, which is because of the combinatorial ambiguity among the jets. The bottom panels show the invariant mass distributions in the $jj$ and $b\bar{b}$ targetted modes. Again, the broad peak structure arises mostly from the combinatorial ambiguity in capturing the correct signal jets to reconstruct the resonance. We note that $b$-tagging efficiency and the combinatorial ambiguity also cause $X\to jj$ events to populate the $m_{bb}$ signal region and vice versa.

The search in the fully leptonic top quark decay channel is very similar. We again select jets and leptons as in the semi-leptonic channel, but we require only 4 jets, at least two of which are $b$-tagged, and also require exactly two isolated leptons of opposite charge. We loosen the MET cut, $\not{E}_{T}>50$ GeV, and keep the $H_{T}$ cut, $H_{T}\geq(4/3)\,m_{X}$. To avoid the anticipated $Z+$jets background we veto events with $m_{\ell\ell}\leq 115$ GeV. Again we distinguish between $b\bar{b}$ and $jj$ dijet resonance decay modes, using the same method as the semi-leptonic search. The corresponding cut flow is presented in Table 2. The final signal rates lead to the same relative signal to background ratios as the semi-leptonic analysis, but the absolute rate is only an $\approx 5\%$ additional contribution compared to the semi-leptonic channel.

III.2 Comparison with current searches

We now compare the projected sensitivity from the mixed $t\bar{t}(JJ)$ searches in combination with the recasted exclusions from ATLAS and CMS for $(JJ)(JJ)$ and $4t$ searches. We also show the single production limits for dijet and ditop resonance searches. Since we assume that our new physics state $X$ only decays to quark pairs, we can express the pair production constraints in the branching fraction vs. mass plane, where

[TABLE]

So, the pair production limits can be translated according to:

[TABLE]

with $\text{Br}_{JJ}=\text{Br}(X\to JJ)$. The $4t$ constraint and the $t\bar{t}(JJ)$ projected exclusion after replacing $(\text{Br}_{JJ})^{2}$ with $(\text{Br}_{tt})^{2}$ and $2(\text{Br}_{JJ}\ \text{Br}_{tt})$, respectively, where $\text{Br}_{tt}=\text{ Br}(X\to t\bar{t})$.

The single dijet resonance limits are determined only after specifying the total width of the resonance. If the total width is narrow, a reference cross section can be rescaled by the partial width into light quarks. We see that

[TABLE]

In the case of $t\bar{t}$ resonances, the above constraint becomes a bounded requirement on the branching fraction to tops. Hence, the $t\bar{t}$ resonance constraint follows

[TABLE]

Note that here we drop the acceptance factor since the experiments unfold the acceptance when presenting their results. We combine all four orthogonal signal regions of the semi-leptonic and fully leptonic search and compute the $95\%$ C.L. limits based on the respective $JJ$ invariant mass shapes, where we assume a 10% systematic uncertainty on signal and background. The result can be found in Fig. 2.

We remark that the width of our resonance, $\Gamma_{X}=5\times 10^{-4}m_{X}$, corresponds to diagonal entries of $\tilde{\lambda}$, $\tilde{\kappa}$, $\tilde{\eta}\lesssim 0.1$. As discussed in Subsec. II.3, such LH quark couplings to $X$ readily satisfy flavor violation bounds from meson oscillation measurements and neutral current transitions given $m_{X}>1$ TeV. On the other hand, these couplings generally arise from mixing the SM quarks with heavy vectorlike states, since otherwise perturbativity in the parent extended color gauge symmetry is violated Dobrescu and Yu (2013). The corresponding flavor violation bounds, collider constraints, electroweak precision observable tests are more model-dependent, but realistic and complete scenarios can be constructed Bai et al. (2011); Haisch and Westhoff (2011); Chivukula et al. (2013).

As mentioned previously, the sensitivity from single production compared to the sensitivity from pair production strongly depends on the ratio $\Gamma_{X}/m_{X}$. Any direct resonance bounds from $JJ$ searches could in principle be evaded completely with a suitably small choice of $\Gamma_{X}$, but choosing $\Gamma_{X}/m_{X}=5\times 10^{-4}$ allows complementary dijet constraints from CMS and ATLAS Collaboration (2017); Aaboud et al. (2017). We remark that the $t\bar{t}$ resonance limits collaboration (2016e) are absent from Fig. 2 for $\Gamma_{X}/m_{X}=5\times 10^{-4}$. These constraints are only relevant once $\Gamma_{X}/m_{X}\gtrsim 7\times 10^{-4}$ for $m_{X}$ around 1 TeV. As shown in Eq. (26), the $t\bar{t}$ resonance limit is symmetric around $\text{Br}_{tt}=50\%$ since the maximum rate in this channel corresponds to equal partial widths to dijets and ditops.

The strongest existing bounds in Fig. 2 are therefore from $(JJ)(JJ)$ collaboration (2017) and $4t$ collaboration (2016d) searches, which are clearly optimal for their respective $\text{Br}_{JJ}$ and $\text{Br}_{tt}$ corners. The mass reach of both searches weakens by about 250 GeV in the intermediate regime, however, leaving significant room for our dedicated $t\bar{t}(JJ)$ search to explore.

In Fig. 2, we assume that all quark couplings are flavour universal except for the top. We relax this assumption in Fig. 3, allowing both branching fractions $\text{Br}(X\to b\bar{b})=\text{ Br}_{bb}$ and $\text{Br}_{tt}$ to float. The results are presented in an equilateral triangle since the sum of the $jj$, $b\bar{b}$ and $t\bar{t}$ branching fractions must equal 100% in our model. The shading in the left panel of Fig. 3 shows the lower limit on the resonance mass $m_{X}$ as a function of the three branching fractions. The right panel of Fig. 3 indicates which particular dedicated search yields the corresponding lower mass limit. We see that our new search channels, $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$, overtake the sensitivity in the central areas of the triangle compared to the existing $(JJ)(JJ)$ and $4t$ searches.

We reiterate that a complete characterization of $t\bar{t}$, $b\bar{b}$, and $jj$ decay channels for pair production colored resonances would necessitate optimizing the current $(JJ)(JJ)$ search into $(b\bar{b})(b\bar{b})$, $(b\bar{b})(jj)$, and $(jj)(jj)$ signal regions. In particular, the $(b\bar{b})(b\bar{b})$ search would bear striking similarities with the current searches for pair production of the 125 GeV Higgs Aaboud et al. (2016); Collaboration (2016), where the main novelty would be varying the $(b\bar{b})$ mass window to test for new resonances. The multijet background, however, is very challenging to simulate and thus substantial statistics in the $b$-tagged backgrounds would be required to suitably smooth paired invariant mass spectrum in the $(b\bar{b})(b\bar{b})$ and $(b\bar{b})(jj)$ channels.

IV Conclusion

In this work, we have highlighted the $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ mixed decay channels of a massive, color octet vector as new targets for ATLAS and CMS searches. Hierarchies in the underlying couplings of the $X$ resonance to light quarks, bottom pairs, or top pairs are entirely consistent with low energy FCNC constraints if the $X$ mass is above $1$ TeV and its couplings to quarks are at most $0.1$. As a result, the LHC provides the leading reach to TeV-scale color octet vectors with variable couplings to heavy and light flavor quarks by virtue of the model independent pair production rate.

In our $t\bar{t}(JJ)$ analyses, we focused on resolving the jet combinatorial ambiguity to reconstruct the dijet or dibottom resonance. In principle, reconstructing the $(t\bar{t})$ resonance is also possible, but our scenario with its many resolved jets did not afford any additional signal discrimination in this regard.

Nevertheless, our results show that new $t\bar{t}(jj)$ and $t\bar{t}(b\bar{b})$ searches will fill a sensitivity gap between the $(JJ)(JJ)$ and $4t$ searches. This gap is clear in the $\text{Br}_{JJ}$ vs. $m_{X}$ plane, which itself provides a useful tool for easily presenting the results from different collider searches of pair-produced resonances. We stress that a post-discovery scenario of a new resonance greatly benefits from this complementary information, where single and pair production modes combined with different decay channels provide direct information about underlying Lagrangian couplings.

Acknowledgments

FY would like to thank R. Sekhar Chivukula and Elizabeth Simmons for helpful discussions during the 2015 Les Houches summer session, and also Susanne Westhoff at the Mainz Institute for Theoretical Physics, “The TeV Scale: A Threshold to New Physics?” 2017 scientific program. FY would also like to acknowledge the hospitality of the theory group at Fermi National Accelerator Laboratory while this work was completed. This research is supported by the Cluster of Excellence Precision Physics, Fundamental Interactions and Structure of Matter (PRISMA-EXC 1098). The work of MB is moreover supported by the German Research Foundation (DFG) in the framework of the Research Unit “New Physics at the Large Hadron Collider” (FOR 2239).