Flavor changing neutral current decays $t\to c X$ ($X=\gamma,\,g,\, Z,\, H$) and $t\to c\bar \ell\ell $ ($\ell=\mu,\,\tau$) via scalar leptoquarks

TL;DR

This paper investigates rare top quark decays involving flavor-changing neutral currents mediated by scalar leptoquarks, providing analytical calculations and exploring parameter space constraints to estimate possible branching ratios.

Contribution

It introduces a renormalizable scalar leptoquark model with detailed analytical results for decay contributions and explores parameter space constraints from current experimental data.

Findings

Branching ratios for $t o c X$ can reach up to $10^{-8}$.

Branching ratios for $t o car au au$ can be up to $10^{-6}$.

Branching ratios for $t o car au au$ and $t o car \muar \mu$ are significantly enhanced in certain parameter regions.

Abstract

The flavor changing neutral current decays $t\to c X$ ($X=\gamma,\,g,\, Z,\, H$) and $t\to c\bar \ell\ell $ ($\ell=\mu,\,\tau$) are studied in a renormalizable scalar leptoquark (LQ) model with no proton decay, where a scalar $SU(2)$ doublet with hypercharge $Y=7/6$ is added to the standard model, yielding a non-chiral LQ $\Omega_{5/3}$. Analytical results for the one-loop (tree-level) contributions of a scalar LQ to the $f_i\to f_j X$ ($f_i\to f_j \bar f_m f_l$) decays, with $f_a=q_a, \ell_a$, are presented. We consider the scenario where $\Omega_{5/3}$ couples to the fermions of the second and third families, with its right- and left-handed couplings obeying $\lambda_R^{\ell u_i}/\lambda_L^{\ell u_i}=O(\epsilon)$, where $\epsilon$ parametrizes the relative size between these couplings. The allowed parameter space is then found via the current constraints on the muon $(g-2)$, the $\tau\to \mu\gamma$ decay, the LHC Higgs boson data, and the direct LQ searches at the LHC. For $m_{\Omega_{5/3}}=1$ TeV and $\epsilon=10^{-3}$, we find that the $t\to c X$ branching ratios are of similar size and can be as large as $10^{-8}$ in a tiny area of the parameter space, whereas ${\rm Br}(t\to c\bar \tau\tau)$ [${\rm Br}(t\to c\bar \mu\mu)$] can be up to $10^{-6}$ ($10^{-7}$).

Tables

Table 1. Table 1: Branching ratios of the t → c ​ X → 𝑡 𝑐 𝑋 t\to cX decays for a few ( λ L μ ​ c , λ L μ ​ t , λ L τ ​ c ​ λ L τ ​ t ) superscript subscript 𝜆 𝐿 𝜇 𝑐 superscript subscript 𝜆 𝐿 𝜇 𝑡 superscript subscript 𝜆 𝐿 𝜏 𝑐 superscript subscript 𝜆 𝐿 𝜏 𝑡 (\lambda_{L}^{\mu c},\lambda_{L}^{\mu t},\lambda_{L}^{\tau c}\lambda_{L}^{\tau t}) points inside the area allowed by the Δ ​ a μ Δ subscript 𝑎 𝜇 \Delta a_{\mu} discrepancy and the experimental bound on the τ → μ ​ γ → 𝜏 𝜇 𝛾 \tau\to\mu\gamma decay. We consider the scenario where λ R ℓ ​ u / λ L ℓ ​ u i = O ​ ( ϵ ) superscript subscript 𝜆 𝑅 ℓ 𝑢 superscript subscript 𝜆 𝐿 ℓ subscript 𝑢 𝑖 𝑂 italic-ϵ \lambda_{R}^{\ell u}/\lambda_{L}^{\ell u_{i}}=O(\epsilon) for ϵ = 10 − 3 italic-ϵ superscript 10 3 \epsilon=10^{-3} and 10 − 1 superscript 10 1 10^{-1} . For the coupling of the Higgs boson to a LQ pair we use λ Ω 5 / 3 = 1 subscript 𝜆 subscript Ω 5 3 1 \lambda_{\Omega_{5/3}}=1 . The t → c ​ X → 𝑡 𝑐 𝑋 t\to cX branching ratios are given in units of 10 − 8 superscript 10 8 10^{-8} for ϵ = 10 − 3 italic-ϵ superscript 10 3 \epsilon=10^{-3} and 10 − 9 superscript 10 9 10^{-9} for ϵ = 10 − 1 italic-ϵ superscript 10 1 \epsilon=10^{-1} .

${\lambda }_R^{\ell u}/{\lambda }_L^{\ell u_i}=O\left({10}^{-3}\right)$
$m_{{\Omega }_{5/3}}$ [GeV]	${\lambda }_L^{\mu c}$	${\lambda }_L^{\mu t}$	${\lambda }_L^{\tau c}$	${\lambda }_L^{\tau t}$	$\gamma$	$g$	$Z$	$H$
1000	$2.54\times {10}^{-4}$	$7.8\times {10}^{-1}$	$2.39\times {10}^{-1}$	$9.48$	$0.416$	$0.665$	$1.02$	$0.68$
	$1.48\times {10}^{-4}$	$6.54\times {10}^{-1}$	$2.06\times {10}^{-1}$	$7.87$	$0.212$	$0.339$	$0.521$	$0.347$
	$2.52\times {10}^{-5}$	$1.02$	$1.72\times {10}^{-1}$	$7.86$	$0.147$	$0.236$	$0.362$	$0.241$
	$1.25\times {10}^{-5}$	$7.68\times {10}^{-1}$	$1.28\times {10}^{-1}$	$9.97$	$0.132$	$0.212$	$0.325$	$0.216$
	$8.47\times {10}^{-5}$	$6.48\times {10}^{-1}$	$2.54\times {10}^{-1}$	$4.97$	$0.129$	$0.207$	$0.318$	$0.211$
1500	$1.31\times {10}^{-3}$	$1.$	$3.49\times {10}^{-1}$	$6.52$	$0.0819$	$0.131$	$0.246$	$0.135$
	$3.45\times {10}^{-4}$	$9.21\times {10}^{-1}$	$2.56\times {10}^{-1}$	$8.46$	$0.0741$	$0.119$	$0.222$	$0.122$
	$1.02\times {10}^{-5}$	$1.04$	$2.58\times {10}^{-1}$	$8.04$	$0.0678$	$0.109$	$0.203$	$0.111$
	$2.17\times {10}^{-6}$	$1.21$	$2.89\times {10}^{-1}$	$6.89$	$0.0624$	$0.1$	$0.187$	$0.103$
	$1.65\times {10}^{-4}$	$1.13$	$2.5\times {10}^{-1}$	$6.84$	$0.0462$	$0.074$	$0.138$	$0.0758$
2000	$2.16\times {10}^{-5}$	$1.1$	$4.48\times {10}^{-1}$	$7.39$	$0.0543$	$0.0872$	$0.186$	$0.0894$
	$7.69\times {10}^{-6}$	$1.29$	$3.29\times {10}^{-1}$	$9.86$	$0.0524$	$0.0841$	$0.179$	$0.0862$
	$2.59\times {10}^{-3}$	$1.07$	$3.69\times {10}^{-1}$	$8.71$	$0.0512$	$0.082$	$0.175$	$0.0843$
	$6.09\times {10}^{-3}$	$1.07$	$3.05\times {10}^{-1}$	$8.34$	$0.0323$	$0.0514$	$0.111$	$0.0532$
	$3.9\times {10}^{-4}$	$1.69$	$2.73\times {10}^{-1}$	$8.98$	$0.0298$	$0.0478$	$0.102$	$0.049$
${\lambda }_R^{\ell u}/{\lambda }_L^{\ell u_i}=O\left({10}^{-1}\right)$
$m_{{\Omega }_{5/3}}$ [GeV]	${\lambda }_L^{\mu c}$	${\lambda }_L^{\mu t}$	${\lambda }_L^{\tau c}$	${\lambda }_L^{\tau t}$	$\gamma$	$g$	$Z$	$H$
1000	$2.54\times {10}^{-4}$	$7.8\times {10}^{-1}$	$2.39\times {10}^{-1}$	$9.48$	$4.51$	$6.73$	$10.2$	$6.92$
	$1.48\times {10}^{-4}$	$6.54\times {10}^{-1}$	$2.06\times {10}^{-1}$	$7.87$	$2.3$	$3.43$	$5.21$	$3.53$
	$2.52\times {10}^{-5}$	$1.02$	$1.72\times {10}^{-1}$	$7.86$	$1.6$	$2.39$	$3.62$	$2.45$
	$1.25\times {10}^{-5}$	$7.68\times {10}^{-1}$	$1.28\times {10}^{-1}$	$9.97$	$1.44$	$2.14$	$3.25$	$2.2$
	$8.47\times {10}^{-5}$	$6.48\times {10}^{-1}$	$2.54\times {10}^{-1}$	$4.97$	$1.4$	$2.09$	$3.17$	$2.15$
1500	$1.31\times {10}^{-3}$	$1.$	$3.49\times {10}^{-1}$	$6.52$	$0.893$	$1.33$	$2.45$	$1.37$
	$3.45\times {10}^{-4}$	$9.21\times {10}^{-1}$	$2.56\times {10}^{-1}$	$8.46$	$0.809$	$1.2$	$2.22$	$1.24$
	$1.02\times {10}^{-5}$	$1.04$	$2.58\times {10}^{-1}$	$8.04$	$0.739$	$1.1$	$2.03$	$1.13$
	$2.17\times {10}^{-6}$	$1.21$	$2.89\times {10}^{-1}$	$6.89$	$0.681$	$1.01$	$1.87$	$1.04$
	$1.65\times {10}^{-4}$	$1.13$	$2.5\times {10}^{-1}$	$6.84$	$0.504$	$0.749$	$1.38$	$0.772$
2000	$2.16\times {10}^{-5}$	$1.1$	$4.48\times {10}^{-1}$	$7.39$	$0.595$	$0.883$	$1.86$	$0.911$
	$7.69\times {10}^{-6}$	$1.29$	$3.29\times {10}^{-1}$	$9.86$	$0.574$	$0.851$	$1.79$	$0.878$
	$2.59\times {10}^{-3}$	$1.07$	$3.69\times {10}^{-1}$	$8.71$	$0.561$	$0.83$	$1.75$	$0.859$
	$6.09\times {10}^{-3}$	$1.07$	$3.05\times {10}^{-1}$	$8.34$	$0.354$	$0.52$	$1.11$	$0.542$
	$3.9\times {10}^{-4}$	$1.69$	$2.73\times {10}^{-1}$	$8.98$	$0.326$	$0.484$	$1.02$	$0.5$

Equations

$${\mathrm{Br}}^{\mathrm{SM}}(t\to cg)$$

$${\mathrm{Br}}^{\mathrm{SM}}(t\to c\gamma )$$

$${\mathrm{Br}}^{\mathrm{SM}}(t\to cZ)$$

$${\mathrm{Br}}^{\mathrm{SM}}(t\to cH)$$

$${\mathcal{L}}_{F=0}=h_{2L}^{ij}{R}_2^T{\bar{u}}_R^{i}i{\tau }_2{L}_L^j+h_{2R}^{ij}{\bar{Q}}_L^i{e}_R^j{R}_2+H.c.,$$

$${\mathcal{L}}_{F=0}={\bar{e}}^i\left({\lambda }_L^{ij}{P}_L+{\lambda }_R^{ij}{P}_R\right)u^j{\Omega }_{5/3}^{\ast }+{\bar{e}}^i{\eta }_R^{ij}{P}_R{d}^j{\Omega }_{2/3}^{\ast }+H.c.,$$

$${\mathcal{L}}_S=\frac{1}{2}(D_{\mu }{R}_2{)}^{†}{D}_{\mu }{R}_2,$$

$$D_{\mu }{R}_2=\left({\partial }_{\mu }+ig\frac{{\tau }^i}{2}{W}^i+i{g}^{\prime }\frac{7}{6}{B}^{\mu }\right)R_2.$$

$${\mathcal{L}}_S$$

$$\mathcal{L}=\left(M_{R_2}^2+{\lambda }_{R_2}{\Phi }^{†}\Phi \right)\left(R_2^{†}{R}_2\right),$$

$$\mathcal{L}\supset {\lambda }_{{\Omega }_{5/3}}vH{\Omega }_{5/3}^{\ast }{\Omega }_{5/3}.$$

$$\bar{f}f{A}_{\mu }$$

$$\bar{f}f{Z}_{\mu }$$

$$\mathcal{M}(f_i\to f_{j}V)$$

$$\Gamma (f_i\to f_{j}V)$$

$$+g_{ij}\left(|L^{{}^{\prime }V}{|}^2+|R^{{}^{\prime }V}{|}^2\right)$$

$$+3\left(m_j^2-m_i^2+m_V^2\right)\left(L^{{}^{\prime }V}{R}^{V\ast }+L^V{R}^{{}^{\prime }V\ast }\right)$$

$$+\frac{3m_j}{m_i}\left(m_i^2-m_j^2+m_V^2\right)\left(L^V{L}^{{}^{\prime }V\ast }+R^V{R}^{{}^{\prime }V\ast }\right)$$

$$-\frac{3m_j{m}_V^2}{m_i}\mathrm{Re}\left(L^V{R}^{V\ast }\right)$$

$$\displaystyle-12m_{i}m_{j}{\rm Re}\left(L^{{}^{\prime}V}R^{{}^{\prime}V*}\right)\Bigg{)},$$

$$\lambda (x,y,z)=x^2+y^2+z^2-2(xy+xz+yz).$$

$$\Gamma (f_i\to f_j\gamma )$$

$$\mathcal{M}(q_i\to q_{j}g)$$

$$\times ϵ(q{)}_{\mu },$$

$$\mathcal{M}(f_i\to f_{j}H)={\bar{f}}_j\left(F_L{P}_L+F_R{P}_R\right)f_i,$$

$$\Gamma (f_i\to f_{j}H)$$

$$\displaystyle+2m_{i}m_{j}{\rm Re}\left(F_{L}F_{R}^{*}\right)\Big{)},$$

$$\Gamma (H\to f_i{f}_j)$$

$$\displaystyle-2m_{i}m_{j}{\rm Re}(F_{L}F_{R}^{*})\Big{)},$$

$$\Gamma (f_i\to f_j{\bar{f}}_m{f}_l)=\frac{m_i}{256{\pi }^3}{\int }_{x_{ji}}^{x_{jf}}{\int }_{x_{li}}^{x_{lf}}|\overline{\mathcal{M}}{|}^{2}d{x}_{j}d{x}_l.$$

$$x_{jf}$$

$$x_{ji}$$

$$x_{li,lf}$$

$$|\overline{\mathcal{M}}{|}^2$$

$$p_l\cdot p_m$$

$$p_j\cdot p_m$$

$$p_j\cdot p_l$$

$${\kappa }_i^2=\frac{\Gamma (H\to i)}{{\Gamma }^{\mathrm{SM}}(H\to i)},$$

Full text

††journal: Eur. Phys. J. C

1 Introduction

The conjecture that lepton number is the fourth color quantum number was put forward long ago in the context of an $SU(4)_{R}\times SU(4)_{L}\times SU(4^{\prime})$ theory Pati:1973uk ; Pati:1974yy , which requires the presence of new gauge and scalar bosons carrying both lepton and baryon number. Such particles, dubbed leptoquarks (LQs) since transform leptons into quarks and vice versa, appear naturally in grand unified theories Georgi:1974sy ; Fritzsch:1974nn ; Ramond:1976jg ; Senjanovic:1982ex ; Frampton:1989fu , but they are also predicted in other well motivated theories, such as technicolor Ellis:1980hz ; Farhi:1980xs ; Hill:2002ap , models with composite fermions Schrempp:1984nj ; Buchmuller:1985nn ; Gripaios:2009dq , superstring-inspired ${\rm E}_{6}$ models Witten:1985xc ; Hewett:1988xc , models with extended scalar sectors Davies:1990sc ; Arnold:2013cva , etc. However, LQs with lepton and baryon number violating interactions can give rise to dangerous effects such as large lepton flavor violating interactions (LFV) or tree-level-induced proton decay. The latter implies that, unless an extra symmetry is invoked to forbid diquark couplings, LQ masses must be as heavy as the Planck scale, thereby rendering unobservable effects on low energy processes. Therefore, only those theories with renormalizable lepton and baryon number conserving LQ interactions are phenomenologically appealing. LQ phenomenology and low-energy constraints on the parameter space of the most representative LQ models have been widely discussed (see Davidson:1993qk for instance). For a more up-to-date review on LQ physics we refer the reader to Ref. Dorsner:2016wpm . It turns out that vector LQ masses and couplings are tightly constrained by experimental data, therefore the study of scalar LQs has been favored in the literature. In this work we are interested in a simple renormalizable LQ model with no proton decay where the presence of relatively light scalar LQs can still be compatible with low energy constraints from experimental data Dorsner:2009cu ; Arnold:2013cva ; Cheung:2015yga . In such a model, scalar LQs are introduced in the standard model (SM) via a doublet of $SU(2)$.

In the SM flavor changing neutral currents (FCNCs) can arise up to the one-loop level or higher orders of perturbation theory, but are additionally suppressed by the so-called Glashow–Iliopoulos–Maiani (GIM) mechanism. On the other hand, the SM forbids LFV effects at any order, though experimental evidences hint that neutrinos are massive, thereby implying that LFV effects should be present in nature indeed. No evidences of large FCNCs transitions have yet been experimentally observed, so the search for this class of effects is a must in the physics program of any particle collider. While FCNC transition between fermions of the second and first family are considerably constrained by experimental data, transitions involving the fermions of the third and second generation have no such strong restrictions. In this regard, FCNC top quark transitions stand out among the most widely studied processes at the CERN LHC. This stems from the fact that in spite of having negligible rates in the SM, they can have a considerable enhancement in beyond the SM theories and could be at the reach of detection.

The scalar particle discovered at the LHC in 2012 seems to be consistent with the SM Higgs boson, but several of its properties remain to be tested more accurately, such as its couplings to light fermions. The LHC also offers great potential to search for some exotic Higgs decay channels that are highly suppressed or forbidden in the SM. Along these lines, there has been considerably interest in the study of the LFV Higgs boson decay $H\to\bar{\ell}_{i}\ell_{j}$, which was first studied in DiazCruz:1999xe ; Han:2000jz and has been the focus of great attention recently. Although an apparent excess of the $H\to\tau\mu$ branching ratio, with a significance of $2.4\,\sigma$, was observed at the LHC Run 1 Khachatryan:2015kon , it was not confirmed in Run-2 data. The LFV Higgs boson decay $H\to\bar{\ell}_{i}\ell_{j}$ has been widely studied in several extension models. In particular, in the LQ model of Ref. Cheung:2015yga , the $H\to\tau\mu$ rate is predicted to be at the reach of experimental detection in some regions of the allowed parameter space. On the other hand, in the quark sector, apart from the FCNC Higgs boson decay into light quarks $H\to qq^{\prime}$, the FCNC top quark decay $t\to cH$ has also been widely studied along with other decays such as $t\to c\gamma$, $t\to cZ$ and $t\to cg$.

At hadron colliders, the dominant top quark production mechanisms are gluon fusion and quark annihilation. The former is the main top quark production process at the LHC (about 90 % at $\sqrt{s}=$14 TeV), with a small percentage due to quark annihilation. The millions of yearly top quark events at the LHC would allow experimentalist to search for its FCNCs decays such as $t\to cX$ ($V=\gamma,\,Z,\,H$), whose rates are negligibly small in the SM Eilam:1990zc ; DiazCruz:1989ub ; Mele:1998ag :

[TABLE]

These rates however can be several orders of magnitude larger in several SM extensions, such as two-Higgs doublet models Eilam:1990zc ; Diaz:2001vj , supersymmetric models Couture:1994rr ; Li:1993mg ; Lopez:1997xv ; Yang:1997dk , left-right supersymmetric models Frank:2005vd , extra dimensions GonzalezSprinberg:2007zz , models with an extra neutral gauge boson CorderoCid:2005kp , 331 models Cortes-Maldonado:2013rca , etc. Therefore, any experimental evidence of these FCNCs top quark and Higgs boson decay channels may shed light on the underlying fundamental theory of particle interactions.

In this work we focus on the study of the FCNC decays $t\to cX$ ($X=\gamma,\,g,\,Z,\,H$) in a simple renormalizable scalar LQ model in which there is no proton decay induced via tree-level LQ exchange, where these processes arise at one-loop level at the lowest order of perturbation theory. As a by-product, we present the exact results for the one-loop LQ scalar contribution to the $H\to\bar{f}_{j}f_{i}$ decay width, which follows easily from the $f_{i}\to f_{j}H$ decay width by crossing symmetry. For completeness, we also consider in our study the tree-level FCNC decay $t\to c\bar{\ell}\ell$ ($\ell=\mu,\,\tau$), which in fact can have larger branching ratios than those of the one-loop induced decays.

The rest of this presentation is as follows. In Sec. 2 we briefly discuss the framework of the LQ model we are interested in. Sec. 3 is devoted to present the general calculation of the FCNC decay amplitudes and decay widths. For the one-loop induced decays we express the amplitudes in terms of both Passarino-Veltman scalar functions and Feynman parameter integrals. We present a discussion on the constraints on the LQ couplings from experimental data in Sec. 5, followed by the numerical analysis of the FCNCs Higgs boson transitions in Sec. 6. The conclusions and outlook are presented in Sec. 7. Finally, a few lengthy formulas for the loop integrals are presented in the Appendices.

2 A simple scalar LQ model

Rather than considering a specific theory, a convenient strategy to study the LQ phenomenology is via a model-independent approach through an effective lagrangian. One can thus focus on the low energy LQ interactions and, without loss of generality, disregard the complex framework of the ultraviolet completion, which is not relevant for the phenomenology below the TeV scale. The most general dimension-four $SU(3)_{c}\times SU(2)_{L}\times U(1)_{Y}$-invariant effective interactions of scalar and vector LQs, respecting both lepton and baryon number was first presented in Buchmuller:1986zs and has been analyzed recently in Dorsner:2016wpm . In this work we consider a simple renormalizable LQ model in which it is not necessary to invoke an extra symmetry to forbid the proton decay. A single $SU(2)$ doublet with hypercharge $7/6$ is added to the SM, giving rise to two LQs with electric charges $5/3e$ and $2/3e$. The former one is a non-chiral LQ that couples to up quarks and charged leptons, thereby giving rise to FCNC top quark and Higgs boson decays at the one-loop level, but also to the $t\to c\bar{\ell}\ell$ decay at the tree-level. The phenomenology of this model was studied in Arnold:2013cva and bounds on its couplings to a lepton-quark pairs from the experimental constraints on the muon anomalous magnetic dipole moment and the LFV tau decay $\tau\to\mu\gamma$ were obtained in Bolanos:2013tda . We first start by discussing the corresponding LQ couplings to quarks and leptons and afterwards we discuss the remaining interactions.

In the model we are interested in, a scalar LQ representation $R_{2}$ with $SU(3)\times SU(2)\times U(1)$ quantum numbers $(3,2,7/6)$ is introduced. This LQ doublet has the following renormalizable zero-fermion-number interactions Buchmuller:1986zs

[TABLE]

where ${L_{L}^{i}}$ and ${Q_{L}^{i}}$ are $SU(2)_{L}$ left-handed lepton and quark doublets, whereas $e^{i}_{R}$ and $q^{i}_{R}$ are singlets, with $i$ and $j$ being generation indices.

After rotating to the LQ mass eigenstates $\Omega_{5/3}$ and $\Omega_{2/3}$, where the subscript denotes the electric charge in units of $e$, we obtain the following interaction Lagrangian

[TABLE]

where $P_{L,R}$ are the chiral projection operators. We are interested in the effects of the non-chiral LQ $\Omega_{5/3}$ on the FCNC decays of the top quark and the Higgs boson. Since there are stringent constraints on the LQ couplings to the fermions of the two first families, in our study below we will consider that $\Omega_{5/3}$ only couples to the second and third generation fermions.

Apart from the LQ interaction to up quarks and charged lepton pairs, which follow easily from the above expression, for our calculation we also need the LQ couplings to both the photon and the $Z$ gauge boson, which are extracted from the LQ kinetic terms:

[TABLE]

where the $SU(2)_{L}\times U(1)_{Y}$ covariant derivative is given by

[TABLE]

Therefore, in the mass eigenstate basis we have

[TABLE]

where $g_{Z\Omega_{5/3}\Omega_{5/3}}=1/2-5/3s_{W}^{2}$.

Finally, we consider the following renormalizable effective LQ interactions to the SM Higgs doublet $\Phi$

[TABLE]

where $M_{R_{2}}$ is the LQ mass. From here we obtain the Higgs boson coupling to $\Omega_{5/3}$:

[TABLE]

For easy reference, we also present the SM Feynman rules for the interaction of the photon and the $Z$ gauge boson with a fermion-antifermion pair:

[TABLE]

where $g_{L}^{f}=2T^{3}_{f}-2Q_{f}s_{W}^{2}$ and $g_{R}^{f}=-2Q_{f}s_{W}^{2}$, with $T^{3}_{f}=1/2(-1/2)$ for up (down) fermions and $Q_{f}$ the fermion charge in units of that of the positron.

The corresponding Feynman rules follow straightforwardly from the above Lagrangians and are shown in Fig. 1. Below we present the calculation of the FCNC $t\to cX$ and $t\to c\bar{\ell}\ell$ decays.

3 LQ contribution to the FCNC $t\to cX$ decays

We now discuss the calculation of the FCNC $t\to cX$ decays, which in our scalar LQ model proceed at the one-loop level at the lowest order of perturbation theory. For the sake of completeness we present the most general expressions for the $f_{i}\to f_{j}X$ decays with $X=\gamma,\,Z,\,H$ and $f_{i,j}$ quarks or leptons. From our result for the $q_{i}\to q_{j}\gamma$ decay, that for the $q_{i}\to q_{j}g$ decay will follow easily as discussed below.

For the calculation of the loop integrals, we use both the Feynman parameter technique and the Passarino-Veltman reduction scheme, which allows one to cross-check the results. For the algebra we used the Mathematica software routines along with the FeynCalc package Shtabovenko:2016sxi . It is worth mentioning that our results are also valid for the contribution of the LQ singlet $\chi_{1/3}$ of the model of Ref. Cheung:2015yga , where the LQ contribution to the $H\to\mu\tau$ was discussed. The Feynman rules for such a LQ are of Majorana-like type (there are two fermion-flow arrows clashing into a vertex as shown in Fig. 1) and they require a special treatment. We have followed the approach of Ref. Moore:1984eg and found that the results for the contribution of LQ $\Omega_{5/3}$ to the $f_{i}\to f_{j}X$ decays are also valid for the contributions of LQ $\chi_{1/3}$ after replacing the respective coupling constants. A similar result was found for the contribution of single and doubly charged scalars to the muon anomalous MDM Moore:1984eg .

3.1 $f_{i}\to f_{j}V$ ($V=\gamma,\,Z$) decays

This decay proceeds at the lowest order via the Feynman diagrams of Fig. 2, where the internal fermion is a lepton (quark) provided that the external fermions are quarks (leptons).

The ultraviolet divergences cancel out when summing over all the partial amplitudes. The most general invariant amplitude can be written as

[TABLE]

where the monopole terms $L^{{}^{\prime}\gamma}$ and $R^{{}^{\prime}\gamma}$ vanish for the $f_{i}\to f_{j}\gamma$ decay due to gauge invariance: the bubble diagrams only give contributions to the monopole terms, which are canceled out by those arising from the triangle diagrams. The corresponding $L^{V}$, $R^{V}$, $L^{{}^{\prime}Z}$, and $R^{{}^{\prime}Z}$ form factors are presented in terms of Passarino-Veltman scalar functions in A.

After averaging (summing) over polarizations of the initial (final) fermion and gauge boson, we use the respective two-body decay width formula, which reduces to

[TABLE]

with $f_{ij}=\frac{1}{m_{i}^{2}}\left(2\left(m_{i}^{2}-m_{j}^{2}\right)^{2}-\left(m_{i}^{2}+m_{j}^{2}\right)m_{V}^{2}-m_{V}^{4}\right)$ and $g_{ij}=\frac{1}{m_{V}^{2}}\left(\left(m_{i}^{2}-m_{j}^{2}\right)^{2}+\left(m_{i}^{2}+m_{j}^{2})\right)m_{V}^{2}-2m_{V}^{4}\right)$. The so-called triangle function is given by

[TABLE]

For the $f_{i}\to f_{j}\gamma$ decay, Eq. (3.1) reduces to

[TABLE]

3.2 $q_{i}\to q_{j}g$ decay

This one-loop FCNC process is induced by Feynman diagrams similar to those shown in Fig. 2, except that there is no contribution from Feynman diagram of type a) as the internal fermion is a lepton. The Feynman rules necessary for the calculation are presented in Fig. 3.

The $q_{i}\to q_{j}g$ amplitude can be written as

[TABLE]

where the $L^{g}$ and $R^{g}$ coefficients can be obtained from Eqs. (A.1.1) and (A.2.1) of A once the replacements $Q_{k}\to 0$, $Q_{S}\to 1$, and $N_{c}e\to g_{s}$ are done. After averaging (summing) over initial (final) polarizations and colors, we obtain the average square amplitude and thereby the corresponding decay width, which has the same form of Eq. (17), though we must multiply the right-hand side by the color factor $C_{F}=4/3$.

3.3 $f_{i}\to f_{j}H$ decay

We now present the invariant amplitude for the LQ contribution to the $f_{i}\to f_{j}H$ decay, which is induced at the one-loop level by Feynman diagrams analogue to those shown in Fig. 2, but with the gauge boson $V$ replaced by the Higgs boson $H$. We have found that while the amplitude of Feynman diagram (d) is ultraviolet finite, that of Feyman diagram (a) has ultraviolet divergences, but they are canceled out by those arising from the bubble diagrams (b) and (c). After some algebra, the invariant amplitude can be cast in the form

[TABLE]

where the $F_{L}$ and $F_{R}$ form factors are presented in B in terms of Passarino-Veltman scalar functions and Feynman parameter integrals.

After summing (averaging) over the polarizations of the final (initial) fermion, we plug the average squared amplitude into the two-body decay width formula to obtain

[TABLE]

with $p_{i}\cdot p_{j}=(m_{i}^{2}+m_{j}^{2}-m_{H}^{2})/2$.

3.4 $H\to\bar{f}_{j}f_{i}$ decay

As a by-product we present the $H\to\bar{f}_{j}f_{i}$ decay width, which follows straightforwardly from the above results by crossing symmetry. Although the scalar LQ contribution to the LFV decay $H\to\tau\mu$ has been already presented in the zero lepton mass approximation Cheung:2015yga ; Kim:2018oih ; Cai:2017wry ; Bauer:2015knc , we now present the exact one-loop calculation for the $H\to\bar{f}_{j}f_{i}$ decay width. It reads

[TABLE]

where $\Gamma(H\to f_{i}f_{j})=\Gamma(H\to\bar{f}_{i}f_{j})+\Gamma(H\to\bar{f}_{j}f_{i})$. Also $p_{j}\cdot p_{j}=(m_{H}^{2}-(m_{i}^{2}+m_{j}^{2}))/2$, and the $F_{L}$ and $F_{R}$ form factors are the same as those presented in B for the $f_{i}\to f_{j}H$ decay as discussed in C.

4 Three-body tree-level decay $f_{i}\to f_{j}{\bar{f}}_{m}f_{l}$

Finally we discuss the calculation of the three-body decay $t\to c\bar{\ell}\ell$. Following our calculation approach, we consider the general decay $f_{i}\to f_{j}{\bar{f}}_{m}f_{l}$, where $f_{l}$ and $f_{m}$ are leptons (quarks) if $f_{i}$ and $f_{j}$ are quarks (leptons). This process is induced by a scalar LQ at the tree-level via the Feynman diagram of Fig. 4. We denote the four-momentum of fermion $f_{a}$ ($a=i,j,l,m$) by $p_{a}$. The corresponding decay width can be written as

[TABLE]

In the center-of-mass frame of the decaying fermion, the scaled variables $x_{a}$ ($a=j,l,m$) are given as $x_{a}=2E_{a}/m_{a}$. From energy conservation, these variables obey $x_{j}+x_{l}+x_{m}=2$. The kinematic limits in Eq. (22) are in turn

[TABLE]

where $\mu_{a}=m_{a}^{2}/m_{i}^{2}$ $(a=j,l,m)$.

The square average amplitude can be expressed as

[TABLE]

where the scalar products can be written as

[TABLE]

The integration of Eq. (22) can be performed numerically.

5 Constraints on the parameter space of the scalar LQ models

We now consider the LQ model introduced above and present an analysis of the constraints on the LQ couplings to SM fermions and the Higgs boson. While the LQ couplings to fermions can be obtained from the muon anomalous MDM and LFV tau decays, the LQ coupling to a Higgs boson pair can be extracted from the constraint on the $H\gamma\gamma$ and $Hgg$ couplings obtained by the ATLAS and CMS collaborations Khachatryan:2016vau .

5.1 Constraints on scalar LQ masses

The phenomenology of the scalar LQ doublet $R_{2}$ has been long studied in the literature Shanker:1982nd ; Davidson:1993qk ; Mizukoshi:1994zy ; Arnold:2013cva ; Dorsner:2013tla ; Bolanos:2013tda , and constraints on their mass and couplings have been derived from the $Z\to b\bar{b}$ decay, the muon anomalous MDM, and LFV decays. Since low energy physics strongly constrains the LQ couplings to the first-generation fermions, it is usually assumed that the only non-negligible couplings are those to the fermions of the second and third generations. The most stringent current constraint on the mass of the scalar LQ doublet $R_{2}$ masses is $m_{\Omega_{2/3}}\gtrapprox 1$ TeV, which was obtained by the ATLAS Aaboud:2019bye and CMS Sirunyan:2018vhk collaborations from the LHC data at $\sqrt{s}=13$ TeV under the assumption that $\Omega_{2/3}$ is a third-generation LQ that decays mainly as $\Omega_{2/3}\to\bar{\tau}b$, though such a bound relaxes up to 800 GeV when it is assumed that $\Omega_{2/3}$ decays into both the $\bar{\tau}b$ and $t\nu_{\tau}$ channels. Also, the LQ search via pair production Sirunyan:2018ryt gives a very stringent upper bound of $1500$ GeV on the mass of second-generation LQs, which we do not consider here as we are interested in a LQ that couples to both second and third-generation fermions. We will then assume the less stringent bound $m_{\Omega_{5/3}}\geq 800$ GeV in our analysis below since $m_{\Omega_{2/3}}$ and $m_{\Omega_{5/3}}$ are mass degenerate, cf. Eq. (10). In fact, a non-degenerate scalar LQ doublet could give dangerous contributions to the oblique parameters Keith:1997fv ,

5.2 Constraints from the LHC data on the Higgs boson

LHC data indicate that the 125 GeV Higgs boson couplings are compatible with those predicted by the SM, which provides a useful approach to constrain the parameter space of SM extension models by means of the so-called Higgs boson coupling modifiers, which are defined as

[TABLE]

where $\Gamma^{\rm SM}(H\to i)$ is the SM Higgs boson decay width and $\Gamma(H\to i)$ is the one including new physics effects. Bounds on the Higgs boson coupling modifiers were obtained by fitting the combined data of the ATLAS and CMS collaborations Khachatryan:2016vau . Since LQs contribute at the one-loop level to the $H\to\gamma\gamma$ and $H\to gg$ decays, to constrain the LQ couplings to a Higgs boson pair $H\Omega_{5/3}\Omega_{5/3}$, we use $\kappa_{\gamma}$ and $\kappa_{g}$, which are given as Dorsner:2016wpm

[TABLE]

and

[TABLE]

where the sum is over the LQs $S_{i}$, $\tau_{a}=4m_{H}^{2}/m_{a}^{2}$, and the $F_{s}(\tau_{a})$ function is given by

[TABLE]

where

[TABLE]

Although in our model FCNCs top quark decays receive contribution from $\Omega_{5/3}$ only, $\Omega_{2/3}$ also contribute to the decays $H\to\gamma\gamma$ and $H\to gg$. As already mentioned, these LQs are mass degenerate: $m_{\Omega_{5/3}}=m_{\Omega_{2/3}}$. We show in the left plot of Fig. 5 the area allowed by the experimental constraints on $\kappa_{\gamma}$ and $\kappa_{g}$ in the $\lambda_{\Omega_{5/3}}$ vs $\lambda_{\Omega_{2/3}}$ plane for two values of $m_{\Omega_{5/3}}$. In general, values of the order of $O(10)$ are allowed for either $\lambda_{\Omega_{2/3}}$ or $\lambda_{\Omega_{5/3}}$, with the largest allowed values obtained for either large $m_{\Omega_{5/3}}$ or $\lambda_{\Omega_{2/3}}=-\lambda_{\Omega_{5/3}}$. We also show the allowed area in the $m_{\Omega_{5/3}}$ vs $\lambda_{\Omega_{5/3}}$ plane in several $\lambda_{\Omega_{2/3}}$ scenarios. We observe that for a particular $m_{\Omega_{5/3}}$ value, the strongest constraints are obtained when $\lambda_{\Omega_{2/3}}=\lambda_{\Omega_{5/3}}$, whereas the less stringent constraints are obtained when $\lambda_{\Omega_{2/3}}=-\lambda_{\Omega_{5/3}}$.

In summary the $\kappa_{\gamma}$ and $\kappa_{g}$ constraints are satisfied for $\lambda_{\Omega_{5/3}}$ of the order of $O(10)$, with the largest values allowed for a heavy LQ. In our analysis below we will use however the conservative value $\lambda_{\Omega_{5/3}}\simeq 1$ as a very large value would violate the perturbativity of the LQ coupling.

5.3 Constraints from the muon anomalous magnetic moment and the LFV decay $\tau\to\mu\gamma$

The experimental bounds on the muon anomalous magnetic dipole moment (MDM) $a_{\mu}$ and the LFV tau decays provide an useful tool to constrain LFV effects Tanabashi:2018oca . In particular, $a_{\mu}$ can be useful to constrain the LQ couplings $\lambda^{\mu u_{i}}_{L,R}$ ($u_{i}=c,t$), whereas the decay $\tau\to\mu\gamma$ allow us to constrain the $\lambda^{\tau u_{i}}_{L,R}$ ones.

5.3.1 Muon anomalous magnetic dipole moment

Currently there is a discrepancy between the experimental and theoretical values of the muon anomalous MDM $\Delta a_{\mu}=a_{\mu}^{\rm Theo.}-a_{\mu}^{\rm Exp.}=268\,(63)\,(43)\times 10^{-11}$ Tanabashi:2018oca . We assume that this discrepancy is due to the LQ contribution, though such a puzzle could be settled in the future once new experimental measurements and more accurate evaluations of the hadronic contributions were available.

The contribution of scalar LQs $\Omega_{5/3}$ to the muon anomalous magnetic dipole moment $a_{\mu}^{\rm LQ}$ arises at the one-loop level from the triangle diagrams of Fig. 2 with $f_{j}=f_{i}=\mu$ and $f_{k}=u_{k}$. It can be written as Bolanos:2013tda

[TABLE]

where $x_{a}=m_{a}^{2}/m^{2}_{\Omega_{5/3}}$. The $F(x,y)$ and $G(x,y)$ functions are presented in D in terms of Feynman parameter integrals and Passarino-Veltman scalar functions. Since $x_{\mu}\ll x_{u_{k}}$, we have the following approximate expression

[TABLE]

Since $\Omega_{2/3}$ is a chiral LQ, its contribution to $a_{\mu}$ is proportional to the muon mass and is thus subdominant. We now consider that the $\Delta a_{\mu}$ discrepancy is due to the LQ contribution $a_{\mu}^{\rm LQ}$ and show in Fig. 6 the allowed area in the ${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)$ vs ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)$ plane for three values of $m_{\Omega_{5/3}}$. We note that a positive contribution from LQs to $a_{\mu}$ is required to explain the discrepancy, therefore there are three possible scenarios:

${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)$ and ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)<0$. 2. 2.

${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)<0$ and ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)>0$. 3. 3.

${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)>0$ and ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)<0$.

In the first scenario (left plot of Fig. 6) we observe that while ${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)$ can range between $10^{-4}$ and $10^{-3}$ for negligible ${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)$, the latter can range between $10^{-3}$ and $10^{-2}$ for negligible ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)$, with the largest allowed values corresponding to heavy $m_{\Omega_{5/3}}$. On the other hand, more large values of the LQ couplings are allowed when ${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)$ and ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)$ are of opposite sign (right plot) as there is a cancellation between the contributions of the $c$ and $t$ quarks. In particular, there is a very narrow band where ${\rm Re}\left(\lambda_{L}^{\mu c}\lambda_{R}^{\mu c}\right)\simeq O(10)$ and ${\rm Re}\left(\lambda_{L}^{\mu t}\lambda_{R}^{\mu t}\right)\simeq O(1)$.

5.3.2 Decay $\tau\to\mu\gamma$

The LQ couplings $\lambda_{L,R}^{\tau u_{i}}$ and $\lambda_{L,R}^{\mu u_{i}}$ can be constrained by the experimental bound on the LFV tau decay $\tau\to\mu\gamma$, which can receive the contributions of loops with $\Omega_{5/3}$ accompanied by the up quarks. Such contributions follow straightforwardly from our result for the $f_{i}\to f_{j}\gamma$ decay width given in Eq. (3.1) after the proper replacements are made. The result is in agreement with previous calculations of the $\ell_{i}\to\ell_{j}\gamma$ decay width Bolanos:2013tda .

If the LQ couples to both the $c$ and $t$ quarks, the $\tau\to\mu\gamma$ decay width acquires the form

[TABLE]

where $\alpha_{LL}^{\mu u_{i}}$, etc. stand for the loop integrals. To simplify our analysis we assume the scenario where $\lambda_{R}^{\ell u_{i}}/\lambda_{L}^{\ell u_{i}}=O(\epsilon)$ ($\ell=\mu,\,\tau$), with $\epsilon=10^{-3}$(predominantly left-handed couplings), $10^{-1}$ (small right handed-couplings), and 1 (purely scalar couplings). We do not analyze the case when $\epsilon>1$ as a similar situation is observed as in the $\epsilon<1$ case but with $\lambda_{L}^{\ell u_{i}}$ replaced by $\lambda_{R}^{\ell u_{i}}$. Thus the parameter $\epsilon$ is a measure of the relative size between the right and left-handed LQ couplings. Under this assumption, the $\tau\to\mu\gamma$ decay width becomes a function of the products $\lambda_{L}^{\mu c}\lambda_{L}^{\tau c}$ and $\lambda_{L}^{\mu t}\lambda_{L}^{\tau t}$. We thus show in Fig. 7 the allowed area in the $\lambda_{L}^{\mu c}\lambda_{L}^{\tau c}$ vs $\lambda_{L}^{\mu t}\lambda_{L}^{\tau t}$ plane for three values of $m_{\Omega_{5/3}}$. We observe that for $m_{\Omega_{5/3}}=1000$ GeV the largest allowed area is obtained in the scenario with $\epsilon=10^{-3}$, which allow $\lambda_{L}^{\mu u_{i}}\lambda_{L}^{\tau u_{i}}$ values as large as $O(10^{-1})$, whereas the smallest area is obtained when $\epsilon=1$, which allows $\lambda_{L}^{\mu u_{i}}\lambda_{L}^{\tau u_{i}}$ values of the order of $O(10^{-4})$. Such bounds are slightly relaxed when $m_{\Omega_{5/3}}$ increases up to 2 TeV.

As far as constraints on the $|\lambda_{L,R}^{\ell u_{i}}|$ couplings from direct LQ searches at the LHC via the Drell-Yan process Sirunyan:2018exx , single production Khachatryan:2015qda , and pair production Sirunyan:2018ryt , an up-to-date discussion is presented in Ref. Schmaltz:2018nls . A restricted scenario (minimal LQ model) is considered where each LQ is allowed to couple to just one lepton-quark pair. In particular, a 95% C.L. limit on $|\lambda^{\mu c}|$ of the order of $O(1)$ is obtained for a LQ with a mass above the 1 TeV level from the Drell-Yan process Sirunyan:2018exx , whereas the bounds obtained from the LHC Run 1 and Run 2 data on single production Khachatryan:2015qda yield less stringent bounds. Although such limit could be relaxed in a more general scenario where the LQ is allowed to couple to more than one fermion pair, below we assume a conservative scenario and consider the bound $|\lambda^{\mu c}|\leq O(1)$, whereas for the remaining couplings we impose the $|\lambda_{L,R}^{\mu t}|,\,|\lambda_{L,R}^{\tau u_{i}}|<4\pi$ bound to avoid the breakdown of perturbativity.

We are interested in the region of the parameter space where the largest $t\to cX$ and $t\to c\bar{\ell}\ell$ branching ratios can be reached, which is the area where either $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$ or $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ reaches their largest allowed values. Again we consider the scenario with $\lambda_{R}^{\ell u_{i}}/\lambda_{L}^{\ell u_{i}}=O(\epsilon)$, with four $\epsilon$ values, and perform a scan of $(\lambda_{L}^{\mu c},\,\lambda_{L}^{\mu t},\,\lambda_{L}^{\tau c},\,\lambda_{L}^{\tau t})$ points consistent with both the $\Delta a_{\mu}$ discrepancy (Fig. 6) and the constraint on the $\tau\to\mu\gamma$ decay (Fig. 7) for two values of $m_{\Omega_{5/3}}$: we consider a large mass splitting to observe how the LQ couplings get constrained by the experimental data. As already discussed, we also impose the bound $|\lambda_{L,R}^{\mu c}|\leq O(1)$ from the direct LQ search at the LHC and, to avoid perturbativity violation, we impose the extra constraint $|\lambda_{L,R}^{\mu t}|,\,|\lambda_{L,R}^{\tau u_{i}}|<4\pi$. The corresponding allowed areas in the $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ vs $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$ plane are shown in Fig. 8. We observe that, for $m_{\Omega_{5/3}}=1000$ GeV, the scenario with $\epsilon=10^{-3}$ (top left plot) allows values of $\lambda_{L}^{\tau c}\lambda_{L}^{\tau t}$ as large as $O(1)$ for $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ of the order of $10^{-1}$, but values of the order of the order of $O(1)$ are allowed for $\lambda_{L}^{\mu c}\lambda_{L}^{\mu t}$ for $\lambda_{L}^{taut}\lambda_{L}^{\tau c}$ of the order of $10^{-2}$. For fixed $\epsilon$, the allowed area expands slightly when the LQ mass increases, which is expected as the loop functions become suppressed for large LQ mass, thereby allowing larger couplings. On the other hand, for fixed $m_{\Omega_{5/3}}$ GeV, the allowed areas shrink significantly in the $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ direction and slightly in the $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$ direction as $\epsilon$ increases. For instance, in the scenario when $\epsilon=1$ (bottom right plot), the largest allowed $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ values for $m_{\Omega_{5/3}}=1000$ GeV are of the order of $O(10^{-3})$ for small $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$, whereas the latter can be as large as $O(10^{-1})$ for very small $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$. We conclude that the scenario with predominantly dominant left-handed couplings ($\epsilon=10^{-3}$) is the one that allows the largest values of the LQ couplings.

6 Numerical analysis of the $t\to cX$ and $t\to c\bar{\ell}\ell$ branching ratios

We now turn to analyze the behavior of the $t\to cX$ and $t\to c\bar{\ell}\ell$ branching ratios in the allowed area of the parameter space. For the numerical evaluation of the one-loop induced decays $t\to cX$ we have made a cross-check by evaluating the Passarino-Veltman scalar functions via the LoopTools package vanOldenborgh:1989wn ; Hahn:1998yk and then comparing the results with those obtained by numerical integration of the parametric integrals. For the tree-level induced decay $t\to c\bar{\ell}\ell$ we have used the Mathematica numerical integration routines to solve the two-dimensional integral of Eq. (22).

6.1 $t\to cX$ branching ratios

We first consider two $\epsilon$ values and present in Table 1 a few sets of allowed $(\lambda_{L}^{\mu c}$, $\lambda_{L}^{\mu t}$, $\lambda_{L}^{\tau c}$, $\lambda_{L}^{\tau t})$ points where the $t\to cX$ decays can reach their largest branching ratios for three LQ masses. In the scenario where $\epsilon=10^{-3}$ we observe that there is a small area where all of the $t\to cX$ branching ratios can be as large as $10^{-9}-10^{-8}$ for $m_{\Omega_{5/3}}=1000$ GeV, though they get suppressed by one order of magnitude when $m_{\Omega_{5/3}}$ increases up to 2000 GeV. In such an area, the LQ couplings $\lambda_{L}^{\mu u_{i}}$ are rather small, whereas the $\lambda_{L}^{\tau t}$ one is very close to the perturbative limit, which means that this possibility would require a large amount of fine-tuning. As for the $\epsilon=10^{-1}$ scenario, we observe that the $t\to cX$ branching ratios are much smaller than in the $\epsilon=10^{-3}$ scenario: they can be of the order of $10^{-9}-10^{-10}$ at most for $m_{\Omega_{5/3}}=1000$ GeV, and decrease by one order of magnitude as $m_{\Omega_{5/3}}$ increases up to 2000 GeV. We refrain from presenting the results for the $\epsilon=1$ scenario as the $t\to cX$ branching ratios are two orders of magnitude than in the $\epsilon=10^{-3}$ scenario.

We also observe in Table 1 that all the branching ratios ${\rm Br}(t\to cX)$ are of similar order of magnitude, with ${\rm Br}(t\to cZ)$ slightly larger. It seems surprising that ${\rm Br}(t\to cg)$ is about the same size than ${\rm Br}(t\to c\gamma)$, whereas in the SM and other of its extensions it is one or two orders of magnitude larger. To explain this result, let us examine the case of the SM, where the $t\to c\gamma$ decay proceeds via a Feynman diagram where the photon emerges off a down-type quark and so the squared amplitude for the analogue $t\to cg$ diagram has an enhancement factor of $c_{F}\left[g_{S}/(-1/3e)\right]^{2}\simeq O(10^{2})$, where $c_{F}=4/3$ is the color factor. On the other hand, in our LQ model the photon emerges off the charge $5/3e$ LQ, which means that the enhancement factor for the squared $t\to cg$ amplitude is just $c_{F}\left[g_{S}/(5/3e)\right]^{2}\simeq O(1)$. Furthermore, in our LQ model the Feynman diagram where the photon emerges off the LQ gives a smaller contribution than that where it emerges off the lepton, which is absent in the $t\to cg$ decay. These two facts conspire to yield ${\rm Br}(t\to cg)\gtrsim{\rm Br}(t\to c\gamma)$. It is also worth mentioning that the $t\to cH$ decay receives its main contribution from the diagram where the Higgs boson is emitted off the LQ line, and thus its decay width is very sensitive to the magnitude of the $\lambda_{\Omega_{5/3}}$ coupling.

Finally we show in Fig. 9 the contours of the $t\to cX$ branching ratios in the allowed area of the $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ vs $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$ plane in the scenario with $\lambda_{R}^{\ell u_{i}}/\lambda_{L}^{\ell u_{i}}=O(10^{-3})$, where the largest values of the $t\to cX$ branching ratios are reached. As already noted, when $m_{\Omega_{5/3}}=1000$ GeV the largest $t\to cX$ branching ratios, of the order of $10^{-9}-10^{-8}$, are obtained in a tiny area where $\lambda_{L}^{\mu t}\lambda_{L}^{\mu c}$ is very small and $\lambda_{L}^{\tau t}\lambda_{L}^{\tau c}$ reaches its largest allowed values (top-left corner of the upper plots), but they decrease as the allowed area expands. It means that the largest branching ratios are obtained in the region where the main contribution arises from the loops with an internal tau lepton, which is due to the fact that the LQ couplings to the tau lepton are less constrained than those to the muon. We also observe that the $t\to cX$ branching ratios decrease by one or two orders of magnitude as $m_{\Omega_{5/3}}$ reaches the 2 TeV level, where they can be as large as $10^{-9}-10^{-10}$. The behavior of the $t\to cX$ branching ratios in the scenarios with $\epsilon=10^{-1}$ and $\epsilon=1$ is rather similar to that observed in Fig. 9, but they are one or two orders of magnitude below: they can only be as large as $10^{-9}$ for $\epsilon=10^{-1}$ and $10^{-10}$ for $\epsilon=1$.

6.2 $t\to c\bar{\ell}\ell$ branching ratios

We now perform the corresponding analysis for the $t\to c\bar{\ell}\ell$ ($\ell=\mu,\,\tau$) branching ratios in the area allowed by the experimental constraints discussed above. In Fig. 10 we show the contours of ${\rm Br}(t\to c\bar{\ell}\ell)$ in the $\lambda_{L}^{\ell t}$ vs $\lambda_{L}^{\ell c}$ plane in the scenario with $\lambda_{R}^{\ell u_{i}}/\lambda_{L}^{\ell u_{i}}=O(10^{-3})$ for two values of the LQ mass. We observe that for $m_{\Omega_{5/3}}=1000$ GeV, ${\rm Br}(t\to c\bar{\tau}\tau)$ can be of as large as $10^{-6}$, whereas ${\rm Br}(t\to c\bar{\mu}\mu)$ is one order of magnitude below, which is due to the fact that the $\lambda_{L}^{\mu q}$ couplings are more constrained than the $\lambda_{L}^{\tau q}$ ones. When $m_{\Omega_{5/3}}$ increases up to 2000 GeV, the $t\to c\bar{\ell}\ell$ branching ratios decrease by about one order of magnitude. As for the $t\to c\bar{\mu}\tau$ decay, its branching ratio is also suppressed as involves the $\lambda_{L}^{\mu q}$ couplings. In conclusion, the three-body tree-level decay $t\to c\bar{\ell}\ell$ can have larger branching ratios than the two-body one-loop decays $t\to cX$.

7 Summary and outlook

The FCNC decays of the top quark $t\to cX$ ($X=\gamma,\,g,\,Z,\,H$) and $t\to c\bar{\ell}\ell$ ($\ell=\mu,\,\tau$) were calculated in a simple LQ model with no proton decay, where the SM is augmented by a $SU(2)$ scalar LQ doublet with hypercharge $Y=7/6$. In such a model there is a non-chiral LQ with electric charge $Q=5/3e$ that couples to charged leptons and up quarks and contribute to the FCNC decays of the top quark.

As far as the analytical results are concerned, we perform a general calculation of the FCNC fermion decays $f_{i}\to f_{j}X$ and $f_{i}\to f_{j}\bar{f}_{m}f_{l}$. The loop amplitudes of the $f_{i}\to f_{j}X$ decays are presented in terms of both Passarino-Veltman scalar functions and Feynman parameter integrals, which can be useful to calculate the contributions of other scalar LQs. On the other hand, an analytical expression is presented for the $f_{i}\to f_{j}\bar{f}_{m}f_{l}$ decay width, which can be numerically evaluated.

As for the numerical analysis, to obtain bounds on the parameter space of the model we assumed that the LQ only couples to the fermions of the last two families and used the experimental constraints on the LHC Higgs boson data, the muon anomalous magnetic dipole moment $a_{\mu}$, the LFV decay of the tau lepton $\tau\to\mu\gamma$, as well as the direct LQ searches at the LHC via the Drell-Yan process, single production, and double production. For the LQ couplings to charged leptons and up quarks $\lambda_{L,R}^{\ell u_{i}}$, a scenario was considered where $\lambda_{R}^{\ell u}/\lambda_{L}^{\ell u_{i}}=O(\epsilon)$, with $\epsilon$ being a measure of the relative size between the right- and left-handed LQ couplings. Afterwards, the $t\to cX$ and $t\to c\bar{\ell}\ell$ branching ratios were evaluated in the allowed region of the parameter space. In particular, we find that in the scenario where the LQ couplings are predominantly left-handed, $\epsilon=O(10^{-3})$, there is a tiny region of the parameter space where the branching ratios of the one-loop induced $t\to cX$ decay can be as large as $10^{-8}$ for $m_{\Omega_{5/3}}=1000$ GeV, with the main contribution arising from the loops with an internal tau lepton, although a large amount of fine-tuning between the LQ couplings would be required. However, for $\epsilon=10^{-1}$ ($\epsilon=1$), the main part of the allowed region yields $t\to cX$ branching ratios of the order of $10^{-9}$ ($10^{-10}$) at most. For $m_{\Omega_{5/3}}\geq 2000$ GeV, the largest $t\to cX$ branching ratios are of the order of $10^{-10}$ in all the scenarios analyzed in this work. Although the $t\to cX$ branching ratios are larger in our LQ model than in the SM, such contributions would be out of the reach of detection in the near future. As for the tree-level induced decays $t\to c\bar{\ell}\ell$, the $t\to c\bar{\tau}\tau$ branching ratio can be as large as $10^{-6}$ for $m_{\Omega_{5/3}}=1000$ GeV in the scenario with $\epsilon=O(10^{-3})$, but ${\rm Br}(t\to c\bar{\mu}\mu)$ is one order of magnitude below. These branching ratios decrease by about one order of magnitude when the LQ mass increases up to 2000 GeV.

It is worth noting that experimental constraints on the LQ mass and couplings obtained from the direct search at the LHC are very stringent, but they rely on several assumptions and may be relaxed, which would yield a slight enhancement of the LQ contribution to the top quark FCNC top quark decays. The magnitude of the $t\to cX$ branching ratios is similar to that recently found for the contributions from a scalar LQ with charge $-1/3e$, which arises in a model with a scalar LQ singlet Kim:2018oih . We do not consider this scenario in our analysis as we are interested in LQ models where no further symmetries must be invoked to forbid the proton decay Arnold:2013cva .

Acknowledgements.

We acknowledge support from Consejo Nacional de Ciencia y Tecnología and Sistema Nacional de Investigadores. Partial support from Vicerrectoría de Investigación y Estudios de Posgrado de la Benémerita Universidad Autónoma de Puebla is also acknowledged.

Appendix A Loop integrals for the $f_{i}\to f_{j}V$ decay

We now present the contribution of the scalar LQ $S$ to the $f_{i}\to f_{j}V$ loop amplitudes. Although in these Appendices $S$ will stand for the $\Omega_{5/3}$ LQ, as already explained, our results are also valid for the contribution of any other scalar LQ.

A.1 Passarino-Veltman results

We first define the following sets of ultraviolet finite Passarino-Veltman scalar function combinations

[TABLE]

The form factors of Eq. (14) are given in terms of these ultraviolet-finite functions as follows.

A.1.1 $f_{i}\to f_{j}\gamma$ decay

There are only dipole form factors as the monopole ones must vanish due to electromagnetic gauge invariance. Although each Feynman diagram has ultraviolet divergences, they cancel out when summing over all the contributions. The results read

[TABLE]

where $N_{c}$ is the color number of the internal fermion and we introduced the following definitions $x_{a}=m_{a}^{2}/m_{S}^{2}$, $\xi_{ab}=x_{a}-x_{b}$, $\eta_{ab}=x_{a}+x_{b}$, $\xi_{a}=x_{a}-1$, and $\eta_{a}=x_{a}+1$. In addition, the right handed form factors can be obtained from the left-handed ones as follows

[TABLE]

A.1.2 $f_{i}\to f_{j}Z$ decay

The amplitude for this decay contains both dipole and monopole form factors. Again the ultraviolet divergences cancel when summing over partial contributions. The $L^{Z}$ and $L^{{}^{\prime}Z}$ form factors are too lengthy and can be written as a sum of partial terms arising from each contributing diagram as follows

[TABLE]

and

[TABLE]

where the superscript stands for the Feynman diagram of Fig. 2 out of which the corresponding term arises, with (cd) standing for the sum of the contributions of diagrams (c) and (d).

The contributions of diagram (a) are given by

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $\delta=x_{i}^{2}-2(x_{j}+x_{Z})x_{i}+(x_{j}-x_{Z})^{2}$. The contributions of diagram (b) are

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Finally Feynman diagrams (c) and (d) only contribute to monopole terms. The corresponding contribution of both diagrams is

[TABLE]

We can observe that the ultraviolet divergent term $B_{kS}\equiv B_{0}(0,m_{k}^{2},m_{S}^{2})$, which appears only in the monopole terms, is canceled out when summing over all the contributions.

Furthermore, the form factors associated with the right-handed terms are given by

[TABLE]

and

[TABLE]

A.2 Feynman parameter results

A.2.1 $f_{i}\to f_{j}\gamma$ decay

The $L^{\gamma}$ form factor of Eq. (14) is ultraviolet finite and is given in terms of Feynman parameter integrals as follows

[TABLE]

where

[TABLE]

The $R^{\gamma}$ form factor is given by Eq. (49), whereas monopole terms $L^{{}^{\prime}\gamma}$ and $R{{}^{\prime}\gamma}$ are zero as already mentioned (one must consider electric charge conservation).

A.2.2 $f_{i}\to f_{j}Z$ decay

The dipole terms of Eq. (50), which only arise from diagrams (a) and (b), are ultraviolet finite and are given by

[TABLE]

[TABLE]

whereas the partial contributions to the monopole terms of Eq. (51) are ultraviolet divergent and read

[TABLE]

[TABLE]

[TABLE]

where $\Delta_{\rm UV}$ stands for the ultraviolet divergence, which cancels out when summing over the partial contributions as it is proportional to $g_{L}^{k}-g_{R}^{j}-g_{VSS}$. We also have defined the following functions

[TABLE]

Appendix B Loop integrals for the $f_{i}\to f_{j}H$ decay

The $F_{L}$ and $F_{R}$ form factors of Eq. (3.3) are given by

[TABLE]

with $f_{L,R}^{(k)}$ ($\rm k=a,b,c,d$) being the contributions of the Feynman diagram analogue to the diagram (k) of Fig. 2, with the $V$ gauge boson replaced by the Higgs boson. Again we present our results in terms of Passarino-Veltman scalar functions and Feynman parameter integrals.

B.1 Passarino-Veltman results

The sum of the contributions of the triangle and bubble diagrams (a), (b) and (c) is ultraviolet finite and reads

[TABLE]

whereas the contribution of triangle diagram (d), which is ultraviolet finite by itself, can be written as

[TABLE]

where we have introduced the auxiliary variable $\chi=\xi_{ji}^{2}+x_{H}^{2}-2x_{H}\xi_{ij}$, and $\zeta=\chi/(x_{j}x_{i}\xi_{ji})$. As for the $C^{\prime}_{j}$ and $\Delta^{\prime}_{j}$ functions, they are given by

[TABLE]

It is thus evident that ultraviolet divergences cancel out. As far as the right-handed terms are concerned, they obey

[TABLE]

B.2 Feynman parameter results

Feynman parametrization yield the following results for the $f_{L,R}^{\rm(k)}$ coefficients:

[TABLE]

[TABLE]

where it is evident that the ultraviolet divergence $\Delta_{\rm UV}$ cancels out when summing over the partial contributions. We also use the following auxiliary variables

[TABLE]

and

[TABLE]

As far as the contribution of Feynman diagram (d) is concerned, it is given by

[TABLE]

with

[TABLE]

Appendix C Loop integrals for the $H\to\bar{f}_{j}f_{i}$ decay

As already mentioned, the form factors $F_{L}$ and $F_{R}$ for the $f_{i}\to f_{j}H$ decay width are also valid for the $H\to f_{j}f_{i}$ decay width given in (3.4). It is interesting to obtain the approximate results in the limit of small $x_{j}$ and $x_{i}$. In the case of the Passarino-Veltman results $\Delta^{\prime}_{1}\to\Delta^{\prime}_{3}+O(x_{j})$ and $\Delta^{\prime}_{2}\to\Delta^{\prime}_{3}+O(x_{i})$ for small $x_{i}$ and $x_{j}$, which means that in the limit of vanishing external fermion masses we have

[TABLE]

and

[TABLE]

which means that in this scenario the $H\to f_{i}f_{j}$ decay width can be written as

[TABLE]

where

[TABLE]

This result agrees with the one presented in Cheung:2015yga ; Kim:2018oih ; Cai:2017wry ; Bauer:2015knc .

As far as the Feynman parameter results, in the vanishing limit of $m_{j}$ and $m_{i}$ one can obtain

[TABLE]

[TABLE]

and

[TABLE]

with

[TABLE]

Appendix D Lepton anomalous magnetic dipole moment

The $F$ and $G$ functions of Eq. (5.3.1) read

[TABLE]

with the $F_{a}$ and $G_{a}$ functions given in terms of Feynman parameter integrals by

[TABLE]

where $\xi_{1}(x)=1-x$ and $\xi_{2}(x)=x$. The integration is straightforward in the limit of a light external fermion and heavy internal fermion and LQ: $x_{i}\ll x_{k}$

[TABLE]

[TABLE]

For completeness we also present the results in terms of Passarino-Veltman scalar functions:

[TABLE]

with

[TABLE]

and $\zeta(x,y)=\left(1+y-x\right)^{2}-4y$.