Flavor violating top decays and flavor violating quark decays of the Higgs boson

TL;DR

This paper explores how extending the MSSM with a vector-like quark generation can significantly enhance flavor-violating decays of the top quark and Higgs boson, making them potentially observable in current or future collider experiments.

Contribution

It demonstrates that adding a vector-like quark generation to MSSM can increase flavor-violating decay branching ratios to observable levels, unlike in the standard model or minimal MSSM.

Findings

Branching ratios for t→h^0 c and t→h^0 u can reach current experimental limits.

Higgs flavor-violating decays h^0→b s̄ + b̄ s can be as large as 1%.

CP phases influence the decay branching ratios significantly.

Abstract

In the standard model flavor violating decays of the top quark and of the Higgs boson are highly suppressed. Further, the flavor violating decays of the top and of the Higgs are also small in MSSM and not observable in current or in near future experiment. In this work we show that much larger branching ratios for these decays can be achieved in an extended MSSM model with an additional vector like quark generation. Specifically we show that in the extended model one can achieve branching ratios for $t\to h^0 c$ and $t\to h^0 u$ as large as the current experimental upper limits given by the ATLAS and the CMS Collaborations. We also analyze the flavor violating quark decay of the Higgs boson, i.e., $h^0\to b\bar s + \bar b s$ and $h^0\to b\bar d + \bar b d$. Here again one finds that the branching ratio for these decays can be as large as $O(1)\%$. The analysis is done with inclusion of the CP phases in the Higgs sector, and the effect of CP phases on the branching ratios is investigated. Specifically the Higgs sector spectrum and mixings are computed involving quarks and mirror quarks, squarks and mirror squarks in the loops consistent with the Higgs boson mass constraint. The resulting effective Lagrangian with inclusion of the vector like quark generation induce flavor violating decays at the tree level. The test of the branching ratios predicted could come with further data from LHC13 and such branching ratios could also be accessible at future colliders such as the Higgs factories where the Higgs couplings to fermions will be determined with greater precision.

Tables

Table 1. Table 1: An exhibition of masses of the vectorlike quarks from diagonalization of the matrices of Eqs.( 7 ) and ( 12 ) consistent with the current lower bounds on exotic quarks. The parameters used are | h 3 | = 1.6 subscript ℎ 3 1.6 |h_{3}|=1.6 , | h 3 ′ | = 6.34 × 10 − 2 subscript superscript ℎ ′ 3 6.34 superscript 10 2 |h^{\prime}_{3}|=6.34\times 10^{-2} , | h 3 ′′ | = 1.97 × 10 − 2 subscript superscript ℎ ′′ 3 1.97 superscript 10 2 |h^{\prime\prime}_{3}|=1.97\times 10^{-2} , | h 4 | = 485 subscript ℎ 4 485 |h_{4}|=485 , | h 4 ′ | = 500 subscript superscript ℎ ′ 4 500 |h^{\prime}_{4}|=500 , | h 4 ′′ | = 410 subscript superscript ℎ ′′ 4 410 |h^{\prime\prime}_{4}|=410 , | h 5 | = 22 subscript ℎ 5 22 |h_{5}|=22 , | h 5 ′ | = 545 subscript superscript ℎ ′ 5 545 |h^{\prime}_{5}|=545 , | h 5 ′′ | = 130 subscript superscript ℎ ′′ 5 130 |h^{\prime\prime}_{5}|=130 , | h 6 | = 550 subscript ℎ 6 550 |h_{6}|=550 , | h 7 | = 280 subscript ℎ 7 280 |h_{7}|=280 , | h 8 | = 450 subscript ℎ 8 450 |h_{8}|=450 , χ 3 = 0.9 subscript 𝜒 3 0.9 \chi_{3}=0.9 , χ 3 ′ = 1 × 10 − 3 subscript superscript 𝜒 ′ 3 1 superscript 10 3 \chi^{\prime}_{3}=1\times 10^{-3} , χ 3 ′′ = 4 × 10 − 3 subscript superscript 𝜒 ′′ 3 4 superscript 10 3 \chi^{\prime\prime}_{3}=4\times 10^{-3} , χ 4 = χ 4 ′ = 2.1 subscript 𝜒 4 subscript superscript 𝜒 ′ 4 2.1 \chi_{4}=\chi^{\prime}_{4}=2.1 , χ 4 ′′ = 0.6 subscript superscript 𝜒 ′′ 4 0.6 \chi^{\prime\prime}_{4}=0.6 , χ 5 = 0.99 subscript 𝜒 5 0.99 \chi_{5}=0.99 , χ 5 ′ = 2.7 subscript superscript 𝜒 ′ 5 2.7 \chi^{\prime}_{5}=2.7 , χ 5 ′′ = 2.4 subscript superscript 𝜒 ′′ 5 2.4 \chi^{\prime\prime}_{5}=2.4 , χ 6 = 0.01 subscript 𝜒 6 0.01 \chi_{6}=0.01 , χ 7 = 3.1 subscript 𝜒 7 3.1 \chi_{7}=3.1 , χ 8 = 0.01 subscript 𝜒 8 0.01 \chi_{8}=0.01 . The input of the diagonal elements are ( 173.21 173.21 173.21 , 420 420 420 , 1.7 1.7 1.7 , 2.3 × 10 − 3 2.3 superscript 10 3 2.3\times 10^{-3} , 700 700 700 ) for Eq.( 7 ) and and are ( 4.18 4.18 4.18 , 560 560 560 , 0.095 0.095 0.095 , 4.8 × 10 − 3 4.8 superscript 10 3 4.8\times 10^{-3} , 640 640 640 ) for Eq.( 12 ). The masses of the three generation of quarks are as given by the PDG  [ 57 ] . All masses are in GeV and all phases in rad.

Heavy Up Quarks		Heavy Down Quarks
Mirror Up Quark	$m_{t^{\prime }}$ = $803$	Mirror Down Quark	$m_{b^{\prime }}$ = $817$
Fourth Generation Up Quark	$m_4^{\mathrm{up}}$ = $917$	Fourth Generation Down Quark	$m_4^{\mathrm{down}}$ = $1044$

Table 2. Table 2: An exhibition of the branching ratios for the flavor violating top decays t → c ​ h → 𝑡 𝑐 ℎ t\to ch and t → h ​ u → 𝑡 ℎ 𝑢 t\to hu and also for the flavor violating Higgs decay h 0 → b ​ s ¯ + b ¯ ​ s → superscript ℎ 0 𝑏 ¯ 𝑠 ¯ 𝑏 𝑠 h^{0}\to b\bar{s}+\bar{b}s and h 0 → b ​ d ¯ + b ¯ ​ d → superscript ℎ 0 𝑏 ¯ 𝑑 ¯ 𝑏 𝑑 h^{0}\to b\bar{d}+\bar{b}d . The results of the table are consistent with the experimental data of Eq.(1). The analysis is for the parameter sets given by (1): tan ⁡ β = 20 𝛽 20 \tan\beta=20 , m 0 = m 0 u = m 0 d = 5000 subscript 𝑚 0 subscript superscript 𝑚 𝑢 0 subscript superscript 𝑚 𝑑 0 5000 m_{0}=m^{u}_{0}=m^{d}_{0}=5000 , | μ | = 650 𝜇 650 |\mu|=650 , | A 0 u | = 400 subscript superscript 𝐴 𝑢 0 400 |A^{u}_{0}|=400 , | A 0 d | = 210 subscript superscript 𝐴 𝑑 0 210 |A^{d}_{0}|=210 ; (2): tan ⁡ β = 30 𝛽 30 \tan\beta=30 , m 0 = m 0 u = m 0 d = 6400 subscript 𝑚 0 subscript superscript 𝑚 𝑢 0 subscript superscript 𝑚 𝑑 0 6400 m_{0}=m^{u}_{0}=m^{d}_{0}=6400 , | μ | = 580 𝜇 580 |\mu|=580 , | A 0 u | = 260 subscript superscript 𝐴 𝑢 0 260 |A^{u}_{0}|=260 , | A 0 d | = 140 subscript superscript 𝐴 𝑑 0 140 |A^{d}_{0}|=140 ; (3): tan ⁡ β = 40 𝛽 40 \tan\beta=40 , m 0 = m 0 u = m 0 d = 7100 subscript 𝑚 0 subscript superscript 𝑚 𝑢 0 subscript superscript 𝑚 𝑑 0 7100 m_{0}=m^{u}_{0}=m^{d}_{0}=7100 , | μ | = 490 𝜇 490 |\mu|=490 , | A 0 u | = 170 subscript superscript 𝐴 𝑢 0 170 |A^{u}_{0}|=170 , | A 0 d | = 150 subscript superscript 𝐴 𝑑 0 150 |A^{d}_{0}|=150 ; (4): tan ⁡ β = 50 𝛽 50 \tan\beta=50 , m 0 = m 0 u = m 0 d = 9400 subscript 𝑚 0 subscript superscript 𝑚 𝑢 0 subscript superscript 𝑚 𝑑 0 9400 m_{0}=m^{u}_{0}=m^{d}_{0}=9400 , | μ | = 540 𝜇 540 |\mu|=540 , | A 0 u | = 150 subscript superscript 𝐴 𝑢 0 150 |A^{u}_{0}|=150 , | A 0 d | = 150 subscript superscript 𝐴 𝑑 0 150 |A^{d}_{0}|=150 . The common parameters are θ μ = 0.1 subscript 𝜃 𝜇 0.1 \theta_{\mu}=0.1 , α A 0 u = 0.4 subscript 𝛼 subscript superscript 𝐴 𝑢 0 0.4 \alpha_{A^{u}_{0}}=0.4 , α A 0 d = 0.02 subscript 𝛼 subscript superscript 𝐴 𝑑 0 0.02 \alpha_{A^{d}_{0}}=0.02 , m A = 600 subscript 𝑚 𝐴 600 m_{A}=600 . The values of the parameters h 3 , h 4 , h 5 , h 3 ′ , h 4 ′ , h 5 ′ , h 3 ′′ , h 4 ′′ , h 5 ′′ , h 6 , h 7 , h 8 subscript ℎ 3 subscript ℎ 4 subscript ℎ 5 superscript subscript ℎ 3 ′ superscript subscript ℎ 4 ′ superscript subscript ℎ 5 ′ superscript subscript ℎ 3 ′′ superscript subscript ℎ 4 ′′ superscript subscript ℎ 5 ′′ subscript ℎ 6 subscript ℎ 7 subscript ℎ 8 h_{3},h_{4},h_{5},h_{3}^{\prime},h_{4}^{\prime},h_{5}^{\prime},h_{3}^{\prime\prime},h_{4}^{\prime\prime},h_{5}^{\prime\prime},h_{6},h_{7},h_{8} and y 1 , y 2 , y 3 , y 4 , y 5 , y 1 ′ , y 2 ′ , y 3 ′ , y 4 ′ , y 5 ′ subscript 𝑦 1 subscript 𝑦 2 subscript 𝑦 3 subscript 𝑦 4 subscript 𝑦 5 superscript subscript 𝑦 1 ′ superscript subscript 𝑦 2 ′ superscript subscript 𝑦 3 ′ superscript subscript 𝑦 4 ′ superscript subscript 𝑦 5 ′ y_{1},y_{2},y_{3},y_{4},y_{5},y_{1}^{\prime},y_{2}^{\prime},y_{3}^{\prime},y_{4}^{\prime},y_{5}^{\prime} are given table  1 . All masses are in GeV and all phases in rad.

	$BR\left(t\to h^0+c\right)\%$	$BR\left(t\to h^0+u\right)\%$	$BR\left(H_1\to s\overline{b}+\overline{s}b\right)\%$	$BR\left(H_1\to d\overline{b}+\overline{d}b\right)\%$	$M_{H_1}$
1	$0.13$ %	$0.015$ %	$0.49$ %	$0.30$ %	$124.8$
2	$0.22$ %	$0.025$ %	$0.49$ %	$0.31$ %	$125.3$
3	$0.40$ %	$0.045$ %	$0.49$ %	$0.31$ %	$125.4$
4	$0.59$ %	$0.066$ %	$0.49$ %	$0.31$ %	$125.4$

Equations

$$BR(t\to h^0+c)$$

$$BR(t\to h^0+u)$$

$$BR(t\to h^0+c)$$

$$BR(t\to h^0+u)$$

$$q_{iL}\equiv \left(\begin{array}{c}{t}_{iL}\\ b_{iL}\end{array}\right)\sim \left(3,2,\frac{1}{6}\right);t_{iL}^c\sim \left(3^{\ast },1,-\frac{2}{3}\right);b_{iL}^c\sim \left(3^{\ast },1,\frac{1}{3}\right);i=1,2,3,4.$$

$$Q^c\equiv \left(\begin{array}{c}{B}_L^c\\ T_L^c\end{array}\right)\sim \left(3^{\ast },2,-\frac{1}{6}\right);T_L\sim \left(3,1,\frac{2}{3}\right);B_L\sim \left(3^{\ast },1,-\frac{1}{3}\right).$$

$${\bar{\xi }}_R^T=\left(\begin{array}{ccccc}{\bar{t}}_R& {\bar{T}}_R& {\bar{c}}_R& {\bar{u}}_R& {\bar{t}}_{4R}\end{array}\right),{\xi }_L^T=\left(\begin{array}{ccccc}{t}_L& T_L& c_L& u_L& {\bar{t}}_{4L}\end{array}\right).$$

$$-{\mathcal{L}}_m^u={\bar{\xi }}_R^T(M_u){\xi }_L+\text{h.c.}.$$

$$M_u=\left(\begin{array}{ccccc}{y}_1^{\prime }{v}_2/\sqrt{2}& h_5& 0& 0& 0\\ -h_3& y_2{v}_1/\sqrt{2}& -h_3^{\prime }& -h_3^{\prime \prime }& -h_6\\ 0& h_5^{\prime }& y_3^{\prime }{v}_2/\sqrt{2}& 0& 0\\ 0& h_5^{\prime \prime }& 0& y_4^{\prime }{v}_2/\sqrt{2}& 0\\ 0& h_8& 0& 0& y_5^{\prime }{v}_2/\sqrt{2}\end{array}\right).$$

$$D_R^{u†}(M_u)D_L^u=\text{diag}(m_{u_1},m_{u_2},m_{u_3},m_{u_4},m_{u_5}).$$

$$\left(\begin{array}{c}{t}_R\\ T_R\\ c_R\\ u_R\\ t_{4R}\end{array}\right)=D_R^u\left(\begin{array}{c}{u}_{1_R}\\ u_{2_R}\\ u_{3_R}\\ u_{4_R}\\ u_{5_R}\end{array}\right),\left(\begin{array}{c}{t}_L\\ T_L\\ c_L\\ u_L\\ t_{4L}\end{array}\right)=D_L^u\left(\begin{array}{c}{u}_{1_L}\\ u_{2_L}\\ u_{3_L}\\ u_{4_L}\\ u_{5_L}\end{array}\right).$$

$${\bar{\eta }}_R^T=\left(\begin{array}{ccccc}{\bar{b}}_R& {\bar{B}}_R& {\bar{s}}_R& {\bar{d}}_R& {\bar{b}}_{4R}\end{array}\right),{\eta }_L^T=\left(\begin{array}{ccccc}{b}_L& B_L& s_L& d_L& b_{4L}\end{array}\right).$$

$$-{\mathcal{L}}_m^d={\bar{\eta }}_R^T(M_d){\eta }_L+\text{h.c.},$$

$$M_d=\left(\begin{array}{ccccc}{y}_1{v}_1/\sqrt{2}& h_4& 0& 0& 0\\ h_3& y_2^{\prime }{v}_2/\sqrt{2}& h_3^{\prime }& h_3^{\prime \prime }& h_6\\ 0& h_4^{\prime }& y_3{v}_1/\sqrt{2}& 0& 0\\ 0& h_4^{\prime \prime }& 0& y_4{v}_1/\sqrt{2}& 0\\ 0& h_7& 0& 0& y_5{v}_1/\sqrt{2}\end{array}\right).$$

$$h_k=|h_k|e^{i{\chi }_k},h_k^{\prime }=|h_k^{\prime }|e^{i{\chi }_k^{\prime }},h_k^{\prime \prime }=|h_k^{\prime \prime }|e^{i{\chi }_k^{\prime \prime }}.$$

$$Y{M}^2{Y}^T=diag(M_{H_1}^2,M_{H_2}^2,M_{H_3}^2).$$

$$t\to h^{0}c,t\to h^{0}u.$$

$$\mathcal{L}={\bar{d}}_i({ϵ}_{ijk}+{\gamma }_5{ϵ}_{ijk}^{\prime })d_j{H}_k+{\bar{u}}_i({\eta }_{ijk}+{\gamma }_5{\eta }_{ijk}^{\prime })u_j{H}_k,$$

$${ϵ}_{ijk}=-\frac{1}{2\sqrt{2}}\{{\varphi }_{ij}(Y_{k2}+i{Y}_{k3}\cos \beta )+{\varphi }_{ji}^{\ast }(Y_{k2}-i{Y}_{k3}\cos \beta )$$

$$+{\alpha }_{ij}(Y_{k1}+i{Y}_{k3}\sin \beta )+{\alpha }_{ji}^{\ast }(Y_{k1}-i{Y}_{k3}\sin \beta )\},$$

$${ϵ}_{ijk}^{\prime }=-\frac{1}{2\sqrt{2}}\{-{\varphi }_{ij}(Y_{k2}+i{Y}_{k3}\cos \beta )+{\varphi }_{ji}^{\ast }(Y_{k2}-i{Y}_{k3}\cos \beta )$$

$$-{\alpha }_{ij}(Y_{k1}+i{Y}_{k3}\sin \beta )+{\alpha }_{ji}^{\ast }(Y_{k1}-i{Y}_{k3}\sin \beta )\},$$

$${\eta }_{ijk}=-\frac{1}{2\sqrt{2}}\{{\varphi }_{ij}^{\prime }(Y_{k2}+i{Y}_{k3}\cos \beta )+{\varphi }_{ji}^{{}^{\prime }\ast }(Y_{k2}-i{Y}_{k3}\cos \beta )$$

$$+{\alpha }_{ij}^{\prime }(Y_{k1}+i{Y}_{k3}\sin \beta )+{\alpha }_{ji}^{{}^{\prime }\ast }(Y_{k1}-i{Y}_{k3}\sin \beta )\},$$

$${\eta }_{ijk}^{\prime }=-\frac{1}{2\sqrt{2}}\{-{\varphi }_{ij}^{\prime }(Y_{k2}+i{Y}_{k3}\cos \beta )+{\varphi }_{ji}^{{}^{\prime }\ast }(Y_{k2}-i{Y}_{k3}\cos \beta )$$

$$-{\alpha }_{ij}^{\prime }(Y_{k1}+i{Y}_{k3}\sin \beta )+{\alpha }_{ji}^{{}^{\prime }\ast }(Y_{k1}-i{Y}_{k3}\sin \beta )\},$$

$${\varphi }_{ij}$$

$${\alpha }_{ij}$$

$${\varphi }_{ij}^{\prime }$$

$${\alpha }_{ij}^{\prime }$$

$$\Gamma (t\to H_1+c)$$

$$\{(m_t+m_c{)}^2|{\eta }_{311}{|}^2+(m_t-m_c{)}^2|{\eta }_{311}^{\prime }{|}^2-M_{H_1}^2(|{\eta }_{311}{|}^2+|{\eta }_{311}^{\prime }{|}^2)\},$$

$$\Gamma (t\to H_1+u)$$

$$\{(m_t+m_u{)}^2|{\eta }_{411}{|}^2+(m_t-m_u{)}^2|{\eta }_{411}^{\prime }{|}^2-M_{H_1}^2(|{\eta }_{411}{|}^2+|{\eta }_{411}^{\prime }{|}^2)\}.$$

$$\Gamma (H_1\to b+\bar{s})$$

$$\{(|{ϵ}_{311}{|}^2+|{ϵ}_{311}^{\prime }{|}^2)(M_{H_1}^2-m_b^2-m_s^2)-(|{ϵ}_{311}{|}^2-|{ϵ}_{311}^{\prime }{|}^2)(2m_b{m}_s)\},$$

$$\Gamma (H_1\to \bar{b}+s)$$

$$\{(|{ϵ}_{131}{|}^2+|{ϵ}_{131}^{\prime }{|}^2)(M_{H_1}^2-m_b^2-m_s^2)-(|{ϵ}_{131}{|}^2-|{ϵ}_{131}^{\prime }{|}^2)(2m_b{m}_s)\}.$$

$$BR(H_1\to b\bar{s}+\bar{b}s)=\frac{\Gamma (H_1\to b\bar{s})+\Gamma (H_1\to \bar{b}s)}{\Gamma (H_1\to \bar{b}s)+\Gamma (H_1\to \bar{s}b)+{\sum }_i\Gamma (H_1\to \overline{f_i}{f}_i)+{\Gamma }_{H_{1}DB}},$$

Full text

**Flavor violating top decays and flavor violating quark decays of the Higgs boson **

Tarek Ibrahima**Email: tibrahim@zewailcity.edu.eg, Ahmad Itanib*†††Email: ahmad.it@gmail.com, Pran Nathc‡‡‡Email: nath@neu.edu, Anas Zorika§§§Email: p-azorik@zewailcity.edu.eg

aUniversity of Science and Technology, Zewail City of Science and Technology,

6th of October City, Giza 12588, Egypt555Permanent address: Department of Physics, Faculty of Science, University of Alexandria, Alexandria, Egypt

bDepartment of Physics, Beirut Arab University, Beirut 11-5020,Lebanon

cDepartment of Physics, Northeastern University, Boston, MA 02115-5000, USA

Abstract

In the standard model flavor violating decays of the top quark and of the Higgs boson are highly suppressed. Further, the flavor violating decays of the top and of the Higgs are also small in MSSM and not observable in current or in near future experiment. In this work we show that much larger branching ratios for these decays can be achieved in an extended MSSM model with an additional vector like quark generation. Specifically we show that in the extended model one can achieve branching ratios for $t\to h^{0}c$ and $t\to h^{0}u$ as large as the current experimental upper limits given by the ATLAS and the CMS Collaborations. We also analyze the flavor violating quark decay of the Higgs boson, i.e., $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$. Here again one finds that the branching ratio for these decays can be as large as $O(1)\%$. The analysis is done with inclusion of the CP phases in the Higgs sector, and the effect of CP phases on the branching ratios is investigated. Specifically the Higgs sector spectrum and mixings are computed involving quarks and mirror quarks, squarks and mirror squarks in the loops consistent with the Higgs boson mass constraint. The resulting effective Lagrangian with inclusion of the vector like quark generation induce flavor violating decays at the tree level. The test of the branching ratios predicted could come with further data from LHC13 and such branching ratios could also be accessible at future colliders such as the Higgs factories where the Higgs couplings to fermions will be determined with greater precision.

Keywords:  Flavor violation, CP phases, top and Higgs decays

PACS numbers: 12.60.-i, 14.60.Fg

1 Introduction

The flavor violating decays provide a window to new physics beyond the standard model. Thus in the standard model the flavor violating top decay $t\to h^{0}c$ has a branching ratio which is rather small, i.e., $O(10^{-15})$ [1, 2, 3, 4]. In the two Higgs doublet model the branching ratio is predicted to be in the range $10^{-5}-10^{-3}$ [5, 6, 7, 8, 9, 10, 11]. Currently experimental results on $t\to h^{0}+c$ and on $t\to h^{0}+u$ from the ATLAS collaboration [12] and from the CMS Collaboration  [13] are as follows (for a review of experiment see  [14]): From the ATLAS Collaboration  [12] one has for the branching ratios

[TABLE]

and from the CMS Collaboration [13] one has

[TABLE]

The flavor violating decays of the Higgs boson offer another window to the discovery of new physics. In a previous work we analyzed the flavor violating Higgs decays into leptons using an extension of MSSM with a vector like leptonic generation [15]. Experimental limits on such decays exist from the ATLAS[16] and from the CMS [17] Collaborations. In [15]. it was shown that in the extended MSSM model one could achieve up to $O(1)\%$ branching ratio for the flavor violating leptonic decay $h^{0}\to\tau\mu$. Here we carry out a similar analysis with inclusion of a vector like quark generation to analyze the flavor violating Higgs couplings to quarks which we utilize to compute the flavor violating decays of the top $t\to h^{0}c$ and $t\to h^{0}u$. Other significant channels for the observation of flavor violating process are the decays $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$. In the standard model the branching ratio for such a processes is order $10^{-7}$ [18] or less and unobservable. In the framework of SUGRA/MSSM model the branching ratio is still small, i.e., $O(10^{-4})$ [19, 20, 21]. In the extended MSSM model discussed here, we find that the branching ratios $BR(h^{0}\to b\bar{s}+\bar{b}s)$ and $BR(h^{0}\to b\bar{d}+\bar{b}d)$ can be as large as $O(1)\%$ which is several orders of magnitude larger than in MSSM and possibly within reach of current and future experiment. More data is expected in the near future which makes an investigation of the flavor violating decays of the top and of the Higgs a timely topic of investigation.

The outline of the rest of the paper is as follows: In section 2 a discussion of the extended MSSM model with a vector like generation is given. Here it is shown that the quark couplings involve flavor violating vertices. In section 3 an analysis of the flavor violating top decays $t\to h^{0}c$ and $t\to h^{0}u$ is given. In section 4 the flavor violating Higgs decays $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$ are discussed. In sec 5 a numerical analysis of sizes of the branching ratios of the flavor violating processes is given. Conclusions are given in sec 6. Further details of the squark mass squared matrices including the vectorlike squarks are given in the appendix A and a brief discussion of CP even -CP odd Higgs mixing is given in appendix B.

2 The Model and Notation

Here we describe the model briefly and further details are given in the appendix A and appendix B. The model we consider is an extension of MSSM with an additional vectorlike multiplet. Vectorlike multiplets appear in a variety of settings which include grand unified models, string and D brane models [22, 23, 24, 25]. Further, they are anomaly free. Several analyses have recently appeared which utilize such multiplets [26, 27, 28, 29, 30, 31, 32, 33, 34, 35]. Here we focus on the quark sector where the vectorlike multiplet consists of a fourth generation of quarks and their mirror quarks. Thus the quark sector of the extended MSSM model has the matter content given by

[TABLE]

[TABLE]

The numbers in the braces in Eq.(3) and Eq.(4) show the properties under $SU(3)_{C}\times SU(2)_{L}\times U(1)_{Y}$ where the first two entries label the representations for $SU(3)_{C}$ and $SU(2)_{L}$ and the last one gives the value of the hypercharge normalized so that $Q=T_{3}+Y$. We allow the mixing of the vectorlike generation with the first three generations. We display now some relevant features of the model. In the up quark sector we choose a basis as follows

[TABLE]

and we write the mass term so that

[TABLE]

The superpotential of the model (as shown in appendix A) leads to the up-quark mass matrix $M_{u}$ where

[TABLE]

This mass matrix is not hermitian and a bi-unitary transformation is needed to diagonalize it. Thus one has

[TABLE]

Under the bi-unitary transformations the basis vectors transform so that

[TABLE]

A similar analysis can be carried out for the down quarks. Here we choose the basis set as

[TABLE]

In this basis the down quark mass terms are given by

[TABLE]

where using the interactions of appendix A, $M_{d}$ has the following form

[TABLE]

In general the parameters $h_{3},h_{4},h_{5},h_{3}^{\prime},h_{4}^{\prime},h_{5}^{\prime},h_{3}^{\prime\prime},h_{4}^{\prime\prime},h_{5}^{\prime\prime},h_{6},h_{7},h_{8}$ appearing in Eqs. (7) and (12) can be complex and we define their phases so that

[TABLE]

The squark sector of the model contains a variety of terms including F -type, D-type, as well as soft mixings terms involving squarks and mirror squarks. The details of these contributions to squark mass square matrices are discussed in appendix A. In addition to the CP phases arising from the mixing parameters as given by Eq. (13) there are CP phases arising from the soft parameters as discussed in appendix A. In general these phases can be large. The CP phases in general contribute to the EDM of the quarks and the leptons. Compatiblity with experiment can be achieved in a variety of ways such as via mass suppression [36, 37] or the cancellation mechanism [38] [39, 40, 41, 42] (for a review see [43]).

We note that the up quark matrix given by Eq. (7) and the down quark matrix given by Eq. (12) contain off diagonal elements in the flavor space. Additionally Eq. (23) and Eq. (28) contain flavor mixings. It is the presence of these mixings that lead to flavor violating decays of the top quark and the flavor violating decays of the Higgs boson. This was done with inclusion of CP violating phases in the Higgs sector. An analysis of the effects of CP phases on the Higgs boson masses and mixings with inclusion of the vector like generation was given in  [44] and we utilize the work of that analysis here. (For previous work on the effects of CP phases on Higgs boson massess and mixings see [45, 46, 47, 48, 49, 50, 51, 52, 53, 54]). The inclusion of CP phases affects the Higgs boson masses and mixings. Specifically the mass eigenstates of the neutral Higgs bosons are no longer CP eigenstates and further their couplings to quarks and leptons are dependent on the phases. In this work we further show that the coupling of the neutral Higgs bosons with the quarks allow flavor violating decays even at the tree level. Specifically we have analyzed the loop corrections to the scalar potential in the Higgs sector using the super trace technique. We have produced the corrected Higgs mass2 matrix and diagonalized it

[TABLE]

A brief discussion of the CP phases on the CP even-CP odd Higgs mixing is given in appendix B.

3 Flavor violating top decays: $t\to h^{0}c$ and $t\to h^{0}u$

In this section we compute the flavor violating decays of the top quark, i.e.,

[TABLE]

As mentioned in section 1 experimental upper limits from the ATLAS  [12] and from and the CMS [17] Collaborations exist on these decays as exhibited in Eq.(1) and Eq.(2). In the analysis presented here we will show that branching ratios for the processes Eq. (15) close to the upper limit exhibited in Eq. (1) can be achieved.

Using the superpotential Eq.(23), one can write the interaction between the mass eigen states of the Higgs bosons $H_{k}$ and the quark mass eigen states so that

[TABLE]

where the couplings are given by

[TABLE]

and the parameters $\phi_{ij}$, $\alpha_{ij}$, $\phi^{\prime}_{ij}$ and $\alpha^{\prime}{ij}$ are given by

[TABLE]

Using the interaction (16), we calculate the decay widths of the top quark into the lightest Higgs and the flavors $c$ and $u$ to be

[TABLE]

To calculate the branching ratio of the top quark to the above channels, we just need to divide the partial widths for these top decays by the total width of the top quark which is $1.41^{+.19}_{-0.15}$ GeV. In the analysis here we take the center value of $1.41$ GeV for the width.

4 Flavor violating Higgs boson decays: $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$

We proceed now to discuss the flavor violating quark decays of the Higgs, and specifically $h^{0}\to b\bar{s}+\bar{b}s$. In future data from collider experiments is likely to either discover such decays or put more stringent constraints on them. This may happen with LHC13 data after the high luminosity upgrade. Further, flavor violating decays would be explored at Higgs factories which are under active consideration such as the International Linear Collider (ILC- Japan), Circular Electron-Positron Collider (CEPC-China) or Future Circular Collider-ee (FCC-ee: CERN). Thus flavor violating decays of the Higgs are of considerable interest.

Using the interaction (16), we calculate the decay widths of the lightest Higgs boson into $b+\bar{s}$ and $\bar{b}+s$ to be

[TABLE]

To calculate the branching ratio of the lightest Higgs to the above channels, we just need to divide the partial decay widths by the total width of the Higgs boson. Thus the flavor violating branching ratio of $H_{1}$ into $b\bar{s}+\bar{b}s$ is given by

[TABLE]

where $f_{i}$ stand for fermionic particles that have coupling with the Higgs boson and have a mass less than half the Higgs boson mass and $\Gamma_{H_{1}DB}$ is the decay width into diboson states which include $gg,\gamma\gamma,\gamma Z,ZZ,WW$. Thus the computation of the branching ratios of Eq. (21) involve the decay widths

[TABLE]

An identical analysis for $H_{1}\to b\bar{d}+\bar{b}d$ holds with $s$ replaced with $d$. In the decays of the lightest Higgs into $ZZ$ and $WW$, these states are off shell and the on-shell final states are dominantly four fermions arising from the decay of the $Z$ and $W$ bosons. We note that $b\bar{s}+\bar{b}s$ and $b\bar{d}+\bar{b}d$ final states do not originate from any of the $ZZ$ and $WW$ diboson decay modes. Further, the Higgs boson observed at $\sim 125$ GeV  [55, 56] is effectively in the decoupling limit. Thus we approximate the diboson decay widths as given by the standard model.

5 Discussion of numerical results

We discuss now the numerical analysis of the flavor violating decays of the top: $t\to h^{0}+c$ and $t\to h^{0}+u$, and the flavor violating Higgs decay $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$. As a first step we diagonalize the $5\times 5$ up and down quark and mirror quark mass matrices. In the diagonalzation the parameters are chosen so as produce the masses of the three generation of up and down quarks as given by the PDG [57]. Further, the inputs are chosen to generate the masses of the remaining quarks and mirror quarks to be consistent with the lower bounds given by PDG [57] for heavy quarks. As discussed in section 2 we include loop corrections to the Higgs boson potential which generate mixings of the CP even-CP odd Higgs leading to the mass eigenstates $H_{1}H_{2}H_{3}$ which are not eigenstates of CP. The analysis of the loop corrections to the Higgs involves masses of the squarks and the mirror squarks which are given in appendix B. In the numerical analysis of the loop corrections we make the following simplifying assumption: $m^{u^{2}}_{0}=M^{2}_{\tilde{T}}=M^{2}_{\tilde{t}_{1}}=M^{2}_{\tilde{t}_{2}}=M^{2}_{\tilde{t}_{3}}=M^{2}_{\tilde{t}_{4}}$ and $m^{d^{2}}_{0}=M^{2}_{\tilde{1}L}=M^{2}_{\tilde{B}}=M^{2}_{\tilde{b}_{1}}=M^{2}_{\tilde{Q}}=M^{2}_{\tilde{2}L}=M^{2}_{\tilde{b}_{2}}=M^{2}_{\tilde{3}L}=M^{2}_{\tilde{b}_{3}}=M^{2}_{\tilde{4}L}=M^{2}_{\tilde{b}_{4}}$. and $m^{u}_{0}=m^{d}_{0}=m_{0}$. Additionally the trilinear couplings are chosen so that: $A^{u}_{0}=A_{t}=A_{T}=A_{c}=A_{u}=A_{4t}$ and $A^{d}_{0}=A_{b}=A_{B}=A_{s}=A_{d}=A_{4b}$.

In table 1 (see caption) we exhibit the inputs for the matrices of Eqs. 7 and  12 which lead to the masses of the three generations quarks as given by the PDG [57] and produce masses for the extra quarks and mirror quarks. The masses of these extra quarks and mirror quarks are exhibited in table 1 and they are consistent with the lower limits of the heavy quarks given by PDG [57]. We utilize the inputs of table 1 in the computation of the flavor violating branching ratios of the top and the Higgs given in table 2. Here, however, we need to specify also the soft parameters. These are chosen so they provide the desired loop correction to the Higgs boson mass to be consistent with the experimental value of $\sim 125$ GeV. The inputs are given in the caption of table 2. For the flavor violating top decays $t\to h^{0}c$ and $t\to h^{0}u$, the branching ratios are orders of magnitude larger than achievable in the standard model or in the MSSM, and come close to the upper limits given by ATLAS [12](see Eq.(1)) and CMS [13](see Eq.(2)). A similar result holds for the flavor violating quark decay of the Higgs into $b\bar{s}+\bar{b}s$ and $b\bar{d}+\bar{b}d$. Again in the standard model and in MSSM, the branching ratio for the flavor violating decay of the Higgs is orders of magnitude smaller than a precent. However, in the extended MSSM model discussed here, the branching ratio can be as large as $O(1)\%$. A branching ratio of this size could be tested with more data from LHC13 and possibly at future colliders such as the Higgs factories.

The dependence of the flavor violating branching ratios on CP phases are discussed in Figs. (1)-(8). In Fig. (1) we exhibit the dependence of $t\to h^{0}c$ and $t\to h^{0}u$ on the CP phase $\chi_{3}$. A similar analysis for these branching ratios on $\chi_{5}$ is given in Fig.(2), on $\chi_{6}$ in Fig. (3), and on $\chi_{8}$ in Fig. (4). In these analyses one finds that the branching ratios are sensitive to the phases. A similar analysis for the flavor violating branching ratio of the Higgs $H_{1}\to b\bar{s}+\bar{b}s$ and $H_{1}\to b\bar{d}+\bar{b}d$ are given in Figs. (5)-(8). In Fig. (5) the analysis is for the dependence on the CP phase $\chi_{3}$, on $\chi_{4}$ in Fig. (6), on $\chi_{6}$ in Fig. (7), and on $\chi_{7}$ in Fig. (8). As in the case of flavor violating decays of the top here too we find that the decays are sensitive to the CP phases. We note that in all these cases the mixings with the vector like generation enter prominently.

6 Conclusion

As is well known flavor violating processes involving the Higgs and the quarks are highly suppressed in the standard model and beyond the reach of experimental observation. They are also suppressed in MSSM and beyond the reach of experiment. In this work we have analyzed such processes in the framework of an extended MSSM with a vector like quark generation. In this framework we first analyze the flavor violating top decays $t\to h^{0}c$ and $t\to h^{0}u$. Here it is shown that branching ratios can be several orders of magnitude larger than in the standard model on in MSSM and could be as large as the upper limits given by the ATLAS and the CMS Collaborations. Next we analyze the Higgs boson decay $h^{0}\to b\bar{s}+\bar{b}s$ and $h^{0}\to b\bar{d}+\bar{b}d$. As in the flavor violating decays of the top, here also we find that the branching ratios in this model can be several orders of magnitude larger than in the standard model or in MSSM could be as large as $O(1)\%$. Such a branching ratio may be testable with more data from LHC13 and may also lie within reach of future colliders specifically the Higgs factories.

**Acknowledgments: ** This research was supported in part by the NSF Grant d PHY-1620575.

7 Appendix A: Squark mass matrices

In this appendix we give further details of the extended MSSM model with vector like generation. As discussed in section 2 we allow for mixing between the vector generation and specifically the mirrors and the standard three generations of quarks. The superpotential allowing such mixings is given by

[TABLE]

Here the couplings are in general complex. Thus, for example, $\mu$ is the complex Higgs mixing parameter so that $\mu=|\mu|e^{i\theta_{\mu}}$. The mass terms for the up quarks (ups), the mirror up quarks, the down quarks, and the mirror down quarks arise from the term

[TABLE]

where $\psi$ and $A$ stand for generic two-component fermion and scalar fields. After spontaneous breaking of the electroweak symmetry, ($\langle H_{1}^{1}\rangle=v_{1}/\sqrt{2}$ and $\langle H_{2}^{2}\rangle=v_{2}/\sqrt{2}$), we have the following set of mass terms written in the four-component spinor notation so that

[TABLE]

where the basis vectors are defined in Eq. (5) and Eq. (10).

Next we consider the mixings among the squarks and mirror squarks. The interactions that contribute to them receive $F$ type and $D$ type contributions as well contributions from soft terms. Thus the terms that contribute to the mixings are given by

[TABLE]

where $-{\cal L}_{F}=V_{F}=F_{i}F^{*}_{i}$ and $F_{i}=\partial W/\partial A_{i}$ while ${\cal L}_{D}$ is given by

[TABLE]

For ${\cal L}_{\rm soft}$ we assume the following form

[TABLE]

Here $M_{\tilde{1}L},M_{\tilde{T}}$, etc are the soft masses and $A_{t},A_{b}$, etc are the trilinear couplings. The trilinear couplings are in general complex and we define their phases so that

[TABLE]

After spontaneous breaking of the electroweak symmetry, when the Higgs bosons develop VEVs, we construct the scalar mass squared matrices using Eqs. (26), (27), and (28). Thus we denote the mass squared matrix for the down squarks and the down mirror squarks in the basis $(\tilde{b}_{L},\tilde{B}_{L},\tilde{b}_{R},\tilde{B}_{R},\tilde{s}_{L},\tilde{s}_{R},\tilde{d}_{L},\tilde{d}_{R},$ $\tilde{b}_{4L},\tilde{b}_{4R})$ by $(M^{2}_{\tilde{d}})_{ij}=M^{2}_{ij}$. This mass squared matrix is hermitian and can be diagonalized by the unitary transformation

[TABLE]

Similarly we write the mass squared matrix in the up squark and up mirror squark sector in the basis $(\tilde{t}_{L},\tilde{T}_{L},$ $\tilde{t}_{R},\tilde{T}_{R},\tilde{c}_{L},\tilde{c}_{R},\tilde{u}_{L},\tilde{u}_{R},\tilde{t}_{4L},\tilde{t}_{4R})$ and denote it by $(M^{2}_{\tilde{u}})_{ij}=m^{2}_{ij}$ which is also a hermitian matrix, and can be diagonalized by the unitary transformation

[TABLE]

We display now the matrix elements $M^{2}_{ij}$ and $m^{2}_{ij}$. First for $M^{2}_{ij}$ we have

[TABLE]

[TABLE]

[TABLE]

Next for $m^{2}_{ij}$ we have

[TABLE]

[TABLE]

[TABLE]

8 Appendix B: CP even-CP odd Higgs mixing with vector like quarks

For completeness we give here a brief discussion of the mixings of CP even Higgs and CP odd Higgs which arise as a consequence of CP phases in the soft SUSY parameters of the theory. While there is no CP violation in the Higgs sector at the tree level, the CP phases from the soft breaking sector induce CP violation at the loop level. Thus at the tree level the scalar potential in the Higgs sector is given by

[TABLE]

This potential has no CP violation. However, there are important loop corrections to the potential. At the one loop level they are given by

[TABLE]

where the super trace sums over bosons and fermions circulating in the loop. In our case the largest contributions to the scalar potential arise from the third generation quarks and squarks, and from the vector like quarks and their super partners in the loop.

The loop corrections introduce a CP phase in the Higgs sector and one may parametrize the Higgs fields so that

[TABLE]

[TABLE]

The minimization of the scalar potential including the loop correction induces mixing of CP even and CP odd Higgs fields. Thus at the tree level one may write the components of the neutral Higgs fields as $\Phi_{a}$ (a=1-4) where $\Phi_{a}=(\phi_{1},\phi_{2},\psi_{1},\psi_{2})$ and where $\phi_{1},\phi_{2}$ are CP even and $\psi_{1},\psi_{2}$ are CP odd. After minimization of the full potential including the loop corrections we find a mixing of CP even and CP odd states and the mass matrix in the neutral Higgs sector then takes the following form

[TABLE]

Here $M_{ab}^{2(0)}$ are the contributions at the tree level and $\Delta M_{ab}^{2}$ are the contributions at the loop level. The loop correction to the mass squared matrix takes the form

[TABLE]

where e=2.718. Eq.(38) gives a $4\times 4$ mass square matrix in the basis $(\phi_{1},\phi_{2},\psi_{1},\psi_{2})$. The $4\times 4$ mass squared matrix can be reduced to a $3\times 3$ mass squared matrix by use of the following linear combinations of $\psi_{1}$ and $\psi_{2}$.

[TABLE]

where $\tan\beta=<H_{2}>/<H_{1}>$. In the new basis $\psi^{\prime}_{2}$ decouples and can be identified as a Goldstone field with a zero mass eigenvalue. The remaining Higgs mass squared matrix now involves only three fields and in the basis $(\phi_{1},\phi_{2},\psi_{1}^{\prime})$ is given by

[TABLE]

Here $s_{\beta}=\sin\beta$, $c_{\beta}=\cos\beta$, $M_{Z}$ is the $Z$-boson mass, and $M_{A}$ is the mass of the CP odd Higgs before mixing. As mentioned in section 2 a detailed analysis of $\Delta_{ij}$ is given in [44] including a vectorlike quark generation and we utilize the results of  [44] in the computation of the Higgs mass eigenstates and mixings in the analysis given in this work.