$K$-sample omnibus non-proportional hazards tests based on right-censored data

TL;DR

This paper introduces new, powerful statistical tests for comparing non-proportional hazard functions in right-censored data, overcoming key challenges posed by censoring and unequal group distributions, with demonstrated superior performance.

Contribution

The paper develops invariant, consistent tests for non-proportional hazards that address censoring and unequal distributions, extending existing methods to censored data.

Findings

Proposed tests outperform existing methods in simulations under non-proportional hazards.

The tests are invariant and consistent, suitable for right-censored data.

Implementation available in R package KONPsurv.

Abstract

This work presents novel and powerful tests for comparing non-proportional hazard functions, based on sample-space partitions. Right censoring introduces two major difficulties which make the existing sample-space partition tests for uncensored data non-applicable: (i) the actual event times of censored observations are unknown; and (ii) the standard permutation procedure is invalid in case the censoring distributions of the groups are unequal. We overcome these two obstacles, introduce invariant tests, and prove their consistency. Extensive simulations reveal that under non-proportional alternatives, the proposed tests are often of higher power compared with existing popular tests for non-proportional hazards. Efficient implementation of our tests is available in the R package KONPsurv, which can be freely downloaded from {https://github.com/matan-schles/KONPsurv

Figures

Tables

Table 1. Table 1: Computation Time (seconds)

Censoring rates	$n$
of groups 1 and 2	100	200	300	400	1000
(25%,25%)	1.7	7.1	16.5	30.0	204.2
(50%,50%)	0.9	3.5	8.0	14.5	97.3
(27%,55%)	0.9	3.8	8.8	16.1	114.1
(40%,55%)	0.8	3.2	7.5	13.7	93.8

Table 2. Table 2: Unequal Censoring Distributions

Group	Small difference	Substantial difference
1	$\min \{U(a,b),Exp\left({\theta }_1\right)\}$	$\min \{U(a,b),Exp\left({\theta }_1\right)\}$
2	$\min \{U(a,b),Exp\left({\theta }_2\right)\}$	$U(a,b)$

Table 3. Table 3: Examples: Gastrointestinal Tumor Study (GST) and Urothelial Carcinoma Data (UCD)

Test	GST $p$-value	UCD $p$-value
KONP Pearson	0.0109	0.0049
KONP LR	0.0108	0.0049
Cauchy combination $Cau$	0.0164	0.0071
Yang–Prentice 1	0.0304	0.0186
Yang–Prentice 2	0.0800	0.0252
Yang–Prentice Invariant	0.0479	0.0219
Logrank	0.6350	0.0673
Pepe–Fleming	0.9464	0.2362
Peto–Peto	0.0465	0.3807
Lee	0.0968	0.0054
MaxCombo	0.0908	0.0061

Table 4. Table A1: Description of the Non-Proportional Hazards Simulation Scenarios

Scenario	Failure time distribution and reference	Graphical description	Censoring distribution
Null	$F_1\left(t\right)=log$-$Logistic(1,1)$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.1,0)$ unequal: $(a,b)=(0,10)$ $({\theta }_1,{\theta }_2)=(0.85,0.25)$
A	$F_2=Weibull(0.849,10)$ Difference in middle times, Uno et al. (2015)		equal: $C\sim Weibull(\alpha ,\beta )$ $(\alpha ,\beta )=(18,16),(1.5,9)$ unequal: $(a,b)=(2,30)$ $({\theta }_1,{\theta }_2)=(0.06,0.04)$
B	$F_2\left(t\right)=Weibull(0.849,10)$ Difference in early times, Uno et al. (2015)		equal: $C\sim Weibull(\alpha ,\beta )$ $(\alpha ,\beta )=(10,15),(3,7.5)$ unequal: $(a,b)=(2,30)$ $({\theta }_1,{\theta }_2)=(0.06,0.04)$
C	Difference in early times, Pepe and Fleming (1989; 1991)		equal: $C\sim U(\alpha ,\beta )$ $(\alpha ,\beta )=(1,2),(0,1.6)$ unequal: $(a,b)=(0.01,3)$ $({\theta }_1,{\theta }_2)=(0.5,0.8)$
D	Difference in early times, Pepe and Fleming (1989; 1991)		equal: $C\sim U(\alpha ,\beta )$ $(\alpha ,\beta )=(1.1,3),(0.1,2.1)$ unequal: $(a,b)=(0.5,3.5)$ $({\theta }_1,{\theta }_2)=(0.5,0.3)$
E	$F_1\left(t\right)=Exp\left(1\right)$ Proportional difference in late times, Pepe and Fleming (1989; 1991)		equal: $C\sim U(\alpha ,\beta )$ $(\alpha ,\beta )=(0.9,1.2),(0.1,1.1)$ unequal: $(a,b)=(0.01,2.3)$ $({\theta }_1,{\theta }_2)=(0.5,0.8)$
F	$F_1\left(t\right)=1-{\left\{1+\exp \left(2\right)t\right\}}^{-\exp \left(1\right)}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice Model (1), Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.35,0.01)$ unequal: $(a,b)=(0.5,5)$ $({\theta }_1,{\theta }_2)=(0.7,0.25)$
G	$F_1\left(t\right)=1-{\left\{1+t/\exp \left(2\right)\right\}}^{-\exp \left(1\right)}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice Model (1), Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.4,0.4)$ unequal: $(a,b)=(0.5,7)$ $({\theta }_1,{\theta }_2)=(0.2,0.4)$
H	$F_1\left(t\right)=Exp\left(1\right)$ U shape hazard ratio, Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(0.25,-0.7)$ unequal: $(a,b)=(0,2.5)$ $({\theta }_1,{\theta }_2)=(0.5,0.25)$
I-1	$F_1\left(t\right)=Exp\left(1\right)$ $F_2\left(t\right)=1-{\left\{4\exp \left(t\right)+3\right\}}^{-0.5}$ Yang and Prentice Model (1),		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.32,1)$ unequal: $(a,b)=(0,4)$ $({\theta }_1,{\theta }_2)=(0.8,0.4)$
I-2	$F_1\left(t\right)=Exp\left(1\right)$ $F_2\left(t\right)=1-{\left[\left\{\exp \left(t\right)+3\right\}/4\right]}^{-2}$ Yang and Prentice Model (1)		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,0.85)$ unequal: $(a,b)=(0,4)$ $({\theta }_1,{\theta }_2)=(0.8,0.25)$
I-3	$F_1\left(t\right)={\left\{5/\left(0.5t+5\right)\right\}}^5$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice Model (1)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.2,0.25)$ unequal: $(a,b)=(0,10)$ $({\theta }_1,{\theta }_2,)=(0.4,0.25)$
J-1	$F_1\left(t\right)=Exp\left(1\right)$ strong monotone hazards ratio assumption does not hold		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,1)$ unequal: $(a,b)=(0,4)$ $({\theta }_1,{\theta }_2)=(0.9,0.5)$
J-2	$F_1\left(t\right)=Exp\left(1\right)$ strong monotone hazards ratio assumption does not hold		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,1)$ unequal: $(a,b)=(0,4)$ $({\theta }_1,{\theta }_2)=(0.9,0.5)$
J-3	$F_1\left(t\right)=Exp\left(1\right)$ strong monotone hazards ratio assumption does not hold		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.25,0.75)$ unequal: $(a,b)=(0,5)$ $({\theta }_1,{\theta }_2)=(0.9,0.3)$
K-1	$F_1\left(t\right)=Exp\left(1\right)$ $F_2\left(t\right)={\left\{4\exp \left(1.7t\right)-3\right\}}^{-1/3.4}$ strong monotone hazards ratio		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,0.9)$ unequal: $(a,b)=(0,5)$ $({\theta }_1,{\theta }_2)=(0.9,0.3)$
K-2	$F_1\left(t\right)=Exp\left(1\right)$ $F_2\left(t\right)={\left\{8\exp \left(2t\right)-7\right\}}^{-0.25}$ strong monotone hazards ratio		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,1)$ unequal: $(a,b)=(0,5)$ $({\theta }_1,{\theta }_2)=(0.9,0.45)$
K-3	$F_1\left(t\right)=Exp\left(0.4\right)$ $F_2\left(t\right)={\left[\left\{2\exp \left(t\right)+64\right\}/66\right]}^{-2.5}$ strong monotone hazards ratio		equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.13,35)$ unequal: $(a,b)=(0,10)$ $({\theta }_1,{\theta }_2)=(0.3,0.15)$

Table 5. Table A2: Additional 2-samples settings: proportional hazards and close to proportional hazards

Scenario	Failure time distribution and reference	Graphical description	Censroing distribution
L	$F_1\left(t\right)=Weibull(0.849,20)$ $F_2\left(t\right)=Weibull(0.849,10)$ Proportional hazards, Uno et al. (2015)		equal: $C\sim Weibull(\alpha ,\beta )$ $(\alpha ,\beta )=(5,24),(1.5,12)$ unequal: $(a,b)=(0,40)$ $({\theta }_1,{\theta }_2)=(0.025,0.05)$
M	Substantial difference in early times, Pepe and Fleming (1989; 1991)		equal: $C\sim Weibull(\alpha ,\beta )$ $(\alpha ,\beta )=(0.9,5.5),(0.35,3.4)$ unequal: $(a,b)=(0,4.5)$ $({\theta }_1,{\theta }_2)=(0.25,0.14)$
N	$F_1\left(t\right)=1-{\left(1+t\right)}^{exp\left(-0.5\right)}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice model, Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(0.75,-0.1)$ unequal: $(a,b)=(0,12)$ $({\theta }_1,{\theta }_2)=(1.5,0.4)$
O	$F_1\left(t\right)=1-{\left(1+t\right)}^{exp\left(-1\right)}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice model, Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(-0.6,-1.8)$ unequal: $(a,b)=(0,8)$ $({\theta }_1,{\theta }_2)=(2,0.5)$
P	$F_1\left(t\right)=1-{\left\{1+t\exp \left(1\right)\right\}}^{-1}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice model, Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(0.6,-0.5)$ unequal: $(a,b)=(0,8)$ $({\theta }_1,{\theta }_2)=(2,0.5)$
Q	$F_1\left(t\right)=1-{\left\{1+t/\exp \left(1\right)\right\}}^{-exp\left(1\right)}$ $F_2\left(t\right)=log$-$Logistic(1,1)$ Yang and Prentice model, Yang and Prentice (2010)		equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(0.7,-0.15)$ unequal: $(a,b)=(0,8)$ $({\theta }_1,{\theta }_2)=(0.9,0.3)$

Table 6. Table S3: K 𝐾 K -sample scenarios with K = 3 , 4 , 5 𝐾 3 4 5 K=3,4,5 : under the null hypothesis and based on scenarios D and J-2 from the main text

Scenario	Failure time	Censroing distribution
Null $K=3$	$F_1\left(t\right)=F_2\left(t\right)=F_3\left(t\right)=$ $log$-$Logistic(1,1)$	equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.1,0)$ 40% and 55%: $C_1,C_2\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_3\sim U(0,10)$ 27% and 55%: $C_1\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_2\sim \min \{Exp\left(0.25\right),U(0,10)\}$ $C_3\sim U(0,10)$
Null $K=4$	$F_1\left(t\right)=F_2\left(t\right)=F_3\left(t\right)=F_4\left(t\right)=$ $log$-$Logistic(1,1)$	equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.1,0)$ 40% and 55%: $C_1,C_2\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_3,C_4\sim U(0,10)$ 27% and 55%: $C_1\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_2\sim \min \{Exp\left(0.25\right),U(0,10)\}$ $C_3\sim U(0,10)$ $C_4\sim log-Normal(1.5,0.5)$
Null $K=5$	$F_1\left(t\right)=F_2\left(t\right)=F_3\left(t\right)=F_4\left(t\right)=F_5\left(t\right)=$ $log$-$Logistic(1,1)$	equal: $C\sim log$-$Normal(\alpha ,0.5)$ $\alpha =(1.1,0)$ 40% and 55%: $C_1,C_2\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_3,C_4,C_5\sim U(0,10)$ 27% and 55%: $C_1\sim \min \{Exp\left(0.85\right),U(0,10)\}$ $C_2\sim \min \{Exp\left(0.25\right),U(0,10)\}$ $C_3\sim U(0,10)$ $C_4\sim log-Normal(1.5,0.5)$ $C_5\sim Exp\left(1.5\right)$
D	$F_2\left(t\right)=$	equal: $C\sim U(\alpha ,\beta )$ $(\alpha ,\beta )=(1.1,3),(0.1,2.1)$ 40% and 55%: $C_1,C_2\sim \min \{Exp\left(0.5\right),U(0.5,3.5)\}$ $C_3\sim U(0.5,3.5)$ 27% and 55%: $C_1\sim \min \{Exp\left(0.3\right),U(0.5,3.5)\}$ $C_2\sim \min \{Exp\left(0.5\right),U(0.5,3.5)\}$ $C_3\sim U(0.5,3.5)$
J-2	$F_1\left(t\right)=Exp\left(1\right)$ $F_2\left(t\right)=$	equal: $C\sim Exp\left(\lambda \right)$ $\lambda =(0.3,1)$ 40% and 55%: $C_1,C_2\sim \min \{Exp\left(0.9\right),U(0,4)\}$ $C_3\sim U(0,4)$ 27% and 55%: $C_1\sim \min \{Exp\left(0.9\right),U(0,4)\}$ $C_2\sim \min \{Exp\left(0.5\right),U(0,4)\}$ $C_3\sim U(0,4)$

Table 7. Table S4: Empirical power of all the 2 2 2 -sample scenarios: KONP-P and KONP-LR are our two proposed tests; YP-1 and YP-2 are Yang-Prentice tests with group 1 as control and 2 as treatment, and vice verse; YP-Inv is Yang–Prentice invariant test; LR - the logrank test; PP - the Peto–Peto test; and PF - the Pepe–Fleming test. In each line of the table, the highest power among the invariant tests is in bold. Evidently, under the non-proportional hazards scenarios (A through K-3) usually KONP-P and KONP-LR are with the highest power. Under proportional hazards (or close to, L through Q) usually the logrank test is with the highest power.

n	Scenario	Censoring	KONP-P	KONP-LR	YP-1	YP-2	YP-Inv	LR	PP	PF
100	Null	25% and 25%	0.047	0.046	0.064	0.066	0.052	0.054	0.051	0.047
100	Null	50% and 50%	0.049	0.049	0.072	0.074	0.057	0.056	0.056	0.051
100	Null	27% and 55%	0.041	0.038	0.061	0.066	0.050	0.048	0.048	0.041
100	Null	40% and 55%	0.046	0.044	0.060	0.061	0.052	0.049	0.048	0.043
100	A	25% and 25%	0.492	0.492	0.513	0.556	0.496	0.479	0.320	0.337
100	A	50% and 50%	0.235	0.235	0.237	0.295	0.217	0.211	0.147	0.169
100	A	27% and 55%	0.342	0.337	0.299	0.338	0.271	0.280	0.178	0.259
100	A	40% and 55%	0.309	0.305	0.273	0.319	0.235	0.255	0.167	0.231
100	B	25% and 25%	0.635	0.647	0.680	0.258	0.634	0.102	0.278	0.192
100	B	50% and 50%	0.582	0.597	0.755	0.438	0.688	0.274	0.459	0.589
100	B	27% and 55%	0.482	0.489	0.705	0.311	0.636	0.156	0.354	0.203
100	B	40% and 55%	0.455	0.464	0.710	0.355	0.632	0.197	0.399	0.229
100	C	25% and 25%	0.866	0.868	0.615	0.444	0.564	0.187	0.525	0.457
100	C	50% and 50%	0.743	0.741	0.669	0.593	0.604	0.431	0.639	0.601
100	C	27% and 55%	0.710	0.702	0.630	0.517	0.557	0.326	0.597	0.386
100	C	40% and 55%	0.682	0.673	0.670	0.601	0.609	0.437	0.655	0.504
100	D	25% and 25%	0.783	0.785	0.520	0.319	0.462	0.145	0.383	0.451
100	D	50% and 50%	0.696	0.692	0.628	0.552	0.565	0.404	0.591	0.633
100	D	27% and 55%	0.646	0.629	0.581	0.448	0.521	0.295	0.506	0.475
100	D	40% and 55%	0.639	0.630	0.624	0.515	0.549	0.366	0.566	0.540
100	E	25% and 25%	0.529	0.526	0.578	0.600	0.538	0.538	0.345	0.332
100	E	50% and 50%	0.273	0.269	0.271	0.314	0.226	0.244	0.160	0.150
100	E	27% and 55%	0.406	0.401	0.409	0.429	0.354	0.376	0.231	0.321
100	E	40% and 55%	0.336	0.338	0.331	0.368	0.267	0.296	0.189	0.245
100	F	25% and 25%	0.247	0.250	0.156	0.320	0.264	0.053	0.090	0.044
100	F	50% and 50%	0.190	0.193	0.158	0.284	0.226	0.114	0.175	0.173
100	F	27% and 55%	0.148	0.149	0.138	0.259	0.199	0.084	0.152	0.079
100	F	40% and 55%	0.160	0.160	0.149	0.282	0.212	0.102	0.177	0.095
100	G	25% and 25%	0.537	0.536	0.614	0.565	0.549	0.490	0.622	0.547
100	G	50% and 50%	0.589	0.584	0.674	0.641	0.609	0.621	0.662	0.655
100	G	27% and 55%	0.485	0.478	0.604	0.575	0.548	0.520	0.601	0.521
100	G	40% and 55%	0.468	0.461	0.599	0.559	0.531	0.530	0.599	0.515
100	H	25% and 25%	0.548	0.548	0.452	0.403	0.394	0.274	0.480	0.404
100	H	50% and 50%	0.583	0.581	0.649	0.621	0.587	0.625	0.629	0.596
100	H	27% and 55%	0.429	0.420	0.440	0.410	0.385	0.328	0.461	0.323
100	H	40% and 55%	0.439	0.436	0.461	0.421	0.389	0.348	0.486	0.346
100	I-1	25% and 25%	0.190	0.194	0.193	0.132	0.144	0.061	0.103	0.056
100	I-1	50% and 50%	0.110	0.109	0.178	0.108	0.131	0.079	0.123	0.074
100	I-1	27% and 55%	0.122	0.123	0.177	0.117	0.137	0.063	0.114	0.065
100	I-1	40% and 55%	0.112	0.113	0.186	0.116	0.138	0.072	0.131	0.069
100	I-2	25% and 25%	0.256	0.253	0.249	0.287	0.235	0.177	0.285	0.176
100	I-2	50% and 50%	0.182	0.177	0.247	0.285	0.238	0.206	0.272	0.203
100	I-2	27% and 55%	0.180	0.173	0.265	0.315	0.248	0.242	0.293	0.230
100	I-2	40% and 55%	0.177	0.169	0.257	0.305	0.239	0.227	0.281	0.222
100	I-3	25% and 25%	0.196	0.197	0.232	0.179	0.197	0.122	0.215	0.154
100	I-3	50% and 50%	0.250	0.250	0.320	0.272	0.266	0.252	0.288	0.287
100	I-3	27% and 55%	0.166	0.158	0.243	0.193	0.199	0.137	0.221	0.114
100	I-3	40% and 55%	0.153	0.150	0.236	0.183	0.187	0.145	0.218	0.126
100	J-1	25% and 25%	0.388	0.388	0.271	0.160	0.211	0.055	0.113	0.048
100	J-1	50% and 50%	0.244	0.244	0.266	0.152	0.205	0.082	0.162	0.076
100	J-1	27% and 55%	0.264	0.261	0.278	0.152	0.220	0.073	0.155	0.063
100	J-1	40% and 55%	0.239	0.239	0.261	0.160	0.193	0.094	0.178	0.083
100	J-2	25% and 25%	0.322	0.325	0.173	0.153	0.139	0.069	0.076	0.067
100	J-2	50% and 50%	0.172	0.172	0.123	0.099	0.086	0.068	0.109	0.059
100	J-2	27% and 55%	0.192	0.189	0.113	0.088	0.083	0.049	0.093	0.041
100	J-2	40% and 55%	0.164	0.165	0.107	0.084	0.077	0.063	0.088	0.056
100	J-3	25% and 25%	0.758	0.759	0.549	0.571	0.518	0.555	0.470	0.577
100	J-3	50% and 50%	0.535	0.526	0.499	0.472	0.417	0.441	0.326	0.421
100	J-3	27% and 55%	0.453	0.444	0.527	0.482	0.439	0.424	0.300	0.372
100	J-3	40% and 55%	0.460	0.455	0.510	0.465	0.417	0.415	0.283	0.369
100	K-1	25% and 25%	0.286	0.289	0.271	0.208	0.222	0.115	0.047	0.120
100	K-1	50% and 50%	0.123	0.126	0.177	0.087	0.114	0.056	0.054	0.050
100	K-1	27% and 55%	0.158	0.163	0.180	0.090	0.129	0.054	0.046	0.046
100	K-1	40% and 55%	0.140	0.140	0.183	0.087	0.123	0.053	0.064	0.056
100	K-2	25% and 25%	0.449	0.459	0.401	0.189	0.325	0.058	0.136	0.053
100	K-2	50% and 50%	0.226	0.225	0.377	0.186	0.300	0.098	0.212	0.088
100	K-2	27% and 55%	0.294	0.290	0.394	0.180	0.308	0.087	0.199	0.068
100	K-2	40% and 55%	0.262	0.262	0.399	0.204	0.310	0.106	0.218	0.092
100	K-3	25% and 25%	0.869	0.873	0.422	0.699	0.608	0.057	0.383	0.102
100	K-3	50% and 50%	0.572	0.572	0.427	0.631	0.544	0.175	0.497	0.281
100	K-3	27% and 55%	0.570	0.571	0.427	0.654	0.556	0.175	0.485	0.324
100	K-3	40% and 55%	0.542	0.542	0.414	0.635	0.548	0.195	0.474	0.311
100	L	25% and 25%	0.605	0.602	0.733	0.724	0.681	0.717	0.657	0.666
100	L	50% and 50%	0.427	0.418	0.545	0.532	0.477	0.529	0.502	0.502
100	L	27% and 55%	0.483	0.476	0.616	0.610	0.545	0.603	0.556	0.580
100	L	40% and 55%	0.422	0.414	0.563	0.549	0.476	0.545	0.492	0.502
100	M	25% and 25%	0.280	0.283	0.491	0.464	0.404	0.458	0.390	0.413
100	M	50% and 50%	0.172	0.174	0.373	0.343	0.268	0.333	0.280	0.272
100	M	27% and 55%	0.209	0.210	0.387	0.372	0.312	0.358	0.333	0.339
100	M	40% and 55%	0.227	0.224	0.382	0.363	0.316	0.349	0.333	0.333
100	N	25% and 25%	0.473	0.475	0.589	0.593	0.538	0.577	0.524	0.550
100	N	50% and 50%	0.396	0.393	0.498	0.511	0.444	0.481	0.460	0.437
100	N	27% and 55%	0.269	0.264	0.430	0.463	0.377	0.428	0.405	0.316
100	N	40% and 55%	0.281	0.276	0.387	0.414	0.324	0.381	0.346	0.326
100	O	25% and 25%	0.968	0.968	0.989	0.990	0.984	0.989	0.976	0.985
100	O	50% and 50%	0.904	0.900	0.940	0.941	0.922	0.944	0.929	0.909
100	O	27% and 55%	0.781	0.772	0.921	0.931	0.894	0.925	0.908	0.866
100	O	40% and 55%	0.790	0.782	0.909	0.917	0.870	0.908	0.892	0.877
100	P	25% and 25%	0.707	0.707	0.777	0.793	0.758	0.753	0.797	0.762
100	P	50% and 50%	0.654	0.648	0.734	0.758	0.697	0.731	0.742	0.721
100	P	27% and 55%	0.488	0.479	0.676	0.710	0.638	0.674	0.686	0.592
100	P	40% and 55%	0.578	0.571	0.711	0.742	0.670	0.710	0.732	0.668
100	Q	25% and 25%	0.148	0.146	0.211	0.212	0.178	0.186	0.125	0.135
100	Q	50% and 50%	0.078	0.077	0.119	0.110	0.088	0.099	0.086	0.069
100	Q	27% and 55%	0.116	0.119	0.158	0.144	0.123	0.121	0.097	0.105
100	Q	40% and 55%	0.105	0.103	0.138	0.127	0.099	0.108	0.079	0.101
200	Null	25% and 25%	0.059	0.058	0.067	0.065	0.054	0.058	0.054	0.055
200	Null	50% and 50%	0.050	0.050	0.061	0.059	0.056	0.053	0.052	0.052
200	Null	27% and 55%	0.044	0.042	0.050	0.058	0.042	0.045	0.043	0.043
200	Null	40% and 55%	0.043	0.041	0.052	0.052	0.048	0.045	0.048	0.046
200	A	25% and 25%	0.850	0.851	0.824	0.849	0.826	0.773	0.546	0.587
200	A	50% and 50%	0.497	0.494	0.445	0.502	0.427	0.388	0.262	0.319
200	A	27% and 55%	0.675	0.670	0.565	0.620	0.561	0.522	0.331	0.487
200	A	40% and 55%	0.610	0.602	0.505	0.561	0.481	0.456	0.294	0.423
200	B	25% and 25%	0.939	0.942	0.896	0.428	0.886	0.150	0.471	0.348
200	B	50% and 50%	0.919	0.925	0.945	0.702	0.933	0.444	0.737	0.891
200	B	27% and 55%	0.858	0.861	0.929	0.592	0.918	0.286	0.659	0.391
200	B	40% and 55%	0.820	0.821	0.916	0.604	0.901	0.314	0.658	0.405
200	C	25% and 25%	0.998	0.998	0.885	0.750	0.867	0.345	0.818	0.768
200	C	50% and 50%	0.977	0.978	0.920	0.885	0.902	0.711	0.917	0.899
200	C	27% and 55%	0.979	0.975	0.901	0.833	0.877	0.586	0.889	0.690
200	C	40% and 55%	0.950	0.946	0.906	0.866	0.888	0.694	0.908	0.768
200	D	25% and 25%	0.984	0.984	0.780	0.549	0.753	0.241	0.636	0.738
200	D	50% and 50%	0.958	0.958	0.885	0.828	0.863	0.671	0.865	0.905
200	D	27% and 55%	0.928	0.922	0.868	0.758	0.833	0.531	0.819	0.799
200	D	40% and 55%	0.929	0.923	0.853	0.775	0.829	0.576	0.822	0.804
200	E	25% and 25%	0.859	0.859	0.875	0.882	0.866	0.831	0.616	0.628
200	E	50% and 50%	0.552	0.552	0.522	0.567	0.495	0.470	0.313	0.310
200	E	27% and 55%	0.736	0.734	0.724	0.744	0.689	0.668	0.421	0.600
200	E	40% and 55%	0.600	0.600	0.583	0.609	0.521	0.508	0.319	0.446
200	F	25% and 25%	0.523	0.525	0.268	0.465	0.428	0.065	0.128	0.057
200	F	50% and 50%	0.350	0.351	0.255	0.385	0.356	0.175	0.288	0.300
200	F	27% and 55%	0.289	0.288	0.233	0.399	0.364	0.124	0.276	0.117
200	F	40% and 55%	0.306	0.307	0.250	0.409	0.373	0.132	0.281	0.123
200	G	25% and 25%	0.841	0.840	0.866	0.852	0.848	0.773	0.881	0.839
200	G	50% and 50%	0.861	0.860	0.900	0.892	0.883	0.879	0.904	0.907
200	G	27% and 55%	0.801	0.796	0.876	0.864	0.860	0.800	0.893	0.819
200	G	40% and 55%	0.750	0.748	0.857	0.838	0.834	0.807	0.866	0.805
200	H	25% and 25%	0.871	0.872	0.686	0.663	0.661	0.467	0.740	0.667
200	H	50% and 50%	0.875	0.874	0.904	0.899	0.883	0.897	0.902	0.889
200	H	27% and 55%	0.765	0.761	0.725	0.709	0.695	0.595	0.771	0.607
200	H	40% and 55%	0.755	0.745	0.731	0.710	0.702	0.614	0.770	0.615
200	I-1	25% and 25%	0.383	0.388	0.240	0.202	0.208	0.049	0.142	0.051
200	I-1	50% and 50%	0.203	0.203	0.242	0.170	0.209	0.107	0.206	0.094
200	I-1	27% and 55%	0.241	0.232	0.224	0.168	0.196	0.083	0.179	0.082
200	I-1	40% and 55%	0.211	0.210	0.238	0.175	0.212	0.093	0.209	0.091
200	I-2	25% and 25%	0.489	0.487	0.405	0.414	0.383	0.250	0.498	0.292
200	I-2	50% and 50%	0.338	0.332	0.431	0.460	0.428	0.352	0.505	0.364
200	I-2	27% and 55%	0.323	0.313	0.471	0.493	0.455	0.387	0.526	0.430
200	I-2	40% and 55%	0.309	0.307	0.438	0.467	0.422	0.365	0.493	0.396
200	I-3	25% and 25%	0.371	0.370	0.337	0.303	0.308	0.166	0.359	0.262
200	I-3	50% and 50%	0.440	0.436	0.496	0.459	0.456	0.421	0.481	0.508
200	I-3	27% and 55%	0.304	0.296	0.350	0.304	0.328	0.201	0.372	0.176
200	I-3	40% and 55%	0.285	0.282	0.357	0.309	0.330	0.216	0.367	0.198
200	J-1	25% and 25%	0.753	0.754	0.429	0.268	0.371	0.049	0.172	0.050
200	J-1	50% and 50%	0.513	0.506	0.400	0.257	0.361	0.120	0.307	0.102
200	J-1	27% and 55%	0.506	0.491	0.399	0.233	0.351	0.091	0.273	0.085
200	J-1	40% and 55%	0.516	0.506	0.391	0.240	0.338	0.117	0.281	0.105
200	J-2	25% and 25%	0.660	0.665	0.261	0.293	0.259	0.075	0.082	0.095
200	J-2	50% and 50%	0.334	0.335	0.141	0.127	0.117	0.075	0.142	0.065
200	J-2	27% and 55%	0.382	0.371	0.158	0.142	0.131	0.061	0.145	0.064
200	J-2	40% and 55%	0.328	0.328	0.134	0.118	0.113	0.070	0.134	0.069
200	J-3	25% and 25%	0.982	0.982	0.782	0.825	0.795	0.821	0.768	0.859
200	J-3	50% and 50%	0.853	0.848	0.745	0.732	0.698	0.711	0.544	0.721
200	J-3	27% and 55%	0.793	0.767	0.784	0.780	0.735	0.725	0.542	0.687
200	J-3	40% and 55%	0.753	0.739	0.748	0.723	0.677	0.664	0.485	0.646
200	K-1	25% and 25%	0.541	0.546	0.434	0.383	0.382	0.176	0.053	0.203
200	K-1	50% and 50%	0.251	0.251	0.224	0.127	0.174	0.058	0.054	0.062
200	K-1	27% and 55%	0.299	0.294	0.253	0.157	0.198	0.067	0.056	0.064
200	K-1	40% and 55%	0.273	0.274	0.235	0.133	0.187	0.054	0.061	0.055
200	K-2	25% and 25%	0.789	0.792	0.593	0.356	0.532	0.046	0.217	0.055
200	K-2	50% and 50%	0.539	0.536	0.606	0.336	0.550	0.154	0.385	0.123
200	K-2	27% and 55%	0.546	0.529	0.590	0.305	0.523	0.108	0.329	0.082
200	K-2	40% and 55%	0.533	0.526	0.606	0.337	0.550	0.132	0.385	0.103
200	K-3	25% and 25%	0.996	0.995	0.895	0.947	0.924	0.059	0.654	0.158
200	K-3	50% and 50%	0.920	0.918	0.720	0.903	0.856	0.274	0.779	0.423
200	K-3	27% and 55%	0.927	0.924	0.748	0.905	0.860	0.240	0.777	0.544
200	K-3	40% and 55%	0.911	0.909	0.714	0.905	0.853	0.292	0.784	0.520
200	L	25% and 25%	0.904	0.904	0.954	0.952	0.942	0.950	0.921	0.933
200	L	50% and 50%	0.747	0.740	0.844	0.843	0.815	0.836	0.809	0.820
200	L	27% and 55%	0.790	0.791	0.890	0.888	0.862	0.884	0.848	0.877
200	L	40% and 55%	0.742	0.737	0.856	0.851	0.815	0.847	0.807	0.834
200	M	25% and 25%	0.542	0.546	0.770	0.759	0.707	0.748	0.664	0.736
200	M	50% and 50%	0.356	0.362	0.625	0.603	0.518	0.586	0.523	0.559
200	M	27% and 55%	0.425	0.420	0.671	0.662	0.611	0.648	0.593	0.638
200	M	40% and 55%	0.407	0.410	0.635	0.625	0.576	0.616	0.569	0.603
200	N	25% and 25%	0.776	0.777	0.884	0.888	0.864	0.880	0.834	0.855
200	N	50% and 50%	0.663	0.663	0.755	0.758	0.731	0.755	0.721	0.705
200	N	27% and 55%	0.493	0.482	0.716	0.724	0.662	0.705	0.667	0.652
200	N	40% and 55%	0.494	0.478	0.664	0.678	0.610	0.659	0.619	0.635
200	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
200	O	50% and 50%	0.997	0.997	1.000	1.000	1.000	1.000	0.998	0.998
200	O	27% and 55%	0.956	0.947	0.998	0.998	0.996	0.998	0.997	0.997
200	O	40% and 55%	0.971	0.965	0.996	0.997	0.995	0.997	0.995	0.996
200	P	25% and 25%	0.950	0.949	0.975	0.974	0.971	0.966	0.982	0.975
200	P	50% and 50%	0.940	0.937	0.962	0.964	0.955	0.958	0.966	0.964
200	P	27% and 55%	0.741	0.732	0.939	0.945	0.924	0.932	0.940	0.915
200	P	40% and 55%	0.859	0.854	0.939	0.942	0.932	0.933	0.943	0.919
200	Q	25% and 25%	0.278	0.278	0.372	0.376	0.344	0.321	0.204	0.238
200	Q	50% and 50%	0.111	0.111	0.157	0.150	0.138	0.134	0.113	0.095
200	Q	27% and 55%	0.183	0.182	0.218	0.227	0.186	0.186	0.131	0.178
200	Q	40% and 55%	0.171	0.171	0.217	0.212	0.185	0.180	0.124	0.179
300	Null	25% and 25%	0.055	0.055	0.064	0.062	0.056	0.056	0.052	0.053
300	Null	50% and 50%	0.057	0.056	0.067	0.066	0.058	0.062	0.062	0.060
300	Null	27% and 55%	0.042	0.043	0.052	0.058	0.051	0.047	0.049	0.041
300	Null	40% and 55%	0.037	0.039	0.049	0.050	0.040	0.043	0.041	0.038
300	A	25% and 25%	0.966	0.966	0.956	0.962	0.957	0.923	0.728	0.770
300	A	50% and 50%	0.686	0.683	0.592	0.661	0.581	0.519	0.320	0.426
300	A	27% and 55%	0.866	0.862	0.736	0.770	0.723	0.691	0.461	0.658
300	A	40% and 55%	0.802	0.798	0.675	0.729	0.665	0.616	0.392	0.590
300	B	25% and 25%	0.996	0.996	0.964	0.590	0.962	0.195	0.638	0.501
300	B	50% and 50%	0.992	0.993	0.975	0.869	0.981	0.613	0.887	0.974
300	B	27% and 55%	0.973	0.973	0.974	0.728	0.971	0.377	0.797	0.514
300	B	40% and 55%	0.975	0.976	0.982	0.786	0.980	0.431	0.834	0.563
300	C	25% and 25%	1.000	1.000	0.971	0.919	0.971	0.476	0.942	0.919
300	C	50% and 50%	0.999	0.999	0.981	0.968	0.976	0.856	0.983	0.976
300	C	27% and 55%	1.000	1.000	0.972	0.945	0.969	0.751	0.967	0.838
300	C	40% and 55%	0.993	0.992	0.975	0.962	0.969	0.849	0.978	0.907
300	D	25% and 25%	1.000	1.000	0.904	0.715	0.896	0.330	0.807	0.895
300	D	50% and 50%	0.995	0.995	0.968	0.949	0.959	0.822	0.962	0.978
300	D	27% and 55%	0.991	0.990	0.950	0.899	0.938	0.672	0.933	0.918
300	D	40% and 55%	0.992	0.991	0.964	0.928	0.947	0.760	0.947	0.938
300	E	25% and 25%	0.960	0.959	0.961	0.966	0.962	0.941	0.786	0.795
300	E	50% and 50%	0.729	0.729	0.670	0.707	0.649	0.603	0.412	0.414
300	E	27% and 55%	0.894	0.893	0.882	0.893	0.847	0.844	0.571	0.785
300	E	40% and 55%	0.789	0.787	0.780	0.793	0.716	0.704	0.444	0.628
300	F	25% and 25%	0.745	0.746	0.362	0.595	0.565	0.059	0.153	0.050
300	F	50% and 50%	0.504	0.503	0.354	0.498	0.481	0.236	0.398	0.417
300	F	27% and 55%	0.468	0.462	0.329	0.499	0.472	0.159	0.378	0.158
300	F	40% and 55%	0.469	0.465	0.319	0.492	0.448	0.166	0.366	0.160
300	G	25% and 25%	0.954	0.953	0.962	0.958	0.957	0.905	0.976	0.951
300	G	50% and 50%	0.964	0.964	0.977	0.976	0.974	0.968	0.980	0.982
300	G	27% and 55%	0.934	0.933	0.964	0.960	0.958	0.924	0.973	0.938
300	G	40% and 55%	0.909	0.905	0.964	0.958	0.956	0.939	0.970	0.937
300	H	25% and 25%	0.969	0.969	0.845	0.844	0.837	0.620	0.898	0.837
300	H	50% and 50%	0.968	0.966	0.977	0.976	0.975	0.976	0.978	0.976
300	H	27% and 55%	0.929	0.924	0.870	0.864	0.856	0.770	0.910	0.786
300	H	40% and 55%	0.916	0.912	0.865	0.857	0.843	0.767	0.904	0.776
300	I-1	25% and 25%	0.602	0.606	0.325	0.300	0.305	0.050	0.193	0.065
300	I-1	50% and 50%	0.311	0.307	0.280	0.223	0.249	0.108	0.265	0.099
300	I-1	27% and 55%	0.343	0.336	0.270	0.209	0.250	0.090	0.254	0.091
300	I-1	40% and 55%	0.331	0.332	0.285	0.217	0.265	0.113	0.267	0.103
300	I-2	25% and 25%	0.672	0.670	0.542	0.540	0.531	0.331	0.677	0.394
300	I-2	50% and 50%	0.490	0.486	0.588	0.605	0.589	0.489	0.670	0.514
300	I-2	27% and 55%	0.484	0.470	0.616	0.634	0.609	0.514	0.681	0.571
300	I-2	40% and 55%	0.481	0.469	0.603	0.621	0.596	0.528	0.665	0.558
300	I-3	25% and 25%	0.555	0.555	0.446	0.423	0.428	0.226	0.507	0.372
300	I-3	50% and 50%	0.584	0.581	0.642	0.624	0.611	0.577	0.648	0.673
300	I-3	27% and 55%	0.479	0.469	0.497	0.471	0.488	0.321	0.548	0.277
300	I-3	40% and 55%	0.453	0.448	0.517	0.493	0.496	0.347	0.562	0.309
300	J-1	25% and 25%	0.928	0.927	0.568	0.360	0.509	0.051	0.238	0.060
300	J-1	50% and 50%	0.738	0.735	0.507	0.345	0.458	0.137	0.411	0.112
300	J-1	27% and 55%	0.712	0.689	0.506	0.327	0.453	0.103	0.359	0.100
300	J-1	40% and 55%	0.715	0.708	0.520	0.342	0.464	0.146	0.394	0.131
300	J-2	25% and 25%	0.843	0.846	0.360	0.432	0.386	0.084	0.106	0.141
300	J-2	50% and 50%	0.528	0.524	0.147	0.150	0.135	0.077	0.175	0.060
300	J-2	27% and 55%	0.555	0.541	0.165	0.168	0.149	0.067	0.184	0.066
300	J-2	40% and 55%	0.493	0.488	0.147	0.143	0.140	0.077	0.179	0.078
300	J-3	25% and 25%	1.000	1.000	0.931	0.941	0.924	0.940	0.908	0.964
300	J-3	50% and 50%	0.968	0.967	0.902	0.891	0.862	0.883	0.716	0.891
300	J-3	27% and 55%	0.917	0.902	0.898	0.887	0.861	0.861	0.700	0.853
300	J-3	40% and 55%	0.914	0.902	0.882	0.861	0.837	0.827	0.671	0.819
300	K-1	25% and 25%	0.770	0.769	0.616	0.556	0.578	0.286	0.052	0.337
300	K-1	50% and 50%	0.406	0.408	0.290	0.172	0.227	0.068	0.065	0.073
300	K-1	27% and 55%	0.407	0.400	0.302	0.195	0.243	0.066	0.066	0.071
300	K-1	40% and 55%	0.395	0.390	0.272	0.169	0.230	0.050	0.057	0.055
300	K-2	25% and 25%	0.946	0.947	0.783	0.527	0.735	0.048	0.318	0.074
300	K-2	50% and 50%	0.742	0.740	0.735	0.455	0.699	0.181	0.519	0.146
300	K-2	27% and 55%	0.740	0.724	0.734	0.431	0.679	0.127	0.481	0.107
300	K-2	40% and 55%	0.758	0.750	0.745	0.483	0.702	0.167	0.542	0.137
300	K-3	25% and 25%	1.000	1.000	0.991	0.995	0.992	0.054	0.823	0.178
300	K-3	50% and 50%	0.992	0.992	0.881	0.976	0.950	0.367	0.916	0.580
300	K-3	27% and 55%	0.988	0.987	0.931	0.983	0.962	0.298	0.897	0.682
300	K-3	40% and 55%	0.984	0.983	0.886	0.982	0.958	0.363	0.912	0.678
300	L	25% and 25%	0.980	0.980	0.993	0.993	0.991	0.992	0.980	0.988
300	L	50% and 50%	0.887	0.886	0.950	0.948	0.936	0.943	0.933	0.941
300	L	27% and 55%	0.934	0.934	0.976	0.974	0.969	0.973	0.962	0.970
300	L	40% and 55%	0.893	0.891	0.954	0.954	0.935	0.950	0.932	0.945
300	M	25% and 25%	0.750	0.755	0.913	0.912	0.886	0.907	0.854	0.902
300	M	50% and 50%	0.510	0.519	0.766	0.762	0.682	0.751	0.680	0.735
300	M	27% and 55%	0.590	0.591	0.810	0.804	0.769	0.796	0.756	0.796
300	M	40% and 55%	0.569	0.570	0.800	0.794	0.751	0.782	0.748	0.779
300	N	25% and 25%	0.922	0.921	0.966	0.966	0.961	0.964	0.939	0.958
300	N	50% and 50%	0.847	0.846	0.901	0.903	0.893	0.896	0.880	0.872
300	N	27% and 55%	0.607	0.590	0.836	0.843	0.807	0.832	0.807	0.808
300	N	40% and 55%	0.664	0.645	0.836	0.837	0.804	0.826	0.803	0.814
300	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
300	O	50% and 50%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
300	O	27% and 55%	0.991	0.983	1.000	1.000	1.000	1.000	1.000	1.000
300	O	40% and 55%	0.995	0.994	1.000	1.000	1.000	1.000	0.999	0.999
300	P	25% and 25%	0.994	0.994	0.999	0.999	0.998	0.997	0.998	0.998
300	P	50% and 50%	0.987	0.987	0.995	0.996	0.995	0.996	0.996	0.995
300	P	27% and 55%	0.850	0.827	0.990	0.990	0.988	0.987	0.992	0.986
300	P	40% and 55%	0.966	0.963	0.991	0.993	0.989	0.990	0.994	0.983
300	Q	25% and 25%	0.436	0.435	0.531	0.540	0.518	0.478	0.305	0.359
300	Q	50% and 50%	0.155	0.155	0.200	0.191	0.176	0.180	0.149	0.126
300	Q	27% and 55%	0.254	0.249	0.300	0.303	0.260	0.247	0.158	0.261
300	Q	40% and 55%	0.231	0.228	0.285	0.286	0.248	0.233	0.155	0.240
400	Null	25% and 25%	0.052	0.052	0.055	0.054	0.052	0.051	0.052	0.048
400	Null	50% and 50%	0.045	0.044	0.052	0.049	0.046	0.045	0.048	0.047
400	Null	27% and 55%	0.043	0.043	0.053	0.054	0.050	0.051	0.050	0.046
400	Null	40% and 55%	0.053	0.051	0.058	0.059	0.049	0.050	0.049	0.049
400	A	25% and 25%	0.991	0.991	0.989	0.991	0.989	0.970	0.830	0.875
400	A	50% and 50%	0.845	0.845	0.735	0.788	0.731	0.646	0.429	0.558
400	A	27% and 55%	0.940	0.939	0.845	0.875	0.840	0.802	0.568	0.782
400	A	40% and 55%	0.911	0.908	0.799	0.831	0.784	0.734	0.495	0.708
400	B	25% and 25%	1.000	1.000	0.981	0.716	0.984	0.252	0.762	0.637
400	B	50% and 50%	1.000	1.000	0.990	0.947	0.994	0.743	0.959	0.995
400	B	27% and 55%	0.999	0.998	0.989	0.849	0.992	0.504	0.902	0.663
400	B	40% and 55%	0.996	0.997	0.993	0.876	0.993	0.556	0.921	0.674
400	C	25% and 25%	1.000	1.000	0.993	0.968	0.993	0.585	0.978	0.963
400	C	50% and 50%	1.000	1.000	0.994	0.990	0.994	0.943	0.993	0.992
400	C	27% and 55%	1.000	1.000	0.995	0.988	0.992	0.855	0.994	0.928
400	C	40% and 55%	1.000	1.000	0.996	0.988	0.993	0.932	0.994	0.967
400	D	25% and 25%	1.000	1.000	0.968	0.843	0.964	0.411	0.916	0.965
400	D	50% and 50%	0.999	0.999	0.991	0.988	0.989	0.908	0.992	0.996
400	D	27% and 55%	0.999	0.999	0.985	0.963	0.978	0.806	0.981	0.977
400	D	40% and 55%	0.998	0.998	0.987	0.976	0.980	0.876	0.986	0.983
400	E	25% and 25%	0.993	0.993	0.994	0.995	0.993	0.987	0.889	0.898
400	E	50% and 50%	0.862	0.861	0.823	0.845	0.797	0.747	0.522	0.531
400	E	27% and 55%	0.955	0.956	0.955	0.959	0.929	0.922	0.677	0.884
400	E	40% and 55%	0.909	0.905	0.881	0.891	0.849	0.822	0.572	0.765
400	F	25% and 25%	0.896	0.897	0.488	0.721	0.703	0.072	0.188	0.060
400	F	50% and 50%	0.679	0.678	0.481	0.623	0.610	0.298	0.527	0.554
400	F	27% and 55%	0.616	0.612	0.403	0.600	0.573	0.188	0.457	0.190
400	F	40% and 55%	0.623	0.620	0.407	0.598	0.571	0.186	0.458	0.176
400	G	25% and 25%	0.990	0.990	0.993	0.994	0.994	0.976	0.998	0.993
400	G	50% and 50%	0.991	0.991	0.998	0.998	0.997	0.996	0.999	0.998
400	G	27% and 55%	0.981	0.981	0.991	0.991	0.990	0.977	0.995	0.984
400	G	40% and 55%	0.962	0.961	0.986	0.985	0.984	0.972	0.992	0.976
400	H	25% and 25%	0.995	0.995	0.935	0.942	0.941	0.772	0.962	0.942
400	H	50% and 50%	0.991	0.991	0.995	0.995	0.995	0.995	0.997	0.995
400	H	27% and 55%	0.984	0.982	0.947	0.945	0.941	0.883	0.973	0.895
400	H	40% and 55%	0.977	0.976	0.945	0.941	0.940	0.882	0.967	0.888
400	I-1	25% and 25%	0.747	0.750	0.395	0.345	0.379	0.046	0.254	0.052
400	I-1	50% and 50%	0.443	0.439	0.367	0.305	0.351	0.138	0.358	0.120
400	I-1	27% and 55%	0.485	0.471	0.347	0.290	0.327	0.112	0.333	0.110
400	I-1	40% and 55%	0.466	0.463	0.375	0.292	0.343	0.136	0.363	0.126
400	I-2	25% and 25%	0.806	0.806	0.674	0.665	0.668	0.419	0.797	0.489
400	I-2	50% and 50%	0.615	0.610	0.693	0.706	0.694	0.577	0.774	0.614
400	I-2	27% and 55%	0.600	0.587	0.726	0.733	0.720	0.601	0.793	0.688
400	I-2	40% and 55%	0.601	0.595	0.728	0.737	0.710	0.630	0.790	0.689
400	I-3	25% and 25%	0.708	0.706	0.564	0.542	0.555	0.309	0.630	0.497
400	I-3	50% and 50%	0.720	0.715	0.774	0.759	0.755	0.711	0.789	0.815
400	I-3	27% and 55%	0.608	0.592	0.587	0.573	0.576	0.369	0.656	0.320
400	I-3	40% and 55%	0.591	0.586	0.618	0.599	0.601	0.437	0.679	0.383
400	J-1	25% and 25%	0.981	0.981	0.686	0.457	0.623	0.056	0.293	0.079
400	J-1	50% and 50%	0.872	0.870	0.634	0.452	0.573	0.168	0.523	0.128
400	J-1	27% and 55%	0.847	0.834	0.623	0.419	0.572	0.137	0.462	0.129
400	J-1	40% and 55%	0.861	0.853	0.637	0.463	0.578	0.175	0.529	0.161
400	J-2	25% and 25%	0.946	0.947	0.455	0.541	0.485	0.108	0.127	0.177
400	J-2	50% and 50%	0.697	0.690	0.189	0.204	0.177	0.093	0.240	0.070
400	J-2	27% and 55%	0.689	0.677	0.184	0.189	0.170	0.071	0.208	0.067
400	J-2	40% and 55%	0.678	0.672	0.187	0.203	0.180	0.092	0.245	0.086
400	J-3	25% and 25%	1.000	1.000	0.980	0.985	0.984	0.985	0.971	0.992
400	J-3	50% and 50%	0.995	0.995	0.945	0.941	0.928	0.933	0.824	0.942
400	J-3	27% and 55%	0.980	0.972	0.956	0.949	0.937	0.936	0.838	0.937
400	J-3	40% and 55%	0.970	0.965	0.951	0.942	0.921	0.926	0.797	0.927
400	K-1	25% and 25%	0.889	0.892	0.738	0.687	0.708	0.355	0.049	0.421
400	K-1	50% and 50%	0.537	0.532	0.332	0.193	0.263	0.068	0.071	0.079
400	K-1	27% and 55%	0.552	0.542	0.377	0.245	0.304	0.078	0.062	0.083
400	K-1	40% and 55%	0.513	0.503	0.329	0.219	0.261	0.072	0.063	0.075
400	K-2	25% and 25%	0.989	0.989	0.878	0.639	0.849	0.047	0.408	0.084
400	K-2	50% and 50%	0.887	0.886	0.839	0.594	0.804	0.246	0.657	0.200
400	K-2	27% and 55%	0.862	0.844	0.820	0.536	0.776	0.155	0.571	0.132
400	K-2	40% and 55%	0.885	0.878	0.825	0.557	0.783	0.201	0.612	0.160
400	K-3	25% and 25%	1.000	1.000	0.999	1.000	0.999	0.064	0.912	0.221
400	K-3	50% and 50%	0.999	0.999	0.968	0.997	0.988	0.480	0.970	0.694
400	K-3	27% and 55%	1.000	1.000	0.984	0.997	0.992	0.334	0.967	0.803
400	K-3	40% and 55%	0.998	0.998	0.972	0.997	0.988	0.455	0.973	0.791
400	L	25% and 25%	0.998	0.998	0.999	0.999	0.999	0.999	0.999	0.999
400	L	50% and 50%	0.953	0.951	0.986	0.984	0.977	0.984	0.973	0.981
400	L	27% and 55%	0.976	0.976	0.992	0.991	0.990	0.992	0.983	0.992
400	L	40% and 55%	0.958	0.957	0.993	0.993	0.988	0.994	0.985	0.991
400	M	25% and 25%	0.876	0.878	0.960	0.960	0.953	0.958	0.930	0.954
400	M	50% and 50%	0.633	0.639	0.873	0.870	0.814	0.866	0.800	0.850
400	M	27% and 55%	0.730	0.733	0.912	0.910	0.889	0.904	0.879	0.900
400	M	40% and 55%	0.715	0.714	0.905	0.903	0.876	0.896	0.870	0.895
400	N	25% and 25%	0.983	0.982	0.992	0.992	0.992	0.993	0.985	0.991
400	N	50% and 50%	0.934	0.932	0.964	0.965	0.960	0.962	0.956	0.950
400	N	27% and 55%	0.714	0.696	0.928	0.931	0.914	0.925	0.909	0.918
400	N	40% and 55%	0.769	0.751	0.919	0.923	0.902	0.918	0.902	0.912
400	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
400	O	50% and 50%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
400	O	27% and 55%	0.995	0.991	1.000	1.000	1.000	1.000	1.000	1.000
400	O	40% and 55%	0.999	0.998	1.000	1.000	1.000	1.000	1.000	1.000
400	P	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
400	P	50% and 50%	0.999	0.999	1.000	1.000	1.000	1.000	1.000	1.000
400	P	27% and 55%	0.900	0.872	1.000	1.000	0.999	1.000	0.999	0.998
400	P	40% and 55%	0.989	0.988	0.997	0.998	0.997	0.997	0.999	0.997
400	Q	25% and 25%	0.534	0.533	0.634	0.644	0.624	0.577	0.378	0.451
400	Q	50% and 50%	0.210	0.208	0.257	0.251	0.236	0.238	0.191	0.159
400	Q	27% and 55%	0.333	0.329	0.372	0.386	0.339	0.312	0.200	0.327
400	Q	40% and 55%	0.319	0.315	0.353	0.356	0.319	0.312	0.211	0.327

Table 8. Table S5: Empirical power of all scenarios with K = 3 , 4 , 5 𝐾 3 4 5 K=3,4,5 groups: KONP-P and KONP-LR are our two proposed KONP-Pearson and KONP-likelihood ratio test, respectively; LR and PP are the logrank the Peto–Peto tests

K	Scenario	n	Censoring	KONP-P	KONP-LR	LR	PP
3	Null	102	equal - 25%	0.049	0.047	0.045	0.050
3	Null	102	equal - 50%	0.051	0.049	0.055	0.057
3	Null	102	unequal - mild	0.051	0.046	0.057	0.059
3	Null	102	unequal - severe	0.049	0.048	0.058	0.052
3	D	102	equal - 25%	0.587	0.581	0.122	0.286
3	D	102	equal - 50%	0.399	0.382	0.304	0.440
3	D	102	unequal - mild	0.351	0.338	0.274	0.403
3	D	102	unequal - severe	0.425	0.407	0.272	0.413
3	J-2	102	equal - 25%	0.207	0.206	0.051	0.068
3	J-2	102	equal - 50%	0.127	0.124	0.062	0.079
3	J-2	102	unequal - mild	0.124	0.121	0.078	0.094
3	J-2	102	unequal - severe	0.113	0.117	0.054	0.071
3	Null	201	equal - 25%	0.056	0.055	0.054	0.052
3	Null	201	equal - 50%	0.046	0.048	0.057	0.054
3	Null	201	unequal - mild	0.050	0.048	0.060	0.055
3	Null	201	unequal - severe	0.055	0.055	0.058	0.060
3	D	201	equal - 25%	0.922	0.922	0.178	0.493
3	D	201	equal - 50%	0.796	0.794	0.483	0.690
3	D	201	unequal - mild	0.740	0.731	0.478	0.700
3	D	201	unequal - severe	0.797	0.790	0.430	0.670
3	J-2	201	equal - 25%	0.483	0.490	0.076	0.090
3	J-2	201	equal - 50%	0.223	0.217	0.063	0.117
3	J-2	201	unequal - mild	0.220	0.216	0.070	0.116
3	J-2	201	unequal - severe	0.208	0.211	0.070	0.110
3	Null	300	equal - 25%	0.052	0.052	0.046	0.046
3	Null	300	equal - 50%	0.048	0.047	0.049	0.048
3	Null	300	unequal - mild	0.039	0.037	0.057	0.062
3	Null	300	unequal - severe	0.049	0.047	0.055	0.062
3	D	300	equal - 25%	0.993	0.993	0.240	0.643
3	D	300	equal - 50%	0.962	0.962	0.642	0.867
3	D	300	unequal - mild	0.922	0.916	0.630	0.853
3	D	300	unequal - severe	0.956	0.951	0.580	0.839
3	J-2	300	equal - 25%	0.668	0.670	0.067	0.095
3	J-2	300	equal - 50%	0.372	0.368	0.076	0.137
3	J-2	300	unequal - mild	0.317	0.309	0.080	0.147
3	J-2	300	unequal - severe	0.343	0.336	0.068	0.149
3	Null	402	equal - 25%	0.052	0.052	0.047	0.049
3	Null	402	equal - 50%	0.053	0.055	0.054	0.055
3	Null	402	unequal - mild	0.042	0.041	0.048	0.051
3	Null	402	unequal - severe	0.051	0.050	0.054	0.053
3	D	402	equal - 25%	1.000	1.000	0.326	0.779
3	D	402	equal - 50%	0.989	0.990	0.776	0.933
3	D	402	unequal - mild	0.982	0.980	0.742	0.924
3	D	402	unequal - severe	0.991	0.990	0.704	0.930
3	J-2	402	equal - 25%	0.836	0.839	0.089	0.115
3	J-2	402	equal - 50%	0.511	0.509	0.070	0.157
3	J-2	402	unequal - mild	0.445	0.434	0.093	0.179
3	J-2	402	unequal - severe	0.435	0.429	0.070	0.154
4	Null	100	equal - 25%	0.057	0.057	0.062	0.059
4	Null	100	equal - 50%	0.038	0.038	0.045	0.044
4	Null	100	unequal - mild	0.051	0.049	0.056	0.047
4	Null	100	unequal - severe	0.041	0.039	0.050	0.045
4	Null	200	equal - 25%	0.047	0.048	0.060	0.056
4	Null	200	equal - 50%	0.055	0.052	0.047	0.045
4	Null	200	unequal - mild	0.052	0.047	0.057	0.055
4	Null	200	unequal - severe	0.057	0.055	0.047	0.050
4	Null	300	equal - 25%	0.058	0.057	0.043	0.044
4	Null	300	equal - 50%	0.044	0.042	0.040	0.042
4	Null	300	unequal - mild	0.040	0.038	0.050	0.052
4	Null	300	unequal - severe	0.050	0.051	0.056	0.058
4	Null	400	equal - 25%	0.051	0.050	0.052	0.052
4	Null	400	equal - 50%	0.042	0.043	0.049	0.047
4	Null	400	unequal - mild	0.044	0.043	0.041	0.045
4	Null	400	unequal - severe	0.054	0.054	0.057	0.057
5	Null	100	equal - 25%	0.052	0.050	0.063	0.053
5	Null	100	equal - 50%	0.049	0.048	0.058	0.057
5	Null	100	unequal - mild	0.050	0.051	0.053	0.050
5	Null	100	unequal - severe	0.045	0.049	0.051	0.048
5	Null	200	equal - 25%	0.050	0.049	0.053	0.049
5	Null	200	equal - 50%	0.042	0.042	0.056	0.050
5	Null	200	unequal - mild	0.044	0.042	0.053	0.061
5	Null	200	unequal - severe	0.051	0.049	0.056	0.056
5	Null	300	equal - 25%	0.060	0.060	0.057	0.045
5	Null	300	equal - 50%	0.052	0.052	0.052	0.053
5	Null	300	unequal - mild	0.049	0.050	0.042	0.046
5	Null	300	unequal - severe	0.062	0.054	0.060	0.060
5	Null	400	equal - 25%	0.057	0.056	0.052	0.054
5	Null	400	equal - 50%	0.047	0.046	0.047	0.050
5	Null	400	unequal - mild	0.059	0.055	0.046	0.049
5	Null	400	unequal - severe	0.054	0.051	0.051	0.048

Table 9. Table S6: Empirical power of KONP-P and KONP-LR, our two proposed KONP-Pearson and KONP-LR tests, respectively; LR - the logrank test, and the robust test based on the Cauchy-combination test, C ​ a ​ u 𝐶 𝑎 𝑢 Cau , of Section 3.4 of the main text; the test of Lee (2007) and the MaxCombo test. Scenarios L through Q are of proportional hazards or close to it.

$n$	Scenario	Censoring	KONP-P	KONP-LR	logrank	$Cau$	Lee (2007)	MaxCombo
100	A	25% and 25%	0.542	0.539	0.515	0.547	0.582	0.584
100	A	50% and 50%	0.238	0.234	0.235	0.252	0.283	0.291
100	A	40% and 55%	0.323	0.322	0.262	0.320	0.333	0.329
100	A	27% and 55%	0.355	0.351	0.300	0.354	0.344	0.350
200	A	25% and 25%	0.858	0.860	0.782	0.856	0.867	0.878
200	A	50% and 50%	0.484	0.482	0.375	0.476	0.537	0.521
200	A	40% and 55%	0.610	0.608	0.465	0.592	0.623	0.604
200	A	27% and 55%	0.665	0.658	0.508	0.652	0.573	0.660
300	A	25% and 25%	0.973	0.974	0.919	0.970	0.971	0.972
300	A	50% and 50%	0.700	0.691	0.512	0.678	0.744	0.746
300	A	40% and 55%	0.797	0.798	0.609	0.781	0.806	0.788
300	A	27% and 55%	0.862	0.854	0.687	0.845	0.760	0.811
400	A	25% and 25%	0.995	0.995	0.976	0.993	0.995	0.995
400	A	50% and 50%	0.852	0.848	0.641	0.836	0.847	0.830
400	A	40% and 55%	0.920	0.918	0.755	0.912	0.904	0.890
400	A	27% and 55%	0.955	0.953	0.818	0.948	0.849	0.922
100	B	25% and 25%	0.621	0.635	0.113	0.555	0.256	0.241
100	B	50% and 50%	0.600	0.616	0.294	0.561	0.378	0.375
100	B	40% and 55%	0.476	0.489	0.206	0.439	0.326	0.332
100	B	27% and 55%	0.478	0.476	0.190	0.435	0.340	0.319
200	B	25% and 25%	0.948	0.951	0.143	0.919	0.499	0.450
200	B	50% and 50%	0.927	0.926	0.482	0.899	0.735	0.690
200	B	40% and 55%	0.840	0.849	0.327	0.812	0.602	0.604
200	B	27% and 55%	0.862	0.865	0.269	0.827	0.613	0.579
300	B	25% and 25%	0.998	0.998	0.208	0.996	0.672	0.672
300	B	50% and 50%	0.994	0.995	0.637	0.992	0.895	0.870
300	B	40% and 55%	0.964	0.964	0.454	0.968	0.787	0.788
300	B	27% and 55%	0.968	0.967	0.393	0.960	0.799	0.768
400	B	25% and 25%	1.000	1.000	0.265	1.000	0.813	0.792
400	B	50% and 50%	1.000	1.000	0.773	1.000	0.969	0.959
400	B	40% and 55%	0.997	0.997	0.577	0.996	0.904	0.918
400	B	27% and 55%	0.996	0.995	0.494	0.992	0.913	0.903
100	C	25% and 25%	0.895	0.898	0.224	0.848	0.481	0.443
100	C	50% and 50%	0.746	0.750	0.456	0.713	0.558	0.544
100	C	40% and 55%	0.679	0.674	0.459	0.658	0.531	0.539
100	C	27% and 55%	0.739	0.727	0.368	0.692	0.567	0.493
200	C	25% and 25%	0.998	0.998	0.337	0.996	0.807	0.787
200	C	50% and 50%	0.975	0.975	0.725	0.960	0.871	0.879
200	C	40% and 55%	0.934	0.934	0.681	0.934	0.854	0.848
200	C	27% and 55%	0.974	0.971	0.584	0.958	0.863	0.824
300	C	25% and 25%	1.000	1.000	0.482	1.000	0.948	0.948
300	C	50% and 50%	0.999	0.999	0.873	0.999	0.970	0.966
300	C	40% and 55%	0.992	0.993	0.855	0.993	0.962	0.961
300	C	27% and 55%	1.000	1.000	0.758	0.998	0.963	0.952
400	C	25% and 25%	1.000	1.000	0.597	0.999	0.989	0.991
400	C	50% and 50%	1.000	1.000	0.950	1.000	0.992	0.994
400	C	40% and 55%	1.000	1.000	0.926	1.000	0.992	0.994
400	C	27% and 55%	1.000	1.000	0.873	1.000	0.992	0.991
100	D	25% and 25%	0.789	0.785	0.166	0.740	0.339	0.342
100	D	50% and 50%	0.681	0.681	0.419	0.645	0.493	0.489
100	D	40% and 55%	0.654	0.643	0.365	0.615	0.453	0.427
100	D	27% and 55%	0.641	0.630	0.314	0.608	0.470	0.405
200	D	25% and 25%	0.978	0.978	0.236	0.968	0.624	0.597
200	D	50% and 50%	0.948	0.948	0.663	0.933	0.814	0.809
200	D	40% and 55%	0.925	0.923	0.574	0.908	0.767	0.757
200	D	27% and 55%	0.924	0.919	0.531	0.910	0.776	0.733
300	D	25% and 25%	0.999	0.999	0.347	0.999	0.948	0.757
300	D	50% and 50%	0.993	0.993	0.842	0.993	0.970	0.940
300	D	40% and 55%	0.990	0.989	0.770	0.989	0.913	0.909
300	D	27% and 55%	0.993	0.993	0.698	0.988	0.914	0.897
400	D	25% and 25%	1.000	1.000	0.443	1.000	0.919	0.896
400	D	50% and 50%	0.999	0.999	0.918	0.999	0.986	0.983
400	D	40% and 55%	0.999	0.999	0.872	0.999	0.979	0.974
400	D	27% and 55%	1.000	0.999	0.825	0.999	0.977	0.974
100	E	25% and 25%	0.570	0.569	0.557	0.589	0.602	0.604
100	E	50% and 50%	0.317	0.314	0.277	0.325	0.333	0.343
100	E	40% and 55%	0.331	0.332	0.296	0.338	0.451	0.392
100	E	27% and 55%	0.435	0.431	0.389	0.443	0.384	0.448
200	E	25% and 25%	0.861	0.861	0.827	0.877	0.885	0.892
200	E	50% and 50%	0.543	0.537	0.438	0.546	0.567	0.593
200	E	40% and 55%	0.595	0.594	0.510	0.606	0.745	0.666
200	E	27% and 55%	0.725	0.723	0.658	0.744	0.635	0.772
300	E	25% and 25%	0.969	0.968	0.943	0.971	0.972	0.980
300	E	50% and 50%	0.735	0.734	0.627	0.730	0.772	0.779
300	E	40% and 55%	0.793	0.788	0.715	0.818	0.901	0.839
300	E	27% and 55%	0.895	0.894	0.846	0.905	0.814	0.922
400	E	25% and 25%	0.990	0.990	0.982	0.993	0.996	0.997
400	E	50% and 50%	0.880	0.879	0.772	0.879	0.884	0.878
400	E	40% and 55%	0.911	0.909	0.819	0.913	0.969	0.923
400	E	27% and 55%	0.965	0.963	0.928	0.971	0.902	0.967
100	F	25% and 25%	0.236	0.240	0.064	0.196	0.220	0.213
100	F	50% and 50%	0.206	0.212	0.131	0.192	0.146	0.129
100	F	40% and 55%	0.178	0.175	0.113	0.167	0.138	0.141
100	F	27% and 55%	0.159	0.158	0.102	0.156	0.146	0.131
200	F	25% and 25%	0.512	0.513	0.069	0.427	0.427	0.409
200	F	50% and 50%	0.361	0.359	0.189	0.319	0.250	0.247
200	F	40% and 55%	0.315	0.314	0.140	0.276	0.264	0.227
200	F	27% and 55%	0.293	0.287	0.133	0.254	0.254	0.212
300	F	25% and 25%	0.764	0.763	0.062	0.679	0.619	0.579
300	F	50% and 50%	0.524	0.527	0.256	0.466	0.358	0.336
300	F	40% and 55%	0.487	0.486	0.185	0.422	0.373	0.349
300	F	27% and 55%	0.449	0.447	0.173	0.393	0.358	0.320
400	F	25% and 25%	0.904	0.905	0.059	0.836	0.741	0.733
400	F	50% and 50%	0.666	0.662	0.313	0.611	0.466	0.461
400	F	40% and 55%	0.626	0.623	0.206	0.572	0.453	0.449
400	F	27% and 55%	0.607	0.602	0.182	0.548	0.457	0.471
100	G	25% and 25%	0.505	0.504	0.471	0.520	0.538	0.527
100	G	50% and 50%	0.558	0.553	0.594	0.587	0.579	0.599
100	G	40% and 55%	0.442	0.433	0.518	0.490	0.518	0.508
100	G	27% and 55%	0.463	0.453	0.486	0.479	0.519	0.513
200	G	25% and 25%	0.823	0.822	0.791	0.832	0.843	0.834
200	G	50% and 50%	0.849	0.849	0.886	0.870	0.866	0.873
200	G	40% and 55%	0.756	0.750	0.804	0.800	0.827	0.829
200	G	27% and 55%	0.784	0.781	0.783	0.807	0.835	0.832
300	G	25% and 25%	0.964	0.964	0.908	0.969	0.964	0.962
300	G	50% and 50%	0.965	0.965	0.973	0.972	0.972	0.969
300	G	40% and 55%	0.900	0.896	0.937	0.930	0.958	0.952
300	G	27% and 55%	0.932	0.930	0.920	0.943	0.951	0.949
400	G	25% and 25%	0.991	0.991	0.972	0.991	0.993	0.993
400	G	50% and 50%	0.993	0.993	0.994	0.996	0.995	0.995
400	G	40% and 55%	0.972	0.972	0.977	0.982	0.990	0.983
400	G	27% and 55%	0.984	0.985	0.982	0.987	0.990	0.989
100	H	25% and 25%	0.518	0.515	0.268	0.465	0.385	0.355
100	H	50% and 50%	0.556	0.550	0.610	0.588	0.573	0.606
100	H	40% and 55%	0.405	0.403	0.342	0.410	0.379	0.383
100	H	27% and 55%	0.419	0.417	0.341	0.416	0.381	0.405
200	H	25% and 25%	0.861	0.860	0.456	0.820	0.677	0.655
200	H	50% and 50%	0.867	0.862	0.888	0.886	0.866	0.881
200	H	40% and 55%	0.763	0.752	0.577	0.737	0.678	0.666
200	H	27% and 55%	0.767	0.761	0.569	0.743	0.711	0.646
300	H	25% and 25%	0.974	0.974	0.627	0.965	0.862	0.847
300	H	50% and 50%	0.968	0.968	0.982	0.979	0.969	0.967
300	H	40% and 55%	0.917	0.917	0.774	0.905	0.865	0.865
300	H	27% and 55%	0.918	0.914	0.756	0.906	0.869	0.840
400	H	25% and 25%	0.998	0.998	0.735	0.996	0.939	0.939
400	H	50% and 50%	0.995	0.993	0.994	0.995	0.996	0.992
400	H	40% and 55%	0.980	0.979	0.874	0.974	0.950	0.939
400	H	27% and 55%	0.979	0.978	0.871	0.977	0.951	0.951
100	I-1	25% and 25%	0.186	0.190	0.058	0.151	0.151	0.144
100	I-1	50% and 50%	0.097	0.100	0.081	0.095	0.122	0.111
100	I-1	40% and 55%	0.130	0.127	0.076	0.122	0.128	0.107
100	I-1	27% and 55%	0.134	0.131	0.067	0.129	0.118	0.105
200	I-1	25% and 25%	0.401	0.408	0.043	0.331	0.289	0.263
200	I-1	50% and 50%	0.211	0.211	0.102	0.186	0.197	0.185
200	I-1	40% and 55%	0.210	0.203	0.104	0.183	0.191	0.172
200	I-1	27% and 55%	0.238	0.233	0.087	0.199	0.190	0.182
300	I-1	25% and 25%	0.613	0.616	0.038	0.524	0.391	0.352
300	I-1	50% and 50%	0.339	0.334	0.140	0.307	0.261	0.273
300	I-1	40% and 55%	0.322	0.317	0.111	0.293	0.268	0.244
300	I-1	27% and 55%	0.364	0.358	0.086	0.312	0.265	0.235
400	I-1	25% and 25%	0.752	0.758	0.048	0.670	0.545	0.468
400	I-1	50% and 50%	0.439	0.440	0.141	0.388	0.353	0.326
400	I-1	40% and 55%	0.464	0.458	0.129	0.403	0.339	0.322
400	I-1	27% and 55%	0.463	0.452	0.092	0.393	0.324	0.306
100	I-2	25% and 25%	0.235	0.233	0.153	0.222	0.223	0.224
100	I-2	50% and 50%	0.188	0.182	0.197	0.192	0.221	0.218
100	I-2	40% and 55%	0.184	0.180	0.233	0.209	0.245	0.227
100	I-2	27% and 55%	0.180	0.178	0.250	0.224	0.237	0.228
200	I-2	25% and 25%	0.462	0.460	0.240	0.434	0.428	0.396
200	I-2	50% and 50%	0.353	0.342	0.356	0.374	0.399	0.393
200	I-2	40% and 55%	0.309	0.300	0.365	0.365	0.435	0.401
200	I-2	27% and 55%	0.309	0.301	0.357	0.362	0.435	0.405
300	I-2	25% and 25%	0.641	0.642	0.314	0.595	0.603	0.555
300	I-2	50% and 50%	0.485	0.478	0.467	0.507	0.573	0.543
300	I-2	40% and 55%	0.457	0.441	0.503	0.515	0.598	0.569
300	I-2	27% and 55%	0.446	0.438	0.500	0.524	0.583	0.576
400	I-2	25% and 25%	0.814	0.814	0.458	0.778	0.726	0.688
400	I-2	50% and 50%	0.642	0.638	0.620	0.685	0.704	0.673
400	I-2	40% and 55%	0.627	0.615	0.636	0.666	0.739	0.703
400	I-2	27% and 55%	0.615	0.603	0.635	0.672	0.738	0.703
100	I-3	25% and 25%	0.182	0.183	0.118	0.173	0.178	0.150
100	I-3	50% and 50%	0.229	0.226	0.229	0.242	0.243	0.223
100	I-3	40% and 55%	0.170	0.167	0.151	0.172	0.174	0.154
100	I-3	27% and 55%	0.172	0.169	0.138	0.168	0.171	0.158
200	I-3	25% and 25%	0.371	0.371	0.150	0.325	0.304	0.284
200	I-3	50% and 50%	0.412	0.407	0.411	0.420	0.410	0.406
200	I-3	40% and 55%	0.301	0.297	0.233	0.300	0.308	0.319
200	I-3	27% and 55%	0.309	0.301	0.200	0.284	0.335	0.302
300	I-3	25% and 25%	0.546	0.546	0.195	0.486	0.446	0.420
300	I-3	50% and 50%	0.583	0.582	0.565	0.602	0.591	0.584
300	I-3	40% and 55%	0.443	0.436	0.334	0.430	0.458	0.454
300	I-3	27% and 55%	0.449	0.435	0.297	0.437	0.461	0.439
400	I-3	25% and 25%	0.706	0.706	0.261	0.657	0.560	0.537
400	I-3	50% and 50%	0.707	0.705	0.684	0.710	0.713	0.721
400	I-3	40% and 55%	0.583	0.577	0.435	0.582	0.579	0.570
400	I-3	27% and 55%	0.591	0.569	0.373	0.572	0.613	0.558
100	J-1	25% and 25%	0.400	0.396	0.058	0.336	0.229	0.240
100	J-1	50% and 50%	0.261	0.257	0.106	0.223	0.172	0.158
100	J-1	40% and 55%	0.229	0.229	0.095	0.205	0.175	0.152
100	J-1	27% and 55%	0.255	0.245	0.085	0.220	0.168	0.158
200	J-1	25% and 25%	0.752	0.751	0.050	0.680	0.429	0.391
200	J-1	50% and 50%	0.530	0.528	0.122	0.456	0.308	0.280
200	J-1	40% and 55%	0.529	0.521	0.132	0.453	0.302	0.287
200	J-1	27% and 55%	0.523	0.507	0.111	0.451	0.295	0.287
300	J-1	25% and 25%	0.912	0.913	0.057	0.871	0.601	0.581
300	J-1	50% and 50%	0.740	0.736	0.154	0.685	0.430	0.387
300	J-1	40% and 55%	0.707	0.698	0.148	0.647	0.434	0.395
300	J-1	27% and 55%	0.713	0.700	0.108	0.620	0.444	0.427
400	J-1	25% and 25%	0.981	0.981	0.062	0.964	0.774	0.712
400	J-1	50% and 50%	0.874	0.873	0.167	0.823	0.558	0.525
400	J-1	40% and 55%	0.869	0.865	0.176	0.813	0.549	0.533
400	J-1	27% and 55%	0.868	0.848	0.134	0.812	0.564	0.551
100	J-2	25% and 25%	0.314	0.320	0.056	0.252	0.205	0.193
100	J-2	50% and 50%	0.169	0.166	0.075	0.146	0.095	0.098
100	J-2	40% and 55%	0.178	0.177	0.066	0.145	0.101	0.101
100	J-2	27% and 55%	0.186	0.184	0.053	0.144	0.103	0.100
200	J-2	25% and 25%	0.684	0.688	0.072	0.602	0.369	0.349
200	J-2	50% and 50%	0.334	0.328	0.066	0.272	0.143	0.135
200	J-2	40% and 55%	0.342	0.339	0.079	0.284	0.153	0.136
200	J-2	27% and 55%	0.355	0.346	0.061	0.280	0.150	0.138
300	J-2	25% and 25%	0.864	0.868	0.090	0.823	0.581	0.526
300	J-2	50% and 50%	0.535	0.527	0.086	0.457	0.199	0.181
300	J-2	40% and 55%	0.523	0.514	0.072	0.443	0.230	0.193
300	J-2	27% and 55%	0.541	0.528	0.057	0.455	0.204	0.193
400	J-2	25% and 25%	0.942	0.942	0.091	0.920	0.715	0.677
400	J-2	50% and 50%	0.704	0.696	0.085	0.618	0.282	0.243
400	J-2	40% and 55%	0.673	0.660	0.076	0.597	0.264	0.237
400	J-2	27% and 55%	0.688	0.675	0.062	0.611	0.252	0.246
100	J-3	25% and 25%	0.765	0.763	0.545	0.746	0.543	0.656
100	J-3	50% and 50%	0.525	0.517	0.434	0.527	0.508	0.540
100	J-3	40% and 55%	0.477	0.472	0.434	0.494	0.582	0.571
100	J-3	27% and 55%	0.495	0.491	0.470	0.531	0.551	0.598
200	J-3	25% and 25%	0.980	0.979	0.808	0.971	0.788	0.928
200	J-3	50% and 50%	0.863	0.857	0.709	0.850	0.741	0.833
200	J-3	40% and 55%	0.775	0.755	0.668	0.779	0.811	0.826
200	J-3	27% and 55%	0.776	0.761	0.705	0.802	0.791	0.856
300	J-3	25% and 25%	0.997	0.997	0.932	0.996	0.920	0.986
300	J-3	50% and 50%	0.957	0.956	0.869	0.961	0.881	0.954
300	J-3	40% and 55%	0.926	0.908	0.831	0.924	0.911	0.940
300	J-3	27% and 55%	0.921	0.907	0.854	0.930	0.897	0.955
400	J-3	25% and 25%	1.000	1.000	0.983	1.000	0.975	1.000
400	J-3	50% and 50%	0.998	0.998	0.946	0.998	0.959	0.985
400	J-3	40% and 55%	0.981	0.976	0.929	0.981	0.959	0.981
400	J-3	27% and 55%	0.984	0.979	0.950	0.988	0.961	0.991
100	K-1	25% and 25%	0.269	0.275	0.115	0.238	0.277	0.256
100	K-1	50% and 50%	0.111	0.109	0.046	0.094	0.139	0.123
100	K-1	40% and 55%	0.165	0.164	0.061	0.147	0.165	0.132
100	K-1	27% and 55%	0.167	0.167	0.058	0.145	0.137	0.148
200	K-1	25% and 25%	0.529	0.536	0.177	0.476	0.510	0.491
200	K-1	50% and 50%	0.255	0.254	0.065	0.217	0.228	0.197
200	K-1	40% and 55%	0.257	0.257	0.053	0.207	0.265	0.223
200	K-1	27% and 55%	0.286	0.281	0.062	0.235	0.236	0.257
300	K-1	25% and 25%	0.757	0.763	0.246	0.709	0.710	0.707
300	K-1	50% and 50%	0.409	0.409	0.045	0.344	0.325	0.295
300	K-1	40% and 55%	0.401	0.398	0.041	0.318	0.372	0.320
300	K-1	27% and 55%	0.427	0.418	0.043	0.345	0.335	0.356
400	K-1	25% and 25%	0.899	0.905	0.372	0.882	0.852	0.832
400	K-1	50% and 50%	0.523	0.520	0.065	0.447	0.429	0.419
400	K-1	40% and 55%	0.534	0.526	0.057	0.464	0.479	0.401
400	K-1	27% and 55%	0.576	0.561	0.066	0.474	0.432	0.462
100	K-2	25% and 25%	0.428	0.427	0.053	0.364	0.283	0.264
100	K-2	50% and 50%	0.215	0.214	0.105	0.188	0.223	0.182
100	K-2	40% and 55%	0.251	0.255	0.093	0.220	0.213	0.181
100	K-2	27% and 55%	0.297	0.285	0.080	0.246	0.217	0.204
200	K-2	25% and 25%	0.796	0.798	0.047	0.737	0.545	0.497
200	K-2	50% and 50%	0.518	0.516	0.159	0.464	0.388	0.379
200	K-2	40% and 55%	0.526	0.525	0.151	0.471	0.397	0.358
200	K-2	27% and 55%	0.536	0.521	0.119	0.460	0.398	0.381
300	K-2	25% and 25%	0.948	0.947	0.044	0.917	0.749	0.695
300	K-2	50% and 50%	0.736	0.730	0.207	0.675	0.571	0.518
300	K-2	40% and 55%	0.755	0.748	0.171	0.683	0.576	0.524
300	K-2	27% and 55%	0.759	0.737	0.131	0.679	0.561	0.562
400	K-2	25% and 25%	0.986	0.986	0.054	0.975	0.894	0.860
400	K-2	50% and 50%	0.857	0.858	0.221	0.821	0.712	0.689
400	K-2	40% and 55%	0.857	0.850	0.194	0.819	0.721	0.685
400	K-2	27% and 55%	0.857	0.835	0.143	0.797	0.702	0.671
100	K-3	25% and 25%	0.897	0.899	0.055	0.844	0.603	0.550
100	K-3	50% and 50%	0.574	0.580	0.180	0.525	0.454	0.427
100	K-3	40% and 55%	0.546	0.536	0.221	0.508	0.512	0.432
100	K-3	27% and 55%	0.589	0.586	0.207	0.552	0.477	0.481
200	K-3	25% and 25%	0.997	0.997	0.051	0.992	0.953	0.940
200	K-3	50% and 50%	0.920	0.921	0.300	0.901	0.809	0.779
200	K-3	40% and 55%	0.899	0.898	0.298	0.876	0.852	0.796
200	K-3	27% and 55%	0.922	0.921	0.234	0.896	0.823	0.837
300	K-3	25% and 25%	1.000	1.000	0.046	1.000	0.998	0.996
300	K-3	50% and 50%	0.990	0.988	0.378	0.983	0.965	0.936
300	K-3	40% and 55%	0.982	0.982	0.391	0.975	0.973	0.946
300	K-3	27% and 55%	0.990	0.990	0.280	0.985	0.963	0.966
400	K-3	25% and 25%	1.000	1.000	0.050	1.000	1.000	1.000
400	K-3	50% and 50%	0.997	0.996	0.500	0.996	0.992	0.995
400	K-3	40% and 55%	1.000	1.000	0.478	0.999	0.996	0.993
400	K-3	27% and 55%	1.000	1.000	0.335	0.997	0.995	0.996
100	L	25% and 25%	0.565	0.562	0.675	0.630	0.701	0.674
100	L	50% and 50%	0.410	0.406	0.514	0.472	0.509	0.512
100	L	40% and 55%	0.386	0.382	0.510	0.463	0.577	0.530
100	L	27% and 55%	0.448	0.440	0.574	0.532	0.517	0.565
200	L	25% and 25%	0.887	0.885	0.948	0.929	0.940	0.941
200	L	50% and 50%	0.723	0.718	0.821	0.781	0.807	0.799
200	L	40% and 55%	0.727	0.720	0.837	0.805	0.857	0.807
200	L	27% and 55%	0.810	0.804	0.891	0.860	0.817	0.863
300	L	25% and 25%	0.982	0.983	0.992	0.988	0.991	0.992
300	L	50% and 50%	0.891	0.888	0.952	0.934	0.922	0.928
300	L	40% and 55%	0.890	0.885	0.949	0.937	0.967	0.937
300	L	27% and 55%	0.936	0.937	0.973	0.963	0.933	0.969
400	L	25% and 25%	0.999	0.999	1.000	0.999	0.997	0.997
400	L	50% and 50%	0.960	0.960	0.984	0.983	0.974	0.983
400	L	40% and 55%	0.961	0.959	0.990	0.985	0.992	0.979
400	L	27% and 55%	0.982	0.982	0.994	0.993	0.988	0.994
100	M	25% and 25%	0.316	0.317	0.474	0.417	0.403	0.417
100	M	50% and 50%	0.221	0.218	0.367	0.302	0.297	0.303
100	M	40% and 55%	0.217	0.220	0.372	0.306	0.341	0.309
100	M	27% and 55%	0.231	0.223	0.384	0.321	0.344	0.339
200	M	25% and 25%	0.529	0.537	0.752	0.686	0.700	0.705
200	M	50% and 50%	0.364	0.364	0.624	0.528	0.529	0.537
200	M	40% and 55%	0.397	0.399	0.614	0.543	0.593	0.571
200	M	27% and 55%	0.416	0.418	0.651	0.564	0.548	0.609
300	M	25% and 25%	0.742	0.749	0.907	0.864	0.870	0.874
300	M	50% and 50%	0.523	0.529	0.775	0.704	0.721	0.697
300	M	40% and 55%	0.568	0.566	0.793	0.726	0.767	0.751
300	M	27% and 55%	0.618	0.614	0.812	0.763	0.732	0.775
400	M	25% and 25%	0.880	0.882	0.965	0.946	0.948	0.958
400	M	50% and 50%	0.669	0.676	0.875	0.815	0.840	0.828
400	M	40% and 55%	0.727	0.724	0.900	0.859	0.875	0.854
400	M	27% and 55%	0.757	0.757	0.916	0.875	0.841	0.874
100	N	25% and 25%	0.510	0.505	0.608	0.569	0.567	0.566
100	N	50% and 50%	0.413	0.409	0.484	0.451	0.432	0.455
100	N	40% and 55%	0.318	0.319	0.418	0.378	0.401	0.367
100	N	27% and 55%	0.291	0.287	0.431	0.385	0.391	0.402
200	N	25% and 25%	0.783	0.782	0.871	0.838	0.846	0.846
200	N	50% and 50%	0.679	0.673	0.757	0.729	0.717	0.730
200	N	40% and 55%	0.489	0.481	0.654	0.613	0.659	0.618
200	N	27% and 55%	0.438	0.432	0.678	0.616	0.612	0.653
300	N	25% and 25%	0.925	0.923	0.971	0.951	0.948	0.955
300	N	50% and 50%	0.846	0.843	0.902	0.869	0.879	0.877
300	N	40% and 55%	0.664	0.653	0.831	0.790	0.807	0.827
300	N	27% and 55%	0.612	0.590	0.848	0.802	0.790	0.826
400	N	25% and 25%	0.977	0.977	0.994	0.991	0.985	0.992
400	N	50% and 50%	0.932	0.932	0.969	0.954	0.957	0.950
400	N	40% and 55%	0.792	0.771	0.924	0.887	0.909	0.897
400	N	27% and 55%	0.731	0.700	0.926	0.903	0.897	0.913
100	O	25% and 25%	0.975	0.975	0.990	0.987	0.986	0.991
100	O	50% and 50%	0.903	0.899	0.943	0.928	0.937	0.931
100	O	40% and 55%	0.793	0.781	0.904	0.884	0.905	0.901
100	O	27% and 55%	0.769	0.759	0.914	0.893	0.902	0.918
200	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000
200	O	50% and 50%	0.993	0.992	0.998	0.998	1.000	1.000
200	O	40% and 55%	0.975	0.966	0.999	0.998	0.996	0.996
200	O	27% and 55%	0.966	0.946	1.000	0.997	0.994	0.998
300	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000
300	O	50% and 50%	1.000	1.000	1.000	1.000	1.000	1.000
300	O	40% and 55%	0.996	0.995	1.000	0.999	1.000	0.999
300	O	27% and 55%	0.989	0.981	1.000	1.000	1.000	1.000
400	O	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000
400	O	50% and 50%	1.000	1.000	1.000	1.000	1.000	1.000
400	O	40% and 55%	1.000	0.998	1.000	1.000	1.000	1.000
400	O	27% and 55%	0.997	0.994	1.000	1.000	1.000	1.000
100	P	25% and 25%	0.743	0.740	0.784	0.779	0.752	0.743
100	P	50% and 50%	0.699	0.696	0.755	0.735	0.690	0.705
100	P	40% and 55%	0.572	0.560	0.695	0.663	0.662	0.661
100	P	27% and 55%	0.493	0.485	0.689	0.648	0.679	0.643
200	P	25% and 25%	0.949	0.949	0.964	0.962	0.970	0.961
200	P	50% and 50%	0.924	0.923	0.949	0.940	0.952	0.945
200	P	40% and 55%	0.860	0.857	0.928	0.909	0.917	0.916
200	P	27% and 55%	0.736	0.724	0.920	0.893	0.926	0.913
300	P	25% and 25%	0.992	0.992	0.997	0.997	0.997	0.996
300	P	50% and 50%	0.990	0.989	0.995	0.992	0.996	0.994
300	P	40% and 55%	0.966	0.964	0.989	0.983	0.982	0.988
300	P	27% and 55%	0.854	0.842	0.986	0.980	0.988	0.982
400	P	25% and 25%	1.000	1.000	1.000	1.000	1.000	1.000
400	P	50% and 50%	1.000	1.000	1.000	1.000	0.998	1.000
400	P	40% and 55%	0.993	0.991	0.999	0.998	1.000	0.999
400	P	27% and 55%	0.900	0.878	0.999	0.996	0.998	0.998
100	Q	25% and 25%	0.167	0.164	0.213	0.187	0.216	0.215
100	Q	50% and 50%	0.086	0.086	0.103	0.088	0.104	0.102
100	Q	40% and 55%	0.116	0.119	0.135	0.138	0.157	0.139
100	Q	27% and 55%	0.136	0.133	0.142	0.149	0.145	0.153
200	Q	25% and 25%	0.281	0.281	0.330	0.303	0.380	0.379
200	Q	50% and 50%	0.110	0.107	0.146	0.132	0.151	0.153
200	Q	40% and 55%	0.162	0.160	0.186	0.190	0.249	0.220
200	Q	27% and 55%	0.184	0.178	0.199	0.212	0.220	0.229
300	Q	25% and 25%	0.441	0.439	0.468	0.464	0.529	0.537
300	Q	50% and 50%	0.164	0.162	0.201	0.182	0.200	0.209
300	Q	40% and 55%	0.239	0.235	0.254	0.255	0.318	0.307
300	Q	27% and 55%	0.261	0.251	0.271	0.290	0.295	0.334
400	Q	25% and 25%	0.552	0.552	0.585	0.589	0.679	0.685
400	Q	50% and 50%	0.221	0.220	0.239	0.228	0.269	0.249
400	Q	40% and 55%	0.307	0.305	0.305	0.336	0.408	0.384
400	Q	27% and 55%	0.346	0.339	0.344	0.373	0.358	0.417

Equations

$$A_{11}(i,j)$$

$$A_{12}(i,j)$$

$$A_{21}(i,j)$$

$$A_{22}(i,j)$$

$$\gamma_{k}=\left\{\begin{array}[]{ll}2\displaystyle\max_{i=1,\ldots,n}\{T_{i}\}-\displaystyle\min_{i=1,\ldots,n}\{T_{i}\}&\quad\mbox{if}\,\,\,\,\displaystyle\max_{i=1,\ldots,n}\{T_{i}\Delta_{i}I(Y_{i}=k)\}=\displaystyle\max_{i=1,\ldots,n}\{T_{i}I(Y_{i}=k)\}\\ \displaystyle\max_{i=1,\ldots,n}\{T_{i}\Delta_{i}I(Y_{i}=k)\}&\quad\mbox{if}\,\,\,\,\displaystyle\max_{i=1,\ldots,n}\{T_{i}\Delta_{i}I(Y_{i}=k)\}\neq\displaystyle\max_{i=1,\ldots,n}\{T_{i}I(Y_{i}=k)\}\end{array}\right.$$

$$A_{11}^{\ast }(i,j)$$

$$A_{12}^{\ast }(i,j)$$

$$A_{21}^{\ast }(i,j)$$

$$A_{22}^{\ast }(i,j)$$

$$S_P(i,j)=\frac{\{n(i,j)-2\}\{A_{12}^{\ast }(i,j)A_{21}^{\ast }(i,j)-A_{11}^{\ast }(i,j)A_{22}^{\ast }(i,j){\}}^2}{A_{1\cdot }^{\ast }(i,j)A_{2\cdot }^{\ast }(i,j)A_{\cdot 1}^{\ast }(i,j)A_{\cdot 2}^{\ast }(i,j)},$$

$$S_{LR}(i,j)=2\sum_{m=1,2}\sum_{k=1,2}A^{*}_{mk}(i,j)\log\bigg{[}\frac{\{n(i,j)-2\}A^{*}_{kl}(i,j)}{A^{*}_{m\cdot}(i,j)A^{*}_{\cdot k}(i,j)}\bigg{]}\,,$$

$$N=\sum _{k=1}^K\sum _{i=1,T_i\in {\mathcal{A}}_k}^n\sum _{j=1,j\ne i}^n{\Delta }_i{\Delta }_{j}I(T_j\le {\tau }_k)I(b_{ij}\le {\tau }_k)$$

$$Q=\frac{1}{N}\sum _{k=1}^K\sum _{i=1,T_i\in {\mathcal{A}}_k}^n\sum _{j=1,j\ne i}^{n}S(i,j){\Delta }_i{\Delta }_{j}I(b_{ij}\le {\tau }_k).$$

$$(T_{i}^{p},\Delta_{i}^{p})=\left\{\begin{array}[]{ll}(T_{i},\Delta_{i})&\quad\mbox{if}\,\,\,\,Y_{i}^{p}=Y_{i}\\ (\widetilde{T}_{i},\widetilde{\Delta}_{i})&\quad\mbox{if}\,\,\,\,Y_{i}^{p}\neq Y_{i}\end{array}\,\,\,\,,\right.$$

$$\widetilde{X}_{i}=\left\{\begin{array}[]{ll}X_{i}&\quad\mbox{if}\,\,\,\,\Delta_{i}=1\\ X_{i}^{*}&\quad\mbox{if}\,\,\,\,\Delta_{i}=0\end{array}\,\,\,\,,\right.$$

$$\text{mbox}HR(t)=\frac{{\lambda }_2(t)}{{\lambda }_1(t)}=\frac{{\theta }_1{\theta }_2}{{\theta }_1+({\theta }_2-{\theta }_1)S_1(t)},0<t<{\tau }_0,{\theta }_1,{\theta }_2>0,$$

$$Cau=[\tan \{(0.5-p\text{mbox}{-value}_P)\pi \}]/3+[\tan \{(0.5-p\text{mbox}{-value}_{LR})\pi \}]/3+[\tan \{(0.5-p\text{mbox}{-value}_{lgrnk})\pi \}]/3,$$

$$p\text{mbox}{-value}_{Cau}=0.5-(\arctan Cau)/\pi .$$

$$\mathcal{B}=\{x:d(x,x_0)<R\}$$

$$\underset{\mathcal{B}}{\min }\{f_{X|Y}(x|g)-f_X^{\ast }(x)\}>0;$$

$${\mathcal{B}}_1$$

$${\mathcal{B}}_2$$

$$\underset{n\to \infty }{\lim }\left\{\frac{A_{11}^{\ast }(i,j)}{n-2}-{\pi }_g{\int }_{{\mathcal{B}}_3}{f}_{X|Y}(x|Y=g)dx\right\}=0,$$

$$\underset{n\to \infty }{\lim }\left\{\frac{A_{1\cdot }^{\ast }(i,j)}{n-2}-{\pi }_g\right\}=0,$$

$$\underset{n\to \infty }{\lim }\left\{\frac{A_{\cdot 1}^{\ast }(i,j)}{n-2}-{\int }_{{\mathcal{B}}_3}{f}_X^{\ast }(x)dx\right\}=0,$$

$${\mathcal{B}}_3=\{x:d(x,X_i)<d(X_i,X_j).\}$$

$$S_P(i,j)=\sum _{m=1,2}\sum _{l=1,2}\frac{\{A_{ml}^{\ast }(i,j)-A_{m\cdot }^{\ast }(i,j)A_{\cdot l}^{\ast }(i,j)/(n-2){\}}^2}{A_{m\cdot }^{\ast }(i,j)A_{\cdot l}^{\ast }(i,j)/(n-2)},$$

$$S_{P_1}(i,j)=\frac{\{A_{11}^{\ast }(i,j)-A_{1\cdot }^{\ast }(i,j)A_{\cdot 1}^{\ast }(i,j)/(n-2){\}}^2}{A_{1\cdot }^{\ast }(i,j)A_{\cdot 1}^{\ast }(i,j)/(n-2)}.$$

$$Q\ge \frac{1}{N}\sum _{k=1}^K\sum _{T_i\in {\mathcal{A}}_k}\sum _{j=1,j\ne i}^n{S}_{P_1}(i,j){\Delta }_i{\Delta }_{j}I(T_j\le {\tau }_k)I(2T_i-T_j\le {\tau }_k)\equiv Q_1.$$

$$\underset{n\to \infty }{\lim }\frac{S_{P_1}(i,j)}{n-2}$$

$$\pi_{g}\Big{[}\int_{\mathcal{B}_{3}}\{f_{X|Y}(x|Y=g)-f_{X}^{*}(x)\}dx\Big{]}^{2}\geq\pi_{g}\Big{\{}c\int_{\mathcal{B}_{3}}dx\Big{\}}^{2}\geq\pi_{g}\Big{\{}c\int_{\mathcal{B}_{1}}dx\Big{\}}^{2}\equiv c^{\prime}$$

$$\lim_{n\to\infty}\mbox{pr}\big{\{}Q-N^{-1}n^{2}(n-2)c^{\prime}p_{1}p_{2}p_{3}p_{4}>0\big{\}}=1\ .$$

$$G^{\rho ,\gamma }=\sqrt{\frac{n_1+n_2}{n_1{n}_2}}{\int }_0^{\infty }{\left\{\hat{S}(t-)\right\}}^{\rho }{\left\{1-\hat{S}(t-)\right\}}^{\gamma }\frac{{\overline{Y}}_1(t){\overline{Y}}_2(t)}{{\overline{Y}}_1(t)+{\overline{Y}}_2(t)}\left\{\frac{d{\overline{N}}_1(t)}{{\overline{Y}}_1(t)}-\frac{d{\overline{N}}_2(t)}{{\overline{Y}}_2(t)}\right\}$$

$${\hat{\sigma }}_{lm}$$

Full text

Consistent test,

Crossing hazards,

Non-parametric test,

Non-proportional hazards,

Permutation test,

Right censoring,

Sample-space partition,

*Both authors contributed equally to this work.,

and

and

1 Introduction

For the task of comparing survival distributions of two or more groups using censored data, the logrank test is the most popular choice. Its optimality properties under proportional-hazard functions are well known. Although the logrank test is asymptotically valid, it may not be powerful when the proportional hazards assumption does not hold. There are a variety of situations in which the hazard functions are of non-proportional shape. For example, a medical treatment might have adverse effects in the short run, yet effective in the long run, or a treatment may be short-term beneficial but gradually lose its effect with time. In such scenarios the hazard functions cross. In general, the longer the follow-up period is, the more likely it is for various non-proportional scenarios to develop (Yang and Prentice, 2010).

Other tests have been proposed that might be better choices for non-proportional hazards under the alternative. Peto and Peto (1972) proposed a test which is similar to the logrank test, but more sensitive for differences in hazards at early survival times than at late ones. Pepe and Fleming Pepe and Fleming (1989, 1991) suggested a weighted Kaplan–Meier (KM) test with a weight function consists of the geometric average of the two censoring survival-function estimators. Yang and Prentice (2010) recently proposed another weighted logrank test whose weights are obtained by fitting their model (Yang and Prentice, 2005), which includes the proportional hazards and the proportional odds models as special cases. In contrast to the logrank and other related tests, the test of Yang and Prentice (2010) uses adaptive weights. Under proportional hazards alternatives, this new adaptively-weighted logrank test is optimal. When the hazards are non-proportional, the adaptive weights typically lead to improvement in power over the logrank test. The test of Yang and Prentice (2010), referred here as the Yang–Prentice test, is currently considered to be the leading one in terms of power, under a wide range of non-proportional hazards alternatives. However, this test is applicable only for two-sample problems. Moreover, it is not invariant to group labeling. Exchanging the group labels between treatment and control would result in a different $p$-value. Thus, in applications with no clear link between the groups to treatment/control labeling, such as in testing the differences between females and males, the Yang–Prentice test in its current form is inappropriate. In Section 3.2 we suggest an invariant version of the Yang–Prentice test.

In the statistical literature of $K$-sample tests for non-censored data, there exist powerful consistent tests that are based on various sample-space partitions. These include the well-known Kolmogorv–Smirnov and Cramer–von Mises tests (Darling, 1957), and the Anderson–Darling (AD) family of statistics (Pettitt, 1976; Scholz and Stephens, 1987). In particular, Thas and Ottoy (2004) showed that the $K$-sample AD test is basically an average of Pearson statistics in $2\times K$ contingency tables that are induced by observation-based partitions of the sample space into two subsets. They suggested an extension of the AD test, by considering higher partitions, up to 4. Heller et al. (2013) proposed the HHG test, a sample-space partition-based non-parametric test for detecting associations between two random vectors of any dimension. When one of the random vectors is a categorical one-dimensional variable, the problem reduces to the $K$-sample problem with an observation-based partition of the sample space into two subsets, using three intervals. This specific partition is adopted in this work and will be described in detail. Heller et al. (2016) extended the work of Thas and Ottoy (2004) by considering sample-space partitions higher than 4, and studied test statistics that aggregate over all partitions by summation or maximization and also by aggregating over different sizes of partitions. They showed, by extensive simulation studies, that increasing the number of partitions can increase power under complex settings in which the density functions intersect 4 times or more.

In this work we present new powerful non-parametric and invariant tests for comparing two or more survival distributions using right-censored data. Our proposed methodology is demonstrated and applied using the specific sample-space partition of Heller et al. (2013), which has been shown to be very powerful with 3 or less densities’ intersections (Heller et al., 2016), under non-censored data. Right-censored data introduce two major difficulties: (i) the actual event times of censored observations are unknown; and (ii) the standard permutation procedure of label shuffling is invalid, in case the censoring distributions of the groups are unequal. We overcome these two obstacles and introduce novel consistent powerful tests for right-censored data.

2 $K$-sample tests based on sample-space partition

2.1 Motivation and Notation

Let $X$ be a one-dimensional non-negative random variable, $X\in\mathbb{R}^{+}$, and let $Y$ be a categorical variable indicating the group label. Under the $K$-sample hypotheses testing, the null hypothesis is $H_{0}:F_{1}(x)=\ldots=F_{K}(x)$ for all $x\in\mathbb{R}^{+}$, and the alternative is $H_{1}:F_{m}(x)\neq F_{k}(x)$ for some $1\leq m<k\leq K$ and some $x\in\mathbb{R}^{+}$, where $F_{k}$ is the cumulative distribution function of group $k$, $k=1,\ldots,K$. We assume that the sample spaces on which these $K$ distributions are defined, coincide.

$K$ random samples $\mathcal{A}_{1},\ldots,\mathcal{A}_{K}$ are drawn from the respective distributions $F_{1},\ldots,F_{K}$. Let $n_{k}$ be the total number of samples in group $k$, $k=1,\ldots,K$, and $n=\sum_{k=1}^{K}n_{k}$. Assume temporarily no censoring, and consider a pair of observations $X_{i}\in\mathcal{A}_{k}$ and $X_{j}$. Then, in the spirit of the HHG test (Heller et al., 2013), we consider the following partition induced by the pair $(i,j)$: $A_{11}(i,j)$ is the number of observations from group $k$ that the distance between the value of $X$ and $X_{i}$ is less than or equal to $|X_{i}-X_{j}|$; $A_{12}(i,j)$ is the number of observations outside group $k$ that the distance to $X_{i}$ is less than or equal to $|X_{i}-X_{j}|$; $A_{21}(i,j)$ is the number of observations from group $k$ that the distance to $X_{i}$ is larger than $|X_{i}-X_{j}|$; and $A_{22}(i,j)$ is the number of observations outside group $k$ that the distance to $X_{i}$ is larger than $|X_{i}-X_{j}|$.

Interestingly, with no censoring, the $A_{lr}$’s, $l,r=1,2$, can be expressed through the empirical cumulative distribution functions. For example, assume $X_{i}>X_{j}$ and $Y_{i}=Y_{j}=k$. Then,

[TABLE]

where $\widehat{F}_{k}(x)=n_{k}^{-1}\sum_{X_{l}\in\mathcal{A}_{k}}I(X_{l}\leq x)$ and $\widehat{F}_{k}(x^{-})=n_{k}^{-1}\sum_{X_{l}\in\mathcal{A}_{k}}I(X_{l}<x)$. The $-2$ above stands for excluding observations $i$ and $j$. In general, all observations with a distances to $X_{i}$ that is less than or equal to $|X_{i}-X_{j}|$ lie inside the interval $[a_{ij},b_{ij}]$, where $a_{ij}=\min(X_{j},2X_{i}-X_{j})$ and $b_{ij}=\max(X_{j},2X_{i}-X_{j})$, as illustrated in Figure 1.

In general, for $Y_{i}=k$, we get

[TABLE]

For each pair $(i,j)$, a $2\times 2$ contingency table can be constructed with $A_{lr}(i,j)$ as the entry of cell $lr$, $l,r=1,2$, and a total sum of $n-2$. Under the null hypothesis of equal distributions, the probability of belonging to cell $lr$ equals the product of the marginal probabilities of the $l$th row and the $r$th column. Therefore, a summary statistic of a such contingency table can be based on either the Pearson chi-squared test statistic, or the log-likelihood ratio statistic. Since it is unknown which pair $(i,j)$ yields the best sample-space partition that provides the largest summary statistic, the final test statistic is defined as a sum over all possible $n(n-1)$ partitions induced by the data. A permutation-based $p$-value can be calculated, under random permutations of the group labels.

To introduce the right-censored data, let $C\in\mathbb{R}^{+}$ be a non-negative random variable indicating the censoring time. Assume that $X$ and $C$ are conditionally independent given $Y$. Define $T$ to be the observed time, namely $T=\min(X,C)$ and let $\Delta=I(X\leq C)$. Hence, the observed data consist of $K$ random samples that can be summarized by $(T_{i},\Delta_{i},Y_{i}),\ i=1,\ldots,n$. Note that the different groups may have different censoring distributions. With right-censored data, our proposed test requires special care in evaluating the $A_{r,l}$’s, $r,l=1,2$, by replacing the empirical distribution functions by their respective Kaplan–Meier (KM) estimators, as well as when applying a permutation test with unequal censoring distributions. Both issues are described in details below.

2.2 The Test Statistic

Let $\widetilde{F}_{k}$ be the KM estimator of the cumulative distribution function using all observations of group $k$. The KM estimator is defined only up to (and including) the last observed failure time. Define $\gamma_{k}$ to be the maximum time in which $\widetilde{F}_{k}$ can be used for the test statistic. If the largest observed time is a failure time, $\widetilde{F}_{k}(t)$ is known for the entire range of $t$. In this case, we define $\gamma_{k}$ to be the maximum possible value of $t$ required for the test, $2\max_{i=1,\ldots,n}\{T_{i}\}-\min_{i=1,\ldots,n}\{T_{i}\}$. However, in case of censoring after the largest failure time, the KM estimator beyond the largest observed failure time is undefined. We thus define the maximum time $\gamma_{k}$ to be the largest observed failure time. Namely,

[TABLE]

Define $\gamma_{-k}$ to be the maximum time in which at least one of the other KM estimators, $\widetilde{F}_{m}$, $m=1,\ldots,K,m\neq k$, can be used for the test statistic, namely $\gamma_{-k}=\max_{m\neq k}\{\gamma_{m}\}$. Define $\tau_{k}$ to be the maximum time point the KM estimator is defined in group $k$ and in at least one more group, namely $\tau_{k}=\min\{\gamma_{k},\gamma_{-k}\}$. Then, for each pair of observed failure times $T_{i}\in\mathcal{A}_{k}$ and $T_{j}$ such that $j\neq i,\ \Delta_{i}=\Delta_{j}=1$ and $b_{ij}\leq\tau_{k}$, a $2\times 2$ contingency table is constructed. The following $A_{lr}^{*}(i,j),\ l,r=1,2$, are the corresponding expressions of $A_{lr}(i,j)$ obtained by replacing $\widehat{F}$ by $\widetilde{F}$, the KM estimators. In case $\gamma_{m}<b_{ij}$, the observations of group $m$ will not be included in the contingency table induced by the pair $(i,j)$. Namely, for $j\neq i,\ \Delta_{i}=\Delta_{j}=1,b_{ij}\leq\tau_{k}$ and $Y_{i}=k$,

[TABLE]

Only pairs of observed failure times are used for the sample-space partitioning (i.e., $\Delta_{i}=\Delta_{j}=1$), while censored observations contribute in $\widetilde{F}_{k}$, $k=1,\ldots,K$. Denote by $n(i,j)$ the number of observations in all the groups included in the contingency table induced by $(i,j)$, namely $n(i,j)=\sum_{k=1}^{K}n_{k}I(\gamma_{k}\geq b_{ij})$. Let $S_{P}$ and $S_{LR}$ be the summary statistic of each contingency table, based on Pearson chi-squared test statistic

[TABLE]

and the log-likelihood ratio statistic,

[TABLE]

respectively, where $A^{*}_{.k}(i,j)=\sum_{m=1,2}A^{*}_{mk}(i,j)$ and $A^{*}_{m.}(i,j)=\sum_{k=1,2}A^{*}_{mk}(i,j)$. In case of at least one zero margin in the contingency table, $S_{P}(i,j)=0$ and $S_{LR}(i,j)=0$. Denote by

[TABLE]

the total number of tables constructed from the data. Then, our proposed sample-space partition test statistic for equality of $K$ distributions based on right-censored data is defined by

[TABLE]

where $S(i,j)$ is either the test statistic $S_{P}(i,j)$ or $S_{LR}(i,j)$. In the case of no right censoring, the number of tables is solely determined by the number of observations. In right-censored data, the number of tables is random and determined also by the actual observed values due to the restrictions $\Delta_{i}=\Delta_{j}=1$ and $b_{ij}\leq\tau_{k}$. This issue is of high importance for the permutation stage of the test, as elaborated in the next subsection.

2.3 The Permutation Procedure

Allegedly, a permuted test can be done based on random permutations of the group labels. However, if the censoring distributions of the $K$ groups are different, such a permutation test is invalid, since a significant result can be yielded under the null due to differences in the censoring distributions. In order to generate random permutations that are independent of $Y$, we adopt the imputation approach suggested by Wang et al. (2010).

The main idea consists of randomly permuting the group labels, while for each observation assigned to a group different from the original one, a censoring time is imputed from the censoring distribution of the new assigned group. If the observation was originally censored, a survival time is also imputed, from the null survival distribution. Let $Y_{1}^{p},...,Y_{n}^{p}$ be a random permutation of the group labels. Define $(T_{i}^{p},\Delta_{i}^{p}),\ i=1,\ldots,n,$ by

[TABLE]

where $\widetilde{T}_{i}=\min(\widetilde{X}_{i},\widetilde{C}_{i})$, $\widetilde{\Delta}_{i}=I(\widetilde{X}_{i}\leq\widetilde{C}_{i})$. Furthermore, $\widetilde{C}_{i}$ is sampled from the estimated censoring distribution of group $Y_{i}^{p}$, based on the KM estimator of the censoring distribution of group $Y_{i}^{p}$, by reversing the roles of event and censoring. The KM estimator, denoted by $\widehat{G}_{C,Y_{i}^{p}}$, is defined up to the largest observed censoring value of that group. Then, each observed censoring time is sampled with probability equals to the jump size of the respective KM estimator. In case $\widehat{G}_{C,Y_{i}^{p}}$ is incomplete, i.e., \widehat{G}_{C,Y_{i}^{p}}\big{[}\max_{j=1,\ldots,n}\{T_{j}(1-\Delta_{j})I(Y_{j}=Y_{i}^{p})\}\big{]}>0, the largest observed time, $\max_{j=1,\ldots,n}\{T_{j}(1-\Delta_{j})I(Y_{j}=Y_{i}^{p})\}$, is sampled with probability \widehat{G}_{C,Y_{i}^{p}}\big{[}\max_{j=1,\ldots,n}\{T_{j}(1-\Delta_{j})I(Y_{j}=Y_{i}^{p})\}\big{]}. $\widetilde{X}_{i}$ is defined by

[TABLE]

where $X_{i}^{*}$ is sampled from an estimator of $\mbox{pr}(X_{i}>x|X_{i}>T_{i})$, the conditional distribution of $X$ under the null hypothesis. In practice, $\mbox{pr}(X_{i}>x|X_{i}>T_{i})$ is replaced by its KM estimator, using all observations from all groups that their observed time is larger than $T_{i}$. Denote this KM estimator by $\widehat{S}_{cond,T_{i}}$. The sampling based on $\widehat{S}_{cond,T_{i}}$ is done in the same fashion as in the above censoring sampling, but in case of incomplete distribution, the value $\max_{i=1,\ldots,n}(T_{i}\Delta_{i})+\varepsilon$ ($\varepsilon$ is any positive number) is sampled with probability $\widehat{S}_{cond,T_{i}}\{\max_{i=1,\ldots,n}(T_{i}\Delta_{i})\}$, and its respective event indicator is set to be $\Delta_{i}^{p}=0$, as there is no empirical evidence for the potential failure times beyond $\max_{i=1,\ldots,n}\{T_{i}\Delta_{i}\}$.

When performing a permutation test, the reported $p$-value can be viewed as an approximation of the true $p$-value, based on all possible permutations. In the above imputation-based permutation procedure, additional variability in a $p$-value is expected due to random imputations. To reduce this variability, multiple imputations can be used, such that for each random imputation, $B$ permutations are generated. Assume $M$ imputations are used. Then the $p$-value is defined as the fraction of the test statistics among the $MB$ test statistics that are at least as large as the observed test statistic $Q$.

In the following theorem it is argued that our proposed tests are consistent against all alternatives. The proof is presented in details in the Supplementary Materials.

Theorem: Let $X$ be a positive failure time random variable, either continuous or discrete, and $Y$ be a categorical random variable with $K$ categories. Let $\pi_{k}=\lim_{n\to\infty}n_{k}/n$, $k=1,\ldots,K$. Assume there are at least two cumulative distribution functions $F_{g}(x)$ and $F_{m}(x),\ g,m\in\{1,\ldots,K\}$, such that $F_{g}(x_{0})\neq F_{m}(x_{0})$ for some $x_{0}\in\mathbb{R}^{+}$, $\pi_{g}>0,\ \pi_{m}>0$, and the conditional censoring distributions are such that $\mbox{pr}(C>x_{0}|Y=g)>0$ and $\mbox{pr}(C>x_{0}|Y=m)>0$. Then, the imputation-based permutation test with the test statistic $Q$ is consistent, namely, the power of the test increases to $1$ as $n\to\infty$.

2.4 Computation Time

Table 1 provides the run time of the proposed tests of one dataset, $K=2$, under the null hypothesis, one imputation, and 1000 permutations, for different total sample sizes $n$ and $n_{1}=n_{2}=n/2$. These results were generated by a 6 years old Intel i7-3770 CPU 3.4 GHz, without paralleling with the different cores of the computer. The first two rows are for identical censoring distributions, and the last two are for different censoring distributions. Evidently, even with $n=1000$ observations and low censoring rates, the run time on such a simple computer is not longer than 3.5 minutes. We expect that an increase of the number of imputations and permutations will increase the run time in a linear fashion.

3 Simulation study

3.1 Simulation Design

An extensive numerical study has performed to systematically examine the behavior of our proposed $K$-sample omnibus non-proportional hazards (KONP) tests under a wide range of alternatives, various sample sizes, and a wide range of censoring distributions, including unequal censoring distributions. The main part of the simulation study was dedicated to the popular 2-sample setting, but settings of $K=3,4,5$ were considered as well.

As competitors under the 2-sample setting, the following tests were included: the logrank test; Peto–Peto weighted logrank test (Peto and Peto, 1972) that uses a weight function that is very close to the pooled KM estimator; Pepe–Fleming weighted KM test (Pepe and Fleming, 1989) with geometric mean of the two KM censoring-distribution estimators as a weight function; and Yang–Prentice test, an adaptive weighted logrank test where the adaptive weights utilize the hazard ratio obtained by fitting the model of Yang and Prentice Yang and Prentice (2005). The tests of Uno et al. (2015) are invalid under unequal censoring distributions (as demonstrated below), and thus are not included in the following power comparisons.

Table 2 (main text) and tables LABEL:scenario_description and LABEL:prop_design in Appendix, provide a comprehensive summary of the 17 non-proportional hazards scenarios and 7 proportional or close to proportional hazards functions, that were studied. For each scenario, the failure and censoring distributions are explicitly provided, and the survival functions of the two groups are plotted. A reference is provided indicating the source of each setting. In short, Scenario A shows differences at mid time points, but similarity in early and late times. Scenarios B–D show differences in early times. Scenario E is of equal survival functions at early times and of proportional hazards at mid and late times. Scenarios F and G are with crossing hazards. Scenario H is of a U-shape hazards ratio. Scenarios I, J and K are with crossing hazards, based on the following hazards-ratio model of Yang and Prentice (2005):

[TABLE]

where $\tau_{0}=\sup\{t:S_{1}(t)>0\}$, $\lambda_{2}$ and $\lambda_{1}$ are the hazard functions of the two groups, and $S_{1}$ is the survival function of group $1$. Under Model (1), $\theta_{1}=\lim_{t\to 0}\mbox{HR}(t)$ and $\theta_{2}=\lim_{t\to\tau_{0}}\mbox{HR}(t)$. Also it is assumed that for a continuous function $S_{1}$, $\mbox{HR}(t)$ is a strong monotone function of $t$, i.e. $\mbox{sign}\{d\mbox{HR}(t)/dt\}$ is the same for all $t\in(0,\tau_{0})$. Scenarios I-1, I-2 and I-3 are with crossing hazards under Model (1). In I-1 the hazard functions cross earlier compared to I-2 and I-3. Scenarios J-1, J-2 and J-3 are with crossing hazards in which the strong monotonicity assumption is violated. In J-1 and J-2 the hazards are piece-wise proportional, and the hazard functions cross in mid time points. In Scenario J-3 the hazard ratio is a continuous function of $t$, and the hazards cross at a late time point. Under Scenarios K-1, K-2 and K-3 Model (1) is violated, but the strong monotonicity assumption of $\mbox{HR}(t)$ holds. In Scenarios K-1 and K-2 the hazards cross at early-mid times, and in Scenario K-3 at mid times.

For each scenario described above, four different censoring distributions were considered, two with equal and two with unequal censoring distributions. Under equal censoring distributions, the censoring distributions were taken to be similar to the corresponding referenced paper. Exponential distributions were used for all other scenarios, with approximately $25\%$ or $50\%$ censoring rates. Under unequal censoring distributions, the censoring distributions of Wang et al. (2010) were used (Table 2). The specific values of $(a,b,\theta_{1},\theta_{2})$ are provided in tables LABEL:scenario_description-LABEL:prop_design. Under small differences, the censoring rates among the two groups are about $40\%$ and $55\%$, where $27\%$ and $55\%$ are the respective values under substantial differences. The various censoring settings are such that the power of a specific test under specific scenario is not necessarily increasing as the censoring rate decreases.

Each of the configurations was studied with $n=100,200,300$ or 400, $n_{1}=n_{2}$, and performances are summarized based on 2000 replications.

A smaller simulation study was done for $K>2$. As competitors, the logrank and Peto-Peto tests were included. For the null scenario $K=3,4,5$ were studied, and under Scenarios D and J-2, $K=3$ was examined. Various sample sizes and a wide range of censoring distributions were considered. A detailed description of these scenarios can be found in the Supplementary Materials.

3.2 The test of Yang and Prentice (2010)

Since the Yang–Prentice test is the strongest competitor in terms of power, for the $2$-sample setting, we highlight some of its properties. The Yang–Prentice test is based on Model (1), where the indices 1 and 2 indicate the control and treatment groups, respectively. Since this model is asymmetric in terms of $F_{1}$ and $F_{2}$, the test is not group-label invariant. By exchanging the group labels between ‘treatment’ and ‘control’, a different $p$-value would be provided. This property is unique to this test, and all other tests considered in this work are invariant to group labeling. Consequently, in applications with no clear link between the two groups to treatment/control status (e.g., comparing females versus males, or young versus old), it is unclear how the Yang–Prentice test should be applied.

In order to make the Yang–Prentice test invariant, we apply a permutation test based on the minimum $p$-value of the two labeling options. Specifically, let $PV_{1}$ and $PV_{2}$ be the $p$-values of the Yang–Prentice tests based on $(X_{i},\Delta_{i},Y_{i}),\ i=1,\ldots,n$, and $(X_{i},\Delta_{i},\widetilde{Y}_{i}),\ i=1,...,n$, respectively, where $Y_{i}\in\{1,2\}$ and $\widetilde{Y}_{i}=I(Y_{i}=2)+2I(Y_{i}=1)$. Then, the Yang–Prentice invariant test statistic is defined by $Q_{YP}=\min\{PV_{1},PV_{2}\}$. The $p$-value of the imputation-based permutation test, based on the statistic $Q_{YP}$, is the fraction of replicates of $Q_{YP}$ under random permutations of the data, as described in Section 2.3, that are at least as small as the observed test statistic. Since our work mainly focuses on invariant tests, we report and discuss in the main text the performance of the Yang–Prentice invariant test. The results of the original Yang-Prentice test, with the two possible options of labeling for treatment and control, can be found in the Supplementary Materials.

The original Yang–Prentice test (implemented in the R package YPmodel) uses the asymptotic distribution of the test statistic under the null hypothesis. Often, its empirical size is greater than the nominal level, especially under small sample size (see the Supplementary Material). In a recent work of Yang (2019), which deals with interim monitoring using adaptive weighted log-rank test, the author suggests using the re-sampling method of Lin et al. (1993) instead of the original asymptotic approach (Yang and Prentice, 2010), for improving the type I error rate. The method of Lin et al. (1993) is also based on asymptotic results. Alternatively, the imputation approach of Wang et al. (2010) works very well in very small sample sizes, and it does not require asymptotic distribution. To make the comparison between the methods consistently, we use the imputation approach for both, our proposed method and the invariant version of Yang-Prentice test.

3.3 Simulation Results

Figure 2 provides the empirical power of the tests under the null hypothesis, with equal and unequal censoring distributions. Evidently, under equal censoring distributions all the tests are valid, as the empirical size of the tests are reasonably close to the nominal value $0.05$. On the other hand, under the null hypothesis and unequal censoring distributions, the empirical sizes of Uno et al. tests are much higher than the nominal value 0.05. For example, under a sample size of $n=400$, and censoring rates of approximately $27\%$ and $55\%$, the empirical size of Uno et al. $V_{2}$ bona fide test is $0.099$; all the other Uno et al. tests’ sizes are even higher. The empirical sizes of all the other tests are reasonably close to their nominal value. Thus, Uno’s tests will not be considered in the rest of this simulation study.

Figure 3 summarizes the empirical power of the tests under settings generated by others, while Figure 4 is based on scenarios generated by us. As expected, the power of each test increases with the sample size. In some scenarios the power of the tests increases as the censoring rate increases, since in these settings the non-censored data are centered mainly at the parts of the hazards which are closer to proportionality and the censored data are mainly located at the non-proportionality area of the hazards. For example, in Scenario I-3 with $n=400$, as the censoring rate increases from about $25\%$ to about $50\%$, the power of Peto–Peto increases by $0.159$, Yang–Prentice by $0.2$, Pepe–Fleming by $0.218$, and logrank by $0.4$. The power increase in our KONP tests is much smaller, 0.012 and 0.009.

Evidently, our KONP tests are often more powerful compared to all other tests. These includes scenarios A, C, D, E, F, J-1, J-2, J-3, I-1, K-1, K-2, K-3. The superiority of KONP over Yang-Prentice under F and I-1 is surprising, since these scenarios follow their Model (1). For example, with $n=400$ and $25\%$ censoring rate, the empirical power is about $90\%$ for KONP tests and only $70\%$ for Yang–Prentice. In Scenario G (close to proportional hazards), which also follows Model (1), the results of Yang-Prentice and logrank are similar and often slightly better than KONP tests. In scenarios J-1 and J-2 our tests are substantially more powerful than all the competitors. For example, in Scenario J-2, $n=400$, and $25\%$ censoring rate, KONP tests with about $95\%$ power, while the second most powerful test is Yang–Prentice with a power of $48\%$.

In scenarios G, I-2 and in some of the censoring rates in I-3, the Peto–Peto and Pepe–Fleming tests tend to be with the highest power. In contrast, in many of the scenarios in which the survival functions cross, their power are much lower than our tests. For example, in Scenario K-1, $n=400$, and $25\%$ censoring rate, the power of KONP is $89\%$, while Pepe–Fleming power is $42\%$ and Peto–Peto is $5\%$.

To conclude, for the $2$-sample setting, in most of the non-proportional hazards settings, our proposed KONP tests tend to be more powerful than the other tests, and the differences between $S_{P}$ and $S_{LR}$ are very small, if any.

Results of settings with $K>2$ can be found in Table S1 and Figure S1 of the Supplementary Material. Based on these results we conclude that all the tests are valid, as the empirical size of the tests are reasonably close to $0.05$. For $K=3$ and scenarios D and J-2, we see similar results to those shown with $K=2$, as KONP tests are often much more powerful than the logrank and Peto–Peto test. For example, for scenario D with $n=200$ and $25\%$ censoring rates in all the three groups, the powers of KONP tests are approximately $92\%$, while of Peto–Peto is $49\%$, and of logrank is $18\%$.

Figure 5 summarizes the power of the 2-sample tests under proportional hazards or close to proportionality. Under these settings the logrank test is often with the highest power among the invariant tests, as expected. The invariant version of Yang–Prentice test is similar to the logrank test, and the proposed KONP tests, sometimes, have less power.

3.4 A Robust Approach

Figures 3 and 4 of the main text and Table S2 of the Supplementary Material indicate that under the non-proportional hazards scenarios and among the invariant tests, usually the proposed KONP tests are with the largest power. Under proportional hazards or close to proportionality, usually the logrank test is with the largest power among the invariant tests. The invariant version of Yang–Prentice test is similar to the logrank test under proportional hazards settings, since their model contains the proportional hazards model ($\theta_{1}=\theta_{2}$).

In case one is interested in a robust powerful test under non-proportional or proportional hazards, the principle of minimum p-value could be adopted based on the elegant Cauchy-combination test of Liu and Xie (2019), which is similar to the test based on the minimum $p$-value. Denote the $p$-values of our KONP tests by $p\mbox{-value}_{P}$ and $p\mbox{-value}_{LR}$, respectively, and the $p$-value of the logrank test by $p\mbox{-value}_{lgrnk}$. Then, based on Liu and Xie (2019) we define a new test statistic

[TABLE]

and its $p$-value is

[TABLE]

The candidate tests to be included in ${Cau}$ are powerful tests under non-proportional (i.e. KONP tests) or proportional hazards (i.e. logrank test and the invariant version of Yang–Prentice test). Due to the similarity in power performances between the logrank test and the invariant version of Yang–Prentice test, under proportional or close to proportional hazards, and due to the high computational burden of Yang–Prentice invariant test, only the logrank test is included.

Figure S3 of the Supplementary Materials provides the empirical Type-I error of the two KONP tests, the logrank test and the robust $Cau$ test. Evidently, the size of the tests are reasonably close to 0.05. Figures S4-S5 of the Supplementary Materials summarize the empirical power, based on 1000 replications, of the two KONP tests, the logrank test and the test statistic $Cau$. Often, the $Cau$ test loses some power comparing to the largest power among KONP and logrank, but the loss is relatively small. Table S4 of the Supplementary Materials provides the power values of these tests under all the studied alternatives, along with two additional combined test: the test of Lee Lee (2007) which is based on the maximum of two weighted logrank test statistics; and the MaxCombo test Lin et al. (2019) which is based on the maximum of the logrank and three weighted logrank test statistics. (See Supplementary Materials for details.) Lee and MaxCombo tests perform very similarly. In general, our $Cau$ test outperforms Lee and the MaxCombo tests, in terms of power, in all the sub-scenarios (i.e., various sample size and censoring patterns) of type B, C, H, J-1, and J-2; and in some of the sub-scenarios of A, F, I-1, I-2, I-3, K-1, K-2, K-3, and P. All of these are non-proportional hazards scenarios. Under proportional hazards or close to proportionality, i.e., settings L, M, N O, P and Q, Lee’s test or MaxCombo outperform $Cau$. Interestingly, the cases where Lee or MaxCombo tests are with higher power than $Cau$, the power loss by using $Cau$ is relatively small; while this is not always the case when $Cau$ outperforms Lee or MaxCombo. For example, under Setting B with $n=200$, the power of $Cau$ equals 0.919 while the power values of Lee and MaxCombo are 0.499 and 0.450, respectively.

Our R package KONPsurv (Schlesinger and Gorfine, 2019) applies the above robust test $Cau$ as well.

4 Real data examples

4.1 The Gastrointestinal Tumor Data

The Gastrointestinal Tumor Study Group (Schein and Group, 1982) compared chemotherapy with combined chemotherapy and radiation therapy, in the treatment of locally unresectable gastric cancer. This dataset was used in Yang and Prentice (2010) to demonstrate the utility of their test. Each treatment arm had 45 patients, and two observations of the chemotherapy group and six of the combination group were censored. The primary outcome measure was time to death. The KM survival curves of time to death, in each treatment group are provided in Figure 6. To apply the Yang–Prentice test, we considered chemotherapy as the control group and chemotherapy plus radiation therapy as the treatment group, which is named Yang–Prentice 1. The Yang–Prentice test with reversed group labeling is denoted by Yang–Prentice 2. Table 3 shows the $p$-values of testing for equality of the survival curves of time to death, of the two treatment groups, against a two-sided alternative, with each of the tests considered in the simulation study. For our tests and the Yang-Prentice invariant test, $10$ imputations and $10^{4}$ permutations for each imputation, were used. Evidently, the smallest $p$-values are observed under our proposed KONP tests and the Cauchy-combination test $Cau$.

4.2 Urothelial Carcinoma

Few options exist for patients with locally advanced or metastatic urothelial carcinoma after progression with platinum-based chemotherapy. Powel et al. Powel et al. (2018) aimed to assess the safety and efficacy of atezolizumab versus chemotherapy in this patient population. Their study consists of a multi-center, open-label, phase 3 randomised controlled trial conducted at 217 academic medical centers and community oncology practices mainly in Europe, North America, and the Asia-Pacific region. The primary endpoint was overall survival. Figure S3 of the Supplementary Material of Powel et al. (2018) provides the overall survival KM curves of atezolizumab versus chemotherapy based on 316 and 309 patients, respectively. Although the detailed survival data are unavailable, inspired by Roychoudhury et al. (2019) we used Guyot et al. Guyot et al. (2012) algorithm that maps from digitised curves back to KM data, by finding numerical solutions to the inverted KM equations, using available information on number of events and numbers at risk. The DigitizeIt software was used for reading the coordinates of the KM curves from the published graph. Figure 7 provides the KM curves by treatment arm, and the last column of Table 3 shows the $p$-values of testing for equality of the survival curves of time to death against a two-sided alternative. Evidently, the smallest $p$-values are observed under our proposed KONP tests and the second best is the test of Lee. Our Cauchy combined test also performs robustly.

5 Discussion and Conclusions

The proposed KONP tests are based on partition of the sample-space into two subsets, corresponding to three intervals, as of the HHG test (Heller et al., 2013). An extensive simulation study shows that when the hazard curves are non-proportional, the KONP tests are often more powerful than all the other tests. In particular, the proposed test is even more powerful than the Yang–Prentice test, under their model with non-proportional hazards. The simulation results show very little differences in power, if any, between the Pearson chi-squared test statistic and the log-likelihood ratio statistic. Since the chi-squared statistic is slightly more powerful, this test statistic is recommended.

Other partitions and summary statistics can be easily adopted. In particular, one may consider the extended Anderson-Darling tests of Thas and Ottoy (2004) and Heller et al. (2016) with higher sample-space partitions and test statistics that aggregate over all partitions by summation or maximization. Nevertheless, for non-censored data, Heller et al. (2016) showed by simulations (see their Table 1), that increasing the number of partitions can improve power over the HHG 2-sample test under settings in which the density functions intersect 4 times or more. Otherwise, the HHG 2-sample test tends to be more powerful. Figure A1 in Appendix 6.1 displays the densities of the 17 non-proportional hazards scenarios studied in this work. Evidently, the survival scenarios considered by others and by us are of less than 4 intersections. Simulation study of the Anderson-Darling test statistic with sample-space partition of two intervals, yields lower power than the proposed KONP tests. A comprehensive comparison with other sample-space partitions and aggregations could be a topic of future research.

This work suggests tests that accommodate right-censored data. Since the tests are based on the KM estimator, it seems that a modification to left truncation might be possible. However, additional work is required to modify the imputation-permutation approach for left-truncated data.

Implementation of our tests, KONP-P, KONP-LR and $Cau$, is available in the R package KONPsurv (Schlesinger and Gorfine, 2019), which can be freely downloaded from CRAN.

Acknowledgement

The authors gratefully acknowledge support from the NIH (R01CA189532) and the U.S.-Israel Binational Science Foundation (2016126), in carrying out this work.

Supplementary materials

Supplementary materials include: (1) The proof of the theorem. (2) Description of some of the tests included in the simulation study. (3) Description and results of $K$-sample simulation settings with $K=3,4,5$. (4) Plots of the empirical power results of the robust test $Cau$. (5) The exact empirical power used for generating all the plots in the main text and the Supplementary Material file.

6 Appendix

6.1 Detailed Description of the Simulation Settings

7 Supplementary Materials

This Supplementary Material file includes, the proof of the theorem, additional simulation results and additional details of the simulation results summarized in the paper.

Proof of the theorem for a continuous survival time

For simplicity, we show the proof using the Pearson chi-squared test statistic. The proof using the likelihood ratio test statistic is very similar and therefore omitted. The reasoning is based on the proofs of Heller et al. (2013). We show in the following that for an arbitrary fixed $\alpha\in(0,1)$, if $H_{0}$ is false then \lim_{n\to\infty}\mbox{pr}\big{(}Q-q_{1-\alpha}>0\big{)}=1, where $q_{1-\alpha}$ is the $1-\alpha$ quantile of the test statistic under the null distribution.

Assume $X\in\mathbb{R}^{+}$ has a continuous distribution given $Y$, denoted by $f_{X|Y}(\cdot|\cdot)$, and let $f_{X}^{*}(x)=\sum_{k=1}^{K}\pi_{k}f_{X|Y}(x|k)$, which is not necessarily the true marginal distribution. If $H_{0}$ is false, there exists at least one pair of points $(x_{0},g)$ such that without loss of generality $f_{X|Y}(x_{0}|g)>f_{X}^{*}(x_{0})$. Assume (for the moment) that $\mbox{pr}(C>x_{0}|Y=k)>0$ for all $k=1,\ldots,K$ and let $d(x,x_{0})=|x-x_{0}|$. Since $f_{X|Y}(\cdot|g)$ and $f_{X}^{*}(\cdot)$ are continuous, there exist a radius, $R>0$, and a set

[TABLE]

with positive probability, such that for $x\in\mathcal{B}$, $f_{X|Y}(x|g)>f_{X}^{*}(x)$ and $\mbox{pr}(C>x+3R|Y=k)>0$ for all $k=1,\ldots,K$. The last condition guarantees that with positive probability $S_{P}(i,j)$ with $n_{ij}=n$ (namely, the table consists of all the groups) is observed, where $Y_{i}=g$, and $X_{i},X_{j}\in\mathcal{B}$. Moreover,

[TABLE]

denote this minimum by $c$. Put

[TABLE]

and let $p_{1}=\mbox{pr}(X\in\mathcal{B}_{1},Y=g)$, $p_{2}=\mbox{pr}(X\in\mathcal{B}_{2})$, $p_{3}=\mbox{pr}(C>x_{0}+R,Y=g)$, and $p_{4}=\min_{k=1,\ldots,K}\mbox{pr}(C>x_{0}+R,Y=k)$. Therefore, we expect to have at least $n^{2}p_{1}p_{2}p_{3}p_{4}$ pairs $(i,j)$ such that $\{X_{i}\in\mathcal{B}_{1},Y_{i}=g,\Delta_{i}=1\}$ and $\{X_{j}\in\mathcal{B}_{2},\Delta_{j}=1\}$. Consider such a pair.

Stute and Wang (1993) showed that the KM estimator converges almost surely to the true survival function under random censorship. Therefore, uniformly almost surely,

[TABLE]

where

[TABLE]

Since

[TABLE]

it is enough to look at the term with $l=m=1$ in $S_{P}(i,j)$, that is,

[TABLE]

It follows that $S_{P}(i,j)\geq S_{P_{1}}(i,j)$ and hence that our test statistic satisfies

[TABLE]

By Slutzky’s theorem and the continuous mapping theorem, in probability

[TABLE]

and we show that this limit can be bounded from below by a positive constant that does not depend on $(i,j)$. Indeed, it can be shown that $\mathcal{B}_{3}\subseteq\mathcal{B}$ and $\mathcal{B}_{1}\subseteq\mathcal{B}_{3}$, by the triangle inequality (see Heller et al. for details). Therefore,

[TABLE]

since $\int_{\mathcal{B}_{3}}f_{X}^{*}(x)dx\leq 1$, it follows that $S_{P_{1}}(i,j)/(n-2)$ converges in probability to a positive constant greater than $c^{\prime}>0$. Therefore, $S_{P_{1}}^{*}(i,j)>(n-2)c^{\prime}$ with probability going to $1$ as $n\to\infty$. As a result, $Q_{1}>N^{-1}n^{2}(n-2)c^{\prime}p_{1}p_{2}p_{3}p_{4}$ with probability going to $1$ as $n\to\infty$. Therefore,

[TABLE]

Since $N<n^{2}$, there exist a constant $\lambda>0$ such that \lim_{n\to\infty}\mbox{pr}\big{(}Q-\lambda n>0\big{)}=1. Under the null hypothesis, for a large enough sample size $n$, $S_{P}(i,j)$ follows the $\chi^{2}$ distribution with one degree of freedom. Therefore, the null expectation of $Q$, which is an average of $N$ different $S_{P}$’s, is approximately 1 and the null variance is bounded above by 2. Consequently, $\lim_{n\to\infty}(\lambda n-q_{1-\alpha})>0$ and \lim_{n\to\infty}\mbox{pr}\big{(}Q-q_{1-\alpha}>0\big{)}=1. Finally, for simplicity of presentation, we required that $\mbox{pr}(C>x_{0}|Y=k)>0$ for all $k=1,\ldots,K$. However this is only required for the two different groups, $g$ and $m$. The proof for the discrete failure time variable $X$ is done in much the same way.

Description of the tests included in the simulation study

Let

[TABLE]

where $\overline{N}_{j}(t)$ is the number of failures before or at time $t$ in group $j$, $\overline{Y}_{j}(t)$ is the number at risk at time $t$ in group $j$, $j=1,2$, and $\widehat{S}$ is the Kaplan-Meier estimator based on the pooled data. Also let

[TABLE]

where $\Delta\overline{N}_{j}(t)=\overline{N}_{j}(t)-\overline{N}_{j}(t-)$, $j=1,2$, $(\rho_{1},\gamma_{1})=(0,0)$ (the logrank test statistic), $(\rho_{2},\gamma_{2})=(1,0)$, $(\rho_{3},\gamma_{3})=(0,1)$, and $(\rho_{4},\gamma_{4})=(1,1)$. Then, four standardized statistics are defined by

[TABLE]

The test statistic of Lee (2007) is defined by

[TABLE]

and the MaxCombo test statistic is

[TABLE]

The $p$values can be easily calculated based on the asymptotic multivariate normal distribution of

[TABLE]

under the null hypothesis, i.e. with a mean zero and a covariance matrix that can be consistently estimated by $\widehat{\sigma}_{lm}$. The R package mvtnorm was used.

The Pepe-Fleming weighted KM test (Pepe and Fleming, 1989) uses the following weight function

[TABLE]

where $\widehat{G}_{j}$ is the KM estimator of the time to censoring in group $j$, $j=1,2$. Details od the variance estimator can be found in Pepe and Fleming (1989). Peto-Peto weighted KM test (Peto and Peto, 1972) uses a weight function that is very close to the pooled KM estimator.