Consensus measure of rankings

TL;DR

This paper proposes a graph-based method to measure consensus among different rankings, applicable to various ranking types, and demonstrates its use in evaluating search engine result quality.

Contribution

It introduces a novel graph representation approach for consensus measurement of rankings, accommodating full, partial, and tied rankings, with algorithms for different consensus aspects.

Findings

Effective in evaluating search engine top-$k$ rankings

Applicable to full, partial, and tied rankings

Provides multiple algorithms for different consensus measures

Abstract

A ranking is an ordered sequence of items, in which an item with higher ranking score is more preferred than the items with lower ranking scores. In many information systems, rankings are widely used to represent the preferences over a set of items or candidates. The consensus measure of rankings is the problem of how to evaluate the degree to which the rankings agree. The consensus measure can be used to evaluate rankings in many information systems, as quite often there is not ground truth available for evaluation. This paper introduces a novel approach for consensus measure of rankings by using graph representation, in which the vertices or nodes are the items and the edges are the relationship of items in the rankings. Such representation leads to various algorithms for consensus measure in terms of different aspects of rankings, including the number of common patterns, the number of common patterns with fixed length and the length of the longest common patterns. The proposed measure can be adopted for various types of rankings, such as full rankings, partial rankings and rankings with ties. This paper demonstrates how the proposed approaches can be used to evaluate the quality of rank aggregation and the quality of top-$k$ rankings from Google and Bing search engines.

Tables

Table 1. TABLE I: A summary of the popular indices for various types of rankings with comparison to our approach of κ p ​ ( ℛ ) subscript 𝜅 𝑝 ℛ \kappa_{p}(\mathcal{R}) and κ ​ ( ℛ ) 𝜅 ℛ \kappa(\mathcal{R}) .

	Full	Partial	Weighted	Ties
$\tau$	x
$\rho$	x
${\tau }_{ap}$	x		x
${\tau }_w$	x		x
${\rho }_w$	x		x
${\rho }_k$		x
RBO		x	x
${\tau }_t$	x			x
${\kappa }_p\left(ℛ\right)$	x	x	x	x
$\kappa \left(ℛ\right)$	x	x	x	x

Table 2. TABLE II: Rankings of clustering algorithms with respect to the validation measures.

Validation	${ℛ}_c$ – The set of rankings for clustering algorithms
measures	1	2	3	4	5	6	7	8	9	10
APN	SM	FN	ST	KM	PM	HR	AG	CL	DI	MO
AD	SM	FN	KM	PM	CL	ST	DI	HR	AG	MO
ADM	FN	SM	ST	KM	CL	PM	DI	HR	AG	MO
FOM	SM	CL	KM	PM	FN	ST	DI	HR	AG	MO
Connectivity	HR	AG	DI	KM	MO	SM	FN	CL	PM	ST
Dunn	HR	AG	KM	PM	DI	SM	CL	MO	FN	ST
Silhouette	HR	AG	KM	SM	CL	PM	ST	DI	FN	MO

Table 3. TABLE III: Rank aggregation of rankings (in Table II ) using GA and CE algorithms.

Algorithm	Aggregation results
GA	SM	HR	KM	FN	AG	PM	CL	DI	ST	MO
CE	KM	SM	PM	FN	HR	AG	CL	DI	ST	MO

Table 4. TABLE IV: The values of κ ​ ( ℛ ) 𝜅 ℛ \kappa(\mathcal{R}) for the rankings of clustering algorithms when γ 𝛾 \gamma and λ 𝜆 \lambda vary from 1 to 0.45.

CE		$\lambda$
$\kappa \left({ℛ}_{CE}\right)$		1	0.95	0.9	0.85	0.8	0.75	0.7	0.65	0.6	0.55	0.5	0.45
$\gamma$	1	19	17.589	16.376	15.336	14.448	13.692	13.05	12.508	12.05	11.666	11.344	11.076
	0.95	17.945	16.534	15.321	14.282	13.394	12.637	11.996	11.453	10.996	10.611	10.29	10.021
	0.9	16.964	15.553	14.34	13.301	12.413	11.657	11.015	10.472	10.015	9.631	9.309	9.04
	0.85	16.055	14.644	13.431	12.391	11.503	10.747	10.105	9.563	9.105	8.721	8.399	8.13
	0.8	15.213	13.803	12.589	11.55	10.662	9.906	9.264	8.721	8.264	7.88	7.558	7.289
	0.75	14.438	13.027	11.814	10.775	9.886	9.13	8.489	7.946	7.489	7.104	6.782	6.514
	0.7	13.726	12.315	11.102	10.063	9.174	8.418	7.777	7.234	6.777	6.392	6.07	5.802
	0.65	13.075	11.664	10.451	9.411	8.523	7.767	7.125	6.583	6.125	5.741	5.419	5.151
	0.6	12.482	11.071	9.858	8.818	7.93	7.174	6.532	5.99	5.532	5.148	4.826	4.557
	0.55	11.944	10.533	9.32	8.281	7.392	6.636	5.995	5.452	4.995	4.61	4.288	4.02
	0.5	11.46	10.049	8.836	7.796	6.908	6.152	5.51	4.967	4.51	4.126	3.804	3.535
	0.45	11.026	9.615	8.402	7.362	6.474	5.718	5.076	4.533	4.076	3.692	3.37	3.101
GA		$\lambda$
$\kappa \left({ℛ}_{GA}\right)$		1	0.95	0.9	0.85	0.8	0.75	0.7	0.65	0.6	0.55	0.5	0.45
$\gamma$	1	19	17.534	16.28	15.211	14.303	13.536	12.889	12.346	11.892	11.514	11.201	10.942
	0.95	17.966	16.5	15.246	14.177	13.27	12.502	11.855	11.312	10.859	10.481	10.167	9.908
	0.9	17.007	15.541	14.287	13.218	12.31	11.543	10.896	10.353	9.899	9.521	9.208	8.949
	0.85	16.119	14.653	13.399	12.33	11.422	10.655	10.008	9.465	9.011	8.634	8.32	8.061
	0.8	15.3	13.834	12.58	11.511	10.603	9.835	9.189	8.646	8.192	7.814	7.501	7.242
	0.75	14.546	13.08	11.826	10.757	9.85	9.082	8.435	7.892	7.439	7.061	6.747	6.488
	0.7	13.856	12.39	11.136	10.066	9.159	8.391	7.744	7.202	6.748	6.37	6.057	5.797
	0.65	13.225	11.759	10.505	9.436	8.528	7.76	7.114	6.571	6.117	5.739	5.426	5.167
	0.6	12.651	11.186	9.931	8.862	7.955	7.187	6.54	5.997	5.544	5.166	4.852	4.593
	0.55	12.132	10.666	9.412	8.343	7.435	6.668	6.021	5.478	5.025	4.647	4.333	4.074
	0.5	11.664	10.199	8.944	7.875	6.968	6.2	5.553	5.011	4.557	4.179	3.865	3.606
	0.45	11.245	9.779	8.525	7.456	6.549	5.781	5.134	4.591	4.138	3.76	3.446	3.187

Table 5. TABLE V: Rankings of top 25 links returned from Google and Bing with the given key words ( B ​ F 𝐵 𝐹 BF for “bond films”, B ​ M 𝐵 𝑀 BM for “bond movies”, 0 ​ M 0 𝑀 0M for “007 movies”, 0 ​ F 0 𝐹 0F for “007 films”, J ​ F 𝐽 𝐹 JF for “james bond films”, and J ​ M 𝐽 𝑀 JM for “james bond movies”). 𝔾 𝔾 \mathbb{G} and 𝔹 𝔹 \mathbb{B} stand for Google and Bing respectively. The numbers are the ids for the links returned from the search engines, and the mapping of the ids to the links can be find on our github repository.

$𝔾$	Top 25 links from Google
$BF$	0,68,9,59,11,5,3,69,79,70,21,4,36,32,76,40,60,51,80,81,42,29,82,83,73
$BM$	5,0,9,11,59,76,3,36,21,79,90,70,4,60,93,35,50,40,42,92,86,87,73,98,94
$0M$	0,9,11,5,3,59,70,84,85,21,76,4,32,42,51,62,12,80,67,60,55,86,87,73,29
$0F$	0,9,3,11,5,2,59,4,32,21,60,35,42,61,62,12,51,17,63,64,55,65,66,40,67
$JF$	5,0,68,9,3,11,59,69,70,71,4,21,60,32,36,61,65,72,73,58,74,75,76,77,78
$JM$	0,9,59,5,11,3,88,60,89,69,90,70,32,91,75,92,93,94,61,40,86,87,95,96,97
$𝔹$	Top 25 links from Bing
$BF$	0,9,1,11,36,8,4,2,22,5,37,15,38,6,28,34,29,19,39,40,35,24,41,32,42
$BM$	0,1,9,2,11,5,50,8,4,56,57,22,3,40,51,7,41,15,19,37,36,55,42,58,38
$0M$	0,7,9,2,10,8,4,6,5,1,11,14,3,40,13,43,19,44,45,46,47,48,49,17,28
$0F$	0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24
$JF$	9,1,25,4,7,2,0,26,5,11,15,8,27,10,28,29,30,31,6,19,32,22,33,34,35
$JM$	0,9,3,1,2,11,50,4,8,7,6,38,40,25,10,51,15,19,52,36,53,14,54,55,28

Table 6. TABLE VI: The values of κ ​ ( ℛ ) 𝜅 ℛ \kappa(\mathcal{R}) for the search rankings from Google and Bing when γ 𝛾 \gamma and λ 𝜆 \lambda vary from 1 to 0.45.

Google		$\lambda$
$\kappa \left(ℛ\right)$		1	0.95	0.9	0.85	0.8	0.75	0.7	0.65	0.6	0.55	0.5	0.45
$\gamma$	1	33	24.502	19.136	15.698	13.438	11.896	10.798	9.982	9.351	8.85	8.446	8.117
	0.95	32.475	23.977	18.611	15.173	12.913	11.371	10.274	9.457	8.826	8.325	7.921	7.592
	0.9	31.982	23.483	18.118	14.68	12.42	10.878	9.78	8.964	8.333	7.832	7.428	7.099
	0.85	31.518	23.02	17.654	14.216	11.956	10.414	9.316	8.5	7.869	7.368	6.964	6.635
	0.8	31.081	22.583	17.217	13.779	11.519	9.977	8.879	8.063	7.432	6.931	6.527	6.198
	0.75	30.669	22.17	16.805	13.367	11.107	9.565	8.467	7.65	7.02	6.519	6.114	5.785
	0.7	30.279	21.781	16.415	12.977	10.717	9.175	8.078	7.261	6.63	6.129	5.725	5.396
	0.65	29.91	21.412	16.046	12.608	10.348	8.806	7.709	6.892	6.261	5.76	5.356	5.027
	0.6	29.56	21.061	15.696	12.258	9.998	8.456	7.358	6.542	5.911	5.41	5.005	4.676
	0.55	29.226	20.728	15.362	11.925	9.664	8.123	7.025	6.208	5.577	5.076	4.672	4.343
	0.5	28.908	20.41	15.044	11.607	9.346	7.805	6.707	5.89	5.259	4.758	4.354	4.025
	0.45	28.604	20.105	14.74	11.302	9.042	7.5	6.402	5.586	4.955	4.454	4.049	3.721
Bing		$\lambda$
$\kappa \left(ℛ\right)$		1	0.95	0.9	0.85	0.8	0.75	0.7	0.65	0.6	0.55	0.5	0.45
$\gamma$	1	23	16.392	12.835	10.887	9.788	9.143	8.745	8.49	8.32	8.207	8.13	8.079
	0.95	22.147	15.539	11.982	10.034	8.935	8.29	7.892	7.637	7.467	7.354	7.277	7.226
	0.9	21.355	14.747	11.189	9.242	8.143	7.497	7.1	6.844	6.675	6.561	6.484	6.433
	0.85	20.62	14.012	10.455	8.507	7.409	6.763	6.365	6.11	5.941	5.827	5.75	5.699
	0.8	19.942	13.334	9.777	7.829	6.731	6.085	5.687	5.432	5.263	5.149	5.072	5.021
	0.75	19.318	12.71	9.152	7.205	6.106	5.46	5.063	4.807	4.638	4.524	4.448	4.396
	0.7	18.745	12.137	8.58	6.632	5.534	4.888	4.49	4.235	4.066	3.952	3.875	3.824
	0.65	18.222	11.614	8.057	6.109	5.01	4.365	3.967	3.712	3.542	3.429	3.352	3.301
	0.6	17.746	11.138	7.581	5.633	4.535	3.889	3.491	3.236	3.067	2.953	2.876	2.825
	0.55	17.316	10.708	7.15	5.203	4.104	3.458	3.061	2.805	2.636	2.522	2.446	2.394
	0.5	16.928	10.321	6.763	4.816	3.717	3.071	2.673	2.418	2.249	2.135	2.058	2.007
	0.45	16.582	9.974	6.417	4.469	3.371	2.725	2.327	2.072	1.903	1.789	1.712	1.661

Equations

$$(x_i,x_{k_1}),(x_{k_1},x_{k_2}),\dots ,(x_{k_p},x_j)$$

$$\kappa (\mathcal{R})=|\mathcal{S}(\mathcal{R})|.$$

$${\kappa }_p(\mathcal{R})=|{\mathcal{S}}_p(\mathcal{R})|$$

$$\kappa ({\mathcal{R}}_y)\le \kappa ({\mathcal{R}}_x)$$

$$\varphi (\mathbf{r})=\left(f_{\mathbf{r}}(y_1),\dots ,f_{\mathbf{r}}(y_m)\right),$$

$$f_{\mathbf{r}}(y_k)=\{\begin{array}{ll}1& y_k⊑\mathbf{r}\\ 0& y_k/⊑\mathbf{r}\end{array}$$

$$\kappa (\mathcal{R})=⟨\varphi ({\mathbf{r}}_1),\dots ,\varphi ({\mathbf{r}}_n)⟩=\sum _{k=1}^m\prod _{\mathbf{r}\in \mathcal{R}}{f}_{\mathbf{r}}(y_k)$$

$$\tau ({\mathbf{r}}_i,{\mathbf{r}}_j)=\frac{|{\mathcal{S}}_2({\mathbf{r}}_i,{\mathbf{r}}_j)|-|{\mathcal{S}}_2(\overleftarrow{{\mathbf{r}}_i},{\mathbf{r}}_j)|}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}$$

$$d_{\tau }({\mathbf{r}}_i,{\mathbf{r}}_j)=\frac{|{\mathcal{S}}_2(\overleftarrow{{\mathbf{r}}_i},{\mathbf{r}}_j)|}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}$$

$$\rho ({\mathbf{r}}_i,{\mathbf{r}}_j)=1-\frac{6{\sum }_{\sigma \in \Sigma }{\left({\eta }_i(\sigma )-{\eta }_j(\sigma )\right)}^2}{n(n^2-1)}$$

$$d_{\rho }({\mathbf{r}}_i,{\mathbf{r}}_j)=\frac{{\sum }_{\sigma \in \Sigma }|{\eta }_i(\sigma )-{\eta }_j(\sigma )|}{n(n^2-1)}$$

$$\mathbf{A}_{6\times 6}=\quad\bordermatrix{~{}&a&b&c&d&e&f\cr a&0&0&0&0&0&0\cr b&0&1&0&0&0&0\cr c&0&1&1&0&0&0\cr d&0&1&0&1&0&0\cr e&0&1&0&1&1&0\cr f&0&1&0&1&1&1\cr}$$

$$E_e$$

$$\displaystyle\qquad(A_{44},A_{55}),(A_{44},A_{66}),(A_{55},A_{66})\big{\}}$$

$$\kappa (\mathcal{R})$$

$$H(x)=\{\begin{array}{ll}1& x>0;\\ 0& \text{otherwise}.\end{array}$$

$${\eta }_k(\sigma )=\{\begin{array}{ll}0,& \sigma /⊑{\mathbf{r}}_k;\\ j,& \sigma =r_{k_j}.\end{array}$$

$$\{\begin{array}{ll}0,& i<j;\\ \theta (r_{x_i}){\prod }_{k=1}^{N}H\left({\eta }_k(r_{x_i})\right),& i=j;\\ \psi (r_{x_i},r_{x_j}){\prod }_{k=1}^{N}H\left({\eta }_k(r_{x_i})-{\eta }_k(r_{x_j})\right)H(A_{ii})H(A_{jj}),& i>j.\end{array}$$

$${\kappa }_p(\mathcal{R})=\{\begin{array}{ll}\mathrm{tr}(\mathbf{A}),& p=1;\\ {\mathbf{z}}^T{\mathbf{L}}^{p-1}\mathbf{z},& p>1.\end{array}$$

$$\mu (\sigma )=\{\begin{array}{ll}-\infty & \sigma /⊑\mathcal{R}\\ \frac{1}{N}{\sum }_{k=1}^N{\eta }_k(\sigma )& \sigma ⊑\mathcal{R}\end{array}$$

$$\theta (\sigma )={\gamma }^d,$$

$$\theta (\sigma )=\prod _{\mathbf{r}\in \mathcal{R}}{f}_{\mathbf{r}}(\sigma ),$$

$$\psi ({\sigma }_i,{\sigma }_j)={\lambda }^g,$$

$$f_{\mathbf{r}}(y_k)=\prod _{i=1}^{p-1}\psi ({\sigma }_{k_i},{\sigma }_{k_{i+1}})$$

$$\mathbf{L}=\left(\begin{array}{cccccc}0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 1& 0& 1& 0& 0\\ 0& 1& 0& 1& 1& 0\end{array}\right),{\mathbf{L}}^2=\left(\begin{array}{cccccc}0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 2& 0& 1& 0& 0\end{array}\right),$$

$${\mathbf{L}}^3=\left(\begin{array}{cccccc}0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\end{array}\right),{\mathbf{L}}^4=\left(\begin{array}{cccccc}0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\end{array}\right).$$

$$\begin{array}[]{ll}\kappa_{1}(\mathcal{R})=\operatorname{tr}(\mathbf{A})=5,&\kappa_{2}(\mathcal{R})=\mathbf{z}^{T}\mathbf{L}\mathbf{z}=7,\\ \kappa_{3}(\mathcal{R})=\mathbf{z}^{T}\mathbf{L}^{2}\mathbf{z}=4,&\kappa_{4}(\mathcal{R})=\mathbf{z}^{T}\mathbf{L}^{3}\mathbf{z}=1.\end{array}$$

$$\max \left\{p:{\kappa }_p(\mathcal{R})>0\right\}.$$

$$\kappa (\mathcal{R})={\kappa }_1(\mathcal{R})+{\mathbf{z}}^T(\mathbf{I}-\mathbf{L}{)}^{-1}\mathbf{z}-n$$

$$\kappa (\mathcal{R})={\kappa }_1(\mathcal{R})+\sum _{p=2}^n{\kappa }_p(\mathcal{R})=\mathrm{tr}(\mathbf{A})+\sum _{i=1}^{n-1}{\mathbf{z}}^T{\mathbf{L}}^i\mathbf{z}.$$

$$\kappa (\mathcal{R})$$

Full text

This paper introduces a novel approach for consensus measure of rankings by using graph representation, in which the vertices or nodes are the items and the edges are the relationship of items in the rankings. Such representation leads to various algorithms for consensus measure in terms of different aspects of rankings, including the number of common patterns, the number of common patterns with fixed length and the length of the longest common patterns. The proposed measure can be adopted for various types of rankings, such as full rankings, partial rankings and rankings with ties. This paper demonstrates how the proposed approaches can be used to evaluate the quality of rank aggregation and the quality of top-$k$ rankings from Google and Bing search engines.

I Introduction

In many information systems, rankings are widely used to represent the preferences over a set of items or candidates, ranging from information retrieval, recommender to decision making systems [1, 2, 3, 4, 5, 6], in order to improve quality of the services provided by the systems. For example, in search engine, the list of the terms suggested by a search engine after a user’s few keystrokes is a typical ranking and such ranking service, widely adopted nowadays, has great impact on user’s search experience; it is also recognized that the list of search results is a ranking after a query is issued.

A ranking is an ordered sequence of items, in which an item with higher ranking score is more preferred than the items with lower ranking scores. The consensus of rankings is the degree to which the rankings agree according to certain common patterns. The consensus measure, can be used in many information systems, in order to uncover how close or related the rankings are. For example, in the group decision making, a group of experts express their preferences over a set of candidates by using rankings and the measure of the degree of consensus is very useful for reaching consensus [2].

In many information system with large volume of items, such as search engines, it is hard to clearly define what ground truth is, which make it more difficult to evaluate and compare the rankings returned from the systems. The consensus measure of rankings, as a tool for understanding how related or close the rankings are, will help engineers and researchers to discern what aspects of a ranking system need to be improved and to detect outliers [7, 8]. For a set of rankings $\mathcal{R}=\{\mathbf{r}_{1},\dots,\mathbf{r}_{n}\}$, one approach to understanding the degree to which rankings agree is to use rank correlation or similarity function by pairwise comparison [3, 9, 10, 11, 12, 13, 14, 15, 16]. The notable functions include the Kendall index $\tau(\mathbf{r}_{i},\mathbf{r}_{j})$ and the Spearman index $\rho(\mathbf{r}_{i},\mathbf{r}_{j})$ [12, 14], which however do not have a weighting scheme so that less important items can be penalized. It is common that in information retrieval, the documents (items) at the top of a ranking list are more important than those at the bottom [17]. As such, it makes sense to reduce the impact from the bottom items with a weighting scheme. For example, the variation of $\tau$ index, denoted by $\tau_{ap}$, with average precision, is able to give greater weight to the top items of the ranking lists [9]. These methods assume rankings are conjoint, meaning that items in the rankings are completely overlapped. Undoubtedly, they cannot be used for partial rankings, in which items may not be mutually overlapped. As a similarity function for two partial rankings, the RBO (rank-biased overlap) proposes to weight the number of common items according to the depth of rankings [15], and it doesn’t take into account the order of items in the rankings.

When one of the correlation or similarity functions is used for consensus measure for a set of $n$ rankings in $\mathcal{R}$, we can aggregate the pairwise comparison values across all rankings for $\binom{n}{2}$ times. Since the pairwise comparison is based on the degree of commonality in two rankings, with respect to features or patterns (e.g, the common items, or the concordant pairs against the disconcordant pairs), the aggregated result is not informative enough to tell the extend to which the rankings agree in $\mathcal{R}$, according to the study by Elzinga et al. [18]. Also, the type of rankings can be full or partial, specially the top-$k$ [17], the existing measures fail to meet the requirements for handling different types of rankings.

In order to effectively evaluate and compare rankings, which could be full or partial and especially in which some items need to be weighted, this paper propose a new approach based on graph representation. The novelty of this paper lies in that fact the new proposed consensus measure of rankings does not need pairwise comparison, which is significantly different from the pairwise approaches using similarity or correlation functions. The contribution of the paper includes:

• •

we introduce a directed acyclic graph (DAG) to represent the relationship between items in the rankings so that such representation can be used to induce efficient algorithms for consensus measure of rankings;

• •

the proposed representation of DAG enables us to approach consensus measure of rankings in terms of different aspects of the common features or patterns hidden in the rankings, including $\kappa(\mathcal{R})$ – the number of common patterns, $\kappa_{p}(\mathcal{R})$ – the number of common patterns with a fixed length $p$, and $\ell(\mathcal{R})$ – the length of the longest or largest common patterns.

• •

the proposed representation of DAG is extended to allow the edges in the graph to have weights so that more “important” features or patterns are assigned with higher values and the features or patterns with less “importance” are penalized.

• •

we also demonstrate that the consensus measure of rankings with graph representation can be extended to calculate consensus measure for duplicate rankings, for rankings with ties and for rankings whose top items need to be weighted.

• •

we show that our approach can be used for different types of rankings, including the full rankings and top-$k$ rankings.

The rest of paper is organized as follows. Section II introduces the important notations and concepts used in the paper, followed by a review of related work in Section III. Section IV presents a directed graph representation approach for consensus measure. Section V shows how the proposed approaches can be used to evaluate rank aggregation and to compare top-$k$ rankings. The paper is concluded in Section VI.

II Preliminaries

This section introduces notations and concepts of graph, ranking sequence, and consensus measure that will be used in the rest of the paper.

II-A Directed graph

A directed graph is a pair $G=(V,E)$, where $V$ is the set of nodes (or vertices) and $E$ is the set of directed edges. A directed edge $(x,y)$ means that the edge leaves node $x\in V$ and enters node $y\in V$. An edge $(x,x)$ is called a loop, which leaves node $x$ and returns to itself. Given a graph $G=(V,E)$ with $n=|V|$ nodes, matrix $\mathbf{A}=\left(A_{ij}\right)_{n\times n}$ is used to denote the adjacency matrix of graph $G=(V,E)$, where $A_{ij}=1$ if there exists edge $(x_{i},x_{j})\in E$; and $A_{ij}=0$, otherwise.

The adjacency matrix $\mathbf{A}$ assumes that all the edges have identical weights of 1, and this can be relaxed in the weighted directed graph. A weighted directed graph $G=(V,E,W)$ is a directed graph, in which $W$ is a set of weights on the edges and each edge $(x_{i},x_{j})\in E$ is assigned a non-zero weight $w(i,j)\in W$. Then, the adjacency matrix $\mathbf{A}$ for $G=(V,E,W)$ is defined as $A_{ij}=w(i,j)$ if $(x_{i},x_{j})\in E$ and $A_{ij}=0$, otherwise.

A path from node $x_{i}$ to $x_{j}$ is a sequence of distinct non-loop edges

[TABLE]

connecting node $x_{i}$ and $x_{j}$.

II-B Ranking sequences

A ranking $\mathbf{r}$ is an ordered sequence $\mathbf{r}=(\sigma_{i_{1}},\sigma_{i_{2}},\ldots,\sigma_{i_{m}})$ of $m$ distinct items drawn from a universe $\Sigma=\{\sigma_{1},\cdots,\sigma_{n}\}$, where $m\leq n$ and $\sigma_{i_{j}}$ is more preferred than $\sigma_{i_{k}}$ if $i_{j}<i_{k}$. The length of $\mathbf{r}$ is denoted by $|\mathbf{r}|$. For notational simplicity, we shall simply write a ranking as a sequence of $\mathbf{r}=\sigma_{i_{1}}\sigma_{i_{2}}\cdots\sigma_{i_{m}}$ in the rest of the paper.

For a ranking $\mathbf{r}=r_{1}\cdots r_{k}$, where $r_{j}\in\Sigma$ for $1\leq j\leq k$, we can define the embedded patterns with respect to subsequences. A sequence $\mathbf{r}^{\prime}=r^{\prime}_{1}\cdots r^{\prime}_{m}$ is called a subsequence of $\mathbf{r}$, denoted by $\mathbf{r}^{\prime}\sqsubseteq\mathbf{r}$, if $\mathbf{r}^{\prime}$ can be obtained by deleting $k-m$ items from $\mathbf{r}$. We denote by $\mathbf{r}^{\prime}\not\sqsubseteq\mathbf{r}$ that $\mathbf{r}^{\prime}$ is not a subsequence of $\mathbf{r}$. For example, $bde\sqsubseteq abcde$, and $bac\not\sqsubseteq abcde$.

A ranking sequence with no items is an empty sequence. We use $\mathcal{S}(\mathbf{r})$ to denote the set of all possible non-empty subsequences of $\mathbf{r}$. $\mathcal{S}(\mathbf{r})$ can be partitioned into subsets $\mathcal{S}_{p}(\mathbf{r})$, where $\mathcal{S}_{p}(\mathbf{r})$ consists of all subsequences of length $p$. For example, if $\mathbf{r}=abcde$, then $\mathcal{S}_{3}(\mathbf{r})=\{abc,abd,abe,acd,ace,ade,bcd,bce,bde,cde\}$, in which each subsequence has length $3$.

The degree to which rankings agree lies in the common patterns or features which are embedded in the rankings. For ranking sequences, the subsequences are the patterns or features. Given a set of $N$ rankings $\mathcal{R}=\{\mathbf{r}_{1},\dots,\mathbf{r}_{N}\}$, consider $\mathcal{S}(\mathcal{R})=\mathcal{S}(\mathbf{r}_{1})\cap\cdots\cap\mathcal{S}(\mathbf{r}_{N})$, each element $x\in\mathcal{S}(\mathcal{R})$ is a common subsequence of $\mathbf{r}_{1},\dots,\mathbf{r}_{m}$, for which we also use the notation $x\sqsubseteq\mathcal{R}$. Similar to $\mathcal{S}_{p}(\mathbf{r})$, we also define $\mathcal{S}_{p}(\mathcal{R})$ to denote the subsets of all common subsequences of length $p$. Therefore, it holds that $\mathcal{S}(\mathcal{R})=\bigcup_{1\leq p\leq l}\mathcal{S}_{p}(\mathcal{R})$, where $l=\min\{|\mathbf{r}|:\mathbf{r}\in\mathcal{R}\}$. In a special case, for two rankings $\mathbf{r}_{i}$ and $\mathbf{r}_{j}$, we will write $\mathcal{S}_{p}(\mathbf{r}_{i},\mathbf{r}_{j})$ to denote the set of $p$-long common subsequences between $\mathbf{r}_{i}$ and $\mathbf{r}_{j}$.

It is clear that $\mathcal{S}(\mathcal{R})$ accommodates all common features (subsequences), which are subsumed by each ranking $\mathbf{r}\in\mathcal{R}$. Let $\kappa(\mathcal{R})$ denote the number of all common subsequences of $\mathcal{R}$, i.e,

[TABLE]

The more common features $\mathcal{S}(\mathcal{R})$ has or the bigger $\kappa(\mathcal{R})$ is, the higher degree of consensus $\mathcal{R}$ has. We also define

[TABLE]

in order to measure the consensus in $\mathcal{R}$ with respect to the number of subsequences of a given length $p$. The length of the longest common subsequences of rankings in $\mathcal{R}$ is denoted by $\ell(\mathcal{R})$ or simply $\ell$. Then, $\ell=\max\{|z|:z\in\mathcal{S}(\mathcal{R})\}$.

Therefore, we have the following properties:

• •

For a set with only one ranking $\mathcal{R}=\{\mathbf{r}\}$, where $n=|\mathbf{r}|$, $\kappa(\mathcal{R})=2^{n}-1$;

• •

For a set of two rankings $\mathcal{R}=\{\mathbf{r}_{x},\mathbf{r}_{y}\}$, where $m=|\mathbf{r}_{x}|$ and $n=|\mathbf{r}_{y}|$, we have $0\leq\kappa(\mathcal{R})\leq 2^{\min\{m,n\}}-1;$

• •

For two sets of rankings $\mathcal{R}_{x}$ and $\mathcal{R}_{y}$, if $\mathcal{R}_{x}\subseteq\mathcal{R}_{y}$, then

[TABLE]

II-C Consensus measure of rankings in feature spaces

For a set of $n$ rankings $\mathcal{R}$, we can form a set of features $\mathbb{F}=\bigcup_{\mathbf{r}\in\mathcal{R}}\mathcal{S}(\mathbf{r})$. Let $m=|\mathbb{F}|$ and $\mathbb{F}=\{y_{1},\dots,y_{m}\}$. Each ranking $\mathbf{r}$ can be represented by a feature vector with a mapping function $\phi:\mathcal{R}\rightarrow\{0,1\}^{m}$:

[TABLE]

where

[TABLE]

It is clear that $\kappa(\mathcal{R})$, defined in Equation (1) can be rewritten using the inner product on $n$-inner product spaces [19, 20] as

[TABLE]

With the generalized inner product, we find that the $\kappa(\mathcal{R})$ is a kernel function [21] when $|\mathcal{R}|=2$. The rewritten $\kappa(\mathcal{R})$ relies on the definition of $f_{\mathbf{r}}(y_{k})$ as defined in Equation (4), whose co-domain is $\{0,1\}$. It is computationally expensive to enumerate all the features and to form $\mathbb{F}$. In this section, we will transform relationship between items to a graph so that efficient algorithms can be found without enumerating features explicitly, which is similar to the kernel trick for kernel functions [21].

III Related work

A ranking can be full or partial ranking, depending on the number of items from $\Sigma$ being ranked. A ranking $\mathbf{r}$ is a full ranking if $|\Sigma|=|\mathbf{r}|$. A ranking $\mathbf{r}$ is called *partial * ranking if the items in $\mathbf{r}$ forms a subset of $\Sigma$. A top-$k$ ranking is a sub-ranking of full ranking but only with the top-$k$ items. Rankings with ties occur when some items share an identical ranking score, which happens very often in the decision making or voting process [16]. For example, in a ranking $\mathbf{r}=\{a\}\{bc\}\{d\}$, both items $b$ and $c$ are assigned with an identical ranking score.

Evaluation or comparison of rankings is an important tasks in many ranking related systems, including decision making, information retrieval, voting and recommender [1, 2]. One approach to evaluating rankings is to use rank correlation between two rankings. The widely used Kendall $\tau$ index [12, 13] is a measure of rank correlation between two rankings $\mathbf{r}_{i}$ and $\mathbf{r}_{j}$ over $n$ items by taking into account 2-long common subsequences between them, which can be formulated as

[TABLE]

where $\overleftarrow{\mathbf{r}_{i}}$ is a reverse ranking of ${\mathbf{r}_{i}}$. In rank aggregation, one could also use the Kendall distance $d_{\tau}(\mathbf{r}_{i},\mathbf{r}_{j})$ – a variation of the Kendall $\tau$:

[TABLE]

The Spearman $\rho$ index is another measure of rank correlation that does not utilize the 2-long common subsequences but instead takes into account of each item positions in $\mathbf{r}_{i}$ and $\mathbf{r}_{j}$ [14]. It is defined as follows

[TABLE]

where $n=|\Sigma|$. The Spearman footrule distance $d_{\rho}$ is an $L_{1}$ distance, which is a variation of the Spearman $\rho$:

[TABLE]

Compared with the Spearman $\rho$, the Kendall $\tau$ ignores the use of items positions, which are in many cases very important factors, e.g, for the top-$k$ rankings. Again, the Spearman $\rho$ can not be used for sensitivity detection and analysis, as studied in [13].

Both Kendall and Spearman can only be used for full rankings. They cannot be used for partial or top-$k$ rankings. Even in full rankings, both of them lack of weighting schemes and are not flexible enough for rankings whose items at the top are more important than the items at the bottom [17]. Therefore, it is necessary to reduce the impact from the bottom items with a down-weighting scheme for those bottom items. For example, the variation of $\tau$ index, denoted by $\tau_{ap}$, with average precision, is able to give greater weight to the top items of the ranking lists [9]. Shieh also developed a weighted metric $\tau_{w}$ based on the Kendall $\tau$ by adding weighting factors to the 2-long subsequences [22]. For full rankings with ties, $\tau_{t}$ was proposed based on the Kendall index [16]. One extension $\rho_{w}$ to the Spearman index by Iman et al. was to assign higher weights to the items at the top [23].

The above methods assume that rankings are full rankings, meaning that items in the rankings are completely overlapped. Therefore, they cannot be used for partial rankings. In information retrieval, it is more interesting to compare the the rankings based on their top-$k$ items. Fagin et al. proposed two measures $\tau_{k}$ and $\rho_{k}$ by adapting both Kendall $\tau$ and Spearman $\rho$ for top-$k$ rankings [17]. As a similarity function for two partial rankings, the RBO (rank-biased overlap) proposes to weight the number of common items according to the depth of rankings [15], but it does not take into account the order of items in the rankings.

These functions are pairwise comparison and they can be transferred into consensus measure for a set $\mathcal{R}$ of $n$ rankings by aggregating the pairwise distance values across all rankings. For example, one can use $\sum_{i=1}^{n}\sum_{j=1,i\neq j}^{n}\tau(\mathbf{r}_{i},\mathbf{r}_{j})$ if the Kendall index is preferred. However, this aggregated result is not informative enough to tell the extend to which the rankings agree in $\mathcal{R}$, according to the study by Elzinga et al. [18].

We summarize the popularly used indices in Table I and we show that our approach of $\kappa_{p}(\mathcal{R})$ and $\kappa(\mathcal{R})$ is more flexible for various types of rankings, which will be demonstrated in the next section. Also, those existing indices shown in Table I cannot be used for sensitivity detection in the consensus measure, while our approach has the ability to discern how the rankings come to agree by varying the parameters to the gaps and positions of items, as pointed out in Section IV-D1 and as verified in Section V.

IV Graph representation for consensus measure of rankings

This section will introduce a graph approach to consensus measure of rankings by calculating $\kappa(\mathcal{R})$ and $\kappa_{p}(\mathcal{R})$.

IV-A A motivating example

Consider a set of rankings $\mathcal{R}=\{\mathbf{r}_{1}=abcdef,\mathbf{r}_{2}=bdcefa,\mathbf{r}_{3}=bcdeghijkf,\mathbf{r}_{4}=badefc\}$. Without loss of generality, we randomly pick $\mathbf{r}_{1}\in\mathcal{R}$ (note that $|\mathbf{r}_{1}|=6$) and form a lower triangle matrix $\mathbf{A}=\left(A_{ij}\right)_{6\times 6}$ of size $6\times 6$, where for $i\geq j$, $A_{ij}=1$ if the $i^{\text{th}}$ item and the $j^{\text{th}}$ item of $\mathbf{r}_{1}$ both occur in the same order in all rankings in $\mathcal{R}$, and $A_{ij}=0$ otherwise. Then we obtain matrix $\mathbf{A}$:

[TABLE]

With matrix $\mathbf{A}$, we can induce a weighted directed graph $G=(V,E_{\ell}\cup E_{e})$ on the diagonal elements of $\mathbf{A}$, where $V=\{A_{11},\dots,A_{66}\}$ is the set of vertices, $E_{\ell}$ is the set of loops and $E_{e}$ is the set of non-loop edges. Later, we may use $V=\{a,b,c,d,e,f\}$ interchangeably without confusion as each $A_{ii}$ stands for an item. Hereinafter in this paper, we shall distinguish loops and non-loops edges, and abuse the notation and simply call the latter edges.

The edges are drawn according to the following: for $1\leq i,j\leq|\mathbf{r}|$, an edge from $A_{ii}$ to $A_{jj}$ is added if the following conditions all hold: (1) $i<j$; (2) $A_{ii}=A_{jj}=1$; (3) $A_{ji}\neq 0$. We also add dashed loops on diagonal elements of value $1$. Figure 1 shows the weighted directed graph for the matrix $\mathbf{A}$, in which there are seven directed (solid) edges, i.e,

[TABLE]

or simply E_{e}=\big{\{}(b,c),(b,d),(b,e),(b,f),(d,e),(d,f),(e,f)\big{\}}. Those edges are the 2-long common subsequences: $bc,bd,be,bf,de,df,ef$, and all of them occur in $\mathbf{r}_{1}$, $\mathbf{r}_{2}$, $\mathbf{r}_{3}$ and $\mathbf{r}_{4}$. As such, $\kappa_{2}(\mathcal{R})=|E_{e}|=7$. Similarly, paths111Recall that our definition of path excludes loop edges. of length $3$ corresponds to common subsequences of length $3$. We find that $\kappa_{3}(\mathcal{R})=4$ with common subsequences being $bde$, $bdf$, $bef$ and $def$. Next, $\kappa_{4}(\mathcal{R})=1$ with the common subsequence being $bdef$. There is no longer common subsequences since the length of the longest path in $G$ is $4$.

In Figure 1, the five dashed loops mean five singletons, i.e., $b$, $c$, $d$, $e$, $f$. As a result, $\kappa_{1}(\mathcal{R})=5$. Therefore, we obtain $\kappa(\mathcal{R})=\kappa_{1}(\mathcal{R})+\kappa_{2}(\mathcal{R})+\kappa_{3}(\mathcal{R})+\kappa_{4}(\mathcal{R})=5+7+4+1=17$.

This process of finding patterns of various lengths with graph representation not only allows us to calculate $\kappa_{p}(\mathcal{R})$, but also makes it easy for us to calculate the number of all common patterns $\kappa(\mathcal{R})$ and the length of the longest common subsequences $\ell(\mathcal{R})$.

IV-B *Consensus measure by graph representation *

The above example shown in Fig. 1 presents an approach with graph representation for consensus measure of rankings when $f_{\mathbf{r}}(y_{k})\in\{0,1\}$. This section will extends the graph representation to the consensus measure of rankings by calculating $\kappa(\mathcal{R})$ and $\kappa_{p}(\mathcal{R})$, when $f_{\mathbf{r}}(y_{k})\in[0,1]$.

In Equation (4), the definition of $f_{\mathbf{r}}(y_{k})\in\{0,1\}$ assumes that features in $\mathbb{F}$ are equally assigned with a weight of 1. However, this is not true in many cases when some features or items in $\mathbb{F}$ is more important than the others [9, 15]. The definition of $f_{\mathbf{r}}(y_{k})\in\{0,1\}$ is not flexible enough to differentiate the importance of the features. As such, we shall extend it to $f_{\mathbf{r}}(y_{k})\in[0,1]$ if $y_{k}\sqsubseteq\mathbf{r}$, and $f_{\mathbf{r}}(y_{k})=0$ if $y_{k}\not\sqsubseteq\mathbf{r}$ so that “important” features will receive higher values of $f_{\mathbf{r}}(y_{k})$ while features with less importance will be “penalized” with lower $f_{\mathbf{r}}(y_{k})$. Therefore, we rewrite Equation (5) as

[TABLE]

for $f_{\mathbf{r}}(y_{k})\in[0,1]$.

In the DAG shown in Figure 1, we assume that the weights on the edges equal to 1, which does not reflect the nature of how each subsequence is embedded in the original rankings. Consider the four rankings in $\mathcal{R}=\{\mathbf{r}_{1}=abcdef,\mathbf{r}_{2}=bdcefa,\mathbf{r}_{3}=bcdeghijkf,\mathbf{r}_{4}=badefc\}$, item $f$ occurs at different positions in the rankings, which is shown in the following table:

[TABLE]

The position for $f$ in $\mathbf{r}_{3}$ is 10, which deviates from the positions of $f$ in the other rankings substantially. In order to incorporate those factors which may affect the degree of consensus, we relax the assumption that the weights are identical to 1 and generalize the induced weighted DAG by introducing two functions $\theta(\sigma)$ and $\psi(\sigma_{i},\sigma_{j})$. Figure 2 shows the new DAG, where each edge is associated with weight $\psi(\sigma_{i},\sigma_{j})$ and each loop is assigned with $\theta(\sigma)$, where $\psi(\sigma_{i},\sigma_{j})\in(0,1]$ and $\theta(\sigma)\in(0,1]$. We will illustrate how the two functions reflect those factors in the following sections, and how they are related to $f_{\mathbf{r}}(y_{k})\in[0,1]$ for Equation (7).

For simpler presentation of our algorithm, we introduce the (left-continuous) Heaviside function

[TABLE]

Now we present the following theorem for measuring the consensus of rankings.

Theorem 1**.**

Given a set $\mathcal{R}=\{\mathbf{r}_{1},\cdots,\mathbf{r}_{N}\}$ of $N$ rankings over a universe $\Sigma$, where each ranking $\mathbf{r}_{k}=r_{k_{1}}\cdots r_{k_{m}}$ is naturally associated with a map $\eta_{k}:\Sigma\rightarrow\{0,1,\dots,|\Sigma|\}$ defined as

[TABLE]

Let $\mathbf{r}_{x}=r_{x_{1}}r_{x_{2}}\cdots r_{x_{n}}$ be an arbitrary ranking from $\mathcal{R}$, $n=|\mathbf{r}_{x}|$, and ${\mathbf{A}}=\left({A}_{ij}\right)_{n\times n}$ be an adjacency matrix of a graph, where ${A}_{ij}=$

[TABLE]

and let ${\mathbf{L}}=\left({L}_{ij}\right)_{n\times n}$ be strictly lower triangle of ${\mathbf{A}}$, and $\mathbf{z}=(1,\dots,1)^{T}$ be a vector of all ones. Then,

[TABLE]

Proof.

Note that $\mathbf{z}^{T}\mathbf{M}\mathbf{z}$ gives the sum of all entries in a matrix $\mathbf{M}$. By definition of $\mathbf{A}$ we know that $\mathbf{A}_{ii}>0$ if $r_{x_{i}}\in\mathcal{S}$ and $\mathbf{A}_{ii}=0$ otherwise. It follows that $\kappa_{1}(\mathbf{A})$ is the number of $1$s on the diagonal line, which equals to, noting that all other entries on the diagonal lines are [math]s, the sum of diagonal entries of $\mathbf{A}$, or $\operatorname{tr}(\mathbf{A})$.

For $p\geq 2$, it is a classical inductive argument that $(\mathbf{L}^{p-1})_{ij}=N_{p}(i,j)$ when $i>j$, where $N_{p}(i,j)$ is the number of common sequences of length $p$ which begin with $r_{x_{i}}$ and end with $r_{x_{j}}$. The advertised result then follows from the fact that all rankings have distinct items and thus the common subsequences also have distinct items. ∎

Theorem 1 shows that the individual items are weighted by $\theta(\sigma)$ and the edges between any two items by $\psi(\sigma_{i},\sigma_{j})$, reflecting the strength of the relationship between two items $\sigma_{i}$ and $\sigma_{j}$.

IV-B1 $\theta(\sigma)$** – weighted by standard deviation of item’s positions**

The position of item $\sigma$ in a ranking is an indication of the strength of being preferred. To show the importance of the position of $\sigma$, we define $\mu(\sigma)$, the average of the positions of $\sigma$ in $\mathbf{r}_{k}$, as follows.

[TABLE]

If an item is placed in a small range of positions throughout all rankings, it is assumed that this item is preferred consistently at the same level by all rankings. On the other hand, if an item has a low position $\eta_{i}(\sigma)$ in one ranking $\mathbf{r}_{i}$ while has a high position $\eta_{j}(\sigma)$ in another ranking $\mathbf{r}_{j}$, the big difference between the positions $|\eta_{j}(\sigma)-\eta_{i}(\sigma)|$ indicates the inconsistency of the preferences over this item. To take into account the differences of item’s positions in consensus measure, we define

[TABLE]

where $d=\frac{1}{N}\sum_{k=1}^{N}|\eta_{k}(\sigma)-\mu(\sigma)|$, in order to weigh the item using the positions $\eta_{k}(\sigma)$. In fact, when feature $y_{k}$ is a singleton (i.e., $y_{k}=\sigma$), it is clear that

[TABLE]

where $f_{\mathbf{r}}(\sigma)=\frac{1}{N}|\eta_{k}(\sigma)-\mu(\sigma)|$ for Equation (7).

IV-B2 $\psi(\sigma_{i},\sigma_{j})$** – weighted by gaps**

The gap between items has been used for pairwise kernel functions or sensitivity detection [24, 21]. Now we extend this to the set-wise consensus measure of $\kappa_{p}(\mathcal{R})$.

The weighted DAG in Figure 2 shows that the edges $(b,f)$ and $(b,c)$, i.e., subsequences $bf$ and $bc$ are quite different in terms of the distance between $b$ and $f$, and between $b$ and $c$, in $\mathbf{r}_{1}$. There are no items between $b$ and $c$, however $b$ and $f$ are separated by three other items $c$, $d$ and $e$, which means that $c$ is much more preferred than $f$. Therefore, for each $\mathbf{r}_{k}$ and every 2-long subsequence $\sigma_{i}\sigma_{j}$, we define the gap $\varpi_{k}(\sigma_{i},\sigma_{j})=\eta_{k}(\sigma_{j})-\eta_{k}(\sigma_{i})$, which indicates how much $\sigma_{i}$ is more preferred than its successor $\sigma_{j}$ in the 2-long subsequence $\sigma_{i}\sigma_{j}$ with respect to the original ranking sequence $\mathbf{r}_{k}$. The following table shows the gaps for $bf$ and $bc$ with respect to the example rankings.

[TABLE]

Clearly, the accumulated gaps for $b$ and $f$ is much bigger than that for $b$ and $c$. This suggests that $f$ is less likely to be preferred over $c$. To take into account this likelihood, we define

[TABLE]

where $0<\lambda\leq 1$ and $g=\frac{1}{N}\sum_{k=1}^{N}|\varpi_{k}(\sigma_{i},\sigma_{j})|$, so that any subsequence $\sigma_{i}\sigma_{j}$ with bigger average gaps will be “penalized”.

Now we relate $\psi(\sigma_{i},\sigma_{j})$ to $f_{\mathbf{r}}(y_{k})$ for Equation (7). For $p>1$, a $p$-long subsequence $y_{k}=(\sigma_{k_{1}},\sigma_{k_{2}},\dots,\sigma_{k_{p}})\in\mathbb{F}$ is represented by a $(p-1)$-long path $(\sigma_{k_{1}},\sigma_{k_{2}}),\dots,(\sigma_{k_{p-1}},\sigma_{k_{p}})$ in the graph. As each edge $(\sigma_{k_{i}},\sigma_{k_{j}})$ has weight $\psi(\sigma_{k_{i}},\sigma_{k_{j}})$, defining

[TABLE]

makes Equation (11) consistent with Equation (7).

IV-B3 An example for $\psi(\sigma_{i},\sigma_{j})=1$ and $\theta(\sigma)=1$

Comparing Fig. 1 and Fig. 2, obviously the example in Section IV-A is a special case for Theorem 1 when $\psi(\sigma_{i},\sigma_{j})=1$ and $\theta(\sigma)=1$.

Now we can use Theorem 1 to calculate the consensus score for the example in Section IV-A.

Example 1**.**

Consider $\mathcal{R}=\{\mathbf{r}_{1}=abcdef,\mathbf{r}_{2}=bdcefa,\mathbf{r}_{3}=bcdeghijkf,\mathbf{r}_{4}=badefc\}$, based on the matrix $\mathbf{A}$ in Equation (6), we have

[TABLE]

[TABLE]

Then,

[TABLE]

IV-C * $\ell(\mathcal{R})$ and $\kappa(\mathcal{R})$ *

In Example 1, we observe that $\mathbf{L}^{p}=0$ for $p\geq 4$, which implies that there are no common subsequences with length more than 4 and hence the length of the longest common subsequences of $\mathcal{R}$ is 4. Based on this fact, the following corollary of Theorem 1 provides an algorithm for calculating $\ell(\mathcal{R})$.

Corollary 1**.**

Under the assumptions in Theorem 1 , the length $\ell(\mathcal{R})$ of the longest subsequences in $\mathcal{S}(\mathcal{R})$ can be obtained by

[TABLE]

Corollary 2**.**

Under the assumptions in Theorem 1 ,

[TABLE]

where $n=|\mathbf{r}_{x}|$ and $\mathbf{I}$ is the identity matrix of size $n\times n$. Consequently $\kappa(\mathcal{R})$ can be computed in $O(n+|E|)$ time.

Proof.

Since the longest possible length of a common subsequence is $n$, Theorem 1 implies that

[TABLE]

Invoking the identity $(\mathbf{I}-\mathbf{L})(\mathbf{I}+\mathbf{L}+\mathbf{L}^{2}+\cdots+\mathbf{L}^{n-1})=\mathbf{I}-\mathbf{L}^{n}$ and the observation that $\mathbf{L}^{n}=0$ since $\mathbf{L}$ is strictly lower triangular, we obtain that

[TABLE]

Now we discuss the runtime. Since $\mathbf{L}$ is a strictly lower triangular matrix, $\mathbf{I}-\mathbf{L}$ is lower triangular. Note that computing $\mathbf{M}^{-1}u$ is equivalent to solving $\mathbf{M}v=u$ and, if $\mathbf{M}$ is lower triangular, can be done efficiently in $O(\operatorname{nnz}(\mathbf{M})+n)$ time using forward elimination (degenerated Gaussian elimination), where $\operatorname{nnz}(\mathbf{M})$ denotes the number of non-zero entries in $\mathbf{M}$. Therefore $(\mathbf{I}-\mathbf{L})^{-1}\mathbf{z}$ can be computed in $O(n+|E|)$ time, and thus $\kappa(\mathcal{R})$ in $O(n+|E|)$ time. ∎

We remark that the runtime in Corollary 2 is significantly faster, by a factor of $n$, than the naïve algorithm to sum up $\kappa_{p}(A)$ over $p$, which would take $O(n^{2}+n|E|)$ time.

IV-D Remarks

Algorithm 1 is the pseudocode for Theorem 1. Generating $\mathbf{A}$ (Line 1–1) takes $O(Nn^{2})$ time. Computing $\kappa_{1}(\mathcal{R})=\operatorname{tr}(A)$ (Line 1) takes $O(n)$ time. For each $p\geq 2$, the matrix-vector multiplication $\mathbf{L}\mathbf{y}$ in Line 1 takes $O(n+|E|)$ time, since $\mathbf{L}$ has at most $O(|E|)$ non-zero entries. In fact, $|E|=\kappa_{2}(\mathcal{R})$. Line 20 takes $O(n)$ time. Overall computing $\kappa_{p}(\mathcal{R})$, after generation of $\mathbf{A}$, takes $O(p(n+|E|))$ time for $p\geq 2$.

IV-D1 Sensitivity detection by gaps and positions of items

There are some differences among $\kappa_{p}(\mathcal{R})$. Note that $\kappa_{1}(\mathcal{R})$ is controlled by the $\gamma$ and $d$ in Equation (13), where $d$ is a factor to reflect the deviation of each item’s positions in the rankings. Higher disagreement of the items positions in the rankings will result in lower $\kappa_{1}(\mathcal{R})$. Hence $\kappa_{1}(\mathcal{R})$ is sensitivity detection of the variation of items positions. Whereas for $p>1$, the measure $\kappa_{p}(\mathcal{R})$ takes into account the extent of the relationship between two items by incorporating $\lambda$ and $g$ in Equation (14).

We demonstrate such ability for sensitivity detection in terms of the gaps and positions of items in Section V.

IV-D2 Duplicate rankings

In the above example of $\mathcal{R}=\{\mathbf{r}_{1}=abcdef,\mathbf{r}_{2}=bdcefa,\mathbf{r}_{3}=bcdeghijkf,\mathbf{r}_{4}=badefc\}$ and its adjacent matrix $\mathbf{A}$ in Equation (6), we assume that there are only distinct rankings in $\mathcal{R}$. However, this is not the case, especially in the group decision making process, where there may be duplicate rankings produced by the experts. For example, we may have a multi-set $\mathcal{R}^{\prime}=\{\mathbf{r}_{1}=abcdef,\mathbf{r}_{2}=bdcefa,\mathbf{r}_{3}=bcdeghijkf,\mathbf{r}_{4}=badefc,\mathbf{r}_{5}=badefc,\mathbf{r}_{6}=badefc$, $\mathbf{r}_{7}=badefc\}$, which contains duplicate ranking of $badefc$. Obviously, $\mathcal{R}^{\prime}$ has higher degree of consensus than $\mathcal{R}$. However, as $\kappa_{p}(\mathcal{R}^{\prime})=\kappa_{p}(\mathcal{R})$, that is, $\kappa_{p}(\cdot)$ cannot discriminate $\mathcal{R}^{\prime}$ and $\mathcal{R}$. In order to distinguish the difference, we let $o(\mathbf{r})$ be the number of occurrences of ranking $\mathbf{r}$ in $\mathcal{R}^{\prime}$, and define

[TABLE]

where $s=\max\{o(\mathbf{r}):\forall\mathbf{r}\in\mathcal{R}^{\prime}\}$ and $\mathcal{R}$ is the set of all distinct rankings in $\mathcal{R}^{\prime}$.

IV-D3 Rankings with ties

Rankings with ties occur when the preference scores over some items are identical. Let $\mathbf{r}_{k}=\mathcal{T}_{k_{1}}\dots\mathcal{T}_{k_{n}}$ be a ranking with ties, where $\mathcal{T}_{k_{i}}$ is a set of items with an identical ranking score, $\mathcal{T}_{k_{i}}\cap\mathcal{T}_{k_{j}}=\emptyset$ if $i\neq j$, and for $i<j$, every $x\in\mathcal{T}_{k_{i}}$ is more preferred than all $y\in\mathcal{T}_{k_{j}}$. If we replace Equation (9) with

[TABLE]

then Theorem 1 can be used, without any modification, to measure consensus of rankings with ties.

IV-D4 Weighting scheme for top-$k$ items

The measure $\kappa_{p}(\mathcal{R})$ is a subsequence-based consensus measure and has the ability to handle top-$k$ rankings. In the case where the top-$k$ items rankings need to be weighted with greater value and the items after top-$k$ are not important (see, e.g. [9, 15, 17]), a slight revision to Equation (13) will accommodate this. Consider a given cut-off value $\zeta$ for items position, we can weigh the items with

[TABLE]

where $0<p<1$. Clearly, if an items is ranked after position $\zeta$ in one of the rankings, $H\!\left(\zeta-\mu(\sigma)-d\right)p^{\mu(\sigma)}\gamma^{d}=0$.

V Experiments

This section will present how the proposed $\kappa_{p}(\cdot)$ and $\kappa(\cdot)$ can be applied to evaluate rank aggregation and compare search engine rankings. The source codes and the experimental data used are available on the github repository 333https://github.com/zhiweiuu/secs.

V-A Evaluation of rank aggregation of full rankings

Rank aggregation, with wide applications in decision making systems, machine learning and social science, is the problem of how to combine many rankings in order to obtain one consensus ranking [25, 2]. Dwork et al. has proved that, even if $|\mathcal{R}|=4$, obtaining an optimal aggregation with the Kendall $\tau$ index is NP-hard [26]. Another problem with rank aggregation is the lack of ground truth for evaluation. Here, we show how we can use the proposed $\kappa_{p}(\cdot)$ to evaluate the ranking aggregation results.

The rankings used in this experiment are the seven rankings (shown in Table II) of 10 clustering algorithms with respect to 7 different validation measures when they are used to cluster microarray data into five clusters [27]. The details for the 7 validation measures of APN, AD, ADM, FOM, connectivity, Dunn and Silhouette and the details for 10 clustering algorithms of SM, FN, KM, PM, HR, AG, CL, DI and MO can be found in [27]. Here, we use $\mathcal{R}_{c}$ to denote the set of the 7 rankings. Two rank aggregation algorithms (the Cross-Entropy Monte Carlo algorithm (CE) and the Genetic algorithm (GA) ) have been used to aggregate the 7 rankings in [27] and the aggregated rankings are shown in Table III. We create two sets of rankings $\mathcal{R}_{GA}$ and $\mathcal{R}_{CE}$ by adding each aggregated ranking into $\mathcal{R}_{c}$,

[TABLE]

Based on the property in Equation (3), clearly $\kappa(\mathcal{R}_{GA})\leq\kappa(\mathcal{R}_{c})$ and $\kappa(\mathcal{R}_{CE})\leq\kappa(\mathcal{R}_{c})$. Therefore, we would reasonably expect that a better aggregated ranking would result less decline of $\kappa(\cdot)$ when adding the aggregated ranking into $\mathcal{R}_{c}$. Table IV shows experimental results for $\kappa(\mathcal{R}_{GA})$ and $\kappa(\mathcal{R}_{CE})$ by varying values of $\gamma$ and $\lambda$ from 1 to 0.45. Figure 3 shows the changes of $\kappa_{p}(\cdot)$ for both GA and CE. We find that the aggregated ranking by the CE algorithm is more sensitive to $\gamma$ while the aggregated ranking by the GA algorithm is more sensitive to $\lambda$.

V-B Evaluation of consensus for top-$k$ rankings

In this section, we show how the proposed consensus measure can be used to evaluate top-$k$ rankings [17]. Here, we are interested in the top-25 items from Google and Bing searches. Twelve top-25 rankings used in this experiment are the search rankings from Google and Bing by using 6 “related” key words: “Bond films”, “Bond Movies”, “007 films”, “007 movies”, “James Bond films” and “James Bond movies”, with 6 rankings from Google and 6 rankings from Bing. Table V shows the twelve rankings. As these key words refer to an identical concept from human perspective, we want to know how close or related these rankings are, which can be evaluated by the proposed consensus measure $\kappa(\mathcal{R})$ and $\kappa_{p}(\mathcal{R})$, in terms of “relatedness” or “closeness”. More details about the extracted rankings can be found on the github repository.

The $\kappa(\mathcal{R})$ values for Bing and Google rankings are shown in Table VI. The results show that for the given key words, Google has consistently higher $\kappa(\mathcal{R})$ values than Bing when both $\lambda$ and $\gamma$ varies from 1 to 0.45. However, it is difficult to discern which search engine results are more sensitive to $\gamma$ and $\lambda$. Therefore, we show the changes of $\kappa_{p}(\mathcal{R})$ in terms of $\gamma$ and $\lambda$ in Figure 4. In Figure 4a, though Bing has 8 links in common ($\kappa_{1}(\mathcal{R})=8$) and Google has only 7 links in common ($\kappa_{1}(\mathcal{R})=7$), Bing’s search results are more sensitive to the $\gamma$, which shows that the deviation of the links positions in Bing’s search results are bigger than that by Google search. Especially, when $\gamma\leq 0.85$, the $\kappa_{1}(\mathcal{R})$ for Google’s rankings is in fact higher than that from Bing’s rankings. Also, as shown in Figure 4, both $\kappa_{2}(\mathcal{R})$ and $\kappa_{3}(\mathcal{R})$ suggest that Google’s rankings have a higher degree of consensus than Bing’s rankings.

VI Conclusion and future work

This paper introduces a novel approach for consensus measure of rankings by using graph representation, in which the vertices are the items and the edges are the relationship of items in the rankings. Such representation leads to various algorithms for consensus measure in terms of different aspects in the rankings, including the number of common patterns, the number of common patterns with fixed length and the length of the longest common patterns. We present how the proposed approaches can be used to evaluate rank aggregation and compare search engines rankings.

In future, we will look into the property, shown in Equation (3) and use it to define a new objective function for rank aggregation so that the proposed approaches can be used to develop elastic algorithms for rank aggregation. A challenging task for future is to extract the common patterns of rankings and use these common pattern to define a probabilistic model for evaluating rankings generated from different systems.