Some impossibilities of ranking in generalized tournaments

TL;DR

This paper demonstrates fundamental impossibilities in ranking players in generalized tournaments when applying axiomatic principles like self-consistency and order preservation simultaneously.

Contribution

It proves that the axiomatic properties of self-consistency and order preservation cannot both be satisfied in generalized tournament rankings.

Findings

Self-consistency and order preservation are incompatible axioms.

The impossibility holds for the universal domain of generalized tournaments.

Highlights limitations of current ranking axioms in complex competitive settings.

Abstract

In a generalized tournament, players may have an arbitrary number of matches against each other and the outcome of the games is measured on a cardinal scale with a lower and upper bound. An axiomatic approach is applied to the problem of ranking the competitors. Self-consistency requires assigning the same rank for players with equivalent results, while a player showing an obviously better performance than another should be ranked strictly higher. According to order preservation, if two players have the same pairwise ranking in two tournaments where the same players have played the same number of matches, then their pairwise ranking is not allowed to change in the aggregated tournament. We reveal that these two properties cannot be satisfied simultaneously on this universal domain.

Equations

$$g_2(N,\mathbf{T})=\delta \left(g_2(N,T^{(1)}),g_2(N,T^{(2)})\right)=\delta \left(g_3(N,T^{(1)}),g_3(N,T^{(2)})\right)=g_3(N,\mathbf{T}).$$

$$(N,R^{\prime \prime },M^{\prime \prime })=\sigma (N,R,M)+\sigma (N,-R^{\prime },M^{\prime })+(N,-R,M)+(N,R^{\prime },M^{\prime }).$$

Full text

JEL** classification number: C44, D71**

MSC** class: 91A80, 91B14**

Keywords: tournament ranking; paired comparison; axiomatic approach; impossibility

1 Introduction

This paper addresses the problem of tournament ranking when players may have played an arbitrary number of matches against each other, from an axiomatic point of view. For instance, the matches among top tennis players lead to a set of similar data: Andre Agassi has played 14 matches with Boris Becker, but he has never played against Björn Borg (Bozóki et al.,, 2016). To be more specific, we show the incompatibility of some natural properties. Impossibility theorems are well-known in the classical theory of social choice (Arrow,, 1950; Gibbard,, 1973; Satterthwaite,, 1975), but our setting has a crucial difference: the set of agents and the set of alternatives coincide, therefore the transitive effects of ’voting’ should be considered (Altman and Tennenholtz,, 2008). We also allow for cardinal and incomplete preferences as well as ties in the ranking derived.

Several characterizations of ranking methods have been suggested in the literature by providing a set of properties such that they uniquely determine a given method (Rubinstein,, 1980; Bouyssou,, 1992; Bouyssou and Perny,, 1992; van den Brink and Gilles,, 2003, 2009; Slutzki and Volij,, 2005, 2006; Kitti,, 2016). There are some excellent axiomatic analyses, too (Chebotarev and Shamis,, 1998; González-Díaz et al.,, 2014).

However, apart from Csató, 2018b , we know only one work discussing impossibility results for ranking the nodes of a directed graph (Altman and Tennenholtz,, 2008), a domain covered by our concept of generalized tournament. We think these theorems are indispensable for a clear understanding of the axiomatic framework. For example, González-Díaz et al., (2014) have found that most ranking methods violate an axiom called order preservation, but it is not known whether this negative result is caused by a theoretical impossibility or it is only due to some hidden features of the procedures that have been considered.

It is especially a relevant issue because of the increasing popularity of sports rankings (Langville and Meyer,, 2012), which is, in a sense, not an entirely new phenomenon, since sports tournaments have motivated some classical works of social choice and voting theory (Landau,, 1895; Zermelo,, 1929; Wei,, 1952). For instance, the ranking of tennis players has been addressed from at least three perspectives, with the use of methods from multicriteria decision-making (Bozóki et al.,, 2016), network analysis (Radicchi,, 2011), or statistics (Baker and McHale,, 2014, 2017). Consequently, the axiomatic approach can be fruitful in the choice of an appropriate sports ranking method. This issue has been discussed in some recent works (Berker,, 2014; Pauly,, 2014; Csató,, 2017; Csató, 2018a, ; Csató, 2018c, ; Csató, 2018d, ; Csató, 2018e, ; Csató, 2018f, ; Csató, 2018g, ; Dagaev and Sonin,, 2017; Vaziri et al.,, 2018; Vong,, 2017), but there is a great scope for future research.

For this purpose, we will place two properties, imported from the social choice literature, in the centre of the discussion. Self-consistency (Chebotarev and Shamis,, 1997) requires assigning the same rank for players with equivalent results, furthermore, a player showing an obviously better performance than another should be ranked strictly higher. Order preservation111 The term order preservation may be a bit misleading since it can suggest that the sequence of matches does not influence the rankings (see Vaziri et al., (2018, Property III)). This requirement obviously holds in our setting. (González-Díaz et al.,, 2014) excludes the possibility of rank reversal by demanding the preservation of players’ pairwise ranking when two tournaments, where the same players have played the same number of matches, are aggregated. In other words, it is not allowed that player $A$ is judged better both in the first and second half of the season than player $B$, but ranked lower on the basis of the whole season.

Our main result proves the incompatibility of self-consistency and order preservation. This finding gives a theoretical foundation for the observation of González-Díaz et al., (2014) that most ranking methods do not satisfy order preservation. Another important message of the paper is that prospective users cannot avoid to take similar impossibilities into account and justify the choice between the properties involved.

The study is structured as follows. Section 2 presents the notion of ranking problem and scoring methods. Section 3 introduces the property called self-consistency and proves that one type of scoring methods cannot satisfy it. Section 4 defines (strong) order preservation besides some other properties, addresses the compatibility of the axioms and derives a negative result by opposing self-consistency and order preservation. Section 5 summarizes our main findings.

2 The ranking problem and scoring methods

Consider a set of players $N=\{X_{1},X_{2},\dots,X_{n}\}$, $n\in\mathbb{N}_{+}$ and a series of tournament matrices $T^{(1)}$, $T^{(2)}$, …, $T^{(m)}$ containing information on the paired comparisons of the players. Their entries are given such that $t_{ij}^{(p)}+t_{ji}^{(p)}=1$ if players $X_{i}$ and $X_{j}$ have played in round $p$ ($1\leq p\leq m$) and $t_{ij}^{(p)}+t_{ji}^{(p)}=0$ if they have not played against each other in round $p$. The simplest definition can be $t_{ij}^{(p)}=1$ (implying $t_{ji}^{(p)}=0$) if player $X_{i}$ has defeated player $X_{j}$, and $t_{ij}^{(p)}=0$ (implying $t_{ji}^{(p)}=1$) if player $X_{i}$ has lost against player $X_{j}$ in round $p$. A draw can be represented by $t_{ij}^{(p)}=t_{ji}^{(p)}=0.5$. The entries may reflect the scores of the players, or other features of the match (e.g. an overtime win has less value than a normal time win), too.

The tuple $\left(N,T^{(1)},T^{(2)},\dots,T^{(m)}\right)$, denoted shortly by $(N,\mathbf{T})$, is called a general ranking problem. The set of general ranking problems with $n$ players ($|N|=n$) is denoted by $\mathcal{T}^{n}$.

The aggregated tournament matrix $A=\sum_{p=1}^{m}T^{(p)}=\left[a_{ij}\right]\in\mathbb{R}^{n\times n}$ combines the results of all rounds of the competition.

The pair $(N,A)$ is called a ranking problem. The set of ranking problems with $n$ players ($|N|=n$) is denoted by $\mathcal{R}^{n}$. Note that every ranking problem can be associated with several general ranking problems, in this sense, ranking problem is a narrower notion.

Let $(N,A),(N,A^{\prime})\in\mathcal{R}^{n}$ be two ranking problems with the same player set $N$. The sum of these ranking problems is $(N,A+A^{\prime})\in\mathcal{R}^{n}$. For example, the ranking problems can contain the results of matches in the first and second half of the season, respectively.

Any ranking problem $(N,A)$ has a skew-symmetric results matrix $R=A-A^{\top}=\left[r_{ij}\right]\in\mathbb{R}^{n\times n}$ and a symmetric matches matrix $M=A+A^{\top}=\left[m_{ij}\right]\in\mathbb{N}^{n\times n}$. $m_{ij}$ is the number of matches between players $X_{i}$ and $X_{j}$, whose outcome is given by $r_{ij}$. Matrices $R$ and $M$ also determine the aggregated tournament matrix through $A=(R+M)/2$, so any ranking problem $(N,A)\in\mathcal{R}^{n}$ can be denoted analogously by $(N,R,M)$ with the restriction $|r_{ij}|\leq m_{ij}$ for all $X_{i},X_{j}\in N$. Despite description with results and matches matrices is not parsimonious, this notation will turn out to be useful.

A general scoring method is a function $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$. Several procedures have been suggested in the literature, see Chebotarev and Shamis, (1998) for an overview of them. A special type of general scoring methods is the following.

Definition 2.1**.**

Individual scoring method (Chebotarev and Shamis,, 1999): A general scoring method $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$ is called individual scoring method if it is based on individual scores, that is, there exist functions $\phi$ and $\delta$ such that for any general ranking problem $(N,\mathbf{T})\in\mathcal{T}^{n}$, the corresponding score vector $\mathbf{s}=g(N,\mathbf{T})$ can be expressed as $\mathbf{s}=\delta(\mathbf{s}^{(1)},\mathbf{s}^{(2)},\dots,\mathbf{s}^{(m)})$, where the partial score vectors $\mathbf{s}^{(p)}=\phi(N,T^{(p)})$ depend solely on the tournament matrix $T^{(p)}$ of round $p$ for all $p=1,2,\dots,m$.

A scoring method is a function $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$. Any scoring method can also be regarded as a general scoring method – by using the aggregated tournament matrix instead of the whole series of tournament matrices –, therefore some articles only consider scoring methods (Kitti,, 2016; Slutzki and Volij,, 2005). González-Díaz et al., (2014) give a thorough axiomatic analysis of certain scoring methods.

In other words, scoring methods initially aggregate the tournament matrices and then rank the players by their scores, while individual scoring methods first give scores to the players in each round and then aggregate them.

3 An argument against the use of individual scoring methods

In this section, some properties of general scoring methods are presented, which will highlight an important failure of individual scoring methods.

3.1 Universal invariance axioms

Axiom 3.1**.**

Anonymity ($ANO$): Let $(N,\mathbf{T})\in\mathcal{T}^{n}$ be a general ranking problem, $\sigma:\{1,2,\dots,m\}\rightarrow\{1,2,\dots,m\}$ be a permutation on the set of rounds, and $\sigma(N,\mathbf{T})\in\mathcal{T}^{n}$ be the ranking problem obtained from $(N,\mathbf{T})$ by permutation $\sigma$. General scoring method $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$ is anonymous if $g_{i}(N,\mathbf{T})=g_{i}\left(\sigma(N,\mathbf{T})\right)$ for all $X_{i}\in N$.

Anonymity implies that any reindexing of the rounds (tournament matrices) preserves the scores of the players.

Axiom 3.2**.**

Neutrality ($NEU$): Let $(N,\mathbf{T})\in\mathcal{T}^{n}$ be a general ranking problem, $\sigma:N\rightarrow N$ be a permutation on the set of players, and $(\sigma(N),\mathbf{T})\in\mathcal{T}^{n}$ be the ranking problem obtained from $(N,\mathbf{T})$ by permutation $\sigma$. General scoring method $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$ is neutral if $g_{i}(N,\mathbf{T})=g_{\sigma(i)}(\sigma(N),\mathbf{T})$ for all $X_{i}\in N$.

Neutrality means that the scores are independent of the labelling of the players.

3.2 Self-consistency

Now we want to formulate a further requirement on the ranking of the players by answering the following question: When is player $X_{i}$ undeniably better than player $X_{j}$? There are two such plausible cases: (1) if player $X_{i}$ has achieved better results against the same opponents; (2) if player $X_{i}$ has achieved the same results against stronger opponents. Consequently, player $X_{i}$ should also be judged better if he/she has achieved better results against stronger opponents than player $X_{j}$. Furthermore, since (general) scoring methods allow for ties in the ranking, player $X_{i}$ should have the same rank as player $X_{j}$ if he/she has achieved the same results against opponents with the same strength.

In order to apply these principles, both the results and strengths of the players should be measured. Results can be extracted from the tournament matrices $T^{(p)}$. Strengths of the players can be obtained from their scores according to the (general) scoring method used, hence the name of the implied axiom is self-consistency. It has been introduced in Chebotarev and Shamis, (1997), and extensively discussed by Csató, 2018b .

Definition 3.1**.**

Opponent multiset: Let $(N,\mathbf{T})\in\mathcal{T}^{n}$ be a general ranking problem. The opponent multiset222 Multiset is a generalization of the concept of set allowing for multiple instances of its elements. of player $X_{i}$ is $O_{i}$, which contains $m_{ij}$ instances of $X_{j}$.

Players of the opponent multiset $O_{i}$ are called the opponents of player $X_{i}$.

Notation 3.1*.*

Consider the ranking problem $(N,T^{(p)})\in\mathcal{T}^{n}$ given by restricting a general ranking problem to its $p$th round. Let $X_{i},X_{j}\in N$ be two different players and $h^{(p)}:O_{i}^{(p)}\leftrightarrow O_{j}^{(p)}$ be a one-to-one correspondence between the opponents of $X_{i}$ and $X_{j}$ in round $p$, consequently, $|O_{i}^{(p)}|=|O_{j}^{(p)}|$. Then $\mathfrak{h}^{(p)}:\{k:X_{k}\in O_{i}^{(p)}\}\leftrightarrow\{\ell:X_{\ell}\in O_{j}^{(p)}\}$ is given by $X_{\mathfrak{h}^{(p)}(k)}=h^{(p)}(X_{k})$.

Axiom 3.3**.**

Self-consistency ($SC$) (Chebotarev and Shamis,, 1997): A general scoring method $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$ is called self-consistent if the following implication holds for any general ranking problem $(N,\mathbf{T})\in\mathcal{T}^{n}$ and for any players $X_{i},X_{j}\in N$: if there exists a one-to-one mapping $h^{(p)}$ from $O^{(p)}_{i}$ onto $O^{(p)}_{j}$ such that $t_{ik}^{(p)}\geq t_{j\mathfrak{h}^{(p)}(k)}^{(p)}$ and $g_{k}(N,\mathbf{T})\geq g_{\mathfrak{h}^{(p)}(k)}(N,\mathbf{T})$ for all $p=1,2,\dots,m$ and $X_{k}\in O_{i}^{(p)}$, then $f_{i}(N,R,M)\geq f_{j}(N,R,M)$, furthermore, $f_{i}(N,R,M)>f_{j}(N,R,M)$ if $t_{ik}^{(p)}>t_{j\mathfrak{h}^{(p)}(k)}^{(p)}$ or $g_{k}(N,\mathbf{T})>g_{\mathfrak{h}^{(p)}(k)}(N,\mathbf{T})$ for at least one $1\leq p\leq m$ and $X_{k}\in O_{i}^{(p)}$.

3.3 Individual scoring methods and self-consistency

In this part, it will be proved that an anonymous and neutral individual scoring method cannot satisfy self-consistency, which is a natural fairness requirement, thus it is enough to focus on ranking problems and scoring methods. For this purpose, the example below will be used.

Example 3.1**.**

Let $\left(N,T^{(1)},T^{(2)}\right)\in\mathcal{T}^{4}$ be a general ranking problem describing a tournament with two rounds.

It is shown in Figure 1: a directed edge from node $X_{i}$ to $X_{j}$ indicates a win of player $X_{i}$ over $X_{j}$ (and a loss of $X_{j}$ against $X_{i}$), while an undirected edge from node $X_{i}$ to $X_{j}$ represents a drawn match between the two players. This representation will be used in further examples, too.

So, player $X_{1}$ has defeated $X_{4}$ in the first round (Figure 1.a), while players $X_{2}$ and $X_{3}$ have not played. In the second round, players $X_{1}$ and $X_{2}$, as well as players $X_{3}$ and $X_{4}$ have drawn (Figure 1.b). The whole tournament is shown in Figure 1.c.

According to the following result, at least one property from the set of $ANO$, $NEU$ and $SC$ will be violated by any individual scoring method.

Proposition 3.1**.**

There exists no anonymous and neutral individual scoring method satisfying self-consistency.

Proof.

Let $g:\mathcal{T}^{n}\to\mathbb{R}^{n}$ be an anonymous and neutral individual scoring method. Consider Example 3.1. $ANO$ and $NEU$ imply that $g_{2}(N,T^{(1)})=g_{3}(N,T^{(1)})$ and $g_{2}(N,T^{(2)})=g_{3}(N,T^{(2)})$, therefore

[TABLE]

Note that $O_{1}^{(1)}=\{X_{4}\}$, $O_{1}^{(1)}=\{X_{2}\}$ and $O_{4}^{(1)}=\{X_{1}\}$, $O_{4}^{(2)}=\{X_{3}\}$. Take the one-to-one correspondences $h_{14}^{(1)}:O_{1}^{(1)}\leftrightarrow O_{4}^{(1)}$ such that $h_{14}^{(1)}(X_{4})=X_{1}$ and $h_{14}^{(2)}:O_{1}^{(2)}\leftrightarrow O_{4}^{(2)}$ such that $h_{14}^{(2)}(X_{2})=X_{3}$. Now $t_{12}^{(2)}=t_{43}^{(2)}$ since the corresponding matches resulted in draws. Furthermore, $t_{14}^{(1)}\neq t_{41}^{(1)}$ since the value of a win and a loss should be different. It can be assumed without loss of generality that $t_{14}^{(1)}>t_{41}^{(1)}$. Suppose that $g_{1}(N,\mathbf{T})\leq g_{4}(N,\mathbf{T})$. Then players $X_{1}$ and $X_{4}$ have a draw against a player with the same strength ($X_{2}$ and $X_{3}$, respectively), but $X_{1}$ has defeated $X_{4}$, so it has a better result against a not weaker opponent. Therefore, self-consistency (Axiom 3.3) implies $g_{1}(N,\mathbf{T})>g_{4}(N,\mathbf{T})$, which is a contradiction, thus $g_{1}(N,\mathbf{T})>g_{4}(N,\mathbf{T})$ holds.

However, $O_{2}^{(1)}=\emptyset$, $O_{2}^{(2)}=\{X_{1}\}$ and $O_{3}^{(1)}=\emptyset$, $O_{3}^{(2)}=\{X_{4}\}$. Consider the unique one-to-one correspondence $h_{14}^{(2)}:O_{2}^{(2)}\leftrightarrow O_{3}^{(2)}$, which – together with $t_{21}^{(2)}=t_{34}^{(2)}$ (the two draws should be represented by the same number) and $g_{1}(N,\mathbf{T})>g_{4}(N,\mathbf{T})$ – leads to $g_{2}(N,\mathbf{T})>g_{3}(N,\mathbf{T})$ because player $X_{2}$ has achieved the same result against a stronger opponent than player $X_{3}$. In other words, $SC$ requires the draw of $X_{2}$ to be more valuable than the draw of $X_{3}$, but it cannot be reflected by any individual scoring method $g$ according to (1). ∎

4 The case of ranking problems and scoring methods

According to Proposition 3.1, only the procedure underlying scoring methods can be compatible with self-consistency. Therefore, this section will focus on scoring methods.

4.1 Axioms of invariance with respect to the results matrix

Let $O\in\mathbb{R}^{n\times n}$ be the matrix with all of its entries being zero.

Axiom 4.1**.**

Symmetry ($SYM$) (González-Díaz et al.,, 2014): Let $(N,R,M)\in\mathcal{R}^{n}$ be a ranking problem such that $R=O$. Scoring method $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ is symmetric if $f_{i}(N,R,M)=f_{j}(N,R,M)$ for all $X_{i},X_{j}\in N$.

According to symmetry, if all paired comparisons (but not necessarily all matches in each round) between the players result in a draw, then all players will have the same score.

Axiom 4.2**.**

Inversion ($INV$) (Chebotarev and Shamis,, 1998): Let $(N,R,M)\in\mathcal{R}^{n}$ be a ranking problem. Scoring method $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ is invertible if $f_{i}(N,R,M)\geq f_{j}(N,R,M)\iff f_{i}(N,-R,M)\leq f_{j}(N,-R,M)$ for all $X_{i},X_{j}\in N$.

Inversion means that taking the opposite of all results changes the ranking accordingly. It establishes a uniform treatment of victories and losses.

Corollary 4.1**.**

Let $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ be a scoring method satisfying $INV$. Then for all $X_{i},X_{j}\in N$: $f_{i}(N,R,M)>f_{j}(N,R,M)\iff f_{i}(N,-R,M)<f_{j}(N,-R,M)$.

The following result has been already mentioned by González-Díaz et al., (2014, p. 150).

Corollary 4.2**.**

$INV$* implies $SYM$.*

It seems to be difficult to argue against symmetry. However, scoring methods based on right eigenvectors (Wei,, 1952; Slutzki and Volij,, 2005, 2006; Kitti,, 2016) violate inversion.

4.2 Properties of independence

The next axiom deals with the effects of certain changes in the aggregated tournament matrix $A$.

Axiom 4.3**.**

Independence of irrelevant matches ($IIM$) (González-Díaz et al.,, 2014): Let $(N,A),(N,A^{\prime})\in\mathcal{R}^{n}$ be two ranking problems and $X_{i},X_{j},X_{k},X_{\ell}\in N$ be four different players such that $(N,A)$ and $(N,A^{\prime})$ are identical but $a_{k\ell}\neq a^{\prime}_{k\ell}$. Scoring method $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ is called independent of irrelevant matches if $f_{i}(N,A)\geq f_{j}(N,A)\Rightarrow f_{i}(N,A^{\prime})\geq f_{j}(N,A^{\prime})$.

$IIM$ means that ’remote’ matches – not involving players $X_{i}$ and $X_{j}$ – do not affect the pairwise ranking of players $X_{i}$ and $X_{j}$.

Independence of irrelevant matches seems to be a powerful property. González-Díaz et al., (2014) state that ’when players have different opponents (or face opponents with different intensities), $IIM$ is a property one would rather not have’. Csató, 2018b argues on an axiomatic basis against $IIM$.

The rounds of a given tournament can be grouped arbitrarily. Therefore, the following property makes much sense.

Axiom 4.4**.**

Order preservation ($OP$) (González-Díaz et al.,, 2014): Let $(N,A),(N,A^{\prime})\in\mathcal{R}^{n}$ be two ranking problems where all players have played $m$ matches and $X_{i},X_{j}\in N$ be two different players. Let $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ be a scoring method such that $f_{i}(N,A)\geq f_{j}(N,A)$ and $f_{i}(N,A^{\prime})\geq f_{j}(N,A^{\prime})$.333 González-Díaz et al., (2014) formally introduce a stronger version of this axiom since only $X_{i}$ and $X_{j}$ should have the same number of matches in the two ranking problems. However, in the counterexample of González-Díaz et al., (2014), which shows the violation of $OP$ by several ranking methods, all players have played the same number of matches. $f$ satisfies order preservation if $f_{i}(N,A+A^{\prime})\geq f_{j}(N,A+A^{\prime})$, furthermore, $f_{i}(N,A+A^{\prime})>f_{j}(N,A+A^{\prime})$ if $f_{i}(N,A)>f_{j}(N,A)$ or $f_{i}(N,A^{\prime})>f_{j}(N,A^{\prime})$.

$OP$ is a relatively restricted version of combining ranking problems, which implies that if player $X_{i}$ is not worse than player $X_{j}$ on the basis of some rounds as well as on the basis of another set of rounds such that all players have played in each round (so they have played the same number of matches altogether), then this pairwise ranking should hold after the two distinct set of rounds are considered jointly.

One can consider a stronger version of order preservation, too.

Axiom 4.5**.**

Strong order preservation ($SOP$) (van den Brink and Gilles,, 2009): Let $(N,A),(N,A^{\prime})\in\mathcal{R}^{n}$ be two ranking problems and $X_{i},X_{j}\in N$ be two players. Let $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ be a scoring method such that $f_{i}(N,A)\geq f_{j}(N,A)$ and $f_{i}(N,A^{\prime})\geq f_{j}(N,A^{\prime})$. $f$ satisfies strong order preservation if $f_{i}(N,A+A^{\prime})\geq f_{j}(N,A+A^{\prime})$, furthermore, $f_{i}(N,A+A^{\prime})>f_{j}(N,A+A^{\prime})$ if $f_{i}(N,A)>f_{j}(N,A)$ or $f_{i}(N,A^{\prime})>f_{j}(N,A^{\prime})$.

In contrast to order preservation, $SOP$ does not contain any restriction on the number of matches of the players in the ranking problems to be aggregated.

Corollary 4.3**.**

$SOP$* implies $OP$.*

It will turn out that the weaker property, order preservation has still unfavourable implications.

4.3 Relations among the axioms

In this part, some links among symmetry, inversion, independence of irrelevant matches, and (strong) order preservation will be revealed.

Remark 4.1*.*

$SYM$ and $OP$ ($SOP$) imply $INV$.

Proof.

Consider a ranking problem $(N,R,M)\in\mathcal{R}^{n}$ where $f_{i}(N,R,M)\geq f_{j}(N,R,M)$ for players $X_{i},X_{j}\in N$. If $f_{i}(N,-R,M)>f_{j}(N,-R,M)$, then $f_{i}(N,O,2M)>f_{j}(N,O,2M)$ due to $OP$, which contradicts to $SYM$. So $f_{i}(N,-R,M)\leq f_{j}(N,-R,M)$ holds. ∎

It turns out that $IIM$ is also closely connected to $SOP$.

Proposition 4.1**.**

A scoring method satisfying $NEU$, $SYM$ and $SOP$ meets $IIM$.

Proof.

Assume to the contrary, and let $(N,R,M)\in\mathcal{R}^{n}$ be a ranking problem, $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ be a scoring method satisfying $NEU$, $SYM$, and $SOP$, and $X_{i},X_{j},X_{k},X_{\ell}\in N$ be four different players such that $f_{i}(N,R,M)\geq f_{j}(N,R,M)$, and $(N,R^{\prime},M^{\prime})\in\mathcal{R}^{n}$ is identical to $(N,R,M)$ except for the result $r^{\prime}_{k\ell}$ and number of matches $m^{\prime}_{k\ell}$ between players $X_{k}$ and $X_{\ell}$, where $f_{i}(N,R^{\prime},M^{\prime})<f_{j}(N,R^{\prime},M^{\prime})$.

According to Remark 4.1, $f$ satisfies $INV$, hence $f_{i}(N,-R,M)\leq f_{j}(N,-R,M)$. Denote by $\sigma:N\rightarrow N$ the permutation $\sigma(X_{i})=X_{j}$, $\sigma(X_{j})=X_{i}$, and $\sigma(X_{k})=X_{k}$ for all $X_{k}\in N\setminus\{X_{i},X_{j}\}$. Neutrality leads to in $f_{i}\left[\sigma(N,R,M)\right]\leq f_{j}\left[\sigma(N,R,M)\right]$, and $f_{i}\left[\sigma(N,-R^{\prime},M^{\prime})\right]<f_{j}\left[\sigma(N,-R^{\prime},M^{\prime})\right]$ due to inversion and Corollary 4.1. With the notations $R^{\prime\prime}=\sigma(R)-\sigma(R^{\prime})-R+R^{\prime}=O$ and $M^{\prime\prime}=\sigma(M)+\sigma(M^{\prime})+M+M^{\prime}$, we get

[TABLE]

Symmetry implies $f_{i}(N,R^{\prime\prime},M^{\prime\prime})=f_{j}(N,R^{\prime\prime},M^{\prime\prime})$ since $R^{\prime\prime}=O$, but $f_{i}(N,R^{\prime\prime},M^{\prime\prime})<f_{j}(N,R^{\prime\prime},M^{\prime\prime})$ from strong order preservation, which is a contradiction. ∎

It remains to be seen whether $NEU$, $SYM$, and $SOP$ are all necessary for Proposition 4.1.

Lemma 4.1**.**

$NEU$, $SYM$, and $SOP$ are logically independent axioms with respect to the implication of $IIM$.

Proof.

It is shown that there exist scoring methods, which satisfy exactly two properties from the set $NEU$, $SYM$, and $SOP$, but violate the third and does not meet $IIM$, too:

1. 1

$SYM$ and $SOP$: the sum of the results of the ’previous’ player, $f_{i}(N,R,M)=\sum_{j=1}^{n}r_{i-1,j}$ for all $X_{i}\in N\setminus\{X_{1}\}$ and $f_{1}(N,R,M)=\sum_{j=1}^{n}r_{n,j}$; 2. 2

$NEU$ and $SOP$: maximal number of matches of other players, $f_{i}(N,R,M)=\max\{\sum_{k=1}^{n}m_{jk}:X_{j}\neq X_{i}\}$;444 It is worth to note that the maximal number of own matches satisfies $NEU$, $SOP$, and $IIM$. 3. 3

$NEU$ and $SYM$: aggregated sum of the results of opponents, $f_{i}(N,R,M)=\sum_{X_{j}\in O_{i}}\sum_{k=1}^{n}r_{jk}$.

∎

Proposition 4.1 helps in deriving another impossibility statement.

Proposition 4.2**.**

There exists no scoring method that satisfies neutrality, symmetry, strong order preservation and self-consistency.

Proof.

According to Proposition 4.1, $NEU$, $SYM$ and $SOP$ imply $IIM$. Csató, 2018b (, Theorem 3.1) has shown that $IIM$ and $SC$ cannot be met at the same time. ∎

4.4 A basic impossibility result

The four axioms of Proposition 4.2 are not independent despite Lemma 4.1. However, a much stronger statement can be obtained by eliminating neutrality and symmetry, which also allows for a weakening of strong order preservation by using order preservation. Note that substituting an axiom with a weaker one in an impossibility statement leads to a stronger result.

We will use a generalized tournament with four players for this purpose.

Example 4.1**.**

Let $(N,R,M),(N,R^{\prime},M^{\prime})\in\mathcal{R}^{4}$ be two ranking problems. They are shown in Figure 2: in the first tournament described by $(N,R,M)$, matches between players $X_{1}$ and $X_{2}$, $X_{1}$ and $X_{4}$, $X_{2}$ and $X_{3}$, $X_{3}$ and $X_{4}$ all resulted in draws (see Figure 2.a). On the other side, in the second tournament, described by $(N,R^{\prime},M^{\prime})$, players $X_{1}$ and $X_{2}$ have lost against $X_{3}$ and drawn against $X_{4}$ (see Figure 2.b). The two ranking problems can be summed in $(N,R^{\prime\prime},M^{\prime\prime})\in\mathcal{R}^{4}$ such that $R^{\prime\prime}=R+R^{\prime}$ and $M^{\prime\prime}=M+M^{\prime}$ (see Figure 2.c).

Theorem 4.1**.**

There exists no scoring method that satisfies order preservation and self-consistency.

Proof.

Assume to the contrary that there exists a self-consistent scoring method $f:\mathcal{R}^{n}\to\mathbb{R}^{n}$ satisfying order preservation. Consider Example 4.1.

1. I.

Take the ranking problem $(N,R,M)$. Note that $O_{1}=O_{3}=\{X_{2},X_{4}\}$ and $O_{2}=O_{4}=\{X_{1},X_{3}\}$.

1. a)

Consider the identity one-to-one correspondences $h_{13}:O_{1}\leftrightarrow O_{3}$ and $h_{31}:O_{3}\leftrightarrow O_{1}$ such that $h_{13}(X_{2})=h_{31}(X_{2})=X_{2}$ and $h_{13}(X_{4})=h_{31}(X_{4})=X_{4}$. Since $r_{12}=r_{32}=0$ and $r_{14}=r_{34}=0$, players $X_{1}$ and $X_{3}$ have the same results against the same opponents, hence $f_{1}(N,R,M)=f_{3}(N,R,M)$ from $SC$. 2. b)

Consider the identity one-to-one correspondences $h_{24}:O_{2}\leftrightarrow O_{4}$ and $h_{42}:O_{4}\leftrightarrow O_{2}$. Since $r_{21}=r_{41}=0$ and $r_{23}=r_{43}=0$, players $X_{2}$ and $X_{4}$ have the same results against the same opponents, hence $f_{2}(N,R,M)=f_{4}(N,R,M)$ from $SC$. 3. c)

Suppose that $f_{2}(N,R,M)>f_{1}(N,R,M)$, which implies $f_{4}(N,R,M)>f_{3}(N,R,M)$. Consider the one-to-one mapping $h_{12}:O_{1}\leftrightarrow O_{2}$, where $h_{12}(X_{2})=X_{1}$ and $h_{12}(X_{4})=X_{3}$. Since $r_{12}=r_{21}=0$ and $r_{14}=r_{23}=0$, player $X_{1}$ has the same results against stronger opponents compared to $X_{2}$, hence $f_{1}(N,R,M)>f_{2}(N,R,M)$ from $SC$, which is a contradiction. 4. d)

An analogous argument shows that $f_{1}(N,R,M)>f_{2}(N,R,M)$ cannot hold.

Therefore, self-consistency leads to $f_{1}(N,R,M)=f_{2}(N,R,M)=f_{3}(N,R,M)=f_{4}(N,R,M)$ in the first ranking problem. 2. II.

Take the ranking problem $(N,R^{\prime},M^{\prime})$. Note that $O_{1}^{\prime}=O_{2}^{\prime}=\{X_{3},X_{4}\}$ and $O_{3}^{\prime}=O_{4}^{\prime}=\{X_{1},X_{2}\}$.

1. a)

Consider the identity one-to-one correspondences $h_{12}^{\prime}:O_{1}^{\prime}\leftrightarrow O_{2}^{\prime}$ and $h_{21}^{\prime}:O_{2}^{\prime}\leftrightarrow O_{1}^{\prime}$. Since $r_{13}^{\prime}=r_{23}^{\prime}=-1$ and $r_{14}^{\prime}=r_{24}^{\prime}=0$, players $X_{1}$ and $X_{2}$ have the same results against the same opponents, hence $f_{1}(N,R^{\prime},M^{\prime})=f_{2}(N,R^{\prime},M^{\prime})$ from $SC$. 2. b)

Consider the identity one-to-one correspondence $h_{34}^{\prime}:O_{3}^{\prime}\leftrightarrow O_{4}^{\prime}$. Since $1=r_{31}^{\prime}>r_{41}^{\prime}=0$ and $1=r_{32}^{\prime}>r_{42}^{\prime}=0$, player $X_{3}$ has better results against the same opponents compared to $X_{4}$, hence $f_{3}(N,R^{\prime},M)>f_{4}(N,R^{\prime},M)$ from $SC$.

Thus self-consistency leads to $f_{1}(N,R^{\prime},M^{\prime})=f_{2}(N,R^{\prime},M^{\prime})$ and $f_{3}(N,R^{\prime},M^{\prime})>f_{4}(N,R^{\prime},M^{\prime})$ in the second ranking problem. 3. III.

Take the sum of these two ranking problems, the ranking problem $(N,R^{\prime\prime},M^{\prime\prime})$.

Suppose that $f_{1}(N,R^{\prime\prime},M^{\prime\prime})\geq f_{2}(N,R^{\prime\prime},M^{\prime\prime})$. Consider the one-to-one mappings $g_{21}:O_{2}\leftrightarrow O_{1}$ and $g_{21}^{\prime}:O_{2}^{\prime}\leftrightarrow O_{1}^{\prime}$ such that $g_{21}(X_{1})=X_{2}$, $g_{21}(X_{3})=X_{4}$ and $g_{21}^{\prime}(X_{3})=X_{3}$, $g_{21}^{\prime}(X_{4})=X_{4}$. Since $r_{21}=r_{12}=0$, $r_{23}=r_{14}=0$ and $r_{23}^{\prime}=r_{13}^{\prime}=-1$, $r_{24}^{\prime}=r_{14}^{\prime}=0$, player $X_{2}$ has the same results against stronger opponents compared to $X_{1}$, hence $f_{2}(N,R^{\prime\prime},M^{\prime\prime})>f_{1}(N,R^{\prime\prime},M^{\prime\prime})$ from $SC$, which leads to a contradiction.

To summarize, self-consistency results in $f_{1}(N,R^{\prime\prime},M^{\prime})<f_{2}(N,R^{\prime\prime},M^{\prime\prime})$, however, order preservation implies $f_{1}(N,R^{\prime\prime},M^{\prime\prime})=f_{2}(N,R^{\prime\prime},M^{\prime\prime})$ as all players have played two matches in $(N,R^{\prime},M^{\prime})$ and $(N,R^{\prime},M^{\prime})$, respectively, which is impossible.

Therefore, it has been derived that no scoring method can meet $OP$ and $SC$ simultaneously on the universal domain of $\mathcal{R}^{n}$. ∎

Theorem 4.1 is a serious negative result: by accepting self-consistency, the ranking method cannot be required to preserve two players’ pairwise ranking when some ranking problems, where all players have played the same number of matches, are aggregated.

Example 4.2**.**

Let $(N,R,M)\in\mathcal{R}^{4}$ be the ranking problem in Figure 3: $X_{1}$ has drawn against $X_{2}$, $X_{2}$ against $X_{3}$ and $X_{3}$ against $X_{4}$.

Theorem 4.1 would be more straightforward as a strengthening of Proposition 4.2 if self-consistency implies neutrality and/or symmetry. However, it is not the case as the following result holds.

Remark 4.2*.*

There exists a scoring method that is self-consistent, but not neutral and symmetric.

Proof.

The statement can be verified by an example where an $SC$-compatible scoring method violates $NEU$ and $SYM$.

Consider Example 4.2 with a scoring method $f$ such that $f_{1}(N,R,M)>f_{2}(N,R,M)>f_{3}(N,R,M)>f_{4}(N,R,M)$, for example, player $X_{i}$ gets the score $4-i$. $f$ meets self-consistency since $X_{1}$ has the same result against a stronger opponent compared to $X_{4}$, while there exists no correspondence between opponent sets $O_{2}$ and $O_{3}$ satisfying the conditions of $SC$.

Let $\sigma:N\to N$ be a permutation such that $\sigma(X_{1})=X_{4}$, $\sigma(X_{2})=X_{3}$, $\sigma(X_{3})=X_{2}$, and $\sigma(X_{4})=X_{1}$. Since $\sigma(N,R,M)=(N,R,M)$, $NEU$ implies $f_{4}(N,R,M)>f_{1}(N,R,M)$ and $f_{3}(N,R,M)>f_{2}(N,R,M)$, a contradiction. Furthermore, $SYM$ leads to $f_{1}(N,R,M)=f_{2}(N,R,M)=f_{3}(N,R,M)=f_{4}(N,R,M)$, another impossibility. Therefore there exists a self-consistent scoring method, which is not neutral and symmetric. ∎

5 Conclusions

We have found some unexpected implications of different properties in the case of generalized tournaments where the players should be ranked on the basis of their match results against each other. First, self-consistency prohibits the use of individual scoring methods, that is, scores cannot be derived before the aggregation of tournament rounds (Proposition 3.1). Second, independence of irrelevant matches (posing a kind of independence concerning the pairwise ranking of two players) follows from three axioms, neutrality (independence of relabelling the players), symmetry (implying a flat ranking if all aggregated comparisons are draws), and strong order preservation (perhaps the most natural property concerning the aggregation of ranking problems). According to Csató, 2018b , there exists no scoring method satisfying self-consistency and independence of irrelevant matches, hence Proposition 4.1 implies that neutrality, symmetry, strong order preservation and self-consistency cannot be met simultaneously (Proposition 4.2). It even turns out that self-consistency and a weaker version of strong order preservation are still enough to derive this negative result (Theorem 4.1), consequently, one should choose between these two natural fairness requirements.

What do our results say to practitioners who want to rank players or teams? First, self-consistency does not allow to rank them in individual rounds, one has to wait until all tournament results are known and can be aggregated. Second, self-consistency is not compatible with order preservation on this universal domain. It is not an unexpected and counter-intuitive result as, according to González-Díaz et al., (2014), a number of ranking methods violate order preservation. We have proved that there is no hope to find a reasonable scoring method with this property. From a more abstract point of view, breaking of order preservation in tournament ranking is a version of Simpson’s paradox, a phenomenon in probability and statistics, in which a trend appears in different groups of data but disappears or reverses when these groups are combined.555 We are grateful to an anonymous referee for this remark. This negative result holds despite self-consistency is somewhat weaker than our intuition suggests: it does not imply neutrality and symmetry, so even a self-consistent ranking of players may depend on their names and without ties if all matches are drawn (Remark 4.2). Third, losing the simplicity provided by order preservation certainly does not facilitate the axiomatic construction of scoring methods.

Consequently, while sacrificing self-consistency or order preservation seems to be unavoidable in our general setting, an obvious continuation of the current research is to get positive possibility results by some domain restrictions or further weakening of the axioms. It is also worth to note that the incompatibility of the two axioms does not imply that any scoring method is always going to work badly, but all can lead to problematic results at times.

Acknowledgements

We are grateful to Sándor Bozóki for useful advice.

Anonymous reviewers provided valuable comments and suggestions on earlier drafts.

The research was supported by OTKA grant K 111797 and by the MTA Premium Post Doctorate Research Program.