Venue Analytics: A Simple Alternative to Citation-Based Metrics

TL;DR

This paper introduces a venue scoring method based solely on publication data, enabling large-scale, transparent evaluation of venues, authors, and institutions over time without relying on citation metrics.

Contribution

It presents a novel venue evaluation approach using regression on publication data, applicable across decades and scalable to large datasets, enhancing transparency and repeatability.

Findings

Venue scores correlate with traditional impact measures

Method enables temporal analysis of conference quality

Applicable to large, open datasets for broad evaluation

Abstract

We present a method for automatically organizing and evaluating the quality of different publishing venues in Computer Science. Since this method only requires paper publication data as its input, we can demonstrate our method on a large portion of the DBLP dataset, spanning 50 years, with millions of authors and thousands of publishing venues. By formulating venue authorship as a regression problem and targeting metrics of interest, we obtain venue scores for every conference and journal in our dataset. The obtained scores can also provide a per-year model of conference quality, showing how fields develop and change over time. Additionally, these venue scores can be used to evaluate individual academic authors and academic institutions. We show that using venue scores to evaluate both authors and institutions produces quantitative measures that are comparable to approaches using citations or peer assessment. In contrast to many other existing evaluation metrics, our use of large-scale, openly available data enables this approach to be repeatable and transparent. To help others build upon this work, all of our code and data is available at https://github.com/leonidk/venue_scores

Tables

Table 1. Table 1. Spearman’s ρ 𝜌 \rho correlation between rankings produced by targeting different metrics of interest.

	Faculty	NSF	Salary
Faculty	1.00	0.91	0.84
NSF	0.91	1.00	0.86
Salary	0.84	0.86	1.00

Table 2. Table 2. Spearman correlation between our model and existing metrics, showing the effect of different normalization schemes. See Sections 4.4 and 4.5 for model details. See Section  6 for evaluation details.

Normalization	influential citations (author)	h-index (author)	h-index (university)	h-index (venue)
None	0.72	0.61	0.60	0.26
Year	0.72	0.66	0.58	0.18
Size	0.74	0.63	0.61	0.25
Year + Size	0.72	0.61	0.60	0.26

Table 3. Table 3. Correlation (Spearman’s ρ 𝜌 \rho ) between our model and Semantic Scholar  (Valenzuela et al . , 2015 ) , showing the properties of different authorship models. For details see section  4.6 .

		Evaluation Author Model
		1	2	3	4
Regression Author Model	1	0.70	0.72	0.65	0.70
	2	0.68	0.71	0.61	0.67
	3	0.71	0.73	0.66	0.71
	4	0.70	0.72	0.65	0.71

Table 4. Table 4. Correlation between our model and traditional measures of scholarly output on the dataset of CMU faculty. For model details see Section  4.7 . For evaluation details see Section  6.4 .

Model	citations	h-index (Hirsch, 2005)	influential citations (Valenzuela et al., 2015)
Faculty	0.59	0.68	0.71
NSF	0.63	0.66	0.67
Salary	0.36	0.36	0.41
Combined	0.69	0.77	0.75

Table 5. Table 5. Correlation between rankings produced by our model against rankings produced by traditional scholarly metrics. Different rows correspond to different hyperparameters choices for our model. Each column is a corresponds to a traditional metric. AI = Author Highly Influential Citations, AH = Author H-index, USN = US News 2018, VH = Venue H-index, VC = Venue Citations.

Years	Metric	AI	AH	USN	VH	VC
$\sigma =4.5$	Faculty	0.73	0.69	0.74	0.63	0.42
10		0.67	0.57	0.76	0.57	0.35
50		0.75	0.68	0.76	0.38	0.21
$\sigma =4.5$	NSF	0.64	0.62	0.62	0.61	0.59
10		0.68	0.68	0.60	0.59	0.60
50		0.67	0.65	0.63	0.64	0.67
$\sigma =4.5$	Salary	0.62	0.58	0.59	0.48	0.55
10		0.65	0.62	0.57	0.45	0.55
50		0.66	0.63	0.56	0.43	0.63

Table 6. Table 6. Section  6.2 . Kendall’s τ 𝜏 \tau correlation across different University Rankings.

Ranking	Correlation with US News 2018
USN2018 (News and Report, 2018)	1.000
USN2010 (Clauset et al., 2015)	0.928
Venue Scores	0.780
ScholarRank (Vucetic et al., 2018)	0.768
ScholarRankFull	0.757
CSMetrics (Blackburn et al., 2018)	0.746
CSRankings (Berger, 2018)	0.724
Times (Education, 2018)	0.721
NRC95 (Clauset et al., 2015)	0.713
t10Sum (Vucetic et al., 2018)	0.713
Prestige (Clauset et al., 2015)	0.666
Citations (Vucetic et al., 2018)	0.665
Shanghai (Rankings, 2015)	0.586
# of papers	0.585
BestPaper (Huang, 2018)	0.559
PageRankA	0.535
PageRankC	0.532
QS (Rankings, 2018)	0.518

Table 7. Table 7. Spearman’s ρ 𝜌 \rho correlation between different journal-level metrics (N=1008). For details see Section  6.3 .

	papers	citations	h-index	PageRank C	venue scores
papers		0.75	0.41	0.91	0.25
citations	0.75		0.60	0.69	0.27
h-index	0.41	0.60		0.37	0.65
PageRankC	0.91	0.69	0.37		0.27
venue scores	0.25	0.27	0.65	0.27

Table 8. Table 8. Correlation between different author-level metrics for a dataset of professors (N=148). Details are in Section  6.4 .

	papers	citations	h-index	i10	CSR (Berger, 2018)	venue score	PageRankA	PageRankC	influence (Valenzuela et al., 2015)
papers		0.66	0.79	0.81	0.71	0.94	0.94	0.89	0.76
citations	0.66		0.93	0.88	0.49	0.66	0.68	0.60	0.81
h-index	0.79	0.93		0.97	0.56	0.75	0.81	0.68	0.80
i10	0.81	0.88	0.97		0.53	0.75	0.82	0.69	0.73
CSR (Berger, 2018)	0.71	0.49	0.56	0.53		0.84	0.64	0.80	0.64
venue score	0.94	0.66	0.75	0.75	0.84		0.86	0.92	0.78
PageRankA	0.94	0.68	0.81	0.82	0.64	0.86		0.83	0.72
PageRankC	0.89	0.60	0.68	0.69	0.80	0.92	0.83		0.67
influence (Valenzuela et al., 2015)	0.76	0.81	0.80	0.73	0.64	0.78	0.72	0.67

Table 9. Table 9. Top 55 Venues

Name	Score	Size
STOC	11.71	128
FOCS	11.04	120
NIPS	10.94	484
Conference on Computational Complexity	10.41	53
SODA	10.17	224
SIAM J. Comput.	9.98	138
HotNets	9.53	46
ICML	9.46	303
TCC	9.06	76
NSDI	8.76	69
CCS	8.76	114
INFOCOM	8.59	434
ITCS	8.56	103
Allerton	8.53	331
ASPLOS	8.43	48
SIGCOMM	8.31	55
CVPR	8.22	606
PLDI	8.05	68
J. ACM	8.03	82
Theory of Computing	7.94	29
USENIX Security Symposium	7.91	62
NDSS	7.86	63
APPROX-RANDOM	7.86	74
COLT	7.78	82
VLDB	7.75	185
IEEE/ACM Trans. Netw.	7.74	226
AAAI	7.73	403
SIGMOD Conference	7.62	98
SIGGRAPH	7.59	125
AISTATS	7.52	138
OOPSLA	7.41	75
SIGGRAPH Asia	7.38	140
CRYPTO	7.24	80
EUROCRYPT	7.24	78
POPL	7.22	68
CHI	7.15	431
IEEE SSP	7.11	54
WWW	6.82	232
HotOS	6.75	27
EMNLP	6.61	263
EC	6.60	63
Commun. ACM	6.59	222
GetMobile	6.50	41
IMC	6.49	70
OSDI	6.46	29
ICDE	6.44	183
SIGSOFT FSE	6.42	93
ECCV	6.42	252
ICALP	6.40	148
ISIT	6.28	1032
KDD	6.23	240
ICCV	6.22	292
Robotics: Science and Systems	6.20	88
MICRO	6.19	62
ISCA	6.18	73

Table 10. Table 10. Top 55 Universities

school	authors	papers	venue score	size normed
Carnegie Mellon University	174	17275	73126	14159
University of California - Berkeley	108	11011	51062	10884
Massachusetts Institute of Technology	108	9613	47037	10026
Univ. of Illinois at Urbana-Champaign	103	11248	44714	9627
Technion	96	8795	39601	8656
Stanford University	69	7028	36169	8513
Georgia Institute of Technology	108	9337	38083	8118
Tsinghua University	150	14643	40662	8104
University of California - Los Angeles	46	6589	30368	7888
University of Michigan	83	7897	33666	7598
Tel Aviv University	48	5468	28816	7404
University of California - San Diego	74	6646	30836	7142
University of Maryland - College Park	76	7751	30795	7089
ETH Zurich	39	6673	26120	7081
Cornell University	81	5918	30278	6871
University of Washington	69	5727	29087	6846
University of Southern California	58	7206	26044	6387
Columbia University	53	5064	25251	6330
EPFL	55	6927	25320	6290
Princeton University	47	4991	24203	6252
HKUST	57	6640	25134	6190
National University of Singapore	75	7210	25123	5801
University of Pennsylvania	53	5340	22696	5690
University of California - Irvine	71	6624	24070	5628
Pennsylvania State University	49	5668	21325	5451
Peking University	147	11858	27050	5413
University of Toronto	99	5955	24759	5376
University of Texas at Austin	52	4485	21225	5346
University of California - Santa Barbara	38	4698	19137	5224
University of Waterloo	105	7316	23781	5099
Max Planck Institute	32	4307	17808	5093
Purdue University	71	5659	21734	5082
New York University	67	4915	20752	4918
Rutgers University	63	4935	20311	4884
Nanyang Technological University	71	9143	20827	4870
University of Wisconsin - Madison	54	3988	18986	4738
University of Massachusetts Amherst	57	4129	19155	4717
Hebrew University of Jerusalem	42	3316	17478	4647
University of California - Riverside	43	4230	17114	4522
Shanghai Jiao Tong University	77	8220	19652	4511
Zhejiang University	89	7531	20232	4496
Ohio State University	42	4121	16740	4451
Duke University	23	2879	13936	4385
Chinese Academy of Sciences	46	5795	16497	4285
University of Minnesota	43	4346	15855	4190
Northwestern University	49	3993	16183	4137
Stony Brook University	47	4565	15819	4086
University of Edinburgh	115	6461	19292	4058
University of Tokyo	80	7524	17762	4042
University of Oxford	69	5549	17160	4039
Chinese University of Hong Kong	32	3927	13843	3959
TU Munich	53	6432	15521	3891
KAIST	81	5591	17114	3884
Harvard University	34	2471	13640	3837
Imperial College London	65	6622	15834	3779

Table 11. Table 11. Top 50 Authors

Name	Score
Philip S. Yu	4354
H. Vincent Poor	3873
Kang G. Shin	3540
Jiawei Han 0001	3143
Micha Sharir	3086
Thomas S. Huang	2992
Don Towsley	2913
Xuemin Shen	2755
Luc J. Van Gool	2700
Noga Alon	2596
Christos H. Papadimitriou	2531
Leonidas J. Guibas	2525
Mahmut T. Kandemir	2467
Jie Wu 0001	2452
Wen Gao 0001	2416
Shuicheng Yan	2373
Rama Chellappa	2286
Alberto L. Sangiovanni-Vincentelli	2266
Georgios B. Giannakis	2266
Yunhao Liu	2257
Mohamed-Slim Alouini	2238
Michael I. Jordan	2213
Christos Faloutsos	2173
Massoud Pedram	2153
Dacheng Tao	2149
Hans-Peter Seidel	2145
Luca Benini	2139
Moshe Y. Vardi	2092
Lajos Hanzo	2081
Yishay Mansour	2073
Sudhakar M. Reddy	2070
Sajal K. Das	2045
Pankaj K. Agarwal	2042
Avi Wigderson	2024
Robert E. Tarjan	2019
Irith Pomeranz	2003
Elisa Bertino	2003
Zhu Han	1992
Shlomo Shamai	1976
David Peleg	1974
Victor C. M. Leung	1966
Kaushik Roy 0001	1949
Xiaoou Tang	1938
Hector Garcia-Molina	1923
Larry S. Davis	1918
Rafail Ostrovsky	1895
Eric P. Xing	1881
Ness B. Shroff	1879
Krishnendu Chakrabarty	1868
Hai Jin 0001	1867

Equations

$$\bordermatrix{&\text{conf}_{1}&\text{conf}_{2}&\ldots&\text{conf}_{n}&\cr\text{auth}_{1}&1&3&\ldots&0&1\cr\text{auth}_{2}&1&0&\ldots&1&1\cr\vdots&\vdots&\vdots&\ddots&\vdots&\vdots\cr\text{auth}_{m}&0&2&\ldots&4&1}\begin{pmatrix}x_{0}\\ x_{1}\\ \vdots\\ x_{n}\end{pmatrix}=\begin{pmatrix}\text{isProf}_{1}\\ \text{isProf}_{2}\\ \vdots\\ \text{isProf}_{m}\end{pmatrix}$$

$$L(Ax,b)+\lambda ||x|{|}^2$$

$$L(\hat{y},y)=\{\begin{array}{ll}\frac{1}{2}(y-\hat{y}{)}^2,& \text{if }|y-\hat{y}|\le \delta \\ \delta |y-\hat{y}|-\frac{1}{2}{\delta }^2,& \text{otherwise}\end{array}$$

$$L(\hat{y},y)=\{\begin{array}{ll}\max (0,1-y\hat{y}{)}^2,& \text{if }y\hat{y}\ge -1\\ -4y\hat{y},& \text{otherwise}\end{array}$$

$$\bordermatrix{&\text{author}_{1}&\text{author}_{2}&\ldots&\text{author}_{n}\cr\text{paper}_{1}&1&1&\ldots&0\cr\text{paper}_{2}&1&0&\ldots&0\cr\vdots&\vdots&\vdots&\ddots&\vdots\cr\text{paper}_{m}&0&1&\ldots&1}\begin{pmatrix}x_{1}\\ x_{2}\\ \vdots\\ x_{n}\end{pmatrix}=\begin{pmatrix}\text{score}_{1}\\ \text{score}_{2}\\ \vdots\\ \text{score}_{n}\end{pmatrix}$$

Full text

To help others build upon this work, all of our code and data is available at https://github.com/leonidk/venue_scores.

††copyright: none

1. Introduction

There exist many tools to evaluate professional academic scholarship. For example Elseiver’s Scorpus provides many author-level and journal-level metrics to measure the impact of scholars and their work (Colledge et al., 2010; da Silva and Memon, 2017). Other publishers, such as the Public Library of Science, provide article-level metrics for their published work (Fenner, 2013). Large technology companies, such as Google and Microsoft provide their own publicly available metrics for scholarship (Butler, 2011). Even independent research institutes, such as the Allen Institute’s Semantic Scholar (Ammar et al., 2018), manage their own corpus and metrics for scholarly productivity. However, these author-based metrics (often derived from citation measurements) can be inconsistent, even across these large, established providers (da Silva and Dobránszki, 2018).

In this work, we propose a method for evaluating a comprehensive collection of published academic work by using an external evaluation metric. By taking a large collection of papers and using only information about their publication venue, who wrote them, and when, we provide a largely automated way of discovering not only venue’s value. Further, we also develop a system for automatic organization of venues. This is motivated by the desire for an open, reproducible, and objective metric that is not subject to some of the challenges inherent to citation-based methods (da Silva and Dobránszki, 2018; Bornmann and Daniel, 2008; Galiani and Gálvez, 2017).

We accomplish this by setting up a linear regression from a publication record to some metric of interest. We demonstrate three valid regression targets: status as a faculty member (a classification task), awarded grant amounts, and salaries. By using DBLP (Ley, 2002) as our source of publication data, NSF grants as our source of awards, University of California data for salaries, and CSRankings (Berger, 2018) for faculty affiliation status, we’re able to formulate these as large data regression tasks, with design matrix dimensions on the order of a million in each dimension. However, since these matrices are sparse, regression weights can be obtained efficiently on a single laptop computer. Details of our method are explained in section 4.

We call our results venue scores and validate their performance in the tasks of evaluating conferences, evaluating professors, and ranking universities. We show that our venue scores correlate highly with other influence metrics, such as h-index (Hirsch, 2005), citations or highly-influential citations (Valenzuela et al., 2015). Additionally, we show that university rankings, derived from publication records correlate highly with both established rankings (News and Report, 2018; Education, 2018; Rankings, 2018) and with recently published quantitative metrics (Berger, 2018; Blackburn et al., 2018; Vucetic et al., 2018; Clauset et al., 2015).

2. Related Work

Quantitative measures of academic productivity tend to focus on methods derived from citation counts. By using citation count as the primary method of scoring a paper, one can decouple an individual article from the authors who wrote it and the venue it was published in. Then, robust citation count statistics, such as h-index (Hirsch, 2005), can be used as a method of scoring either individual authors, or a specific venue. Specific critiques of h-index scores arose almost as soon as the h-index was published, ranging from a claimed lack-of-utility (Lehmann et al., 2006), to a loss of discriminatory power (Tol, 2008).

Citations can also be automatically analyzed for whether or not they’re highly influential to the citing paper, producing a ”influential citations” metric used by Semantic Scholar (Valenzuela et al., 2015). Even further, techniques from graph and network analysis can be used to understand systematic relationships in the citation graph (Bergstrom, 2007). Citation-based metrics can even be used to provide a ranking of different universities (Blackburn et al., 2018).

Citations-based metrics, despite their wide deployment in the scientometrics community, have several problems. For one, citation behavior varies widely by fields (Bornmann and Daniel, 2008). Additionally, citations often exhibits non-trivial temporal behavior (Galiani and Gálvez, 2017), which also varies greatly by sub-field. These issues highly affect one’s ability to compare across disciplines and produce different scores at different times. Recent work suggests that citation-based metrics struggle to effectively capture a venue’s quality with a single number (Walters, 2017). Comparing citation counts with statistical significance requires an order-of-magnitude difference in the citation counts (Kurtz and Henneken, 2017), which limits their utility in making fine-grained choices. Despite these quality issues, recent work (Vucetic et al., 2018) has demonstrated that citation-based metrics can be used to build a university ranking that correlates highly with peer assessment; we show that our method provides a similar quality of correlation.

Our use of straightforward publication data (Section 3) enables a much simpler model. This simplicity is key, as the challenges in maintaining good citation data have resulted in the major sources of h-index scores being inconsistent with one another (da Silva and Dobránszki, 2018).

While there exist many forms of ranking journals such as Eigenfactor (Bergstrom, 2007) or SJR (Falagas et al., 2008), these tend to focus on journal-level metrics while our work focuses on all venues, including conferences.

2.1. Venue Metrics

We are not the first to propose that scholars and institutions can be ranked by assigning scores to published papers(Ren and Taylor, 2007). However, in prior work the list of venues is often manually curated and assigned equal credit, a trend that is true for studies in 1990s (Geist et al., 1996) and their modern online versions (Berger, 2018). Instead, we propose a method for obtaining automatic scores for each venue, and in doing so, requires no manual curation of valid venues.

Previous work (Yan and Lee, 2007) has developed methods for ranking venues automatically, generating unique scores based only on author data by labeling and propagating notions of ”good” papers and authoritative authors. However, this work required a manually curated seed of what good work is. It was only demonstrated to work on a small sub-field of conferences, as new publication cliques would require new labeling of ”good” papers. Recent developments in network-based techniques for ranking venues have included citation information (Zhang and Wu, 2018), and are able to produce temporal models of quality. In contrast, our proposed model doesn’t require citation data to produce sensible venue scores.

Our work, in some ways, is most similar to that of CSRankings (Berger, 2018). CSRankings maintains a highly curated set of top-tier venues; venues selected for inclusion are given 1 point per paper, while excluded venues are given 0 points. Additionally, university rankings produced by CSRankings include a manually curated set of categories, and rankings are produced via a geometric mean over these categories. In comparison, in this work, we produce unique scores for every venue and simply sum together scores for evaluating authors and institutions.

In one of the formulations of our method, we use authors status as faculty (or not faculty) to generate our venue scores. We are not alone in this line of analysis, as recent work has demonstrated that faculty hiring information can be used to generate university prestige rankings (Clauset et al., 2015).

Many existing approaches either focus only on journals (Falagas et al., 2008), or do not have their rankings available online. For our dataset, we do not have citation-level data available, so we are unable to compare against certain existing methods on our dataset. However, as these methods often deploy a variant of PageRank (Page et al., 1999), we describe a PageRank baseline in Section 6.1 and report its results.

3. Data

Our primary data is the dblp computer science bibliography (Ley, 2002). DBLP contains millions of articles, with millions of authors across thousands of venues in Computer Science. We produced the results in this paper by using the dblp-2019-01-01 snapshot. We restricted ourselves to only consider conference and journal publications, skipping books preprints, articles below 6 pages and over 100 pages. We also merged dblp entries corresponding to the same conference. This lead to a dataset featuring 2,965,464 papers, written by 1,766,675 authors, across 11,255 uniquely named venues and 50 years of publications (1970 through 2019).

Our first metric of interest is an individual’s status as a faculty member at a university. For this, we used the faculty affiliation data from CSRankings (Berger, 2018), which are manually curated and contain hundreds of universities across the world and about 15,000 professors. For evaluation against other university rankings, we used the ScholarRank (Vucetic et al., 2018) data to obtain faculty affiliation, which contains a more complete survey of American universities (including more than 50 not currently included in CS Rankings). While CSRankings data is curated to have correct DBLP names for faculty, the ScholarRank data does not. To obtain a valid affiliation, the names were automatically aligned with fuzzy string matching, resulting in about 4,000 faculty with good seemingly unique DBLP names and correct university affiliation. Although those two methods are manually curated, automatic surveys of faculty affiliations have recently been demonstrated (Morgan et al., 2018b).

Our second metric of interest was National Science Foundation grants, where we used awards from 1970 until 2018. This data is available directly from the NSF (Foundation, 2018). We adjusted award amounts using annual CPI inflation data. We restricted ourselves to awards that had finite amount, where we could match at least half the Principal Investigators on the grant to DBLP names and the grant was above $20,000. Award amounts over 10 million dollars were clipped in a smooth way to avoid matching to a few extreme outliers. This resulted in 407,012 NSF grants used in building our model.

Our third and final metric of interest was University of California salary data (Institute, 2018). This was inspired by a paper that predicted ACM/IEEE Fellowships for 87 professors and used salary data (Nocka et al., 2014). We looked at professors across the entire University of California system, matching their names to DBLP entries in an automated way. We used the maximum salary amount for a given individual across the 2015, 2016 and 2017 datasets, skipping individuals making less than $120,000$ or over $800,000$ dollars. This resulted in 2,436 individuals, down from 3,102 names that we matched and an initial set of about 20,000 initial professors. As DBLP contains some Chemistry, Biology, and Economics venues, we expect that some of these are likely not Computer Science professors.

4. Method

Our basic model is that a paper in a given publication venue (either a conference or a journal), having passed peer review, is worth a certain amount of value. Certain venues are more prestigious, impactful or selective, and thus should have higher scores. Other venues have less strict standards, or perhaps provide less opportunity to disseminate their ideas (Morgan et al., 2018a) and should be worth less. While this model explicitly ignores the differences in paper quality at a publication venue, discarding this information enables the use of a large quantity of data to develop a statistical scoring system.

This methodology is not valid for all fields of science, nor all models of how impactful ideas are developed and disseminated. Our method requires individual authors have multiple publications across many different venues, which is more true in Computer Science than the natural sciences or humanities, where publishing rates are lower (Lubienski et al., 2018). If instead we assume that all research ideas produce only a single paper, or that passing peer-review is a noisy measurement of quality (Shah et al., 2018), then our proposed method would not work very well. Instead, the underlying process which supports our methodology is that good research ideas produce multiple research publications in selective venues; better ideas would produce more individual publications in higher quality venues. The concept of All models are wrong, but some are useful is our guide here. This assumption allows us to obtain venue scores in an automatic way, and then use these scores to evaluate both authors and institutions.

Our approach to ranking venues is to construct a regression task, apply an optimization method to solve it, and use the resulting weights. Optimization’s ability to generate powerful representations is well-studied in machine learning (Rumelhart et al., 1986) and statistics (Huber, 1985). The resulting weights are often useful (Coates and Ng, 2011), and this methodology is widely used in natural language processing (Mikolov et al., 2013; Pennington et al., 2014).

4.1. Formal Setup

In general, we will obtain a score for each venue by setting up a linear regression task in the form of equation 1.

[TABLE]

Authors are listed along the rows, and venues are listed along the columns. The number of publications that an author has in a conference is noted in the design matrix. There is an additional column of 1s to learn a bias offset in the regression.

Different forms of counting author credit are discussed in section 4.6, while different regression targets are discussed in section 3. In equation 1, the regression target is shown as a binary variable indicating whether or not that author is currently a professor. If this linear system, $Ax=b$ is solved, then the vector $x$ will contain real-valued scores for every single publishing venue. Since our system is over-determined, there is generally no exact solution.

Instead of solving this sparse linear system directly, we instead solve a regularized regression, using a robust loss and $L_{2}$ regularization. That is, we iteratively minimize the following expression via stochastic gradient descent (Robbins and Monro, 1951)

[TABLE]

The $L_{2}$ regularization enforces a Gaussian prior on the learned conference scores. We can perform this minimization in Python using common machine learning software packages (Pedregosa et al., 2011). We tend to use a robust loss function, such as the Huber loss in the case of regression (Huber, 1964), which is quadratic for errors of less than $\delta$ and linear for errors of larger than $\delta$. It can be written as

[TABLE]

In the case of classification, we have labels $y\in\{-1,1\}$ and use the modified Huber loss (Zhang, 2004),

[TABLE]

We experimented with other loss functions, such as the logistic loss, and while they tended to produce similar rankings and results, we found that the modified Huber loss provided better empirical performance in our test metrics, even though the resulting curves looked very similar upon qualitative inspection.

4.2. Metrics of Interest

As detailed in section 3, we targeted three metrics of interest: status as a faculty member, NSF award sizes, and professor salaries. Each of these metrics came from a completely independent data source, and we found that they each had their own biases and strengths (more in section 5).

For faculty status classification, we used the modified huber loss and CSRankings (Berger, 2018) faculty affiliations. To build venue scores that reward top-tier conferences more highly, we only gave professors in the top-$k$ ranked universities positive labels. We tried $k=5,16,40,80$, and found that we got qualitatively different results with quantitative similar performance. Unless otherwise stated, we used $k=40$. The university ranking used to select top-$k$ was CSRankings itself, and included international universities, covering the Americas, Europe and Asia. This classification is performed across all authors, leading to 1.7 million rows in our design matrix.

For the NSF awards, every Principal Investigator on the award had their papers up to the award year as the input features. We used a Huber loss, $\lambda=0.03$, and experimented with normalizing our award data to have zero mean and unit variance. Additionally, we built models for both raw award sizes and log of award sizes; the raw award sizes seem to follow a power-law while the log award sizes seem distributed as approximately a Gaussian. Another model choice is whether to regress each NSF grant as an independent measurement or instead a marginal measurement which tracks the cumulative total of NSF grants received by the authors. If not all authors on the grant were matched to DBLP names, we only used the fraction of the award corresponding to the fraction of identified authors. This regression had $\sim\frac{1}{2}$ million rows in its design matrix .

For the salary data, we found that normalizing the salary data to have zero mean and unit variance led to a very poor regression result, while having no normalization produced a good result. This regression only had $\sim 2,400$ datapoints, and thus provided information about fewer venues than the other metrics of interest.

4.3. Modeling Change Over Time

In modeling conference values, we wanted to build a model that could adjust for different values in different years. For example, a venue may be considered excellent in the 1980s, but may have declined in influence and prestige since then. To account for this behavior, we break our dataset into chunks of $n$ years and create a different regression variable for each conference for each chunk. The non-temporal model is obtained simply by setting $n\geq 50$.

We also examine a model that creates an independent variable for each year, for each conference. After setting $n=1$ in the block model, we splat each publication as a Gaussian at a given year. By modifying $\sigma$, we can control the smoothness of the obtained weights. The Gaussian is applied via a sparse matrix multiply of the design matrix $A$ with the appropriate band diagonal sparse matrix $G$. The use of a truncated Gaussian (where $p<0.05$ is clipped and the Gaussian is re-normalized) enables our matrix to maintain sparsity. We used a $\sigma=4.5$, which produced an effective window size of about 10 years. This can be seen visually in Figure 2.

Our different temporal models are compared in Table 5 by correlating against existing author-level, journal-level and university-level metrics. For evaluation details see Section 6.

4.4. Normalizing Differences Across Years

The temporal models described in the previous section have an inherent bias. Due to the temporal window of the DBLP publication history, there is variation in distributed value due to changes annual NSF funding, the survivorship of current academic faculty, etc. To adjust for this bias, we scale each conference-year value by the standard deviation of conference values in that year. This scaling can help or hurt performance, depending on which metric of interest the model is built against. It generally produces flatter value scores over time but leads to some artifacts. The effects of this normalization are shown in Figure 3 and Table 2. In our final version, we instead normalize by the average of the top 10 conferences in each year, which produced flatter results over time,

4.5. Normalizing Differences In Venue Size

Our model uses $L_{2}$ regularization on the venue scores, which tends to squash the value of variables with less explanatory power.This process often resulted in the under-valuing of smaller, more selective venues. To correct for this size bias, instead of giving each paper $1$ point of value in the design matrix, we give each paper $\frac{1}{M}$ credit, where $M$ is the number of papers at that venue in that year; this step is performed before Gaussian splatting. This produced rankings which emphasized small venues and provided a good notion of efficient venues. In practice, we instead used a blended result with a multiplier of $\frac{1}{\sqrt[\alpha]{M}}$ with $\alpha=1.5849$, the Hausdorff dimension of the Sierpinski triangle, an arbitrary constant.

4.6. Modeling Author Position

Another question to consider is how credit for a paper is divided up amongst the authors. We consider four models of authorship credit assignment:

1. (1)

Authors get $\frac{1}{n}$ credit for each paper, where $n$ is the number of authors on the paper. Used by  (Berger, 2018). 2. (2)

All authors get full credit (1 point) for each paper 3. (3)

Authors receive less credit for later positions ($\frac{1}{1},\frac{1}{2},\frac{1}{3}\cdots\frac{1}{n}$), normalized so total credit sums to 1). This model assigns more credit to earlier authors and is used by certain practitioners (Sekercioglu, 2008; Huang, 2018). 4. (4)

The same as (3), except the last author is explicitly assigned equal credit with the first author before normalization.

Using Spearman correlation with Semantic Scholar’s ”highly influential citations”, an evaluation metric described in depth in Sec. 6.4, we can evaluate each of these models. Specifically, there are two places where venues scores require a selection of authorship model. The first is how much credit is assigned to each paper when performing regression (in the case of our faculty metric of interest). The second is when evaluating authors with the obtained regression vector. See table 3 for a summary of experimental results.

For the purposes of evaluation, assigning full credit to authors (model 2) produced the best results, while model 3 consistently produced the lowest quality correlations. On the other hand, for the purposes of performing the classification task, the roles are flipped. Assigning full credit (model 2) consistently produces the worst quality correlations while using model 3 produces the highest quality correlations.

4.7. Combining Models

Since the proposed metrics of interest (faculty status, NSF awards, salaries) were generated from different independent regression targets, with different sized design matrices, there may be value in combining them to produce a joint model. The value of ensemble models is well documented in both theory (Freund and Schapire, 1997) and practice (Bell et al., 2009). In the absence of a preferred metric with which to cross-validate our model, we simply perform an unweighted average of our models to obtain a gold model. To ensure that the weights are of similar scale, the conference scores are normalized to have zero mean and unit variance before combining them. Venue scores that are too large or too small are clipped at 12 standard deviations. For the temporal models, this normalization is performed on a per-year basis. While table 4 shows results for a simple combination, one could average together many models with different choices of hyperparameters, regression functions, datasets, filters to scrub the data, etc.

5. Results

A visual example of some of venue scores is shown in Figure 1. We kept the y-axis fixed across all the different Computer Science sub-disciplines to show how venue scores can be used to compare different fields in a unified metric. There are additional results in our qualitative demonstration of normalization methods, Figure 3.

Due to the variation in rankings produced by one’s choice of hyperparameters, and the large set of venues being evaluated, we do not have a canonical set of rankings that can be presented succinctly here. Instead, we will focus on quantitative evaluations of our results in the following section.

6. Evaluation

To validate the venue scores obtained by our regression methods, our evaluation consists of correlating our results against existing rankings and metrics. We consider three classes of existing scholarly measurements to correlate against: those evaluating universities, authors, and venues. Each of these classes has different standard techniques, and a different evaluation dataset, so they will be described separately in Sections 6.2, 6.4, and 6.3.

In the case of our proposed method, venue scores, we have a simple way to turn them from a journal-based to an author-based or institution-based metric. Venues are evaluated directly with the scores. Authors are evaluated as the dot product of venue scores and the publication vector of an author. Universities are evaluated as the dot product of venue scores and the total publication vector of all faculty affiliated with that university.

6.1. PageRank Baseline

Many existing approaches build on the idea of eigenvalue centrality (Bergstrom, 2007; Yan and Lee, 2007; Zhang and Wu, 2018). We implemented PageRank (Page et al., 1999) using the power iteration method to compute a centrality measure to use for both author-level and venue-level metrics. Unlike most versions of PageRank, which use citation counts, we implement two variants based solely on co-authorship information.

Author-level PageRank (PageRankA) is computed on the 1.7M x 1.7M sized co-authorship graph, where an edge is added for every time two authors co-author a paper. We found that the authors with highest centrality measures are often common names with insufficient disambiguation information in DBLP.

Journal-level PageRank (PageRankC) is computed on the 11,000 x 11,000 co-authorship graph, where an edge is added for every author who publishes in both venues. When ran on the unfiltered DBLP data, the highest scoring venue was arXiv, an expected result.

6.2. University Ranks

For this work, we produce university rankings simply as an evaluation method to demonstrate the quality and utility of our venue scoring system. The reader is cautioned that university ranking systems can tend to produce undesirable gaming behavior (Johnes, 2018), and are prone to manipulation.

We obtained and aligned many existing university rankings for Computer Science departments. These include rankings curated by journalistic sources, such as the US News Rankings (News and Report, 2018), the QS Rankings (Rankings, 2018), Shanghai Ranking (Rankings, 2015), Times Higher Education Rankings (Education, 2018) and the National Research Council report (Clauset et al., 2015). In addition, we consider purely quantitative evaluation systems such as ScholarRank (Vucetic et al., 2018), CSRankings (Berger, 2018), CSMetrics (Blackburn et al., 2018), and Prestige Rankings (Clauset et al., 2015). We additionally include ScholarRank’s t10sum across the matched faculty that our venue scores result uses.

We follow a recent paper (Vucetic et al., 2018), which demonstrated the efficacy of a citation-based metric in producing rankings with large correlation against US News rankings. We extend these experiments to include more baselines. In contrast with  (Vucetic et al., 2018), we use a rank correlation metric (namely Kendall’s $\tau$), which naturally handles ordinal ranking systems. While ScholarRank (Vucetic et al., 2018) claimed a correlation of $>0.9$ with US News, this was under Pearson’s correlation coefficient, and the result under Kendall’s $\tau$ is 0.768 in the published version and 0.757 using full precision ScholarRank.

Our faculty-based regression is able to generate a result with the highest correlation against the US News rankings. We perform even better than ScholarRank, which was designed to optimize this metric (although under a non-rank correlation metric).

6.3. Journal-level metrics

To evaluate the fidelity of our venue scores for journals and conferences, we obtain the h-index (Hirsch, 2005) and citation count for 1,308 conferences from Microsoft Academic Graph (Schauerte, 2014). We continue to use Spearman’s $\rho$ as our correlation metric, even though rank-correlation metrics can be highly impacted by noisy data (Abdullah, 1990).

Under this metric, venue scores correlated highly with h-index. Notably, h-index and venue scores are highly correlated with each other, even though venue scores are much less correlated with raw conference size. See Figure 7 for detailed results.

6.4. Author-level Metrics

To evaluate our venue scores in the application of generating author-level metrics, we will use rank correlation (also known as Spearman’s $\rho$) (Spearman, 1904) between our venue scores and traditional author-level metrics such as h-index. Google Scholar was used to obtain citations, h-index (Hirsch, 2005), i10-index, and Semantic Scholar used to obtain highly influential citations (Valenzuela et al., 2015). Prior work has critiqued the h-index measure (Yong, 2014) and proposed an alternative metric, derived from a citation count. However, our use of a rank correlation  means that monotonically transformed approximations of citation counts would lead to identical scores.

For evaluation, we collected a dataset for the largest Computer Science department in CSRankings ($n=148$). The results are shown in Table 8. We can see that venue scores highly correlate with h-index, influential citations, i10-scores and CSRankings scores. The results from the author-based PageRank are surprisingly similar to our venue scores. However, the conference-based PageRank performed worse than venue scores on every correlation metric.

7. Discussion

Our results show a medium to strong correlation of venue scores against existing scholarly metrics, such as citation count and h-index. For author metrics, venue scores correlate with influential citations (Valenzuela et al., 2015) or h-index about as well as such measures correlate against each other or raw citation counts (see Table 8). For venue metrics, venue scores correlate with h-index (0.64) and citations (0.61) nearly as well as citations correlate with h-index (0.66). For university metrics, venue scores correlate as well with measures of peer assessment as citation-based metrics do (Vucetic et al., 2018).

As h-index and citation counts have their flaws, obtaining perfect correlation is not necessarily a desirable goal. Instead, these strong correlations serve as evidence for the viability of venue scores.

Venue scores have been shown to be robust against hyperparameter choices (Tables 2, 3, 4, 5). Even venue scores produced from completely different data sources tend to look very similar (Table 1). Additionally, venue scores can naturally capture the variation of conference quality over time (Figures 1, 3).

As with any inductive method, venue scores are data-driven and will be subject to past biases. For example, venue scores can clearly be biased by hiring practices, pay inequality and NSF funding priorities. As these are the supervising metrics, bias in those datasets will be encoded in our results. For example, we found that the faculty hiring metric prioritized Theoretical Computer Science, while using NSF awards prioritized Robotics. The faculty classification task may devalue publishing areas where candidates pursue industry jobs, while the NSF grant regression task may devalue areas with smaller capital requirements. By using large datasets and combining multiple metrics in a single model (Section 4.7), the final model could reduce the biases in any individual dataset.

Each of our metrics of interest has an inherent bias in timescale, which our temporal normalization tries to correct for, but likely does an incomplete job of. Salaries are often higher for senior faculty. NSF Awards can have a long response time and a preferences towards established researchers. Faculty classification prioritizes the the productive years of existing faculty. Additionally, faculty hiring as a metric will have a bias towards work from prestigious universities (Clauset et al., 2015) and their venue preferences. Some of these issues also exist in citation metrics, and may be why our uncorrected models correlated better with them (Table 2).

8. Similarity Metrics

While the previous sections of this paper have focused on evaluation, the same dataset can be used to organize venues into groups. For organization, we use a much smaller dataset, using data since 2005 and only evaluating the 1,155 venues that have at least 20 R1 universities with faculty publishing in them. We then build the venue $\times$ author matrix, counting the number of papers each that author published in each venue. Performing a $d$-dimensional Latent Dirichlet Allocation (Blei et al., 2003), we obtain a 50-dimensional vector representing each conference in a meaningful way.

These vectors can then be clustered (Arthur and Vassilvitskii, 2007) to produce automatic categories for each conference. These high dimensional vectors can also be embedded (van der Maaten and Hinton, 2008) into two dimensions to produce a visual map of Computer Science. See Figure  5 for our result. These clusters represent natural categories in Computer Science. For example, it is easy to see groups that could be called Theory, Artificial Intelligence, Machine Learning, Graphics, Vision, Parallel Computing, Software Engineering, Human-Computer Interaction, among many others.

While some clusters are distinct and repeatable, others are not. When datasets contain challenging cases, ideal clustering can be hard to estimate (Ben-David, 2015). Using silhouette scores (Rousseeuw, 1987), we can estimate how many natural clusters of publishing exist in Computer Science. In our experiments, silhouette scores were maximized with 20 to 28 clusters. As the clustering process is stochastic, we were unable to determine the optimal cluster number with statistical significance.

By embedding each author with the weighted average of their publication’s vectors, we can also obtain a fingerprint that shows which areas of Computer Science each university focuses on. See Figure  6 for an example of such fingerprints for many top departments. The same clustering method can be used to analyze the focus areas of a single department, for an example see Figure 7.

9. Conclusion

We have presented a method for ranking and organizing a scholarly field based only on simple publication information- namely a list of papers, each labeled with only their published venue, authors, and year. By regressing venue scores from metrics of interest, one obtains a plausible set of venue scores. These scores can be compared across sub-fields and aggregated into author-level and institution-level metrics. The scores provided by this system, and their resulting rankings, correlate highly with other established metrics. As this system is based on easily obtainable, publicly available data, it is transparent and reproducible. Our method builds on simple techniques and demonstrates that their application to large-scale data can produce surprisingly robust and useful tools for scientometric analysis.

Acknowledgements.

Martial Hebert suggested the use of correlation metrics as a technique for quantitative evaluation, thereby providing the framework for every table of results in this paper. Emery Berger developed CSRankings (Berger, 2018), which was highly influential in the design and implementation of this project. Joseph Sill (Sill, 2010), Wayne Winston, Jeff Sagarin and Dan Rosenbaum developed Adjusted Plus-Minus, a sports analytics technique that partially inspired this work. Kevinjeet Gill was unrelenting in advocating for a year-by-year regression model to avoid sampling and quantization artifacts.

Appendix A Credit Assignment

In order to address issues of collinearity raised by having authors who publish papers together, we wanted to solve a credit assignment problem. We adapted a well-known method for addressing this by adapting regularized plus minus (Sill, 2010) from the sports analytics literature. In our case, we simply regress each publications values from its authors, as shown below.

[TABLE]

This technique produced scores that correlated highly with total value scores. While this may be valuable for understanding an individual’s contribution, but we were unable to find an evaluation for this method. The form of this model that considers n-wise or pairwise relationships was also considered, but these matrices were too large for us to run trials.

Appendix B Aging Curve

To evaluate if our model makes a sensible prediction over the timescale of a scholar’s career, we built a model to see what an average academic career looks like, given that the author is still publishing in those years. See Figure 9. Our model suggests a rise in productivity for the first 20 years of one’s publishing history, and then a steady decline.