Metrics of research impact in astronomy: Predicting later impact from metrics measured 10-15 years after the PhD

TL;DR

This study develops and calibrates metrics derived from publication data to predict the future impact of astronomy researchers, aiding resource allocation decisions in academia.

Contribution

It introduces a method to combine multiple impact metrics for more accurate long-term impact prediction in astronomy careers.

Findings

Averaging three metrics improves prediction accuracy.

Predictions made up to 17 years after PhD are reliable.

Calibration based on expert votes achieves RMS uncertainty of 1/8 of impact range.

Abstract

This paper calibrates how metrics derivable from the SAO/NASA Astrophysics Data System can be used to estimate the future impact of astronomy research careers and thereby to inform decisions on resource allocation such as job hires and tenure decisions. Three metrics are used, citations of refereed papers, citations of all publications normalized by the numbers of co-authors, and citations of all first-author papers. Each is individually calibrated as an impact predictor in the book Kormendy (2020), "Metrics of Research Impact in Astronomy" (Astron Soc Pac Conference Series Monograph 8, San Francisco). How this is done is reviewed in the first half of this paper. Then, I show that averaging results from three metrics produces more accurate predictions. Average prediction machines are constructed for different cohorts of 1990-2007 PhDs and used to postdict 2017 impact from metrics measured 10, 12, and 15 years after the PhD. The time span over which prediction is made ranges from 0 years for 2007 PhDs to 17 years for 1990 PhDs using metrics measured 10 years after the PhD. Calibration is based on perceived 2017 impact as voted by 22 experienced astronomers for 510 faculty members at 17 highly-ranked university astronomy departments world-wide. Prediction machinery reproduces voted impact estimates with an RMS uncertainty of 1/8 of the dynamic range for people in the study sample. The aim of this work is to lend some of the rigor that is normally used in scientific research to the difficult and subjective job of judging people's careers.