# A protocol for tracking scholarly output to evaluate the impact of the RADx-UP program on community-engaged COVID-19 research

## TL;DR

This paper presents a method for tracking scholarly output to assess the impact of a large-scale research program focused on underserved populations during the COVID-19 pandemic.

## Contribution

The paper introduces an enhanced search strategy for publication tracking in multi-project research programs.

## Key findings

- The enhanced search strategy had higher sensitivity (62.6%-68.0%) than the simple search (21.5%-27.4%).
- Survey response rates declined over time, from 61.3% to 32.8%.
- The enhanced search improved specificity but surveys were still needed to refine strategies.

## Abstract

Portfolio-level publication tracking collects research output from related programs. Tracking publications is imperative to evaluate the scholarly impact of a program, synthesize program findings, and document impact to funders. A valid tracking protocol increases data quality for accurate impact assessment, but there is little literature on publication tracking methods appropriate for assessing impact across multiple programs.

We tracked, managed, and evaluated publications from the National Institutes of Health-funded Rapid Acceleration of Diagnostics - Underserved Populations, which included over 137 projects and a Coordination and Data Collection Center. During the four-year project, we deployed a quarterly self-report survey to project leads and conducted twice-monthly searches for grant-related publications. Search strategies comprised a simple search of project grant numbers and an enhanced search. We evaluated the sensitivity and positive predictive value of search strategies compared to the surveys.

Compared to the survey, the simple search was 21.5% to 27.4% sensitive with a positive predictive value between 81.1% and 95.8%. The enhanced search was 62.6% to 68.0% sensitive with a positive predictive value between 76.2% and 96.9%. Response rates declined over time from a maximum of 61.3% to a minimum of 32.8%.

The enhanced search increased specificity in identifying publications, but the survey was necessary to refine strategies and identify missed products. However, the enhanced search may have relieved participant burden in entering citations. These findings may be valuable for coordinating centers, academic departments, working groups, and other academic entities that must quantify the impact of their publications.