Calibration of multi‐site raters for prospective visual read of amyloid PET scans acquired across the ADRC Consortium for Clarity in ADRD Research Through Imaging (CLARiTI)

TL;DR

This study develops a calibration method to ensure accurate amyloid PET scan readings across multiple sites and raters in a large Alzheimer's research project.

Contribution

The novel contribution is a calibration process for multi-site raters using diverse imaging tools and amyloid tracers to ensure reliable visual reads.

Findings

Calibration scans showed 97% concordance and almost perfect agreement between readers (κ=0.932).

Intra-rater agreement was 98% with almost perfect agreement (κ=0.966).

Discordant scans were mostly borderline cases with Centiloid values between 8-21.

Abstract

PET visual reads for multi‐center studies are conducted centrally by one or a few experts. In CLARiTI, a broader network of raters will perform the reads, necessitating methods to ensure their accuracy and reliability.

All ADRC sites (N = 37) will prospectively acquire amyloid PET scans for 800 cognitively unimpaired and 1200 cognitively impaired participants. Each site can use one of four amyloid PET tracers ([18F]florbetaben, [18F]florbetapir, [11C]PIB, [18F]NAV4694). PET scans will be visually interpreted by ten PET neuroimaging experts located across eight ADRCs, using their preferred hardware, software, and file format (DICOM, NIFTI). Calibration requires readers to perform blinded, independent visual interpretation of 180 amyloid PET scans (30 unique and 15 duplicate scans per tracer), previously read and selected by an expert (Table 1). An even mix of visually non‐elevated and elevated scans were selected, with quantitation spanning ‐26‐123 Centiloids (Figure 1), and 15% were visually borderline (similar proportion to ADNI4), with quantitation spanning 4‐48 Centiloids (mean 21). Kruskal‐Wallis H test showed no significant difference in the distribution of Centiloid values for scans in each tracer group, Χ2(3)=2.133, p = 0.545.

Calibration scans were read by a second expert not serving in CLARiTI. There was 97% concordance and almost perfect agreement between readers (κ=0.932, p <0.001; Figure 2). Discordantly read scans were visually borderline elevated (N = 2), borderline non‐elevated (N = 1), and negative (N = 1), with quantitation of 8‐21 Centiloids (mean 14). Intra‐rater reads were 98% concordant and had almost perfect agreement (κ=0.966, p <0.001). These findings hold promise for scaling‐up visual reads in CLARiTI, and we expect ≥80% concordance and substantial to almost perfect agreement (κ≥0.7) for inter/intra‐rater reliability.

Calibration is essential for accurate/reliable amyloid PET visual reads in large multi‐center studies with multiple raters. Through the calibration process, we will develop methods to improve the adjudication of borderline/challenging scans and formalize methods for visual reads of amyloid PET tracers that are investigational (i.e., [11C]PIB, [18F]NAV4694). Involving multiple raters across different sites, who use diverse hardware/software/file format combinations, will mirror the natural heterogeneity present in real‐world clinical practice. This calibration set can also serve as a standardized training tool for other studies incorporating amyloid PET visual reads.

Figures