Generalized reliability based on distances

TL;DR

This paper introduces a new distance-based intraclass correlation coefficient (dbICC) for assessing measurement reliability in complex data types, with bias correction and applications to brain imaging.

Contribution

The paper proposes a novel distance-based ICC (dbICC) that generalizes reliability assessment to arbitrary data types and introduces bias correction for better confidence interval coverage.

Findings

dbICC effectively measures reliability for complex data.

Bias correction improves bootstrap confidence intervals.

Application to brain imaging demonstrates practical utility.

Abstract

The intraclass correlation coefficient (ICC) is a classical index of measurement reliability. With the advent of new and complex types of data for which the ICC is not defined, there is a need for new ways to assess reliability. To meet this need, we propose a new distance-based intraclass correlation coefficient (dbICC), defined in terms of arbitrary distances among observations. We introduce a bias correction to improve the coverage of bootstrap confidence intervals for the dbICC, and demonstrate its efficacy via simulation. We illustrate the proposed method by analyzing the test-retest reliability of brain connectivity matrices derived from a set of repeated functional magnetic resonance imaging scans. The Spearman-Brown formula, which shows how more intensive measurement increases reliability, is extended to encompass the dbICC.