Validation of an Adaptive Assessment of Executive Functions (Adaptive Cognitive Evaluation-Explorer): Longitudinal and Cross-Sectional Analyses of Cognitive Task Performance

TL;DR

This study validates a mobile tool for assessing executive functions, showing it is reliable and can be used across different age groups.

Contribution

The study introduces and validates ACE-X, a mobile, adaptive measure of executive functions with strong reliability and structural consistency.

Findings

ACE-X tasks showed moderate to high reliability across repeated assessments.

A network model best described ACE-X task performance, with consistent structure across age groups.

ACE-X task performance was grouped into three cognitive communities: set reconfiguration, attentional control, and interference resolution.

Abstract

Executive functions (EFs) predict positive life outcomes and educational attainment. Consequently, it is imperative that our measures of EF constructs are both reliable and valid, with advantages for research tools that offer efficiency and remote capabilities.

The objective of this study was to evaluate reliability and validity evidence for a mobile, adaptive measure of EFs called Adaptive Cognitive Evaluation-Explorer (ACE-X).

We collected data from 2 cohorts of participants: a test-retest sample (N=246, age: mean 35.75, SD 11.74 y) to assess consistency of ACE-X task performance over repeated administrations and a validation sample involving child or adolescent (5436/6052, 89.82%; age: mean 12.78, SD 1.60 years) and adult participants (484/6052, 8%; age: mean 38.11, SD 14.96 years) to examine consistency of metrics, internal structures, and invariance of ACE-X task performance. A subset of participants (132/6052, 2.18%; age: mean 37.04, SD 13.23 years) also completed a similar set of cognitive tasks using the Inquisit platform to assess the concurrent validity of ACE-X.

Intraclass correlation coefficients revealed most ACE-X tasks were moderately to very reliable across repeated assessments (intraclass correlation coefficient=0.45-0.79; P<.001). Moreover, in comparisons of internal structures of ACE-X task performance, model fit indices suggested that a network model based on partial correlations was the best fit to the data (χ228=40.13; P=.06; comparative fit index=0.99; root mean square error of approximation=0.03, 90% CI 0.00-0.05; Bayesian information criterion=5075.87; Akaike information criterion=4917.71) and that network edge weights are invariant across both younger and older adult participants. A Spinglass community detection algorithm suggested ACE-X task performance can be described by 3 communities (selected in 85% of replications): set reconfiguration, attentional control, and interference resolution. On the other hand, Pearson correlation coefficients indicated mixed results for the concurrent validity comparisons between ACE-X and Inquisit (r=–.05-.62, P<.001-.76).

These findings suggest that ACE-X is a reliable and valid research tool for understanding EFs and their relations to outcome measures.

Figures