Validation of the Observational Assessment Tool for Tailoring (OATT)

TL;DR

The paper introduces and validates a new tool to assess how well behavioral health interventions are tailored to individuals, showing it works across different groups and languages.

Contribution

The OATT is a novel observational tool validated for measuring fidelity in tailored behavioral interventions.

Findings

The OATT demonstrated strong reliability and validity across English and Spanish-speaking sessions.

Fidelity to individualized treatment planning predicted improved engagement and health behaviors.

The tool's two-factor structure was confirmed with strong fit indices in two trials.

Abstract

Individually tailored interventions can address the myriad multi-level determinants of chronic health conditions. Limited measurement modalities to quantify tailoring disallow examining “active ingredient” effects on outcomes and implementation fidelity. The objective of this study is to develop and validate the Observational Assessment Tool for Tailoring (OATT) for behavioral prevention interventions. We developed the OATT and coded n = 172 videorecorded sessions from two trials of the Family Check-Up® 4 Health (FCU4Health), an individually tailored prevention and management program for behavioral health and obesogenic behaviors with English and Spanish-speaking participants. The sample was culturally diverse (> 65% Hispanic/Latino). Confirmatory factor analysis (CFA) tested the two-factor model. McDonald’s Omega estimated internal consistency. Discriminant and predictive validity tests were conducted with FCU4Health fidelity, engagement, and health behavior outcomes, informed by the Implementation Cascade Model. CFA confirmed a two-factor structure for both trials (i.e., RMSEA ≤ 0.06, CFI and TLI of ≥ 0.95, SRMR < 0.08 chi-square p ≥ 0.05). Reliability and inter-rater reliability were good (ICC > 0.77) for both trials and English and Spanish videos. The OATT was not correlated (p > 0.05) with discriminant validity variables. Path analysis for predictive validity indicated that fidelity to the Individualized Treatment Planning factor directly predicts improvements in participant engagement (B = 0.16, p = 0.01, 95% CI [0.03–0.29]), which directly predicts improvements in parent health behaviors 12 months post-baseline (B = 0.18, p = 0.01, 95% CI [0.02–0.34]). The development of the OATT is a critical step to measure and guide tailored intervention development, implementation, and evaluation. Future studies are needed to replicate predictive validity findings and test the OATT factor structure with larger samples and different prevention initiatives.

The online version contains supplementary material available at 10.1007/s11121-026-01879-2.

Figures