# PRO-Angoff method for remote standard setting: establishing clinical thresholds for the upper digestive disease tool

## TL;DR

This study uses the Angoff method to set clinical thresholds for a tool that tracks symptoms in upper digestive disease, using virtual methods for standard setting.

## Contribution

The study introduces a new virtual standard setting approach for establishing clinical thresholds in the UDD Tool.

## Key findings

- Cut scores for symptom control categories were established using the extended Angoff method.
- Panelists found the virtual standard setting process effective and confident in the resulting cut scores.
- Good/moderate and moderate/poor cut scores varied across domains with specific ranges identified.

## Abstract

The Upper Digestive Disease (UDD) Tool™ is used to monitor symptom frequency, intensity, and interference across nine symptom domains and includes two Patient-Reported Outcome Measurement Information System (PROMIS) domains assessing physical and mental health. This study aimed to establish cut scores for updated symptom domains through standard setting exercises and evaluate the effectiveness and acceptability of virtual standard setting.

The extended Angoff method was employed to determine cut scores. Subject matter experts refined performance descriptions for symptom control categories and achieved consensus. Domains were categorized into good, moderate, and poor symptom control. Two cut scores were established, differentiating good vs. moderate and moderate vs. poor. Panelists estimated average scores for 100 borderline patients per item. Cut scores were computed based on the sum of the average ratings for individual questions, converted to 0-100 scale.

Performance descriptions were refined. Panelists discussed that interpretation of the scores should take into account the timing of symptoms after surgery and patient populations, and the importance of items asking symptom frequency, severity, and interference with daily life. The good/moderate cut scores ranged from 21.3 to 35.0 (mean 28.6, SD 3.6) across domains, and moderate/poor ranged from 47.5 to 71.3 (mean 54.5, SD 7.0).

Panelists were confident in the virtual standard setting process, expecting valid cut scores. Future studies can further validate the cut scores using patient perspectives and collect patient and physician preferences for displaying contextual items on patient- and physician-facing dashboard.

The online version contains supplementary material available at 10.1186/s41687-024-00707-x.