Stakeholder input on the CAHPS ambulatory surveys

TL;DR

Experts evaluated and provided feedback on healthcare experience surveys to improve their content and relevance.

Contribution

Systematic stakeholder feedback was gathered to guide revisions of CAHPS ambulatory surveys using a modified Delphi method.

Findings

Stakeholders rated the essentialness of survey items, showing improved reliability in ratings after discussion.

Digital access, communication, and coordination were highlighted as increasingly essential topics.

New topics like mental health integration and patient safety were suggested for inclusion.

Abstract

The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys are widely used to evaluate patients’ experiences with healthcare. Although the surveys have been extensively assessed and periodically updated, concerns persist regarding their content, length, and score distributions. This study aimed to gather systematic stakeholder feedback to inform future revisions of CAHPS ambulatory surveys.

A modified Delphi method was employed using the ExpertLens™ online platform. A panel of 20 members representing a broad stakeholder community, including survey sponsors, survey experts, patient experience advocates, and federal representatives, participated in three phases. The first phase was an initial rating of the essentialness (required, optional, not essential) of 46 existing item topics using a 1 (Not Essential) to 9 (Very Essential) scale with scores of 1–3 used for a topic that should not be included, 4–6 used for a topic that should be optional, and 7–9 for a topic that should be required in a CAHPS survey of health plans, clinicians, or group practices. The second phase was an asynchronous online discussion of the initial ratings, and the third phase was a final rating of the 46 existing item topics. The reliability of ratings was assessed using a mixed-effects analysis of variance model. Means and standard deviations of essentialness ratings were also analyzed. Verbatim comments from the experts were summarized to provide additional insights.

Reliability of expert 1–9 essentialness ratings improved from the initial round (reliability = 0.63, intraclass correlation = 0.08) to the final round (reliability = 0.70, intraclass correlation = 0.10). While most existing items were deemed essential by most stakeholders, there were noteworthy (0.08 or larger) increases from the initial to final rating phases in essentialness for items related to digital access, medication reconciliation, provider communication, and appeals processes, and notable decreases for specialist care ratings, access to medical questions during off-hours, and provider knowledge of chronic conditions. Stakeholders emphasized the importance of access to care, communication and coordination, respectful interactions with staff and providers, and clear cost information. Several potential topics missing from current surveys were identified, including unfair treatment, mental health integration, maternity care, language concordance, trust, self-management, patient safety, continuity of care, care coordination, and claims processing.

This study provides valuable insights into stakeholder perspectives on the relevance and potential improvements to CAHPS ambulatory survey content. The findings support revisions to existing items to enhance their clarity and actionability, as well as the inclusion of new topics that reflect evolving healthcare priorities and patient needs. The identified areas for expansion offer opportunities to create more comprehensive and impactful assessments of patient experiences in ambulatory care settings.

The online version contains supplementary material available at 10.1186/s41687-025-00983-1.

Figures