Usability and Acceptance Testing of an Electronic Patient-Reported Outcome Symptom Monitoring System for People Receiving Immune Checkpoint Inhibitors: Mixed Methods Study

TL;DR

A new electronic system for tracking immune-related side effects in cancer patients was tested and found to be user-friendly and well accepted by patients and clinicians.

Contribution

The first formal usability and acceptance testing of an ePRO system for immune-related adverse events in cancer care.

Findings

The ePRO system showed high usability scores across patients, caregivers, and clinicians.

User acceptance was strong, with themes of empowerment and improved communication.

Limitations included navigation issues and concerns about digital equity and workload.

Abstract

Immune checkpoint inhibitors are widely used in oncology but can cause immune-related adverse events (irAEs), which may be severe or life-threatening if not detected early. Electronic patient-reported outcome (ePRO) symptom monitoring systems may facilitate timely recognition and management of irAEs. Usability testing is a critical stage in ePRO system development, yet no published examples of formal usability and acceptance testing exist.

This study aims to assess the usability and acceptance of a co-designed ePRO symptom monitoring prototype for irAEs embedded within the Epic electronic medical record.

Testing was conducted at an Australian quaternary cancer center. Eligible participants were patients who had received or were receiving immune checkpoint inhibitors, their caregivers, or clinicians (oncologists and nurse specialists). Participants completed baseline digital literacy assessments (16-item Mobile Device Proficiency Questionnaire [MDPQ-16] and 12-item Computer Proficiency Questionnaire [CPQ-12]) before a structured testing session. Each session involved role-specific tasks using the patient-facing Health Hub or the clinician-facing Epic electronic medical record. Usability was assessed using the System Usability Scale (SUS). Acceptance was assessed using a customized Unified Theory of Acceptance and Use of Technology (UTAUT) questionnaire. Semistructured interviews were used to capture qualitative feedback.

A total of 30 participants (7 patients, 3 caregivers, 10 oncologists, and 10 nurse specialists) completed 10 testing sessions. Median MDPQ-16 and CPQ-12 scores were higher for clinicians compared to patients and caregivers. Median SUS scores indicated high usability—patients and caregivers: 77.5% (IQR 70.0%‐86.3%), oncologists: 82.5% (IQR 80.0%‐90.0%), and nurse specialists: 80.0% (IQR 75.6%‐94.4%). Median UTAUT scores demonstrated strong user acceptance—patients or caregivers: 4.27 (IQR 4.09‐4.58), oncologists: 4.33 (IQR 4‐4.63), and nurse specialists: 4.23 (IQR 3.87‐4.57). Health Hub usability themes highlighted overall ease of navigation and efficiency of reporting, but a need for clearer survey navigation, simplification of the actions page, and improved organization of trend graphs. For clinicians, themes included efficient side effect capture and intuitive system design, but a need to improve navigation to results, optimize data display, and facilitate team-based alert management. Health Hub acceptance themes highlighted patient empowerment to self-manage, enhanced patient-clinician communication, and reinforcement of existing care. However, concerns were raised about digital equity for vulnerable groups. Clinicians reported that the system streamlined side effect management between visits, aligned with existing Epic workflows, and could be tailored to personal preferences. Concerns remained regarding additional workload and medico-legal responsibilities associated with real-time alerts.

The ePRO prototype demonstrated high levels of usability and acceptance across patients, caregivers, and clinicians. Limitations around navigation and data visualization, alongside equity and workload concerns, will guide refinements prior to implementation. These findings emphasize the value of rigorous formative usability and acceptance testing to optimize ePRO systems prior to deployment in routine cancer care.

Figures