The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool’s psychometric properties in different specialties

TL;DR

This study evaluates the psychometric properties of the O-RON tool across three medical specialties, finding it valid but noting challenges in form completion due to workload and interprofessional dynamics.

Contribution

The study extends the validation of the O-RON tool beyond Orthopedic Surgery to other clinical specialties.

Findings

The O-RON demonstrated strong validity evidence across three specialties at the University of Ottawa.

Reliability of the O-RON was 0.82 with four forms per resident.

Exit interviews highlighted clinical workload and interprofessional dynamics as barriers to form completion.

Abstract

Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors’ feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident’s performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses’ assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool’s performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa.

O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed.

179 O-RONs were completed on 30 residents. With four forms per resident, the ORON’s reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001).

Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Figures