# Leveraging feedback mechanisms to improve the quality of objective structured clinical examinations in Singapore: an exploratory action research study

## TL;DR

This study explores using psychometric feedback to improve fairness in medical exams by helping examiners score more consistently.

## Contribution

The study introduces a novel, psychometric-informed feedback mechanism for OSCE examiners, designed to enhance transparency and consistency.

## Key findings

- Examiners found feedback reports more credible and neutral after refinements like adding glossaries and visual graphs.
- Examiners shifted from resistance to reflective engagement and requested longitudinal feedback reports.
- The study shows feasibility of feedback reports but calls for larger samples and quantitative data to confirm effectiveness.

## Abstract

Variability in examiner scoring threatens the fairness and reliability of objective structured clinical examinations (OSCEs). While examiner standardization exists, there is currently no structured, psychometric-informed, individualized feedback mechanism for examiners. This study explored the feasibility and perceived value of such a mechanism using an action research approach to co-design and iteratively refine examiner feedback reports.

Two exploratory cycles were conducted between November 2023 and June 2024 with phase 4 OSCE examiners at the Yong Loo Lin School of Medicine. In cycle 1, psychometric analyses of examiner scoring for a phase 4 OSCE informed the design of individualized reports, which were evaluated through interviews. Revisions were made to the format of the report and implemented in cycle 2, where examiner responses were again collected. Data were analyzed thematically, supported by reflective logs and field notes.

Nine examiners participated in cycle 1 and 7 in cycle 2. In cycle 1, examiners highlighted challenges in interpreting complex terminology, leading to report refinements such as glossaries and visual graphs. In cycle 2, examiners demonstrated greater confidence in applying feedback, requested longitudinal reports, and shifted from initial resistance to reflective engagement. Across cycles, the reports improved credibility, neutrality, and examiner self-regulation.

This exploratory study suggests that psychometric-informed feedback reports can facilitate examiner reflection and transparency in OSCEs. While the findings highlight feasibility and examiner acceptance, longitudinal delivery of feedback, collection of quantitative outcome data, and larger samples are needed to establish whether such reports improve scoring consistency and assessment fairness.