# Evaluation of Consecutive Guided Training to Improve Interrater Agreement in Identifying Elements of Situation Awareness in Objective Structured Clinical Examination Assessments

## TL;DR

This study explores how guided training improves consistency in evaluating medical students' situation awareness during clinical exams.

## Contribution

The study introduces consecutive guided training to enhance interrater agreement in assessing situation awareness in OSCEs.

## Key findings

- Consecutive guided training improved interrater agreement for identifying SA elements.
- Generalisability theory identified key factors affecting variance in SA assessment.
- Moderate to very good agreement was achieved among raters after training.

## Abstract

This article was migrated. The article was marked as recommended.

Introduction: Little is known about the medical student’s cognitive ability in diagnostic and therapeutic accuracy. Literature does not suggest a methodology to quantify students' cognitive processing. Situation Awareness (SA) is described as having the proficiency to obtain awareness of the surrounding and to integrate this consciousness into the situational context and potential forthcoming development. OSCEs might be a suitable instrument to evaluate students’ awareness of the situation.

Methods: Consecutive guided training was provided to obtain a consistent comprehension of the model of SA. 4 independent researchers consecutively examined 6 randomised OSCE forms in a qualitative and quantitative method. Final interrater agreement was expressed as Cohens kappa. Generalisability theory determined the impact of the main facets on the variation in disagreement.

Results: Evaluation of identifying and categorising elements of SA within OSCE forms demonstrated a moderate to very good interrater agreement. The G-Theory revealed key facets for variance: OSCE forms, Levels of SA, Items embedded in the Levels, Interaction between Forms and Levels and Forms and Items embedded within Levels.

Conclusion: Consecutive guided training improved the identification of elements of SA within OSCE assessments. Further research is necessary to improve the assessment of SA in undergraduate medical curricula.