# Medical Students' Perception of Automated Note Feedback After Simulated Encounters

## TL;DR

Medical students found automated feedback on patient notes visually helpful, but it didn't replace traditional methods like model notes.

## Contribution

This study explores medical students' perceptions of automated feedback on patient notes using natural language processing.

## Key findings

- Automated feedback was visually appealing and helped students compare found vs. missing items in patient notes.
- Students found model notes more trustworthy compared to automated feedback.
- Automated systems can support formative feedback but have limitations in accuracy and generalizability.

## Abstract

Grading medical student patient notes (PNs) is resource‐intensive. Natural language processing (NLP) offers a promising solution to automatically grade PNs. We deployed an automated grading system that uses NLP and explored the perceived value of PN feedback.

The automated system graded written notes after two standardized patient encounters by third‐year medical students. The system generated an individualized report on ‘items found’ and ‘items not found’ in the history, physical examination, and diagnosis sections, which was shared with students for feedback via a web‐based interface. By rotation, block students received either the automated case feedback first or the faculty‐written model note feedback first (the pre‐intervention baseline).

After reviewing feedback, students completed surveys for both automated feedback and model note feedback and participated in follow‐up focus groups. In total, 44 students received feedback, 37 completed surveys, and 28 participated in focus groups. Qualitative themes that emerged suggested the automated feedback was visually appealing and allowed for easy comparison of items found vs. missing, which would help improve students' documentation skills. Model note appeared trustworthy.

We found automated systems can be a potential tool for formative feedback on note writing activity although in terms of quality it does not surpass the pre‐existing feedback methods, such as model note feedback used in our study. Order effects may have influenced these perceptions and the small sample size limits generalizability. Tested software had occasional errors in recognizing a phrase or showing a false positive.