Assessing Group-level Gender Bias in Professional Evaluations: The Case of Medical Student End-of-Shift Feedback

TL;DR

This study investigates gender bias in medical student evaluations by analyzing written feedback using advanced NLP models, finding no overall bias but noting gendered differences in family-related terms.

Contribution

It introduces a novel application of fine-tuned BERT to assess gender bias in real-world medical evaluations, moving beyond traditional wordlist methods.

Findings

No evidence of group-level gender bias in overall feedback

Women receive more family and children-related terms in evaluations

BERT-based analysis provides nuanced insights into gendered language use

Abstract

Although approximately 50% of medical school graduates today are women, female physicians tend to be underrepresented in senior positions, make less money than their male counterparts and receive fewer promotions. There is a growing body of literature demonstrating gender bias in various forms of evaluation in medicine, but this work was mainly conducted by looking for specific words using fixed dictionaries such as LIWC and focused on recommendation letters. We use a dataset of written and quantitative assessments of medical student performance on individual shifts of work, collected across multiple institutions, to investigate the extent to which gender bias exists in a day-to-day context for medical students. We investigate differences in the narrative comments given to male and female students by both male or female faculty assessors, using a fine-tuned BERT model. This allows us to examine whether groups are written about in systematically different ways, without relying on hand-crafted wordlists or topic models. We compare these results to results from the traditional LIWC method and find that, although we find no evidence of group-level gender bias in this dataset, terms related to family and children are used more in feedback given to women.