# Exploring how complex multiple-choice questions could contribute to inequity in introductory physics

## TL;DR

This study finds that complex multiple-choice questions in physics exams may contribute to inequity, as they are harder for all students and could widen equity gaps.

## Contribution

The study identifies complex multiple-choice questions as a potential source of inequity in physics assessments.

## Key findings

- Students performed 7.9 percentage points worse on complex multiple-choice questions than on non-CMC questions.
- CMC questions may contribute to equity gaps, particularly for male and female students.
- CMC questions do not effectively differentiate top-performing students from others.

## Abstract

High-stakes exams significantly impact introductory physics students’ final grades and have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Certain types of exam questions may contribute more than other types to the observed equity gaps.

The primary objective of this study was to determine whether complex multiple-choice (CMC) questions may be a potential cause of inequity.

We used four years of data from Problem Roulette, an online, not-for-credit exam preparation program, to address our objective. This data set included 951 Physics II (Electricity and Magnetism) questions, each of which we categorized as CMC or non-CMC. We then compared student performance on each question type and created a multi-level logistic regression model to control individual student and question differences.

Students performed 7.9 percentage points worse on CMC questions than they did on non-CMC questions. We find minimal additional performance differences based on student performance in the course. The results from mixed-effects models suggest that CMC questions may be contributing to the observed equity gaps, especially for male and female students, though more evidence is needed.

We found CMC questions are more difficult for everyone. Future research should examine the source of this difficulty and whether that source is functionally related to learning and assessment. Our data does not support using CMC questions instead of non-CMC questions as a way to differentiate top-performing students from everyone else.