Discretion in the Loop: Human Expertise in Algorithm-Assisted College Advising

TL;DR

This study investigates how human advisors in algorithm-assisted college advising use contextual information to improve student outcomes, highlighting the importance of discretion and expertise in such systems.

Contribution

It introduces a causal framework for understanding human discretion in algorithm-assisted advising and provides empirical evidence of its impact on student success.

Findings

Approximately two-thirds of advisor interventions were expertly targeted using non-algorithmic context.

Advisors incorporate diverse contextual factors like personal circumstances and financial issues.

Different advising styles influence the amount of holistic information gathered and are linked to graduation rates.

Abstract

In higher education, many institutions use algorithmic alerts to flag at-risk students and deliver advising at scale. While much research has focused on evaluating algorithmic predictions, relatively little is known about how discretionary interventions by human experts shape outcomes in algorithm-assisted settings. We study this question using rich quantitative and qualitative data from a randomized controlled trial of an algorithm-assisted advising program at Georgia State University. Taking a mixed-methods approach, we examine whether and how advisors use context unavailable to an algorithm to guide interventions and influence student success. We develop a causal graphical framework for human expertise in the interventional setting, extending prior work on discretion in purely predictive settings. We then test a necessary condition for discretionary expertise using structured advisor logs and student outcomes data, identifying several interventions that meet the criterion for statistical significance. Accordingly, we estimate that 2 out of 3 interventions taken by advisors in the treatment arm were plausibly ``expertly targeted'' to students using non-algorithmic context. Systematic qualitative analysis of advisor notes corroborates these findings, showing a pattern of advisors incorporating diverse forms of contextual information--such as personal circumstances, financial issues, and student engagement--into their decisions. Finally, we document heterogeneity in advising styles, finding that one style elicits more holistic information about students and is associated with improved graduation rates. Our results offer theoretical and practical insight into the real-world effectiveness of algorithm-supported college advising, and underscore the importance of accounting for human expertise in the design, evaluation, and implementation of algorithmic decision systems.