Ablation Study of a Fairness Auditing Agentic System for Bias Mitigation in Early-Onset Colorectal Cancer Detection

TL;DR

This paper investigates an agentic AI system with retrieval capabilities to improve fairness auditing of biomedical models in early-onset colorectal cancer detection, addressing bias and safety risks in clinical AI.

Contribution

It introduces a two-agent architecture with retrieval-augmented generation to enhance fairness auditing in biomedical AI models, demonstrating improved semantic similarity to expert assessments.

Findings

Agent with RAG outperforms other configurations in semantic similarity

Retrieval-augmented systems better identify demographic disparities

Scaling models with RAG enhances fairness auditing accuracy

Abstract

Artificial intelligence (AI) is increasingly used in clinical settings, yet limited oversight and domain expertise can allow algorithmic bias and safety risks to persist. This study evaluates whether an agentic AI system can support auditing biomedical machine learning models for fairness in early-onset colorectal cancer (EO-CRC), a condition with documented demographic disparities. We implemented a two-agent architecture consisting of a Domain Expert Agent that synthesizes literature on EO-CRC disparities and a Fairness Consultant Agent that recommends sensitive attributes and fairness metrics for model evaluation. An ablation study compared three Ollama large language models (8B, 20B, and 120B parameters) across three configurations: pretrained LLM-only, Agent without Retrieval-Augmented Generation (RAG), and Agent with RAG. Across models, the Agent with RAG achieved the highest semantic similarity to expert-derived reference statements, particularly for disparity identification, suggesting agentic systems with retrieval may help scale fairness auditing in clinical AI.