Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI

TL;DR

This paper argues that contestability, enabling humans to challenge and correct AI decisions, is essential for trustworthy multi-agent healthcare systems, addressing limitations of explainability alone.

Contribution

It introduces a human-in-the-loop framework with structured argumentation and role-based contestation to enhance trust and accountability in multi-agent care AI.

Findings

Highlights limitations of current XAI in multi-agent settings

Proposes a framework integrating contestation mechanisms

Emphasizes preserving human agency and clinical responsibility

Abstract

Multi-agent systems (MAS) are increasingly used in healthcare to support complex decision-making through collaboration among specialized agents. Because these systems act as collective decision-makers, they raise challenges for trust, accountability, and human oversight. Existing approaches to trustworthy AI largely rely on explainability, but explainability alone is insufficient in multi-agent settings, as it does not enable care partners to challenge or correct system outputs. To address this limitation, Contestable AI (CAI) characterizes systems that support effective human challenge throughout the decision-making lifecycle by providing transparency, structured opportunities for intervention, and mechanisms for review, correction, or override. This position paper argues that contestability is a necessary design requirement for trustworthy multi-agent algorithmic care systems. We identify key limitations in current MAS and Explainable AI (XAI) research and present a human-in-the-loop framework that integrates structured argumentation and role-based contestation to preserve human agency, clinical responsibility, and trust in high-stakes care contexts.