Architecting Trust in Artificial Epistemic Agents

TL;DR

This paper discusses the importance of designing trustworthy epistemic AI agents that align with human knowledge norms to support reliable, collaborative, and resilient knowledge ecosystems, emphasizing evaluation, governance, and socio-epistemic infrastructure.

Contribution

It introduces a normative framework for building and maintaining trustworthiness in epistemic AI agents, focusing on alignment, competence, and socio-epistemic support systems.

Findings

Trustworthy AI agents must demonstrate epistemic competence.

Robust falsifiability is essential for epistemic reliability.

Supporting socio-epistemic infrastructure enhances human-AI knowledge collaboration.

Abstract

Large language models increasingly function as epistemic agents -- entities that can 1) autonomously pursue epistemic goals and 2) actively shape our shared knowledge environment. They curate the information we receive, often supplanting traditional search-based methods, and are frequently used to generate both personal and deeply specialized advice. How they perform these functions, including whether they are reliable and properly calibrated to both individual and collective epistemic norms, is therefore highly consequential for the choices we make. We argue that the potential impact of epistemic AI agents on practices of knowledge creation, curation and synthesis, particularly in the context of complex multi-agent interactions, creates new informational interdependencies that necessitate a fundamental shift in evaluation and governance of AI. While a well-calibrated ecosystem could augment human judgment and collective decision-making, poorly aligned agents risk causing cognitive deskilling and epistemic drift, making the calibration of these models to human norms a high-stakes necessity. To ensure a beneficial human-AI knowledge ecosystem, we propose a framework centered on building and cultivating the trustworthiness of epistemic AI agents; aligning AI these agents with human epistemic goals; and reinforcing the surrounding socio-epistemic infrastructure. In this context, trustworthy AI agents must demonstrate epistemic competence, robust falsifiability, and epistemically virtuous behaviors, supported by technical provenance systems and "knowledge sanctuaries" designed to protect human resilience. This normative roadmap provides a path toward ensuring that future AI systems act as reliable partners in a robust and inclusive knowledge ecosystem.