Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment

TL;DR

This paper introduces Frictive Policy Optimization (FPO), a novel framework for training language models to manage epistemic and normative risks through explicit control actions and risk-sensitive decision-making.

Contribution

FPO formalizes alignment as an epistemic control problem, introducing a taxonomy of interventions and a unified family of methods for improved epistemic conduct in language models.

Findings

FPO enables explicit management of belief, commitment, and uncertainty over time.

The framework includes a new evaluation method measuring epistemic competence.

FPO spans multiple techniques like reward shaping and trust regions for alignment.

Abstract

We propose Frictive Policy Optimization (FPO), a framework for learning language model policies that regulate not only what to say, but when and how to intervene in order to manage epistemic and normative risk. Unlike standard alignment methods that optimize surface-level preference or task utility, FPO treats clarification, verification, challenge, redirection, and refusal as explicit control actions whose purpose is to shape the evolution of belief, commitment, and uncertainty over time. We formalize alignment as a risk-sensitive epistemic control problem in which intervention decisions are selected based on their expected effect on downstream epistemic quality rather than on immediate reward alone. We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions. We further propose an evaluation framework that measures epistemic competence directly through clarification behavior, calibration, contradiction repair, refusal proportionality, and information efficiency. Together, these results provide a formal and algorithmic foundation for learning agents that are aligned not only in outcome, but in epistemic conduct.