Reinforcing privacy reasoning in LLMs via normative simulacra from fiction

TL;DR

This paper introduces a novel approach to improve privacy reasoning in large language models by extracting normative norms from fiction and using reinforcement learning to align models with contextual privacy expectations.

Contribution

It proposes extracting structured normative simulacra from fiction novels and integrating them into LLM training via supervised learning and reinforcement learning, enhancing privacy reasoning.

Findings

GRPO with normative grounding outperforms other methods on law compliance benchmarks.

Fiction-derived norms improve models' recognition of privacy-relevant situations.

Models trained with normative simulacra better align with human privacy expectations.

Abstract

Information handling practices of LLM agents are broadly misaligned with the contextual privacy expectations of their users. Contextual Integrity (CI) provides a principled framework, defining privacy as the appropriate flow of information within context-relative norms. However, existing approaches either double inference cost via supervisor-assistant architectures, or fine-tune on narrow task-specific data. We propose extracting normative simulacra (structured representations of norms and information flows) from fiction novels and using them to fine-tune LLMs via supervised learning followed by GRPO reinforcement learning. Our composite reward function combines programmatic signals, including task clarity (subsuming schema validity, construct discrimination, and extraction confidence), structural completeness, internal consistency, and context identification, with an LLM judge that evaluates whether the model's privacy reasoning is grounded in the held-out normative universe of the source text. To mitigate overfitting, we introduce per-completion contrastive scoring: each completion is evaluated against both the correct normative universe and a randomly selected wrong one, teaching the model to condition on context rather than memorize source-specific norms. We evaluate on five CI-aligned benchmarks spanning distinct societal contexts and ablate the contributions of RL and normative grounding. Across seven models, SFT introduces a conservative prior toward restricting information flow, improving recognition of privacy-relevant situations but not the correctness of privacy judgments. GRPO with normative grounding achieves the highest score on a law compliance benchmark and strongest correlation with crowdsourced human privacy expectations, demonstrating that fiction-derived normative simulacra can teach contextual privacy reasoning that transfers to real-world domains.