The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

TL;DR

This paper reveals fundamental limitations in self-evolving multi-agent AI systems, showing that safety alignment inevitably degrades over time due to intrinsic dynamical risks, both theoretically and empirically.

Contribution

It introduces an information-theoretic framework to formalize safety in self-evolving AI societies and demonstrates the impossibility of maintaining safety invariance under continuous self-evolution.

Findings

Safety degrades irreversibly in self-evolving AI systems

Theoretical proof of the impossibility of safety invariance in isolated systems

Empirical evidence from Moltbook and other systems supports safety erosion prediction

Abstract

The emergence of multi-agent systems built from large language models (LLMs) offers a promising paradigm for scalable collective intelligence and self-evolution. Ideally, such systems would achieve continuous self-improvement in a fully closed loop while maintaining robust safety alignment--a combination we term the self-evolution trilemma. However, we demonstrate both theoretically and empirically that an agent society satisfying continuous self-evolution, complete isolation, and safety invariance is impossible. Drawing on an information-theoretic framework, we formalize safety as the divergence degree from anthropic value distributions. We theoretically demonstrate that isolated self-evolution induces statistical blind spots, leading to the irreversible degradation of the system's safety alignment. Empirical and qualitative results from an open-ended agent community (Moltbook) and two closed self-evolving systems reveal phenomena that align with our theoretical prediction of inevitable safety erosion. We further propose several solution directions to alleviate the identified safety concern. Our work establishes a fundamental limit on the self-evolving AI societies and shifts the discourse from symptom-driven safety patches to a principled understanding of intrinsic dynamical risks, highlighting the need for external oversight or novel safety-preserving mechanisms.