Agent Alignment in Evolving Social Norms

TL;DR

This paper introduces EvolutionaryAgent, an evolutionary framework for aligning AI agents with evolving social norms, emphasizing adaptation through environmental feedback and survival principles to improve alignment over time.

Contribution

It proposes a novel evolutionary approach to agent alignment that accounts for dynamic social norms and environmental feedback, moving beyond passive alignment methods.

Findings

EvolutionaryAgent improves alignment with social norms over time.

The approach maintains general task proficiency during adaptation.

Effective across multiple LLM-based agents and environments.

Abstract

Agents based on Large Language Models (LLMs) are increasingly permeating various domains of human production and life, highlighting the importance of aligning them with human values. The current alignment of AI systems primarily focuses on passively aligning LLMs through human intervention. However, agents possess characteristics like receiving environmental feedback and self-evolution, rendering the LLM alignment methods inadequate. In response, we propose an evolutionary framework for agent evolution and alignment, named EvolutionaryAgent, which transforms agent alignment into a process of evolution and selection under the principle of survival of the fittest. In an environment where social norms continuously evolve, agents better adapted to the current social norms will have a higher probability of survival and proliferation, while those inadequately aligned dwindle over time. Experimental results assessing the agents from multiple perspectives in aligning with social norms demonstrate that EvolutionaryAgent can align progressively better with the evolving social norms while maintaining its proficiency in general tasks. Effectiveness tests conducted on various open and closed-source LLMs as the foundation for agents also prove the applicability of our approach.