TrustAgent: Towards Safe and Trustworthy LLM-based Agents

TL;DR

TrustAgent is a framework that enhances the safety and trustworthiness of LLM-based agents in high-stake environments by integrating safety strategies at multiple stages of plan generation.

Contribution

The paper introduces an Agent-Constitution-based framework with pre-, in-, and post-planning strategies to improve safety and helpfulness of LLM agents.

Findings

Effective safety enhancement across multiple domains.

Improved agent helpfulness through safety measures.

Highlighting the role of LLM reasoning in safety adherence.

Abstract

The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent's safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.