Towards Trustworthy Multi-Turn LLM Agents via Behavioral Guidance

TL;DR

This paper introduces a framework for improving the reliability and verifiability of multi-turn LLM agents by guiding their behavior through a structured, reinforcement learning-inspired approach.

Contribution

It proposes a novel task completion framework with integrated components for behavioral guidance, reasoning, and output validation to enhance trustworthiness of LLM agents.

Findings

Components co-evolve to produce trustworthy behavior

Framework enables explicit behavioral guidance in LLM agents

Improves reliability and verifiability in multi-turn tasks

Abstract

Large Language Models demonstrate strong reasoning and generation abilities, yet their behavior in multi-turn tasks often lacks reliability and verifiability. We present a task completion framework that enables LLM-based agents to act under explicit behavioral guidance in environments described by reinforcement learning formalisms with defined observation, action, and reward signals. The framework integrates three components: a lightweight task profiler that selects reasoning and generation strategies, a reasoning module that learns verifiable observation - action mappings, and a generation module that enforces constraint-compliant outputs through validation or deterministic synthesis. We show that as the agent interacts with the environment, these components co-evolve, yielding trustworthy behavior.