AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data

TL;DR

AgentSGEN introduces a multi-agent LLM-based framework that iteratively collaborates to generate semantically rich synthetic safety-critical scenes, addressing data scarcity in safety-critical AI applications.

Contribution

The paper presents a novel multi-agent system employing LLMs for iterative, semantic-aware synthetic data generation tailored to safety-critical scenarios.

Findings

Effective generation of safety-critical scenes with semantic depth

Balances safety constraints with visual realism

Outperforms prior synthetic data methods in semantic consistency

Abstract

The scarcity of data depicting dangerous situations presents a major obstacle to training AI systems for safety-critical applications, such as construction safety, where ethical and logistical barriers hinder real-world data collection. This creates an urgent need for an end-to-end framework to generate synthetic data that can bridge this gap. While existing methods can produce synthetic scenes, they often lack the semantic depth required for scene simulations, limiting their effectiveness. To address this, we propose a novel multi-agent framework that employs an iterative, in-the-loop collaboration between two agents: an Evaluator Agent, acting as an LLM-based judge to enforce semantic consistency and safety-specific constraints, and an Editor Agent, which generates and refines scenes based on this guidance. Powered by LLM's capabilities to reasoning and common-sense knowledge, this collaborative design produces synthetic images tailored to safety-critical scenarios. Our experiments suggest this design can generate useful scenes based on realistic specifications that address the shortcomings of prior approaches, balancing safety requirements with visual semantics. This iterative process holds promise for delivering robust, aesthetically sound simulations, offering a potential solution to the data scarcity challenge in multimedia safety applications.