Collaboration Dynamics and Reliability Challenges of Multi-Agent LLM Systems in Finite Element Analysis

TL;DR

This study investigates how multi-agent LLM systems perform in finite element analysis, revealing that role diversity and validation strategies are crucial for reliable engineering computations.

Contribution

The paper introduces a framework analyzing multi-agent LLM collaboration in FEA, identifying key factors affecting solution correctness and proposing design principles for trustworthy systems.

Findings

Role complementarity improves solution accuracy

Redundant reviewers can reduce success rates

Systematic failure modes include affirmation bias and premature consensus

Abstract

Large Language Model (LLM)-based multi-agent systems are increasingly applied to automate computational workflows in science and engineering. However, how inter-agent dynamics influence reasoning quality and verification reliability remains unclear. We study these mechanisms using an AutoGen-based multi-agent framework for linear-elastic Finite Element Analysis (FEA), evaluating seven role configurations across four tasks under a fixed 12-turn conversation limit. From 1,120 controlled trials, we find that collaboration effectiveness depends more on functional complementarity than team size: the three-agent Coder-Executor-Critic configuration uniquely produced physically and visually correct solutions, while adding redundant reviewers reduced success rates. Yet three systematic failure modes persist: (1) affirmation bias, where the Rebuttal agent endorsed rather than challenged outputs (85-92% agreement, including errors); (2) premature consensus caused by redundant reviewers; and (3) a verification-validation gap where executable but physically incorrect code passed undetected. No agent combination successfully validated constitutive relations in complex tasks. Building on theories of functional diversity, role differentiation, and computational validation, we propose actionable design principles: (i) assign complementary agent roles, (ii) enforce multi-level validation (execution, specification, physics), and (iii) prevent early consensus through adversarial or trigger-based interaction control. These findings establish a principled foundation for designing trustworthy LLM collaborations in engineering workflows.