Detecting Non-Optimal Decisions of Embodied Agents via Diversity-Guided Metamorphic Testing

TL;DR

This paper introduces NoD-DGMT, a framework using diversity-guided metamorphic testing to detect non-optimal decisions in embodied agents, improving evaluation of their decision quality beyond functional correctness.

Contribution

It formalizes the problem of non-optimal decisions in embodied agents and proposes four metamorphic relations along with a diversity-guided selection strategy for effective detection.

Findings

Achieves 31.9% violation detection rate on average.

Diversity-guided filtering improves detection rates by 4.3%.

Outperforms six baseline methods with 16.8% relative improvement.

Abstract

As embodied agents advance toward real-world deployment, ensuring optimal decisions becomes critical for resource-constrained applications. Current evaluation methods focus primarily on functional correctness, overlooking the non-functional optimality of generated plans. This gap can lead to significant performance degradation and resource waste. We identify and formalize the problem of Non-optimal Decisions (NoDs), where agents complete tasks successfully but inefficiently. We present NoD-DGMT, a systematic framework for detecting NoDs in embodied agent task planning via diversity-guided metamorphic testing. Our key insight is that optimal planners should exhibit invariant behavioral properties under specific transformations. We design four novel metamorphic relations capturing fundamental optimality properties: position detour suboptimality, action optimality completeness, condition refinement monotonicity, and scene perturbation invariance. To maximize detection efficiency, we introduce a diversity-guided selection strategy that actively selects test cases exploring different violation categories, avoiding redundant evaluations while ensuring comprehensive diversity coverage. Extensive experiments on the AI2-THOR simulator with four state-of-the-art planning models demonstrate that NoD-DGMT achieves violation detection rates of 31.9% on average, with our diversity-guided filter improving rates by 4.3% and diversity scores by 3.3 on average. NoD-DGMT significantly outperforms six baseline methods, with 16.8% relative improvement over the best baseline, and demonstrates consistent superiority across different model architectures and task complexities.