Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task

TL;DR

This paper conducts an ablation study on a multimodal human-robot interaction system, analyzing the impact of language models, perception configurations, and controllers on performance.

Contribution

It systematically isolates and evaluates the contribution of each module to optimize the end-to-end system performance.

Findings

Identifies which modules most influence execution time and success rate.

Provides insights into optimal combinations of language models, perception, and control.

Guides future engineering improvements for human-robot interaction systems.

Abstract

This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance: the large language model used for action extraction, the perception system used for visual grounding, and the controller used for motion execution. The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end. We therefore compare three language models, five perception configurations, and three controllers, followed by a second-stage factorial study over the best candidates. The resulting analysis is intended to clarify which choices primarily affect execution time, which primarily affect success rate, and where the largest engineering gains are likely to come from in future revisions of the system.