FaGeL: Fabric LLMs Agent empowered Embodied Intelligence Evolution with Autonomous Human-Machine Collaboration

TL;DR

FaGeL is an embodied agent integrating smart fabric technology and LLMs to enable autonomous, adaptive human-agent collaboration through implicit feedback and interpretability tools, advancing open space exploration and AI alignment.

Contribution

Introduces FaGeL, a novel embodied agent with smart fabric integration, and the DualCUT algorithm for improved AI alignment and autonomous behavior evolution.

Findings

FaGeL can autonomously generate and refine tasks based on multimodal sensor data.

The DualCUT algorithm enhances token-level alignment and interpretability.

Experimental results show FaGeL's effective adaptation in cooperative tasks.

Abstract

Recent advancements in Large Language Models (LLMs) have enhanced the reasoning capabilities of embodied agents, driving progress toward AGI-powered robotics. While LLMs have been applied to tasks like semantic reasoning and task generalization, their potential in open physical space exploration remains underexplored. This paper introduces FaGeL (Fabric aGent empowered by embodied intelligence with LLMs), an embodied agent integrating smart fabric technology for seamless, non-intrusive human-agent interaction. FaGeL autonomously generates tasks using multimodal data from wearable and ambient sensors, refining its behavior based on implicit human feedback in generated text, without explicit ratings or preferences. We also introduce a token-level saliency map to visualize LLM fine-tuning, enhancing the interpretability of token-level alignment. The system leverages dual feedback mechanisms to improve token-level alignment and addresses challenges in non-intrusive human-machine interaction and cognition evolution. Our contributions include FaGeL's development, the DualCUT algorithm for AI alignment, and experimental validation in cooperative tasks, demonstrating FaGeL's ability to adapt and evolve autonomously through implicit feedback. In the future, we plan to explore FaGeL's scalability in dynamic environments and its integration with other AI systems to develop AGI agents that adapt seamlessly to diverse human needs.