ACTLLM: Action Consistency Tuned Large Language Model

TL;DR

ACTLLM is a new large language model designed for robot manipulation that uses language-based scene descriptors and an action consistency constraint to improve visual perception and task execution in dynamic environments.

Contribution

It introduces a novel action consistency constraint and reformulates manipulation as a multi-turn visual dialogue, enhancing long-term task modeling and adaptability.

Findings

Outperforms existing methods in vision-based robot manipulation tasks.

Enhances visual representations with action alignment.

Effective in diverse and challenging scenarios.

Abstract

This paper introduces ACTLLM (Action Consistency Tuned Large Language Model), a novel approach for robot manipulation in dynamic environments. Traditional vision-based systems often struggle to learn visual representations that excel in both task execution and spatial reasoning, thereby limiting their adaptability in dynamic environments. ACTLLM addresses these challenges by harnessing language to craft structured scene descriptors, providing a uniform interface for both spatial understanding and task performance through flexible language instructions. Moreover, we introduce a novel action consistency constraint that aligns visual perception with corresponding actions, thereby enhancing the learning of actionable visual representations. Additionally, we have reformulated the Markov decision process for manipulation tasks into a multi-turn visual dialogue framework. This approach enables the modeling of long-term task execution with enhanced contextual relevance derived from the history of task execution. During our evaluation, ACTLLM excels in diverse scenarios, proving its effectiveness on challenging vision-based robot manipulation tasks.