Human-Object Interaction from Human-Level Instructions

TL;DR

This paper presents a comprehensive system that interprets human instructions to generate realistic, physically plausible human-object interactions, including detailed finger movements, for complex environments using language models and reinforcement learning.

Contribution

It introduces the first system capable of synthesizing detailed, long-horizon human-object interactions driven by instructions, integrating language understanding with physics-based motion generation.

Findings

Successfully generates realistic human-object interactions in complex environments

Capable of detailed finger-object interaction synthesis

Ensures physical plausibility through reinforcement learning

Abstract

Intelligent agents must autonomously interact with the environments to perform daily tasks based on human-level instructions. They need a foundational understanding of the world to accurately interpret these instructions, along with precise low-level movement and interaction skills to execute the derived actions. In this work, we propose the first complete system for synthesizing physically plausible, long-horizon human-object interactions for object manipulation in contextual environments, driven by human-level instructions. We leverage large language models (LLMs) to interpret the input instructions into detailed execution plans. Unlike prior work, our system is capable of generating detailed finger-object interactions, in seamless coordination with full-body movements. We also train a policy to track generated motions in physics simulation via reinforcement learning (RL) to ensure physical plausibility of the motion. Our experiments demonstrate the effectiveness of our system in synthesizing realistic interactions with diverse objects in complex environments, highlighting its potential for real-world applications.