Learning Human Skill Generators at Key-Step Levels

TL;DR

This paper introduces a new task called Key-step Skill Generation (KS-Gen) to generate key steps of human skills in videos, along with a dataset and a multi-stage framework to improve the synthesis of complex, multi-step human actions.

Contribution

The paper proposes the KS-Gen task, a novel framework combining multimodal language models, key-step image generation, and video synthesis to generate human skill videos more effectively.

Findings

The framework improves temporal consistency in skill video generation.

A new dataset and evaluation metrics for key-step skill generation are introduced.

Results demonstrate better handling of complex, multi-step human actions.

Abstract

We are committed to learning human skill generators at key-step levels. The generation of skills is a challenging endeavor, but its successful implementation could greatly facilitate human skill learning and provide more experience for embodied intelligence. Although current video generation models can synthesis simple and atomic human operations, they struggle with human skills due to their complex procedure process. Human skills involve multi-step, long-duration actions and complex scene transitions, so the existing naive auto-regressive methods for synthesizing long videos cannot generate human skills. To address this, we propose a novel task, the Key-step Skill Generation (KS-Gen), aimed at reducing the complexity of generating human skill videos. Given the initial state and a skill description, the task is to generate video clips of key steps to complete the skill, rather than a full-length video. To support this task, we introduce a carefully curated dataset and define multiple evaluation metrics to assess performance. Considering the complexity of KS-Gen, we propose a new framework for this task. First, a multimodal large language model (MLLM) generates descriptions for key steps using retrieval argument. Subsequently, we use a Key-step Image Generator (KIG) to address the discontinuity between key steps in skill videos. Finally, a video generation model uses these descriptions and key-step images to generate video clips of the key steps with high temporal consistency. We offer a detailed analysis of the results, hoping to provide more insights on human skill generation. All models and data are available at https://github.com/MCG-NJU/KS-Gen.