Language-free Compositional Action Generation via Decoupling Refinement

TL;DR

This paper introduces a language-free framework for compositional 3D action generation that leverages energy models, CVAE, and self-supervised refinement, eliminating the need for extensive language annotations.

Contribution

It proposes a novel, language-free approach with a decoupling refinement process and new datasets, advancing compositional action generation without relying on language auxiliary data.

Findings

Effective compositional action generation demonstrated

New datasets HumanAct-C and UESTC-C created

Quantitative and qualitative results validate approach

Abstract

Composing simple elements into complex concepts is crucial yet challenging, especially for 3D action generation. Existing methods largely rely on extensive neural language annotations to discern composable latent semantics, a process that is often costly and labor-intensive. In this study, we introduce a novel framework to generate compositional actions without reliance on language auxiliaries. Our approach consists of three main components: Action Coupling, Conditional Action Generation, and Decoupling Refinement. Action Coupling utilizes an energy model to extract the attention masks of each sub-action, subsequently integrating two actions using these attentions to generate pseudo-training examples. Then, we employ a conditional generative model, CVAE, to learn a latent space, facilitating the diverse generation. Finally, we propose Decoupling Refinement, which leverages a self-supervised pre-trained model MAE to ensure semantic consistency between the sub-actions and compositional actions. This refinement process involves rendering generated 3D actions into 2D space, decoupling these images into two sub-segments, using the MAE model to restore the complete image from sub-segments, and constraining the recovered images to match images rendered from raw sub-actions. Due to the lack of existing datasets containing both sub-actions and compositional actions, we created two new datasets, named HumanAct-C and UESTC-C, and present a corresponding evaluation metric. Both qualitative and quantitative assessments are conducted to show our efficacy.