Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models

TL;DR

This paper introduces a novel approach that combines temporal vision semantics derived from large language models with subspace clustering to improve unsupervised human motion segmentation, achieving superior results on benchmark datasets.

Contribution

It proposes a new method to incorporate temporal semantic information from LLMs into subspace clustering for human motion understanding, which was not explored before.

Findings

Outperforms state-of-the-art methods on four datasets

Effective integration of LLM-derived semantics improves segmentation accuracy

Feedback mechanism enhances subspace embedding optimization

Abstract

Unsupervised human motion segmentation (HMS) can be effectively achieved using subspace clustering techniques. However, traditional methods overlook the role of temporal semantic exploration in HMS. This paper explores the use of temporal vision semantics (TVS) derived from human motion sequences, leveraging the image-to-text capabilities of a large language model (LLM) to enhance subspace clustering performance. The core idea is to extract textual motion information from consecutive frames via LLM and incorporate this learned information into the subspace clustering framework. The primary challenge lies in learning TVS from human motion sequences using LLM and integrating this information into subspace clustering. To address this, we determine whether consecutive frames depict the same motion by querying the LLM and subsequently learn temporal neighboring information based on its response. We then develop a TVS-integrated subspace clustering approach, incorporating subspace embedding with a temporal regularizer that induces each frame to share similar subspace embeddings with its temporal neighbors. Additionally, segmentation is performed based on subspace embedding with a temporal constraint that induces the grouping of each frame with its temporal neighbors. We also introduce a feedback-enabled framework that continuously optimizes subspace embedding based on the segmentation output. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on four benchmark human motion datasets.