"Tidy Up the Table": Grounding Common-sense Objective for Tabletop Object Rearrangement

TL;DR

This paper presents a grounding approach that combines language and visual-spatial reasoning to improve object arrangement for table tidiness, leveraging LLMs and synthetic data for effective, scalable tidiness assessment.

Contribution

It introduces a novel method that decomposes tidiness into semantic and visual-spatial aspects, integrating language models with visual scoring for improved object rearrangement.

Findings

Successfully generates tidy 2D arrangements for various objects

Uses synthetic data from random walks to train the tidiness score

Demonstrates potential for 3D stacking in object arrangement

Abstract

Tidying up a messy table may appear simple for humans, but articulating clear criteria for tidiness is challenging due to the ambiguous nature of common sense reasoning. Large Language Models (LLMs) have proven capable of capturing common sense knowledge to reason over this vague concept of tidiness. However, they alone may struggle with table tidying due to the limited grasp on the spatio-visual aspects of tidiness. In this work, we aim to ground the common-sense concept of tidiness within the context of object arrangement. Our survey reveals that humans usually factorize tidiness into semantic and visual-spatial tidiness; our grounding approach aligns with this decomposition. We connect a language-based policy generator with an image-based tidiness score function: the policy generator utilizes the LLM's commonsense knowledge to cluster objects by their implicit types and functionalities for semantic tidiness; meanwhile, the tidiness score function assesses the visual-spatial relations of the object to achieve visual-spatial tidiness. Our tidiness score is trained using synthetic data generated cheaply from customized random walks, which inherently encode the order of tidiness, thereby bypassing the need for labor-intensive human demonstrations. The simulated experiment shows that our approach successfully generates tidy arrangements, predominately in 2D, with potential for 3D stacking, for tables with various novel objects.