Robotic Desk Organization: A Multi-Primitive Approach to Manipulating Heterogeneous Objects via Environmental Constraints

TL;DR

This paper introduces a task-oriented framework for robotic desktop organization that utilizes environment-assisted primitives and perception enhancements to manipulate heterogeneous objects effectively.

Contribution

It presents a novel multi-primitive approach combining perception, environment constraints, and task planning for complex multi-object desktop organization.

Findings

The framework effectively organizes heterogeneous objects on desks in real-world experiments.

Environment-assisted primitives improve robustness in manipulation tasks.

Perception pipeline enhances pose and keypoint estimation for desktop items.

Abstract

Desktop organization remains challenging for service robots because of heterogeneous objects and diverse manipulation objectives, such as collection and stacking. In this article, a task-oriented framework is presented for organizing planar rigid and deformable objects on desks. A perception pipeline was developed that augments existing datasets with uncommon desktop items and makes geometry-based pose and keypoint estimation possible, along with the detection of environmental constraints, such as table edges. To handle diverse manipulation requirements, environment-assisted primitives are used, including contact-based grasping for small objects, edge-based push-grasping for planar rigid objects, and levering-based grasping for planar deformable objects. These primitives leverage environmental and interobject constraints to improve robustness. A task planner was designed to integrate these primitives into multiobject organization. Sufficient real-world experiments demonstrate the effectiveness and robustness of the proposed framework. This research provides practical manipulation primitives for planar rigid and deformable objects, highlighting the role of environmental and interobject constraints in complex multiobject manipulation tasks. Code and video are available online.