A Grasping-centered Analysis for Cloth Manipulation

TL;DR

This paper introduces a novel grasp analysis framework for textile objects, abstracting from hand shape and enabling better classification, understanding, and design of cloth manipulation tasks and tools.

Contribution

It proposes a new textile grasp definition based on neuroscience concepts, classifies manipulation primitives, and reviews existing practices to guide future gripper design and benchmarks.

Findings

Most cloth manipulations rely on a single grasp type

Certain tasks require more grasp variety for success

The framework aids in designing better grippers and benchmarks

Abstract

Compliant and soft hands have gained a lot of attention in the past decade because of their ability to adapt to the shape of the objects, increasing their effectiveness for grasping. However, when it comes to grasping highly flexible objects such as textiles, we face the dual problem: it is the object that will adapt to the shape of the hand or gripper. In this context, the classic grasp analysis or grasping taxonomies are not suitable for describing textile objects grasps. This work proposes a novel definition of textile object grasps that abstracts from the robotic embodiment or hand shape and recovers concepts from the early neuroscience literature on hand prehension skills. This framework enables us to identify what grasps have been used in literature until now to perform robotic cloth manipulation, and allows for a precise definition of all the tasks that have been tackled in terms of manipulation primitives based on regrasps. In addition, we also review what grippers have been used. Our analysis shows how the vast majority of cloth manipulations have relied only on one type of grasp, and at the same time we identify several tasks that need more variety of grasp types to be executed successfully. Our framework is generic, provides a classification of cloth manipulation primitives and can inspire gripper design and benchmark construction for cloth manipulation.

Tables

Table 1. TABLE I: Grasps used in literature

Reference	Gripper in Fig. 1	Grasps used
Cuén Rochín et al. 2008 [8]	a	P${\Pi }_e$
Maitin-Shepard et al. 2010 [9]	b	PP, 2PP, 2$\Pi {\Pi }_e$, 2PP+${\Pi }_e$
Bersch et al. 2011 [45]	b	PP, 2PP, 2$\Pi {\Pi }_e$
Miller et al. 2012 [46, 47, 48, 49, 50]	b	PP+${\Pi }_e$, 2PP+${\Pi }_e$
Lee at al. 2015 [51]	b	2PP, 2PP+${\Pi }_e$
Yuba et al. 2017 [10]	c	PP, 2PP, 2PP+${\Pi }_e$
Koishihara et al. 2017 [52]	c	PP, PP+Li1
Willimon et al. 2011 [11]	d	PP+${\Pi }_e$
Moriya et al. 2018 [12]	e	PP
Li et al. 2015 [53, 13]	f	PP, 2PP, 2PP+${\Pi }_e$, PP+${\Pi }_e$
Kita et al. 2011 [14]	g	PP, 2PP,
Osawa et al. 2007 [15]	h	PP, 2PP,
Doumanoglou et al. 2016 [16]	i	PP, 2PP, 2PP+${\Pi }_e$, P${\Pi }_e$+L${\Pi }_e$
Sun et al. 2015 [54]	i	PP+${\Pi }_e$, 2PP+${\Pi }_e$
Sahari et al. 2010 [55, 18, 56]	j,k	PP, 2PP, sh 2PP
Hamajima et al. 2000 [20]	l	PP, sh 2P${\Pi }_e$
Kabaya et al. 1998 [19]	l	PP, sh 2P${\Pi }_e$
Jia et al. 2017 [21]	m	2LL, LL+${\Pi }_e$
Shibata et al. 2012 [57, 22]	n	sh 2PP, sh 2PP+${\Pi }_e$
Colome and Torras 2018 [23]	o	2LL, 2LL+${\Pi }_e$
Sugiura et al. 2010 [24]	p	LL+${\Pi }_e$
Yamakawa et al. 2011 [26]	q	2PP
Ruan et al. 2018 [27]	r	2PP+${\Pi }_e$
Balaguer and Carpin 2011 [30]	s	2PP, 2PP+${\Pi }_e$
Monso et al. 2012 [29]	s	PP, sh2PP, L$\Pi$2
Ramisa et al. 2012 [28]	s	PP
Twardon et al. 2015 [31]	t	PP, 2PP,
Koustoumpardis et al. 2014 [32, 58]	u	PP, sh 2P${\Pi }_e$
Ono and Takase 2007 [34, 33]	v	PP, sh 2P${\Pi }_e$, P${\Pi }_e$, $\Pi {\Pi }_e$
Total grasp instances		63

Table 2. TABLE II: Description of tasks

Task Name	Start	Steps	End
1. (a) Unfold in the air	(H1) PP at p	$\mathrm{𝟏})$ (H2) (RG) Gr v: PP $\mathrm{𝟐})$ (H1) (RG) Rlse PP $\mathrm{𝟑})$ (H1) (RG) Gr v: PP	2PP unfd
1. (b) Unfold in the air	(H1) PP at p	$\mathrm{𝟏})$ (H1) (Ex) Add ${\Pi }_e$: PP+${\Pi }_e$ $\mathrm{𝟐})$ (H1) (S) Slide to reveal corner $\mathrm{𝟑})$ (H2) (RG) Gr v: 2PP+${\Pi }_e$ $\mathrm{𝟒}$ (Ex) Rm ${\Pi }_e$: 2PP $\mathrm{𝟓})$ Repeat from $\mathrm{𝟏})$ to $\mathrm{𝟒})$ changing hands	2PP unfd
1. (c) Unfold in the air tracing	${\Pi }_e$ crumpled	$\mathrm{𝟏})$ (H1) (G) Gr v: PP +${\Pi }_e$ $\mathrm{𝟐}$ (H1) (Ex) Rm ${\Pi }_e$: PP $\mathrm{𝟑})$ (H2) (RG) Gr e near H1: 2PP $\mathrm{𝟒})$ (S) Trace edge until H2 finds v	2PP unfd
2. Fold on table	${\Pi }_e$ flat or fd	$\mathrm{𝟏})$ (G) Gr 2 v: 2PP + ${\Pi }_e$ $\mathrm{𝟐})$ (GM) Fold $\mathrm{𝟑})$ (R) Rlse 2PP on tbl	${\Pi }_e$ fd
3. (a) Fold in the air with table	2LL unfd	$\mathrm{𝟏})$ (GM) Pronation rot. each H $\mathrm{𝟐})$ (Ex) Add ${\Pi }_e$: 2LL+${\Pi }_e$ folding $\mathrm{𝟑})$ (R) Rlse 2LL from inside fold	${\Pi }_e$ fd
3. (b) Fold in the air dynamic	2PP unfd	$\mathrm{𝟏})$ (GM) Fast move cloth until bottom vertices move up $\mathrm{𝟐})$ (RG) Regrasp 2PP to add the bottom vertexs	2PP fd
4. Place on table flat or folding	2PP unfd	$\mathrm{𝟏})$ (Ex) Add ${\Pi }_e$: 2PP + ${\Pi }_e$ $\mathrm{𝟐})$ (R) Rlse 2PP on tbl	${\Pi }_e$ flat or fd
5. (a) Flatten cloth	${\Pi }_e$ p.flat	$\mathrm{𝟏})$ (G) Gr 2{L${\Pi }_e$, P${\Pi }_e$} $\mathrm{𝟐})$ (S) Slide H1, hold H2	${\Pi }_e$(p)flat
5. (b) Flatten cloth	${\Pi }_e$ p.flat	$\mathrm{𝟏})$ (G) Gr 2$\Pi {\Pi }_e$ $\mathrm{𝟐})$ (S) Slide H1 & H2 in opp. dir.	${\Pi }_e$ (p)flat
5. (c) Flatten cloth	${\Pi }_e$ p.flat	$\mathrm{𝟏})$ (G) Gr {v, e}: $\{1,2\}$PP+${\Pi }_e$ $\mathrm{𝟐})$ (GM) Pull H(s) to flatten $\mathrm{𝟑})$ (R) Rlse $\{1,2\}$PPon table	${\Pi }_e$(p)flat
5. (d) Flatten cloth	${\Pi }_e$p.flat (1 v folded)	$\mathrm{𝟏})$ (H1) (G) Gr p sh 2P${\Pi }_e$ to hold cloth $\mathrm{𝟐})$ (H2) (RG) Gr v: PP while H1 holds $\mathrm{𝟑})$ (H1) (R) Rlse sh 2P${\Pi }_e$ $\mathrm{𝟒})$ (H2) (GM) Unfold v and rls PP	${\Pi }_e$ (p)flat
6. Separate from pile (pick & place)	${\Pi }_e$ crumpled	$\mathrm{𝟏})$ (G) Gr p w. PP or $\Pi \Pi$ $\mathrm{𝟐})$ (R) Rlse	${\Pi }_e$ crumpled
7. Pick & place folded cloth	${\Pi }_e$ fd	$\mathrm{𝟏})$ (G) Gr 2PP from 2 sides and lift $\mathrm{𝟐})$ (R) Rlse 2PP on table	${\Pi }_e$ fd
8. Clamp (p)flat cloth (pick up)	${\Pi }_e$ (p)flat	$\mathrm{𝟏})$ (GM) Move gripper finger under cloth $\mathrm{𝟐})$ (G) Grasp cloth at e/v: PP + ${\Pi }_e$ $\mathrm{𝟑})$ (Ex) Rm ${\Pi }_e$	PP at e/v
9. Pinch (p)flat cloth (pick up)	${\Pi }_e$ (p)flat	$\mathrm{𝟏})$ (G) Gr sh 2P${\Pi }_e$ $\mathrm{𝟐})$ (GM) Bring P contacts together $\mathrm{𝟑})$ (RG) Rgr to PP + ${\Pi }_e$ $\mathrm{𝟒})$ (Ex) Rm ${\Pi }_e$	PP at p
10. Pick single layer from stack	${\Pi }_e$ flat	$\mathrm{𝟏})$ (H1) (G) Gr p P${\Pi }_e$ near edge $\mathrm{𝟐})$ (H1) (GM) Push and slide inwards cloth to create crease and hold $\mathrm{𝟑})$ (H2) (RG) Grasp edge PP on crease and (R) Rlse H1	PP at p
11. Grasp gaiting	(H1) PP at p	$\mathrm{𝟏})$ (H2) (RG) Gr p PP near H1 $\mathrm{𝟐})$ (H1) (RG) Rlse PP $\mathrm{𝟑})$ Iterate	PP at p
12. Hang	2PP unfd	$\mathrm{𝟏})$ (Ex) Add Pe: 2PP + Pe $\mathrm{𝟐})$ (R) Rlse 2PP	Pe
13. Put hanger in T-shirt	(H1) PP at p	$\mathrm{𝟏})$ (H2) (RG) Insert side of hanger in neck hole, adding a L $\mathrm{𝟐})$ (H2) (GM) Pull between L and PP to insert other side of hanger	L
14.Japanese style T-shirt fold	${\Pi }_e$ flat	$\mathrm{𝟏})$ (G) Gr2: 2PP + ${\Pi }_e$ $\mathrm{𝟐})$ (GM) H1 moves to opposite edg, H2 holds $\mathrm{𝟑})$ (H1) (RG) Add edge p to PP    $\mathrm{𝟒})$ (Ex) Rm ${\Pi }_e$ $\mathrm{𝟓})$ (GM) shake3    $\mathrm{𝟔})$ (Ex) Add ${\Pi }_e$    $\mathrm{𝟕})$ (R) Rlse 2PP on tbl	${\Pi }_e$ fd

Table 3. TABLE III: Distribution of tasks in literature

Tasks

Osawa et al. 2007 [15]

Maitin-Shepard et al. 2010 [9]

Kita et al. 2011 [14]

Bersch et al. 2011 [45]

Li et al. 2015 [53, 13]

Doumanoglou et al. 2016 [16]

Customano-Towner et al. 2011 [47]

Sahari et al. 2010 [55, 18, 56]

Shibata et al. 2012 [57, 22]

Yuba et al. 2017 [10]

Sugiura et al. 2010 [24]

Miller et al. 2012 [46, 48, 49, 50]

Lee at al. 2015 [51]

Jia et al. 2017 [21]

Colome and Torras 2018 [23]

Yamakawa et al. 2011 [26]

Balaguer and Carpin 2011 [30]

Ruan et al. 2018 [27]

Cuén Rochín et al. 2008 [8]

Willimon et al. 2011 [11]

Sun et al. 2015 [54]

Hamajima et al. 2000 [20]

Monso et al. 2012 [29]

Ramisa et al. 2012 [28]

Moriya et al. 2018 [12]

Kabaya et al. 1998 [19]

Koustoumpardis et al. 2014 [32, 58]

Ono and Takase 2007 [34, 33]

Twardon et al. 2015 [31]

Koishihara et al. 2017 [52]

Bell et al. 2010 [60]

1. (a) Unfold in the air

6

x

x

x

x

x

x

1. (b) Unfold in the air

1

x

1. (c) Unfold sliding

3

x

x

x

2. Fold on table

8

x1

x

x

x

x

x3

x

3. Fold in the air

1

x1

x

3. Dynamic fold

1

x

4. Place on table

5

x1

x1

x

x

x

x

5. (a) Flatten cloth

2

x

x4

5. (b) Flatten cloth

1

x1

5. (c) Flatten cloth

2

x

x

5. (d) Flatten cloth

1

x

6. Separate from pile

3

x

x

x

7. Pick & place fd cl

2

x1

x5

8. Clamp (p)flat cl

5

x

x2

x

x

x

9. Pinch (p)flat Clamp

3

x

x

x

10. Pick single layer

1

x

11. Grasp gaiting

1

x

12. Hang

1

x

13. Put hanger

1

x

14.Japanese fold

1

x

TT

1

5

1

3

3

4

1

2

2

2

1

1

1

1

1

1

1

1

1

2

1

1

1

1

1

2

3

3

1

1

1

Table 4. TABLE IV: Describing manipulation primitives in Table II

Primitive	#
(Ex) Adding/removing extrinsic contact
Add Pe to 2PP	1
Add ${\Pi }_e$ to 2LL, 2PP or PP	1,2,2
Rm ${\Pi }_e$ from 2PP+${\Pi }_e$	2
Rm ${\Pi }_e$ from PP+${\Pi }_e$	2
Total	10
(G) Grasping
Gr P${\Pi }_e$ or sh 2P${\Pi }_e$	1,2
Gr 2P${\Pi }_e$ or 2$\Pi {\Pi }_e$	1,1
Gr $\Pi \Pi$	1
Gr 2PP	1
Gr 2PP + ${\Pi }_e$	2
Gr PP + ${\Pi }_e$	4
Total	13

Full text

I Introduction

Robot manipulation in human environments has experienced great progress in recent years  [1]. However, efforts have been focused mostly on rigid objects, and core capabilities such as grasping, placing, or handing to a person still remain a hard and unsolved problem when dealing with challenging objects such as textiles. Indeed, textiles present many additional challenges with respect to rigid object manipulation, including difficult perception, complexity in modeling the object and predicting its behavior  [2], but it is of special importance in the case of domestic and service robotics as textiles are present in many contexts of our daily lives.

As a consequence, this topic is gaining a lot of attention from EU Research committees. This work is framed in the ERC advanced grant CLOTHILDE, started in 2018, which aims to develop a theory of cloth manipulation based on a topology-rich, compact cloth representation, advanced cloth perception, probabilistic motion planning and learning from demonstrations. In the project, we are studying how to define manipulation complexity and/or performance measures in terms of used grasps, computational time and quality of goal attainment. However, for that to be possible, we need a benchmark. Finding a proper benchmark and replicable experiments is challenging for robot manipulation in general  [3, 4] and, in particular for cloth-like objects, benchmarks are almost inexistent. We believe that a proper classification of the types of grasps and manipulation primitives used in handling textile objects is one of the steps necessary to develop a future such benchmark.

In this work, we analyze and classify the types of grasps and the manipulation primitives that have appeared in literature, focusing on manipulations for handling clothes, including sorting, folding, unfolding, picking & placing, hanging, etc. This is one of the three demanding applications addressed in the CLOTHILDE project, the other two being fitting elastic covers and helping handicapped people to dress, which we leave out of the scope of this paper.

Despite being a relatively novel topic, there is already an extensive literature about robots perceiving and dealing with clothes in household tasks. A good recent review covering cloth-like objects, among others, can be found in  [5]. One of the main conclusions of the study is that manipulation skills still remain underdeveloped compared to vision or control for cloth manipulation and that improved end-effectors need to be developed to better handle this kind of objects. Indeed, there are only a few works focusing specifically on grasping of cloth-like objects, that is, on how and in how many different ways cloth can be grasped, and most of them have been developed for an industrial application and are not suitable for domestic environments  [6, 7].

On the contrary, grasping has been studied extensively for rigid objects  [35, 36]. However, most works assume not only that the object is rigid, but also that it is fixed inside the hand and can be held in any direction. This is not true for clothes in almost any case. Furthermore, grasp classification has been highly dependent on object shape and size, while this distinction does not make much sense for clothes because cloth will conform to the shape of the gripper. Note also that rigid-object grasping research has started only recently to consider extrinsic contacts  [37, 38, 39], that is, the use of contacts with the environment to improve grasp success. Instead, cloth manipulation has been utilizing extrinsic contacts since the very beginning, because they are badly needed to deal with this type of extremely deformable materials. It is worth highlighting that, in the context of rigid objects, environmental contacts are viewed as exploitable constraints to ease their grasping  [39], but not as grasps per se as they don’t fix their pose. Instead, such contacts can be considered grasps for nonrigid objects, since they just partly confine their shape in the same way as grippers do. Finally, cloth manipulation is mostly bimanual and most of the grasps have to be used in couples. In conclusion, we believe that further study of grasping centered on clothes, and considering all the aspects mentioned above, is necessary and could improve the robustness, autonomy and versatility of cloth manipulation by robots.

Another interesting insight of the review in  [5] is that the literature shows a clear tendency to develop techniques focused on a specific task and, thus, finding general solutions remains a major open issue. Identifying manipulation primitives that are common across different tasks can be of crucial relevance to move forward in this direction.

In this work we pave the way towards this end by analyzing previous attempts at robotizing specific cloth manipulation tasks under a common framework derived from three distinguishing traits of textiles: 1) their conforming to the gripper geometry (dual of soft manipulation), 2) the possibility to treat environmental contacts as grasps, thus allowing for a unified treatment, and 3) the need of two simultaneous grasps (bimanual handling) to accomplish most tasks. These traits permit abstracting from the particularities of existing grippers and defining some handy geometries and their combinations, inspired by the concepts of virtual fingers and opposition spaces from early neuroscience works on hand prehension  [40]. The proposed framework delineates a well-defined landscape, which permits identifying grasps missing in previous research that may supply the versatility we are aiming at, leading to alternative ways of performing a given task or even the capacity to tackle new tasks. Other far-reaching possibilities are the design of grippers suitable to accomplish a given repertoire of tasks, and building benchmarks to assess the performance of perception, control and manipulation algorithms using some type of gripper to perform a precisely specified subset of tasks.

The paper is structured as follows. Section II analyzes all the grippers used for cloth manipulation. In Section III we propose a framework to characterize textile grasps and manipulations, which is then applied in Section IV to systematize grasps and manipulations tackled in literature. In Section V we show an experiment to assess the influence of grasp type on task performance, and discuss the implications of our analysis and several possible applications. Finally, in Section VI we provide conclusions and outline our future work.

II Gripper designs for manipulating textiles

Fig. 1 shows, as far as the authors know, all the grippers and robotic hands used in the literature focusing on cloth manipulation and presenting robotic experiments. Next we will describe how these grippers are used and what kinds of grasps they can perform.

First, we observe that when cloth is manipulated by sliding a point contact on a table, no real gripper is needed (Fig. 1.a). Of course, then the functionality is limited to flattening clothes on a table. Most papers tackling general cloth manipulation use simple off-the-shelf parallel grippers like Fig. 1. b to g, from different robot platforms like PR2 robot, HIRO, Nekonote robot arm, Baxter and HRP2 robot, respectively. They all have relatively small planar fingertips and perform pinch grasps. Others use generic robotic hands such as Fig. 1-q to t, corresponding to the RobotiQ, Barret and Shadow hands respectively. In these cases, the hands usually perform a pinch grasp between the thumb and 2 fingers, except in the case of  [28, 29] that use different finger configurations from the Barrett hand to pick and place crumpled clothes, discussed in Table I.

It is interesting to note how those tasks that require a bit more of support use grippers that perform a grasp between two lines. These are Fig. 1. m, o and p. The linear grip in p is used to grasp all the side of a T-shirt, therefore simplifying a bimanual folding operation to using a single hand. The gripper in Fig. 1. m is used in  [21] to maintain tension between grasps and therefore a larger part of the cloth can be controlled. Finally, the gripper in Fig. 1.o was used for the task in  [23]. Colomé and Torras apply learning techniques to fold a T-shirt by rotating the shoulders of the T-shirt in the air. The gripper in the figure performs a grasp between two lines of very short length, and it could almost be considered as a pinch grasp. However, later they redesigned the gripper into the version shown at the top left of Fig. 3. The grasp between longer lines enables to transmit more torque effectively to the cloth, reducing the training time. A comparison of manipulation motions using different gripper geometries will be presented in Section V-A.

Finally, we will pay a bit more of attention to the grippers specially designed for manipulating clothes in domestic environments (Fig. 1.h to p and u and v, that is, the right column of the table). Some of these are parallel grippers with some modification at the fingertips, like Fig. 1.h that has a ball bearing to ease sliding, or Fig. 1.k and l that use rolling fingertips to allow sliding along or pulling upwards a pinched cloth, respectively. Fig. 1.i is effectively a parallel gripper with a very thin finger to slide under flat clothes, and a touch sensor at the other fingertip. A linear brush was added at the side to be able to perform a flattening operation by sliding it on top of the cloth, as shown in the left image in Fig. 1.i. Designs j and n show a similar idea: to have small grippers at the fingertips of a parallel gripper in j or joined by a prismatic actuator in n. These designs allow in-hand manipulation such as grasp gaiting, or edge tracing with a single gripper in the case of n, although such a big long gripper would not be practical in domestic environments. Fig. 1.u shows an under-actuated gripper specially designed for pinching and clamping edges and corners of a flat cloth laying on a table. In addition, it could perform the same pinch grasp as the multifingered robot hands that we mentioned before. Finally, the hand in Fig. 1.v is a humanoid-like 3-fingered hand with a palm, and it is specially designed to be able to slide on a cloth with a flat hand, perform a pinch between the palm and one finger or a pinch between two fingers. These operations are identified as requirements for cloth manipulation in  [33].

In conclusion, off-the-shelf grippers can only perform pinch grasps, which suffice to execute most of the tasks. Other specially-designed grippers exhibit a higher versatility, but more testing is needed to validate their usefulness for a wide range of cloth manipulation tasks.

III A framework proposal for robotized clothes handling

III-A Characterizing grasps in terms of opposing geometric entities

Grasping analysis and taxonomies have been highly influenced by neuroscience works in the 80’s that defined parameters to characterize hand prehensile postures in terms of virtual fingers and opposition spaces. Iberall’s works  [40, 41] established that, to grasp an object, at least two opposing forces against the object surface are needed. The opposition space is defined by the possible directions of the two forces. The concept of virtual finger (VF) groups fingers or hand surfaces that apply forces in the same direction into functional units. In other words, each finger and hand surfaces working in a grasp are assigned to one of the VF, and most grasps can be defined by just two VFs exerting opposing forces. This permits the abstraction from the mechanics of the hand and its fingers.

Iberall’s works were centered on the human hand, and they described hand postures as collections of VFs able to apply effective forces against the object. She showed that standard prehensile classifications like Cutkosky’s  [42] could be captured with this framework and classified in terms of VFs and opposition directions, as done later by Feix et al.  [35].

Although traditionally the concept of VF has been applied to the human hand, the framework could be much widely used to characterize grasping by artificial means as well  [43]. Indeed, it is shown in  [44] that a grasp could be fully described and analyzed in terms of the parameters that describe VFs: parameters of the grasping contact surface patch (length, width and orientation w.r.t. the palm), the orientation of the force the VF could exert and the sensor information.

We propose to define textile grasps in terms of the number and geometry of VFs. In other words, geometric virtual fingers define the shape of the contact patch and the orientation with respect to each VF. Implicitly, this also determines the possible directions of the opposition forces. For textiles, we can characterize most of the grasps using only three geometric entities: points (P), lines (L) and planes ($\Pi$); and we define a grasp as a set of opposing geometric entities. We believe this definition makes sense in the context of pure shapeless objects that will conform to the geometry of the grasping device, also implying the shape of the object will not influence the definition of the grasp.

We can easily see at this point that the environment geometries play a very relevant role as grasping agents to increase functionalities of the grippers. Therefore, we will consider environmental contacts as extra fingers, so that a geometric VF can be either intrinsic, that is, defined by the contact patch of the gripper/hand, or extrinsic, that is, defined by contact surfaces from the environment. Any extrinsic VF will be denoted with the subindex ”e”. In this context, and as done also in  [37], we consider as a grasp an object lying on a table. That is, the table is an extrinsic planar VF, denoted $\Pi_{e}$.

Because cloth manipulation is intrinsically bimanual, we will denote bimanual grasps with the addition symbol, in a way that the flattening operation in Fig. 1.i is represented by P$\Pi_{e}$+L$\Pi_{e}$, and if the grasps are symmetric in both hands, like for instance two hands holding a cloth in the air with two pinch grasps, we simply write 2PP. We assume that any double grasp is bimanual, and we mark it as a single handed one otherwise, using ”sh”. For instance, the grasp in Fig. 1.n is defined as sh 2PP. Similarly, a single grasp is assumed to be single handed, but can be marked as bimanual using ”bm” if it is performed by two hands.

In the context of our framework, we define both prehensile and non-prehensile grasps. Any combination of geometries of different dimensions will lead to a non-prehensile grasp, while the same dimensions lead to prehensile ones. In general, according to definitions in literature  [59], even when the two geometries are the same, like a PP grasp or a $\Pi\Pi$ grasp, they perform a non-prehensile grasp because generally they will not fix the position of the entire object with respect to the hand. However, in the context of our framework, we will consider these as prehensile grasps (prehensile inside the limits of the hand), to distinguish them from the other ”less” prehensile grasps. Finally, any grasp involving only extrinsic geometries, like $\Pi_{e}$, is also non-prehensile because it assumes the second opposition force is gravity.

In Section IV-A we will apply this framework to describe all the grasps used by the grippers in Fig. 1, summarized and classified in Fig. 2.

III-B Manipulation primitives as grasp transitions

Similarly as in  [37], after having established the repertoire of grasps we will deal with, we can consider as manipulation primitives either regrasping actions or motions with a fixed grasp. Within this framework, we define six types of manipulation primitives:

• •

(Ex) Adding/removing extrinsic contacts. These are regrasp actions where the origin/destination has an extrinsic VF while the destination/origin doesn’t, respectively.

• •

(G) Grasping. A regrasp from a single extrinsic VF grasp (first row in Fig. 2) to any other grasp in the figure.

• •

(R) Releasing. The opposite of grasping, that is, a regrasp from any grasp to a single extrinsic VF grasp.

• •

(RG) Regrasping. Pure regrasp action, that is, any transition between grasps that does not belong to the two previous types.

• •

(S) Sliding. Manipulation where the initial and end grasps are the same, but the cloth has moved with respect to the grasp without losing contact completely. Examples include edge tracing or flattening on a table.

• •

(GM) In-grasping motions. Single hand or bimanual grasps that are fixed and moved by the arms without performing any regrasp but changing the state of the cloth. Examples include applying tension between two PP grasps or the motion to fold a cloth after is grasped.

Then, we can define a task as a sequence of these types of manipulations. In Section IV-B we will describe all the tasks addressed in literature as sequences of manipulation primitives of the above six types.

IV Analyzing cloth manipulation tasks using the proposed framework

IV-A Gripper capabilities in terms of the proposed framework

The use of geometric virtual fingers is general in the sense that it can be applied to the human hand by mapping its five fingers and palm to a geometric arrangement of VFs, but it can also be applied to other generic grippers like the ones listed in the previous section. Revisiting the grippers in Fig. 1 we realize that if we define a point as a relatively small flat surface, most of the grippers above perform a PP grasp (a pinch) (Fig. 1.b-g, q-t, i and u) or a sh 2PP grasp (Fig. 1.j and n). Some of them perform a PP grasp with additional abilities to roll over clothes (Fig. 1.h, k and l). In addition, the grippers in Fig. 1.m and p perform an LL grasp.

If we consider the extrinsic geometries they use, the grippers in Fig. 1.b and v can perform a $\Pi\Pi_{e}$ grasp, the gripper in Fig. 1.i a L$\Pi_{e}$ and P$\Pi_{e}$ grasps and the gripper in Fig. 1.u, l and v a sh 2P$\Pi_{e}$ grasp, necessary to perform a pinch of a cloth that is lying flat on a table. The gripper Fig. 1.u is specially designed to be able to perform simultaneously a PP and a sh 2P$\Pi_{e}$ grasp, used to grasp a folded corner while holding the cloth under. The most versatile gripper in terms of possible textile grasps is the one in Fig. 1.v from  [34] that can perform the grasps $\Pi\Pi_{e}$, P$\Pi$, PP and sh 2P$\Pi_{e}$. Table I summarizes the relationship between all the reviewed works, what gripper they use and how many grasps they can realize.

Fig. 2 shows all these grasps represented in a schematic way, as well as the grasp combinations appearing in the reviewed bibliography. Note that bimanual robots performing cloth manipulation tasks in the literature have dealt with up to 5 virtual fingers, leading to up to three couples of opposition forces.

IV-B Encoding previously tackled tasks within the framework

In Table II we list all the tasks that we have identified in literature related to cloth folding. For each task, we define the corresponding sequence of primitives and label each primitive with its type, using the nomenclature proposed in Section III. In addition, Table III shows what works address which tasks.

Unfold in the air (task 1(a)) consist in grasping a crumpled cloth and regrasp it until its gasped flat by two significant points. It has been solved in  [15, 9, 14, 45, 53, 16]. These are all vision-based solutions where grasping points have to be located to regrasp the cloth by the corners (or other grasping points) to permit its recognition. This task relies on vision and recognition algorithms, more than on the manipulation itself. A good review of such algorithms can be found in  [5]. As regards to manipulation, what is important is the gripper ability to grasp the correct point without changing the configuration of the cloth during the approach. In a similar way (task 1(b)), some authors ease the recognition and grasping of cloth corners by sliding the cloth on a table  [47], leading to a slightly modified strategy of the same task (task 1(b)). The same objective of tasks 1(a) and (b) is accomplished in task 1(c) simplifying the vision requirements and increasing the challenge of manipulation. It is based on localizing only one corner on a table (provided it is visible), grasp it, and reach for another corner with the second hand by sliding the gripper along the edge of the cloth without losing it  [10]. To increase the chances of having at least one corner visible, the authors previously slide the cloth on the table, similarly as in task 1(b). This edge tracing manipulation can be faster because it reduces the number of grasping points that need to be localized, but the manipulation of ”pinching and sliding” is of high complexity, in many cases requiring specially designed grippers  [55, 22]. Unfolding in the air is a crucial task to recognize the manipulated cloth and set the canonical cloth state to start any other manipulation, therefore, it is one of the most popular tasks, appearing 10 times in Table III.

Once the cloth is unfolded in the air, placing it on the table (task 4) can be done with an open-loop procedure, but challenges may arise at planning time if relying on a cloth model that needs to accurately deal with friction and collisions  [27]. Moreover the task may require taking into account complex dynamics to ensure the cloth is laid flat  [30].

Only three of the tasks involve folding itself: fold on table, fold in the air and dynamic fold in the air (tasks 2, 3(a) and 3(b)). The first has been the second most attempted task in literature. Fold on table consists in grasping two vertices and folding the cloth until the desired fold is attained  [16, 13, 46]. Planning such folds requires recognizing the cloth and deciding where the folding lines should go  [46] but also to compute the hand trajectories to realize the manipulation  [13]. A big linear gripper can simplify the manipulation  [24] but is not general enough to deal with several pieces of cloth of different sizes. Another challenge is to monitor how the cloth is being folded in a controlled manner, for instance, with vision  [21] or with force control manipulation maintaining tension between hands  [51]. The task of folding in the air is tackled in  [23] and  [45]. It presents several manipulation challenges that are not fully solved in the papers; for instance, the grasping and releasing is not addressed in  [23] and the task is done as an open-loop procedure in  [45] without assessing if the T-shirt is properly folded. This approach to folding promises to be much faster than the other, but it may require different procedures depending on the cloth item. The dynamic fold done in  [26] is also fast, but entails a very complex and risky manipulation, and it requires a high-velocity hardware.

Cloth flattening on the table has been attempted by 7 authors, making it the third most popular task, which has been tackled with several different approaches (task 5(a to d)). Some works slide surfaces of the grippers on top of the cloth  [9, 16, 8] and others grasp edges or vertices and pull them  [11, 54]. The challenges are to understand the state of the cloth, localize the creases and define a direction of sliding or pulling to flatten the cloth. A more challenging task from the manipulation point of view would be that of unfolding a corner that has remained folded. Such grasp is difficult because it is easy to grasp at the same time the cloth that lies under the corner. A specially designed gripper for this task is presented in  [32] (Fig. 1.u) which uses one finger to hold the cloth underneath and another finger to grasp the corner on top.

Another important task is pick and place. Regarding crumpled clothes, pick and place becomes challenging when only one cloth needs to be picked up from a bunch of several tangled clothes (task 6). Again, the main challenge in this task comes from vision  [28, 20]. Picking up folded clothes, a task that would be very relevant in any domestic environment, was done in  [9] using an open-loop procedure and it is approached in more detail in  [12], but only using one hand. The paper focuses on localizing the edge of the folded piece that doesn’t have loose layers and, therefore, it can be picked without being unfolded.

Other popular tasks are those of clamping and pinching (tasks 8 and 9). A pinch consists in grasping the cloth from one side, while in a clamp each finger is in a different side of the cloth. The main challenge for the clamp is to go under the cloth before grasping while, for a pinch, a sh 2P$\Pi_{e}$ grasp needs to be performed and then the cloth dragged by bringing the two point contacts together. The same strategy for pinching could also lead to a clamp  [32]. Works like  [22] create first a crease in the cloth to facilitate clamping. Note that a pinch or a clamp is necessary to grasp the cloth to fold it on a table (task 2), but not all the reviewed works addressing task 2 appear in Table III for clamping or pinching because some of them don’t care if the grasp was a clamp or a pinch as long as it was close to the edge  [9, 46, 13]. However, this distinction may be more relevant when the quality of the end result becomes more important. Nowadays, it is taken into account in papers whose main focus is to design a gripper for textile manipulation  [32, 19, 17].

The last five tasks in the list appear only in one paper. Picking a single layer is approached in  [33] by pressing on the pile to create more spacing allowing to clamp one single layer. However, this approach has been solved in industry with vacuum grippers quite efficiently. Grasp gaiting could substitute the ”pinch and slide” to trace an edge, so as to reduce the risk of losing the cloth  [18]. However, it will probably turn out to be too slow compared to sliding to constitute a practical solution.

Finally, the tasks of hanging clothes or putting hangers inside garments are of obvious utility in domestic and commercial environments and they will probably attract more attention in the near future.

V Applications of the framework

V-A Study of manipulation parameters under different grasp types when folding a T-shirt in the air

So far, we assume and claim that choosing one type of grasp or another among those we have defined could greatly influence the performance of a task. To study this influence in more detail we have designed a simple experiment where we recorded motion data of a subject wearing different types of grippers, using the XSens suit and performing a cloth manipulation task. The hands were rigidly holding the grippers, preventing relative motion between hand and gripper.

We compared the three grippers that are shown at the top of Fig. 3. The first is the evolution of the gripper that appears in Fig. 1-o, it is a parallel gripper where each closing part consists of two points. Assuming two points form a line, we can define that grasp as a LL. The next to the right gripper consists of 2 rigid lines at the bottom, forming a plane, and a linear finger on top. Therefore, that gripper can perform a L$\Pi$ grasp. Finally, the last gripper performs a simple PP grasp. All the grippers are used passively with a human-in-the-loop manipulation, and we assume the cloth is pre-grasped. The task performed is described as the task 3 in Table II, ”Fold a T-shirt in the air”, but in this case there is no release step.

We repeated the tasks 10 times, wearing the three grippers. We collected upper body motion data with the Xsens suit, using an extra sensor attached to the right gripper to directly record its position, velocity and acceleration. This extra sensor is part of the Xsens suit and is usually used to record motion of a manipulated object or a tool, like for instance, a sword. The bottom graphics in Fig. 3 show example trajectories corresponding to one trial of each gripper, each point colored with the corresponding linear acceleration norm. In Fig. 4-(top) we show the different velocity profiles for each gripper. Each curve and shaded area represents the mean and standard deviation of all the trial curves, aligned to match the maximum and minimum peaks. At the bottom, we compare the maximum acceleration peaks for the trials performed with the different grippers.

The results of the experiment show evidence on how grasp type can influence the execution of a task. When the gripper provides more support surface along the T-shirt shoulders, the task of folding requires less acceleration, indicating that the dynamics of the task is greatly simplified. We cannot observe a significant difference between the LL gripper and the L$\Pi$ gripper, they both provide sufficient support and therefore, the acceleration peak is similar. Instead, we can appreciate differences in the trajectories (bottom of Fig. 3) because the rotation pivot point is aligned with the rotation axis of the wrist for the LL gripper, while with the other gripper the user tends to rotate it about the abducted fingertip.

Finally, at the last frames of the task shown in Fig. 3, we can observe that the quality of the fold is also different depending on the gripper used, the results being clearly worst for the PP grasp. We could observe this, but we haven’t included the results because we do not have a fair measure to compare qualities of fold.

V-B Increasing versatility and task repertoire

With the experiment in the previous section we have seen how the dynamics of the task of ”Fold a T-shirt in the air” can be greatly simplified by just using the appropriate grasp type. This can reduce the training times when applying learning techniques, and also change the quality of the result. But there are more examples where improving manipulation skills can reduce the complexity of many of the perception and pose estimation algorithms. For instance, the task ”Unfold in the air” may seem relatively easy by looking at the manipulation primitives used, but finding and localizing the two corners and successfully grasping them is a difficult endeavor that usually involves complex vision and recognition algorithms. A way of reducing the complexity of recognizing the second grasp corner is for the second hand to grasp an edge point close to the first grasp point, and then trace the edge until the next corner is reached, which can potentially be faster than looking for the second corner. This is at the expense of gaining manipulation complexity, meaning that more attention has to be put on touch sensors and increasing contact surfaces to gain more tactile feedback.

There are many other examples of the importance of versatile grasping abilities. The task ”Fold on table”, for instance, has been always addressed in a very similar way: grasping two corners laying on the table and performing a coordinated motion with the arms. However, this may not be the best approach and sometimes it doesn’t yield very good end results. Other approaches could consist in folding with one hand while the other holds the cloth by the folding line. However, for that to happen we probably need a linear grasp on the edge and a planar or linear hold along the folding line, which is not possible with most of the off-the-shelf existing grippers.

Moreover, this approach to folding can’t be applied to large clothes like bed sheets or tablecloths. We humans usually fold them in the air, starting from an unfolded 2PP grasp, to bring the two corners together, regrasp them in a single hand and use the other hand to trace the edge until the end of the fold, to end up again in a 2PP grasp where the cloth has one extra fold. When several of these folds have been done, the last one to finish can be done on the table. For big bed sheets, this same strategy is started with 4 hands (2 people). This is of course a complex manipulation that requires more versatile grippers to be executed successfully and more manipulation skills. However, this kind of operations, or something equivalent, will be needed eventually if robots are to fold big textile objects.

Alternative grasps could also be needed depending on the cloth material. As an example, the task of picking and placing a folded garment (task 7 in Table II) has been implemented in  [9] using a 2PP grasp. This was successful for the folded towels used in the paper, but they required to be flattened after being placed on the table. We believe that attempting this task with only point contacts would not work for a thinner textile material, as it would unfold while transferring it. Using a planar contact under the garment can provide better support to achieve the task, for instance a 2P$\Pi$ or 2L$\Pi$ grasp, and prevent it from unfolding while transferring it or having to flatten it after begin placed. In  [12] the task is performed with a single PP grasp using a parallel gripper. Provided the correct edge is grasped, one can lift the folded cloth without unfolding it, but to place it back one needs to perform a motion similar to ”Place flat on table” (task 4) to prevent unfolding the cloth. This motion requires free space to allow the arm to move. However, if the task requires to place the folded clothes inside a shelf, as it may be expected for folded clothes, that space for the motion may not be available. Therefore, here using a 2P$\Pi$ or 2L$\Pi$ grasp becomes again relevant to reduce the workspace volume needed to perform a task.

Extrinsic contacts are fundamental in cloth folding, for instance to perform a pinch. Other strategies have been explored for rigid objects in  [39] and they should be explored and evaluated in the context of cloth manipulation. For instance, bringing the object to the table edge to facilitate clamping. This has only been considered in  [32] as a scenario to evaluate a novel gripper for clothes, but should receive more attention to evaluate its usefulness with respect to other strategies.

In conclusion, we believe that to achieve a more varied repertoire of manipulations, increase success ratios and improve the quality of results, we need, among other things, to be capable of executing more diverse and sophisticated grasps.

V-C Providing guidance for gripper design

The definition of grasps for textiles using geometric virtual fingers permits expanding the taxonomy in Fig. 2 with as many combinations of geometries one can think of. In the previous section we have already identified some tasks that could use other geometries, like ”Pick & place folded cloth” could use a plane to support the safe transfer of the object or ”Fold a T-shirt in the air” benefits from a linear geometry that can transmit more torque to bend in the shoulders of the T-shirt.

Table IV collects all the manipulation primitives appearing as task steps in Table II, which add up to 62 primitives. The table classifies them by type and counts the multiplicity of each one. We can see how most of the primitives appear only once, indicating the complexity and particularity of each manipulation task. This may explain the lack of general solutions that are found in cloth manipulations, as pointed out in  [5].

It is striking that the implementation of manipulation primitives has so strongly relied on the PP grasp so far. This is probably due to the fact that generic parallel-jaw grippers are the most available and, when designing grippers to handle textiles, they were tailored to a very specific application. More versatile grippers able to deal with cloth could achieve many more primitives with different and more tailored functionalities to ease reaching the objective of every task.

Identifying all the individual primitives that we want a gripper to execute can help to identify the requirements each grasp will need in terms of friction, force or precision, which are to be taken into account when we design grippers. For instance, sliding on cloth or releasing from inside a fold require low friction, but hanging a cloth with a PP grasp may require a lot of force, or increased friction. Grasping a cloth corner requires precision while a 2P$\Pi$ to pick up a folded cloth may not require to grasp a particular point.

In our work  [61, 62] we defined gripper requirements using our framework, selecting three tasks and extracting their required grasp geometries. We selected to implement a LL geometry with one linear finger on top, and two bottom finger that can abduct to morph the gripper into a L$\Pi$ geometry, similar to the gripper in the top middle of Fig. 3. We aimed at providing the skills for performing three tasks: picking and placing folded clothes using a L$\Pi$ grasp, the edge tracing task and folding a T-shirt in the air with this same L$\Pi$ grasp. To execute all the primitives we needed variable friction capacities, so we took inspiration from the design in  [63]. Our design method consisted in implementing progressive simple motor-less prototypes to test the geometries, like the one shown in the top-middle picture in Fig. 3, and to better define the requirements for each primitive. Finally, we implemented a fully functional prototype that could execute the required tasks.

Aiming at versatility in the design of grippers to handle textiles is a very challenging goal which, up to the authors’ knowledge has been addressed only by very few authors before and was solved only partially  [34, 32], probably because there hasn’t been a deep study of the grasping and manipulation variety needed to manipulate clothes. Our framework offers a systematic way to explore grasp combinations, with potential to identify the most useful to enhance manipulation primitives, thus suggesting versatile end-effector designs, including multigrip and reconfigurable grippers able to reshape themselves into virtual fingers with different geometric contacts.

V-D Identifying and characterizing manipulation primitives

Following Section III-B, a given task can be segmented according to the used grasps, to obtain different segments that correspond to manipulation primitives like the ones in Table IV. Combined with motion capture approaches, this can provide large amounts of labeled manipulation data, provided we can either recognize grasp type from the data (using data gloves) or by manually labeling them.

Assigning semantic tags for each of the primitives, as it is done in Table IV, allows to learn semantic sequencing of steps within tasks, with applications to autonomous high-level task planning. On the lower level, having data about trajectories, velocities and accelerations for each of the primitives enables to learn motion representations, for instance DMPs, for each of them. Therefore, this permits defining motion building blocks that can be sequenced to perform a given task. Similar approaches have been successfully applied in literature for rigid object manipulation  [64, 65] or whole-body human motions  [66].

Our framework opens the door to apply these kind of methods to cloth manipulation tasks. In order to successfully implement this kind of approaches fully from collecting data to the execution level, several additional problems need to be solved, including efficient action representation for each of the primitives and efficient cloth state representation, perception and estimation. In our research group we are working in parallel on several of these problems in the context of the ERC Clothilde project.

V-E Benchmarking manipulation

One of the great challenges of benchmarking manipulation is to understand the complexities of all the possible skills that need to be evaluated, thus classification efforts are needed to set up the grounds for a proper benchmark. Several examples exist in literature where classifications like grasping taxonomies  [35], ontologies of actions  [67], or task taxonomies  [4] are used to evaluate robot performance in carrying out tasks, mostly involving rigid objects.

We are working on using the proposed framework to classify and evaluate the complexity of a given task. Assessing the complexity of a task is useful to rank tasks in a benchmark and to enable their evaluation, specially if the evaluation has to be done qualitatively. In other words, if we can just state if a task was successfully completed or not, a quantification of the complexity of the task can attribute value to a solution.

In addition, our framework puts special effort in abstracting from the human hand shape and the robot embodiment, which is of special importance when benchmarking manipulation, as many research labs have different robotic systems solving similar problems, and comparison of results must be independent of the used platform.

VI Conclusions and future research

We have proposed a framework to characterize and systematize grasps, manipulation primitives and tasks for the versatile handling of clothes by robots. Grasps have been redefined in terms of prehension geometries, that we called geometric virtual fingers. This generic definition abstracts from the particular robotic embodiment that is used and can easily inspire gripper designs. Because cloth will conform to the shape of the grasping geometry, this grasp definition is not influenced by object shape or size. Instead, our definition focuses on the geometric shape of the gripper contact area where the robot has to transmit forces to the hanging part of the cloth. In the specialized literature only some combinations of geometries have appeared, the PP being the most prominent one by far. Similarly, the most used grippers can only implement a basic pinch.

We have identified several tasks that are solved in previous works using pinch grasps, which could be improved if more versatile grippers were used. The following types of improvements are envisaged:

• •

More efficient learning and execution of a given task. For example, when folding a T-shirt in the air, an L contact geometry could help bend each shoulder, thus transmitting the appropriate dynamics to the hanging part of the shirt, and reducing required dynamics of the robot arm.

• •

Higher quality of the end result. A clothing item could be better folded if a folding line were rigidly maintained, either intrinsically or extrinsically, or tension were applied between parts of the cloth.

• •

Wider range of textile materials for which a task could be completed. Providing larger support surfaces or multiple well-placed contacts could ensure a suitable end result for objects of varied fabrics.

• •

Increased repertoire of robotized tasks. For instance, it is not possible to fold a big piece of cloth (e.g., a bed sheet) or turn inside out a shirt with state-of-the-art procedures.

• •

Reduced workspace requirements. Clothes could be placed on a tight shelf or a drawer if a gripper implementing 2P$\Pi$ or 2L$\Pi$ grasps were available, capable of compressing a pile of folded clothes.

• •

Reduced auxiliary technical requirements. Dependency on complex vision methods could be reduced by relying on robust manipulation primitives such as sliding to trace edges or flatten a cloth lying on a table.

In addition, our framework considers both intrinsic and extrinsic geometries, allowing for the definition of extrinsic dexterous manipulations in a natural way, just as additional re-grasping operations, that is, particular instances of manipulation primitives as appearing in Table IV.

Our analysis has shown that many relevant cloth manipulation tasks have been only barely addressed so far or require major improvements. Identifying the manipulation primitives and the grasps required by each task following the presented framework will help determine the kind of gripper and the actions a robot needs to be able to perform. Moreover, such systematic task specification is a step towards establishing a benchmark for cloth manipulation, in which alternative strategies could be explored and evaluated in terms of success ratios and quality of end results.

We are currently carrying out a human study –using a motion capture system and some hand-held grippers– to evaluate what grasps humans use to execute a range of cloth handling tasks. Such grasps will then be encoded in terms of geometric virtual fingers and the study will serve to validate and refine our framework, by perhaps expanding the list of grasps and tasks, so as to ultimately generate a complete benchmark of cloth manipulation primitives.