Tactile Mapping and Localization from High-Resolution Tactile Imprints

TL;DR

This paper presents methods for reconstructing object shapes and localizing objects using high-resolution tactile sensors, enabling precise tactile feedback and reactive manipulation in robotic systems.

Contribution

It introduces novel algorithms for shape reconstruction and localization from tactile imprints, applicable to various 3D objects with high accuracy.

Findings

High-accuracy shape reconstruction from tactile data

Real-time object localization and identification

Effective tactile feedback for in-hand manipulation

Abstract

This work studies the problem of shape reconstruction and object localization using a vision-based tactile sensor, GelSlim. The main contributions are the recovery of local shapes from contact, an approach to reconstruct the tactile shape of objects from tactile imprints, and an accurate method for object localization of previously reconstructed objects. The algorithms can be applied to a large variety of 3D objects and provide accurate tactile feedback for in-hand manipulation. Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accuracy and do on-line object identification and localization, opening the door to reactive manipulation guided by tactile sensing. We provide videos and supplemental information in the project's website http://web.mit.edu/mcube/research/tactile_localization.html.

Figures

Tables

Table 1. TABLE I: Cross-validation between objects.

	Sphere	Cone1	Cone 2	Hollow	Pyramid
All	0.066	0.077	0.062	0.070	0.067
Removed	0.108	0.167	0.073	0.145	0.137

Table 2. TABLE II: Parameter estimation using shape reconstruction

	Cylinder	Semicone		Double-cylinder			Cuboid		Semipyramid
Object
Tactile shape
Parameters	Radius	Base radius	Slope	Big radius	Small radius	Joint	Side 1	Side 2	Base side	Slope
Real Value	25.0	20.0	7.5∘	16.0	11.0	60.0	25.0	20.0	20.0	7.5∘
Estimation	24.8	19.8	7.5∘	15.3	10.4	61.9	26.0	20.9	19.7	7.2∘
Relative error	2.9%	2.2%	0.1%	4.4%	5.5%	3.2%	4%	4.5%	1.5%	4%

Table 3. TABLE III: Localization error for different objects

	Scissors	Tape	Brush	Flashlight	Gum Box
Object
Tactile shape
RANDOM	46.4	38.4	51.3	62.2	36.1
CTI	10.0	10.0	10.0	20.0	20.0
CTI-ICP-1	7.7	7.5	9.7	22.1	20.5
CTI-ICP-5	6.4	6.1	10.0	22.6	20.4
RMSE distribution

Full text

Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accuracy and do on-line object identification and localization, opening the door to reactive manipulation guided by tactile sensing. We provide videos and supplemental information in the project’s website web.mit.edu/mcube/research/tactile_localization.html.

I Introduction

The correlation between hand dexterity and the spatial-and-pressure resolution of its tactile sensors has been of interest for a long time [1]. In the 19th century, Weber explored spatial acuity with the “two-point touch threshold”, i.e., the shortest distance that can be perceived as two separate pressure points. Later, Max von Frey studied the sensitivity to different levels of applied pressure [2]. It comes with no surprise that the regions with finer spatial sensor resolution, and those that are more sensitive to pressure, are the tips of our fingers and the tip of our tongue, both known for their dexterity.

This work builds from a recent interest in image-based tactile sensors such as GelSlim [3] or GelSight [4] which, by virtue of using a soft gel skin and a camera as transducer, achieve very high spatial acuity and pressure sensitivity, yielding highly discriminative tactile signals.

In this paper we study the use of tactile imprints as dense descriptors of contact, and demonstrate an approach to reconstructing the tactile shape of an object to facilitate tactile localization (Fig. 1). This work is part of an approach to robotic manipulation that has at its core:

• $\cdot$

Dense tactile descriptors. High-resolution tactile imprints are dense local descriptors of touch. This opens the door to a large set of classification and regression techniques from machine learning.

• $\cdot$

Crisp tactile memory. Touching something and recognizing with confidence that it has been touched before. This is essential for data association in estimation/reconstruction problems.

• $\cdot$

Fine differentiation. Detection of small differences between very similar tactile imprints can be used for part verification or comparison.

• $\cdot$

Contact force distribution. The deformation of the sensor skin under contact encodes the spatial distribution of internal contact forces. These are key for precise object manipulation [5, 6].

These suggest an approach to manipulation where tactile information does not play a complementary role, but rather a main driver of dexterous manipulation.

In this paper we demonstrate that we can combine tactile imprints with robot kinematics to build a tactile map of an object for localization. To do so, we present 3 contributions:

Local shape estimation: we use tactile imprints to estimate the shape of the contact patch using CNNs. We validate the algorithm in Section IV with known contact geometries that yield sub-millimeter accuracy.

• 2.

Global tactile mapping: we fuse the tactile imprints and the kinematics (gripper pose and opening) of multiple grasps of a fixed object to reconstruct its global tactile shape. This includes the object geometric shape as well as a discrete representation of its tactile imprints. We validate the algorithm in Section V by recovering the main dimensions of known objects with an error lower than 5%.

• 3.

Object tactile localization: Figure 1 illustrates how we combine tactile imprints with an estimation of the shape of the contact patch to identify and localize a grasped object. Our ICP-based algorithm uses tactile imprints for coarse data association, and contact shape for fine refinement. We validate the process in Section VI with controlled quantitative grasping experiments and in Section VII with qualitative results for unstructured picking.

II Related Work

Effective robotic manipulation usually requires a good understanding of the manipulated objects. To this end, many estimation techniques have been proposed to recover object properties or to track and identify them. With good visibility, visual information can be enough to recover and track most objects [7, 8, 9]. However, in the context of robotic manipulation, occlusions are often unavoidable specially when the robot actively interacts and manipulates objects. To mitigate this problem, some works have explored tactile sensing to better estimate object properties or motion [10, 11, 12, 13, 14]. Most of these works heavily rely on visual information and tend to use tactile information for some form of contact detection.

Recent works such as Luo et al. [13] and Falco et al. [14] provide good shape reconstructions or do object localization, but remain restricted to mostly planar objects or use tactile sensors that are bulky and impede dexterous manipulations. To enable more realistic robotic manipulation, some works have used tactile sensors that are more naturally integrated into robotic arms and hands to explore object properties.

There is extensive work based on tactile sensing that aims at recovering the shape of manipulated objects [15, 16, 17, 18, 19, 20, 21, 22, 23]. Among the most recent works, Jamali et al. [18] and Driess et al. [20] actively search for regions that need to be explored based on tactile readings. However, their sensors are low resolution which makes the reconstructed shapes hard to use for other applications. Our approach instead uses a high resolution sensor that not only reconstructs the local contact shape, but reuses their tactile map to enable object identification and localization. We refer the reader to Luo et al. [24] for a more extensive review on tactile estimation of objects properties such as shape.

There has also been work on estimating the location of objects using tactile sensing [25, 26, 27, 28, 29, 30]. While some of these works use tactile sensing only as an indicator of contact [26], others such as Aggarwal and Kirchner [30] or Pezzementi et al. [28] aim to recover both the object motion and its shape. These works are tailored to mostly planar objects and use low resolution tactile sensors. Our approach instead generalizes to 3D objects without relying on prior knowledge.

Our work is also related to the SLAM estimation problem [31, 32, 33]. The application of SLAM to manipulation of 3D objects however has been challenging because of lack of clean information for data association. Our approach relies on building a dense tactile map of each object offline that enables coarse data association and can later be used for multiple manipulation tasks to both identify objects and locate their pose w.r.t. to the tactile sensor.

In this work we use GelSlim [3], a tactile sensor based on Gelsight [34] that provides high resolution tactile imprints in the form of images. The original Gelsight sensor has been used for object localization of small objects [35], to complement a vision-based tracker [36], or recently to recover 3D shapes using also vision and prior shape models [37]. However, its design is bulky for practical use in complex manipulation tasks. Instead, GelSlim is integrated in a slim finger that facilitates manipulation [38, 39]. Leveraging GelSlim’s high resolution, we show that our approach can reconstruct tactile maps of objects and use them efficiently to identify and recover their location.

III Approach: Tactile mapping and localization

We present a framework to reconstruct a tactile map of an object to identify and locate it in-hand. Figure 1 illustrates the process where we combine an off-line reconstructed tactile shape of a flashlight with an on-line imprint to locate its position after being grasped. The framework is composed of three steps to address three main challenges:

• $\cdot$

Local shape estimation. Given a tactile imprint we estimate the local shape of an object during contact. We address this with a self-supervised approach that trains a CNN (Convolutional Neural Net) to map tactile imprints to local contact shapes. The process is described in Section IV.

• $\cdot$

Global tactile and shape mapping. Given a set of tactile imprints from an object and their respective local shapes, we use the kinematics of the robot to reconstruct a global tactile shape. Section V explains this off-line controlled process.

• $\cdot$

Identification and localization. Given an on-line tactile imprint, we compare it to previously reconstructed tactile shapes to identify the object, and to initialize an ICP-based algorithm to locate it within its global tactile map (Sections VI and VII).

IV Local shape estimation

In this section we explain how to recover the local shape of an object from a tactile imprint. The local shape is given as a heightmap and aims to represent the local geometry of an object at contact. In Fig. 2, we use a data-driven approach to build a map between tactile imprints and local shapes.

We automate data collection using a setup that builds upon the system described in [38, 39] which features a robotic arm (ABB IRB 1600id) with a parallel jaw gripper (WSG-50 Weiss) and a GelSlim tactile sensor at each finger [3]. In this work, we only use one tactile sensor, but our approach readily extends to both. During data collection, objects are rigidly attached to an external platform (Fig. 2) and palpated at different locations and orientations.

To find a map between tactile images and local shapes, we used the 5 controlled geometries in Fig. 2 to ensure a diverse set of tactile imprints. These objects provide ground truth heightmaps as they produce identifiable local shapes such as the section of a sphere, a semicone, a semipyramid or a semicone that has a cavity at its center (hollow) that can be easily localized in the imprint to facilitate the autonomous labelling of each tactile imprint. For each object, we collected 600 pairs of tactile imprints and heightmaps, while holding out 100 for testing.

Network Architecture. Given that tactile images and heightmaps are 2D arrays, we leverage standard CNNs. The basic architecture of the CNN we use is a sequential model of 10 convolutional layers with 64 filters each and a 3-by-3 sized kernel. To improve the robustness to illumination changes, we augment the data including random variations in the 3 channels of the tactile images. We also account for translations by adding two extra channels to the input with the $x$ and $y$ position of each pixel. A more in detail explanation of the data collection and training process can be found in the project’s website [40].

Evaluation. We evaluate the quality of the heightmaps on 100 test images per object. The error reported is the RMSE (Root Mean Squared Error) between the actual and predicted heightmaps w.r.t. to the size of the contact patch. To first order, we see that the RMSE decreases by adding more training data and that the average reconstruction accuracy reaches 0.1mm on the test data with only 500 datapoints and 0.060$\pm$0.016mm using 2000 datapoints.

Table I shows the average RMSE (mm) of the heightmap reconstruction on novel shapes by a cross-validation approach. For each training object, we remove it entirely from the training set, train a CNN with 2000 datapoints, and evaluate it both on the whole test set and just on the images from the removed object. Even for hold-out objects, the reconstruction accuracy is on the order of 0.1mm. We observe experimentally that objects like semicone1 or hollow that have unique geometric features, such as edges and holes, are more important to the training set. Smaller objects that leave a more reliable imprint on the sensor are also more informative for the training set.

Finally, Fig. 3 shows three examples of how we recover the local shape of novel complex objects. Results indicate that the map built to translate tactile imprints to heightmaps is accurate at recovering the local geometry of objects and can be extended to reconstruct their global shapes.

V Global Tactile And Shape Mapping

This section describes and analyzes how to combine a set of tactile imprints from an object to recover its tactile shape. This method for shape recovery relies on the accuracy of the robot kinematics and the gripper, and the precision of the heightmaps described in the previous section.

We first explain how each tactile imprint of the tactile map can be localized in the world frame. From the heightmap, we construct a point cloud in the sensor’s frame using an accurate calibration of the intrinsic parameters of the sensor’s camera. We get this calibration by probing with the tactile sensor on known points w.r.t. the robot base. Then we localize the point cloud in the world’s frame by assuming a rigid and calibrated transformation between sensor, gripper and robot arm. Finally, we stitch all the point clouds obtained from the same object by adding them together into a single point cloud. The project’s website [40] provides a more detailed description of these steps.

To test this approach, we reconstructed the tactile shapes of the controlled objects in table II. The data collection process is similar to the one in Section IV where we fix the position and orientation of each object, and grasp it along many locations and orientations while recording the tactile imprints. For uniformity of resolution, the grasps follow an equispaced grid of 10mm of resolution along the planes defined by the gripper orientation w.r.t. the fixed object. We considered 3 orientations for the grasp: 0 and $\pm$20∘ in the vertical axis to promote diversity in the tactile imprints.

We evaluate the accuracy of the tactile maps by estimating the main dimensions of each object. We see in table II that the error in most parameters is less than a few millimeters. From our experience, a denser grid of tactile imprints improves the accuracy of the tactile map for some objects at the cost of longer exploration times. We opted for keeping the exploration equal for all the analyzed tactile maps to enable fair comparison among objects.

VI Tactile Localization

Given the tactile map of an object, our goal is to effectively use it for robotic manipulation. To show this, we examine and evaluate how to localize the object based on correspondences between the tactile shape and local tactile imprints.

The proposed approach is described in Fig. 4. We first recover the local shape in the sensor’s frame as a point cloud using its tactile imprint as in Section IV. Then we stitch this local point cloud to the global tactile shape of the object and infer how the resulting point cloud is located w.r.t to the tactile sensor. Finally, given the robot kinematics and the gripper opening we estimate the actual pose of the object in the world frame. We consider 3 algorithms to stitch the local point cloud: RANDOM, CTI, and our proposed approach CTI-ICP-N which uses CTI to provide a coarse approximation of the object pose and ICP to refine it.

RANDOM. Assumes that the sensor’s pose during contact is the same as one of the poses (randomly selected) associated with the tactile imprints used to build the global shape. This approach is naive but sets a baseline to assess the performance of other methods.

CTI. The pose of the tactile sensor for the new imprint corresponds with the closest tactile imprint (CTI) used to construct the global shape. To compute the similarity between touches, we first map each tactile image to a feature vector using as encoder the predictions of a ResNet50 trained on ImageNet [41] without its last fully connected layers. Next we measure the cosine distance between images and only consider those whose gripper opening differs in less than 2mm from the current gripper opening. As a result, the new touch is stitched to the global shape as if it was recorded at the same location as its closest tactile imprint.

CTI-ICP-N This approach combines CTI with ICP (Iterative Closest Point) as described in Fig. 4. We start by finding the N closest images to the new tactile imprint, using the same metric as CTI. Then we build an auxiliary point cloud that includes the shapes from all those N closest imprints of the global shape. Finally we do ICP to relocate the local point cloud (initially at the pose of the closest imprint) w.r.t to the auxiliary point cloud to better estimate its pose.

The case where N is the total number of tactile imprints is equivalent to doing ICP between the local point cloud and the global one, but in practice does not provide better results than considering small values of N and it is more time consuming. We hypothesize this is because of the sparsity and size of the global shape that can be confusing to ICP. By cropping the global shape we indirectly provide focus to the ICP’s exploration which reduces the chance that it will get stuck in a local minimum. Figure 4 exemplifies how using low values of N prioritizes comparisons of the new point cloud with regions that are more similar to it making the approach more desirable both in terms of accuracy and computation.

Evaluation. To evaluate the approach, we considered five common objects and built their global maps as in Section V by rotating them 0, 90, 180 and 270∘. Table III shows the objects and their reconstructed tactile shapes. The objects vary in several dimensions including weight, length, width, symmetries and texture. For instance, the gum box is small and symmetric while the brush is longer and exhibits different tactile imprints depending on the grasp.

We use a cross-validation approach to evaluate the accuracy of the algorithms. For each object, we remove one of the touches used to build its global shape and aim to localize it back. We measure the error in the relocation by computing the RMSE between the original point cloud and its final position after estimating its pose. Results in table III show the median values of the RMSEs after removing each local point cloud once from the global tactile map. CTI, i.e., finding the closest tactile imprint to the removed one, has high accuracy as the median RMSE is on the order of 10mm for 3 of the 5 objects and 20mm for the other two. This is encouraging because the distance between grasps for the same object orientation is at least 10mm. For the gum box, we observed that even with a more dense grid it is difficult to achieve an error lower than 20mm because of its symmetries: tactile imprints separated by 20mm look identical at plain sight which makes their local shapes very similar.

From the results, we also conclude that for objects with a wide distribution of contact patches, doing ICP can boost the localization accuracy. The median of RMSEs for the scissors and the tape already shows that applying CTI-ICP-5 leads to lower RMSEs than CTI. Figure 5 compares the distribution of RMSEs for RANDOM, CTI and CTI-ICP-5 for the hair brush. As expected, the distribution of errors is tighter and closer to zero for CTI and CTI-ICP-5. However, we observe that CTI-ICP-5 leads to a greater improvement in those cases where there is an initial good guess as its distribution is more titled to the left compared to CTI. While the improvements of CTI-ICP-5 against CTI can not be seen directly through the median of the RMSEs, the histograms show that CTI-ICP-5 accurately refines the pose of some local point clouds.

Size of the tactile shape. We study how the number of tactile imprints used to build the tactile shape affects the accuracy. To tackle this question, we built several global maps for the scissors using only a percentage (10, 25, 50 or 75%) of the touches and evaluated the accuracy when stitching one of the discarded touches to the incomplete tactile maps. Figure 6 shows the RMSE median for CTI, CTI-ICP-1 and CTI-ICP-5 depending on the number of point clouds. As expected, adding more point clouds helps reduce the error in localization. However, after considering a 25% percent of the point clouds the median RMSE for CTI stays at 10mm. In comparison, CTI-ICP-N approaches improve their accuracy with more touches as they adjust the position of the stitched point cloud reducing almost by half the error.

VII In-hand identification and localization

Given a set of tactile shapes from explored objects and a new tactile imprint, our goal is to recover both the identity of the object and its location in-hand. We identify it by comparing the new tactile imprint to the ones used to create the global maps of each object, and assigning the identity from the most similar tactile imprint. Once identified, we stitch the new tactile imprint to its global shape as in Section VI and obtain its pose w.r.t. to the tactile sensor.

Figure 7 shows 3 examples of grasped objects and their tactile imprints. Our approach correctly identifies each object and estimates its location in-hand. The solution is fast enough to provide real time estimations of the pose as the only steps are: one CNN pass, a similarity comparison in a feature vector space and ICP with small point clouds.

VIII Discussion and Future work

This paper presents an approach to build a tactile shape and use it to identify and localize an object in-hand. Given a tactile imprint, we learn an accurate model of the local shape at contact. By combining several tactile imprints we build a tactile map of an object and use it for localization. We compare several algorithms for object location given a new tactile imprint and show that the accuracy of the estimates is tightly related to the resolution of the tactile shape, but can be improved combining CTI for coarse localization and ICP for fine refinement. As a result, this approach yields accurate tactile feedback that can be used in real manipulation tasks.

While our approach has the potential to significantly enhance in-hand manipulation through tactile feedback, there are many opportunities for improvement:

Using visual information. The approach presented is purely tactile-based, but that answers to the desire to prove a point: tactile is an enabler of manipulation. A practical system would also exploit visual feedback to create more complete maps. We believe this is possible as our tactile maps provide an estimation of the global shape in the form of point cloud.

Considering prior information. For many tasks, object information is already available offline. Using shape priors or object properties such as mass or texture can help our approach produce a more accurate tactile map.

Dealing with hard-to-localize objects. Some objects are harder or impossible to accurately localize with a single tactile imprint because of their symmetries or lack of texture. In these situations, our approach can still provide a set of possible poses for the object. The combination of multiple imprints could help to better estimate its actual pose.

Using multiple tactile sensors. We only use one tactile sensor, but extending to multiples is relatively straightforward. For a two-jaw gripper, two sensors would likely make our approach more discriminative and accurate at location.

Handling negative information. Our high resolution sensor does not only provide information about where contact happens, but also where it does not. However, our current solution does not use this information despite we believe adding these constraints could greatly enhance performance.

Improving tactile similarity. Super-sizing tactile data collection can provide many more tactile imprints and allow to train a better feature space to discriminate between them.

Improving stitching technique. Using ICP on the point clouds from the tactile imprints has proven more challenging than what we expected. We believe there are ways to mitigate this issue by choosing the right metrics to deal with the sparsity of tactile point clouds.

To make sure our approach is robust and useful for complex manipulation tasks, our next steps include addressing many of the points discussed above. We believe our method provides a reliable and accurate source of tactile feedback that will open a lot of research possibilities within a tactile approach to dexterous manipulation.