Grasping Using Tactile Sensing and Deep Calibration

TL;DR

This paper presents a tactile sensing feedback approach combined with deep calibration to improve robot grasping, reducing reliance on visual perception and addressing sensor bias for more effective manipulation.

Contribution

It introduces a novel tactile feedback framework with deep calibration to enhance robot grasping accuracy and robustness.

Findings

Effective tactile feedback improves grasp success.

Deep calibration reduces sensor bias impact.

Framework demonstrated successfully on a real robot.

Abstract

Tactile perception is an essential ability of intelligent robots in interaction with their surrounding environments. This perception as an intermediate level acts between sensation and action and has to be defined properly to generate suitable action in response to sensed data. In this paper, we propose a feedback approach to address robot grasping task using force-torque tactile sensing. While visual perception is an essential part for gross reaching, constant utilization of this sensing modality can negatively affect the grasping process with overwhelming computation. In such case, human being utilizes tactile sensing to interact with objects. Inspired by, the proposed approach is presented and evaluated on a real robot to demonstrate the effectiveness of the suggested framework. Moreover, we utilize a deep learning framework called Deep Calibration in order to eliminate the effect of bias in the collected data from the robot sensors.

Tables

Table 1. TABLE I: Parameters

Parameter	Value
$v_{dz}$	0.0055
${\alpha }_{vx}$	0.0025
${\alpha }_{vy}$	0.0025
${\alpha }_{vz}$	0.4
${\alpha }_{\omega x}$	0.25
${\alpha }_{\omega y}$	0.0025
${\alpha }_{\omega z}$	1
$\beta \omega z$	0.025
$b_x$	1
$b_y$	1
$b_z$	1
$b_{\omega x}$	1
$b_{\omega y}$	1
$b_{\omega z}$	1

Equations

$$sign(f)=-sign(v)$$

$${\tau }_x={\tau }_f$$

$${\tau }_x=\vec{r}\times \vec{f}=||r||||f||sin\theta$$

$${\tau }_z=\vec{r}\times \vec{f}=\mu ||r||||f||sin\theta$$

$$F_e=\left[\begin{array}{c}{f}_e\\ {\tau }_e\end{array}\right]$$

$${\tau }_{int}=J^{T}F$$

$$F=\left[\begin{array}{c}{f}_b\\ {\tau }_b\end{array}\right]$$

$$F=(J{J}^T{)}^{-1}J{\tau }_{int}$$

$$\left[\begin{array}{c}{f}_e\\ {\tau }_e\end{array}\right]=\left[\begin{array}{cc}{R}_b^e& 0\\ 0& R_b^e\end{array}\right]\left[\begin{array}{c}{f}_b\\ {\tau }_b\end{array}\right]$$

$${\tau }_{int}={\tau }_s-{\tau }_{bias}$$

$$\dot{v}+b_{v}v=u_v$$

$$\dot{\omega }+b_{\omega }\omega =u_{\omega }$$

$$u_v={\alpha }_v(f_f-f_e)$$

$$u_{vz}={\alpha }_{vz}(f_{zf}-f_z)$$

$$v_{dz}=\frac{{\alpha }_{vz}{f}_{zf}}{b_z}$$

$$u_{vz}=b_v{v}_{dz}(1+{\alpha }_{vz}{f}_z)$$

$$f_{zf}=-\frac{1}{{\alpha }_{vz}}.$$

$$u_{\omega }={\alpha }_{\omega }({\tau }_d-{\tau }_e)$$

$$u_{\omega z}={\alpha }_{\omega z}({\tau }_{dz}-{\tau }_z)$$

$${\omega }_z=\frac{{\alpha }_{\omega z}{\tau }_{dz}}{b_{\omega z}}$$

$$u_{\omega z}={\alpha }_{\omega z}({\beta }_{\omega z}\ast f_z-{\tau }_z)$$

$$u_{xz}={\alpha }_{vx}{f}_x$$

$$u_{yz}={\alpha }_{vy}{f}_y$$

$$u_{\omega x}={\alpha }_{\omega x}({\tau }_x)$$

$$u_{\omega y}={\alpha }_{\omega y}({\tau }_y)$$

$${\dot{v}}_z+b_z{v}_z=b_z{v}_{dz}(1+{\alpha }_{vz}{f}_z)$$

$${\dot{v}}_z+b_z(v_z-v_{dz})=0$$

$$(v_z-v_{dz})\to 0$$

$$f_z=-K_z\Delta z$$

$${\dot{v}}_z+b_z{v}_z=b_z{v}_{dz}(1-{\alpha }_{vz}{K}_z\Delta z)$$

Full text

I Introduction and Related Work

In the world of active agents, an agent desires to adjust its behavior during interaction with its environment. Such an interaction consists of three main components: sensation, perception, and action. Depending on the task and the sensed modality, these three elements have to be implemented in coordination with each other to have the best performance. Perception is an intermediate step and is responsible for extracting useful information from the sensed data. The quality of extracted information from the sensed data is dependent on the power of designed perception algorithm to mine the data effectively. On the other hand, more powerful algorithms can consume a lot of time and energy from a light robot either in training phase or in inference phase. A promising solution to address these issues is integrated interactive perception approach. In such an approach, the main attention is in minimizing the perception computational processing by coordination of the three components of sensation, perception, and action. Such a coordination highlights the topic of active perception and is referred as a learned policy in the context of reinforcement learning. As is well know, learning this interactive perception from gathered experiments is computationally expensive for light robots; moreover, the learned policy can not be interpreted easily.

In this paper, we aim to propose an interactive tactile perception with use of force-torque sensing for a grasping task. Grasping task can be initiated by either visual or tactile perception. Although the first one is powerful in terms of object recognition, a continuous processing of visual data during interaction is not a simple task for a light robot. In real world, a human also barely utilizes vision ability in close proximity of the object to be grasped; indeed, tactile perception ability is more useful in the vicinity of object [1], [2].

Grasping is an essential and complex daily activity. Through this task, humans show an intention to affect surrounding environment in a controllable manner. Humans primarily utilize a combination of control strategy and learning from repetitive experiments to anticipate grasping in different situations [3]. In this regard, the properties of an object such as size, shape, and contact surface are important parameters during grasping task [4].

Tactile sensation is a very informative feature to recognize object properties. In [5], the authors proposed a tactile perception strategy to measure tactile features for mobile robots. Tactile sensing also is used to propose a robust controller for reliable grasping [6] and slipping avoidance [7]. Visual sensing and tactile sensing are complementary in robot grasping. A combination of both of them through deep architecture is a promising solution in [8], [9]. However, processing these high-dimensional data is not an easy task and a meaningful compact representation would be needed [10], [11]. A robot can learn the manipulation using tactile sensation through demonstrations [12], [10], [13].

A lower dimensional representation of tactile data is also more useful for object material classification [14]. With more processing approaches such as bag-of-words, identification of objects would be possible in advance [15]. Extraction of object pose via touch based perception can be used for manipulation [16]. Moreover, localization will be improved by contact information gathered by tactile sensor [17], [18]. The robot can control and adjust the pose of hand with stability consideration after evaluating the tactile experiences [19], [20].

Tactile feedback and interactive perception are very important components of the grasping task. An introduction of predictive force control and reactive control strategies in this domain is provided in recent years [21]. Interactive perception has been introduced as a potential field of study in recent years [22]. Such importance emerges from the fact that the perception can be facilitated by interaction with the environment.

In this paper, we pursue following objectives and contributions: 1) We show that it is feasible to propose a more human-like grasping using tactile touch sensing. 2) We describe the useful mathematical framework to extract the important tactile sensing information from robot joints. 3) We present the procedure to accurately calibrate the robot joints data in order to successfully mine the tactile sensing. 4) We propose an approach by avoiding sequential logical rules.

This paper is organized as follow. The approach on how to extract tactile sensation data is presented in Section II with grasping steps in Section III. Experimental results are demonstrated in Section IV while conclusions are provided in Section V.

II Tactile Sensing

To implement our proposed approach on automatic grasping based on tactile sensing, we first describe how to extract the tactile data from the robot joint sensors data.

II-A The Six-axis Force/Torque Tactile Data

Grasping is a physical interaction with environment. During such interaction, the exchanged data between the object and robot would be the force and torque data. The robot experiences torque data $\tau_{f}$ as a consequence of inserted force $f$ as shown in Fig. 1.

In this figure, three directions for the sensed force are marked to be recognizable. The direction of this force ( 1 , 2 and 3 ) is dependent of the direction of robot hand movement toward the object $v$ and it is in opposite direction of movement after contact with the object surface:

[TABLE]

where the sign function returns the sign of the data. The sensed force vector can be decomposed into the robot hand frame $x$, $y$ and $z$ in Fig. 1. So, the 3-axes of tactile force data are the set of $f_{e}=\{f_{x},f_{y},f_{y}\}$.

The remaining part of tactile sensation is the torque data. From Fig. 1, the resulting sensed torque $\tau_{f}$ tends to rotate the hand in the clockwise direction. This direction would be valid for all the marked force directions in the figure and since it is actually around the $x$ coordinate:

[TABLE]

This generated torque can be expressed by cross product of the lever arm vector and the sensed force vector as follow:

[TABLE]

where $\theta$ is the angle between the lever arm vector and the force vector. In three dimensional space, the interaction force $f$ can also result in torque around $y$ direction denoted by $\tau_{y}$. The last part of torque tactile sensing set is $\tau_{z}$. Any movement in $x$ and $y$ coordinates during contact with object would generate a torque around $z$ axis. This torque is the consequence of friction force during movement on the object surface and the friction force is dependent on the magnitude of normal force inserted on the surface $f$. If we assume a constant friction coefficient $\mu$ for surface we can expand (3) as:

[TABLE]

As a consequence, the torque tactile data consists of the set of $\tau_{e}=\{\tau_{x},\tau_{y},\tau_{z}\}$. All the six-axis force/torque tactile data set is essential for grasping task. These set of tactile data $F_{e}$ are useful to guide the robot hand during interaction with the object surface:

[TABLE]

II-B Force/Torque Tactile Data and Robot Joint Sensors

The external interaction with robot hand tends to act against the changes in Cartesian position and/or orientation via force and torque response. The force prevents the robot from further changes in the Cartesian position whilst the torque is against the change in orientation. Such external effects can sensed as torque in each joint of the robot. The relationship between the robot measured joint torque sensors and the interaction force torque in robot base frame (the frame attached to the first joint holding the whole robot arm) can be stated by the transpose of the Jacobian matrix $J$:

[TABLE]

where $\tau_{int}$ is the interaction torque data sensed by robot joint torque sensors and $F$ is the vector of six elements encapsulating the end-effector interaction force $f_{b}$ and torque $\tau_{b}$ expressed in the base frame:

[TABLE]

Assuming a Jacobian matrix with number of rows equal or greater than 6, the interaction force-torque $F$ on end-effector can be retrieved by the Moore-Penrose inverse of the Jacobian matrix:

[TABLE]

Since the Jacobian is a joint dependent matrix, the inverse term in some specific joint positions known as singularities is not defined. In our case, we assume that our robot with 6 joints never works in such configurations of the robot arm. Although the force $f_{b}$ and the torque $\tau_{b}$ magnitude are independent of the frame, their directions are completely dependent on the expressed frame. As shown in Figure 2, transformation of these data from base frame $(b)$ to end-effector frame $(e)$ is possible using the relative Rotation Matrix $R_{b}^{e}$:

[TABLE]

where $f_{e}$ and $\tau_{e}$ are the force and the torque in end-effector frame obtained as force torque tactile data in (5) as shown in Fig. 1. Using both (8) and (9), it is possible to retrieve the six-axis force/toque tactile data from the joint sensed interaction torques data $\tau_{int}$.

II-C Deep Calibration: Robot Joint Sensors Calibration Using Deep Learning

In light of (8), we assume that the robot joint sensors data $\tau_{s}$ is purely related to the interaction with object. However, this data needs extra processing of calibration to accurately calculate the tactile data. In this section, we propose to utilize a deep learning technique to calibrate the joint torque sensors. These torques capture many effects, related to the robot motion and gravity as well as the interaction with the object. As a consequence, this bias is required to be removed to extract the interaction tactile data successfully.

To remove the motion-related torque data, we proposed to model the motion bias torque data. The interaction bias-free joint torque data will be obtained by subtracting the measured joint torque data $\tau_{s}$ from this bias model $\tau_{bias}$:

[TABLE]

In this regard, we record the robot joints sensor torque data as well as the joints position $q$ and velocity $\dot{q}$ during robot free-motion in space. Our primary experiments demonstrate that the motion-related torque data change considerably with two parameters: joint movement direction $\dfrac{\dot{q}}{|\dot{q}|}$ and the joint angle $q$. These two variables are fed as inputs to a three-layer fully-connected sequential multilayer perceptron deep network to train and inference the bias value from the output of the model. In fact, the model captures the significant changes in the collected bias data through fitting a regression to the bias data [23].

III Grasping Using Tactile Sensing

In this section, we present the procedure to define the grasping steps for a robot using the six-axis tactile data acquired in the previous section.

III-A Feedback Control Command

In this subsection, we present the procedure to define the grasping steps for a robot using the six-axis tactile data in previous section. We can use the interaction force/torque data to guide the robot hand in the vicinity of the object. These data are informative enough to define the required grasping steps. Our approach is based on feedback tactile sensation to adjust the robot hand velocity in corresponding direction during its interaction with the object (see Fig. 1). With this goal, we assume dynamics for linear velocity $v$ and angular velocity $\omega$ of robot hand as follows [24]:

[TABLE]

[TABLE]

where $u_{v}$ is the command to adjust the linear velocity and $u_{\omega}$ to adjust the angular velocity in which the direction of linear and angular velocities are expressed in the robot hand frame itself (Fig. 1). The constant coefficients $b_{v}$ and $b_{\omega}$ are the system damping to inertia ratio and determine the time profile of linear and angular velocity in response to the adjustment command respectively.

According to Fig. 1, any change in hand location around the object surface will be effected by commanding the linear velocity. Meanwhile, any rotation around the object to point the hand fingers toward the object can be accomplished via influencing the angular velocity. As the robot fingers are interacting with the object surface, the adjustment in the commands is needed to change the hand location or the angle between the hand and the object. Our approach is based on implementing the tactile sensation to adjust the hand linear and angular velocity during its interaction with the object. The control command for linear velocity is defined as:

[TABLE]

where $f_{e}$ can be any element of force tactile sensation set $f_{e}=\{f_{x},f_{y},f_{z}\}$ and $\alpha_{v}$ is a necessary scaling constant to scale two different domains (force in several $Newton$ and velocity typically in less than several $mm/s$). The desired value $f_{f}$ is the desired final value that we consider for that direction. For instance, if the robot needs to touch the object, it has to move directly toward the object (see Fig. 1). In such case, the control command has to be designed for linear velocity in hand $z$ direction with $\alpha_{vz}<1$:

[TABLE]

Executing (11) with this command, the robot hand moves with a constant velocity $v_{dz}$ given as

[TABLE]

toward the object until the force tactile sensation $f_{z}$ converges to $f_{zf}$. Nevertheless, the velocity of movement is dependent on the final touch force $f_{dz}$. With a small modification, we replace the control command with following alternative:

[TABLE]

where $v_{dz}$ can be selected freely and the final contact force is adjusted by $\alpha_{vz}$ as follows:

[TABLE]

Similarly, to rotate the hand around each axis, the corresponding control command can be defined by a scaling coefficient

[TABLE]

where $\tau_{e}=\{\tau_{x},\tau_{y},\tau_{z}\}$ is the feedback torque tactile sensation. As another example, if we need to align the hand by rotating the hand around its $z$ axis, we can consider:

[TABLE]

Inserting this command into (12) after a transient time imposed by the system damping to inertia ratio in the direction $b_{\omega z}$, the rotation with a constant angular velocity $\omega_{z}$ equals

[TABLE]

which will continue till the corresponding tactile sensing measurement $\tau_{z}$ reaches the level of desired torque $\tau_{dz}$. We also can define a desired value as a function of other sensed data. As an example, if the robot needs to rotate around the $z$ direction as a function of force in $z$ direction, (19) changes to:

[TABLE]

where $\beta_{\omega z}$ is scaling factor and the rotation is effected until the desired force in $z$ direction converges to the desired value $f_{dz}$. If the desired value is related to other tactile sensing data, the adjustments would be coupled to one another. This is a very important property since we can define a sequence of actions for the grasping task. We define an intuitive grasping paradigm according to Fig. 1 as follows:

• •

Move directly toward the object in $z$ direction

• •

Stop movement when the fingers touch the object

• •

Keep a constant inserted force in the $z$ direction

• •

Rotate around the contact point using the torque tactile data

All these defined steps can be executed by both proposed commands in (16) and (19) with some minor modifications. The three first steps are encapsulated in (16). To make sure that the inserted force is in $z$ direction, we need to modify (16) as follow for both $x$ and $y$ directions:

[TABLE]

[TABLE]

where $\alpha_{vx}$ and $\alpha_{vy}$ are scaling coefficients in those directions. Under the implementation of these two extra commands, the robot always inserts the force in $z$ direction. Finger elasticity is the main reason for sensing force in $x$ and $y$ directions. Such forces can be eliminated easily to help the robot contact with the object in just the $z$ direction. As a result, we can plan for $x$ and $y$ directions to facilitated the grasping in advance settings.

To rotate around the contact point using tactile torque data, we propose the following commands:

[TABLE]

[TABLE]

where $\alpha_{\omega x}$ and $\alpha_{\omega y}$ are the scaling factors for the corresponding directions. The sensed torques facilitate alignment of each direction with the object surface at the end of the rotation.

In the next section, we prove the stability of above dynamic equations with designed control commands.

III-B Stability Analysis

Since the proposed approach is based on feedback data, we need to prove the stability of the real-time dynamic system under implementation of the proposed control commands. We first investigate the stability of (11) in $z$ direction with (16):

[TABLE]

Before interaction, the force sensing data is zero and the system follows the following dynamic equation to move the robot toward the object:

[TABLE]

This is a stable dynamic equation and the error converges to zero after a settling time greater than $t_{s}=\dfrac{4}{b_{z}}$:

[TABLE]

which implies $v_{z}\,\to\,v_{dz}$ when the sensed force $f_{z}=0$. After contact with the object, the sensed force is no longer zero. Due to flexibility of fingers, the sensed force can be modeled as a spring force with coefficient $K_{z}$. It is proportionally dependent on the level of displacement in the finger joints as well as the object deformation expressed into a unified term $\Delta z$ in $z$ direction:

[TABLE]

We also can model the unified term $\Delta z$ as the displacement from equilibrium point of the surface (see Fig. 3) and insert it into (26):

[TABLE]

[TABLE]

where $z$ is the location of finger tip and $z_{0}$ is the equilibrium point of surface, both of them in the $z$ direction. Rearranging (30), we have the following:

[TABLE]

Moreover, we know that the velocity is the rate of change in location

[TABLE]

such that we we can modify (32):

[TABLE]

This is a second order stable dynamic equation. The velocity error and the acceleration error become zero after a settling time around $t_{s}=\dfrac{4}{b_{z}}$ and the sensed force converges to

[TABLE]

where $z_{d}$ is the final value when the robot hand finger can not move any further. This proof can be easily extended to other directions. The results are extendable for sensed torque as well. In such a case, we utilize the mathematical relationship for sensed torque when there is an angle $\theta$ between the force vector $\vec{f}$ and lever arm vector $\vec{d}$:

[TABLE]

For rotation around the hand $z$ direction, the sensed force $\vec{f}$ is perpendicular to the lever arm vector $\vec{d}$. With a friction coefficient $\mu_{z}$ during rotation on the surface, the sensed torque would be:

[TABLE]

After utilizing this equation in (19) and inserting the sensed torque into (12), we have

[TABLE]

According to this equation, rotation around the $z$ direction will be effected as long as the sensed force $f_{z}\neq 0$. The rotation by commanding the angular velocity $\omega_{z}$ continues on the surface until it detects the edges. In this case, the robot stops rotating around the $z$-axis when the following condition is satisfied

[TABLE]

If we select $\beta$ bigger than $d\mu_{z}$, the robot only reaches the edges where $\mu_{z}$ is bigger than the surface friction coefficient. In this case, the sensed torque converges to:

[TABLE]

when the object is inside the robot hand and $f_{z}\,\to\,f_{zf}$.

IV Experimental Results

In this section, we present the experimental settings, the time profile of tactile sensation set for a typical grasping, and the successful rate for a set of different objects.

IV-A Settings

To validate the proposed approach, we use a two fingered Mico robot with 6 degrees of freedom. Except for singular configurations and some specific configurations in which the robot is unable to reach, the Jacobian matrix used in (8) results in valid tactile sensation data. The deep learning is trained with ReLU activation function for the neurons during 50 epochs and a batch size of 20 to minimize the mean square error for each joint torque bias data. The configuration of final layer is the output estimating the bias value corresponding to the current joint angle and movement direction.

The sensor data is collected through robot movement without any interaction with external environment. By this way, the gathered data is only related to the current joint angle and the direction of the movement completely reflecting the dependency of bias torque data to these inputs.

IV-B Grasping Paradigm Using Tactile Sensing

We first evaluate the deep calibration performance. In Fig. 4 through Fig. 9, the recorded torque data and the model are visualized for all six joints to illustrate the performance of proposed deep calibration in details. There are two blue lines on the vibrating noisy data and each of them is related to the direction of joint movement. In prediction phase, the torque data is extracted from the network corresponding to the joint angle and its movement direction.

Next, we apply the grasping paradigm in this paper to grasp a canned. In Table I, we provide parameters involved in the design. Using these parameters, the commands executed in the grasping process are as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

According to these commands, the tactile force sensing in approaching direction $z$ would be

[TABLE]

which is consistent with the final value of force tactile sensing in Fig. 10. Moreover, the final values of force sensing in $x$ and $y$ directions are zero. As shown in the figure, the robot reaches the object surface after 6000 time samples which is equal to $6$ $sec$ since the sample time is $0.001$ $sec$. Nevertheless, the grasping task is completed after $35$ $sec$ of touching the object surface since the robot struggles to adjust the hand with the object surface before grasping the object completely. It is hard for the robot to keep a constant contact with the object while is suffering from mechanical vibration during movement. Several jumps in Fig. 10 demonstrate this phenomenon.

During this typical interaction, the torque tactile sensing reaches zero for both $x$ and $y$ directions and it converges to a constant final value:

[TABLE]

in the $z$ direction when it detects the object edges. All these explanations can be observed in Fig. 11.

As can be inferred from these figures, the sensed data is prone to mechanical vibrations during the robot movement as well as noise. We applied a threshold filter on the data to reduce this undesirable effect after removing the bias using deep calibration. This is the reason that the sensed data is zero before contact with the object. This filter has no effect on the data vibration after it passes the threshold. According to presented results, our proposed approach grasp an unknown object similar to human being. Deep calibration is useful to provide the accurate tactile sensing data. This data can be mined from robot joint sensors by a mathematical formulation and the grasping strategy implemented by feedback tactile data works smoothly without implementation any logical rules. A video of the experiment is available online https://youtu.be/y7ZBFr1IpVw.

V Conclusions

In this paper, we proposed a human-like automatic grasping of an object using tactile sensing without vision. The tactile sensing data is extracted from the robot joint torque sensors and calibration process is needed to rid the tactile data of the bias related to motion. To calibrate the bias data, we provided a customized deep learning structure with effective inputs to fit a model on bias data. The suggested grasping paradigm in this paper is applied to grasp an unknown shaped canned and the time profile of tactile sensing data is presented during execution of grasping with deep learning alongside the feedback tactile data.

As part of future works, our goal is to propose enhancement strategies for manipulation capability of robots using tactile sensing. In this regard, harder manipulation tasks will be addressed through a touch-based interactive perception.