Modeling Grasp Motor Imagery through Deep Conditional Generative Models

TL;DR

This paper explores how deep generative models can be used to enable robots to synthesize complex, multimodal grasp configurations by translating high-level motor imagery concepts into robotic grasp actions.

Contribution

It introduces a novel approach using deep conditional generative models to learn integrated object-action representations for robotic grasp synthesis.

Findings

Successfully generated multimodal grasp configurations in simulation

Demonstrated the model's capacity to capture complex grasp variations

Showed potential for applying deep generative models to robotic manipulation

Abstract

Grasping is a complex process involving knowledge of the object, the surroundings, and of oneself. While humans are able to integrate and process all of the sensory information required for performing this task, equipping machines with this capability is an extremely challenging endeavor. In this paper, we investigate how deep learning techniques can allow us to translate high-level concepts such as motor imagery to the problem of robotic grasp synthesis. We explore a paradigm based on generative models for learning integrated object-action representations, and demonstrate its capacity for capturing and generating multimodal, multi-finger grasp configurations on a simulated grasping dataset.

Tables

Table 1. TABLE I : Overview of dataset statistics

	# Objects	# Instances
Training files	161	42,351
Testing files - Similar	20	4,848
Testing files - Different	53	8,851

Table 2. TABLE II : Negative CLL for test sets composed of similar or different objects (relative to the training set).

	Similar objects (n=4,848)			Different objects (n=8,851)
Train size	16,384	32,768	42,351	16,384	32,768	42,351
CNN	24.721	24.833	24.577	26.910	26.920	26.599
GSNN	22.827	22.292	21.831	27.945	32.513	33.461
CVAE	15.325	13.531	13.216	18.356	18.808	16.525
R-CVAE	13.670	13.024	12.511	14.277	14.128	13.514

Table 3. TABLE III : Percentage of successful simulated grasps.

	Similar objects (n=4,848)			Different objects (n=8,851)
Train size	16,384	32,768	42,351	16,384	32,768	42,351
CNN	0.155	0.202	0.199	0.106	0.138	0.145
GSNN	0.169	0.177	0.190	0.115	0.145	0.147
CVAE	0.344	0.346	0.347	0.302	0.301	0.295
R-CVAE	0.318	0.323	0.362	0.288	0.304	0.315

Equations

$$\log p_{\theta }(\mathbf{x})$$

$$+$$

$$\mathbf{z}=\mu +\sigma ϵ$$

$${\mathcal{L}}_{\text{VAE}}(\theta ,\varphi ;\mathbf{x})$$

$$+$$

$${\mathcal{L}}_{\text{CVAE}}(\theta ,\varphi ;\mathbf{x},\mathbf{y})$$

$$+$$

$$p_{\theta }(y|x)$$

$${\mathbf{z}}^{(s)}\sim p_{\theta }(z|x)$$

$$p_{\theta }(y|x)$$

$${\mathbf{z}}^{(s)}\sim q_{\varphi }(z|x,\mathbf{y})$$

$$\underset{\alpha ,\beta ,\gamma ,T_x,T_y,T_z}{\min }\sum _{i=1}^N(C_i-Y_i{)}^2$$

Full text

©2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Humans have an innate ability for performing complex grasping maneuvers. Often times, these maneuvers are performed unconsciously, where object dynamics are unknown or continuously changing through time. This ability also manifests where objects themselves may be either similar or novel to those previously encountered. Given some prior experience on grasping an object, it seems highly unlikely that humans learn from scratch how to grasp each new object that is presented to them. Rather, we posit that this ability is driven through both motor and object representations, allowing for abstract generalizations and efficient transference of skills among objects.

In robotics, grasping and manipulation is a critical and challenging problem. Its difficulty stems from variability in an object’s shape and physical properties, the gripper capabilities, and task requirements. As such, most industrial applications require robots to use myriad gripping fixtures or end-of-arm tools to grasp various objects. But as robots expand to applications in non-structured environments (both in industry and beyond), advanced grasping skills are needed.

Currently there are several difficulties in actually learning how to grasp. First, the problem is fundamentally a many-to-many mapping. An object can be grasped in many ways that are all equivalent, while the same grasp configuration can be used to grasp many objects. There is a need to maintain this many-to-many mapping to enable the robot to grasp objects under uncertainty and in highly cluttered environments. Second, while the object shape and location can be obtained from perception, grasping is often challenged by inherent characteristics of the object such as surface friction, weight, center of mass, and finally the actual functionality of the object. All of these factors are only known after the object is touched and the grasp is started.

In this paper, we propose to learn a new concept that we refer to as the grasp motor image (GMI). The GMI combines object perception and a learned prior over grasp configurations, to synthesize new grasps to apply to a different object. We liken this approach to the use of motor representations within humans. Specifically, we focus on the use of motor imagery for creating internal representations of an action, which requires some knowledge or intuition of how an object may react in a given scenario.

We show that using recent advances in deep learning (DL), specifically deep conditional generative models [27] and the Stochastic Gradient Variational Bayes (SGVB) framework [7], we can capture multimodal distributions over the space of grasps conditioned on visual perception, synthesizing grasp configurations with minimal additional complexity compared to conventional techniques such as convolutional neural networks (CNNs). We quantitatively compare our approach to the discriminative CNN baseline and other generative models and qualitatively inspect samples generated from the learned distribution.

I-A Contributions

Most work within deep learning and robotic grasp synthesis has focused in one form or another on the prediction of grasp configurations given visual information. The goal of this work is to show how having an idea of the properties characterizing an object, and an idea of how a similar object was grasped previously, a unified space can be constructed that allows grasps to be generated for novel objects.

A second contribution of this work is a probabilistic framework, leveraging deep architectures to learn multimodal grasping distributions for multi-fingered robotic hands. Grasping is inherently a many-to-many mapping, yet as we show in this paper, naïvely applying mainstream deep learning approaches (e.g. convolutional neural networks) may fail to capture these complex distributions or learn in these scenarios without some stochastic components. Here, we demonstrate the feasibility of deep generative models for capturing multimodal distributions conditional on visual input, and open the door to future work such as the integration of other sensory modalities.

II Background and Related work

Within robotics, our work shares some similarities to the notion of experience databases (ED), where prior knowledge is used to either synthesize grasps partially (i.e. through constraints or algorithmic priors [14]) or fully, by using previously executed grasps. Our work is also similar in spirit to the concept of Grasp Moduli Spaces [21], which define a continuous space for grasp and shape configurations.

Yet, instead of classical approaches with EDs (which require manual specification of storage and retrieval methods), our approach allows these methods to be learned automatically, for highly abstract representations. Further, our approach to constructing this “grasp space” is to collect realistic data on object and grasp attributes using sensory modalities and hand configurations, and learn the encoding space as an integrated object-action representation.

II-A Motor imagery

Motor imagery (MI) and motor execution (ME) are two different forms of motor representation. While motor execution is an external representation (physical performance of an action), motor imagery is the use of an internal representation for mentally simulating an action [5, 17].

As MI is an internal process, assessing MI performance is typically done by analyzing behavioural patterns (such as response time for a given task), or by visualizing neural activity using techniques such as functional magnetic resonance imagining (fMRI). Using these techniques, many studies have reported overlapping activations in neural regions between both MI and ME (e.g. [4, 26], review: [18]), suggesting that at some level, participating in either MI or ME affects some amount of shared representation. These findings lead credence to the hypothesis that, for example, mentally simulating a grasping task shares some measure of similar representation to actually performing the grasp itself.

Among many studies that have examined this phenomenon, we highlight one by Frak et al. [3] who explored it in the context of which frame of reference is adopted during implicit (unconcious) MI performance. The authors presented evidence that even though MI is an internal process, participants mentally simulating a grasp on a water container did so under real-world biomechanical constraints. That is, grasps or actions that would have been uncomfortable to perform in the real world (e.g. due to awkward joint positioning) were correlated with responses of the mentally simulated action.

II-B Learning joint embeddings and object-action interactions

Learning joint embeddings of sensory and motor data is not new. It has been proposed, e.g., by Uno et al. [30], who attempted to learn a joint embedding between visual and motor information. In the field of robotics, other works that have used multimodal embeddings include Sergeant et al. [25] for control of a mobile robot, Noda et al. [19] for behaviour modeling in a humanoid robot, and recently Sung, Lenz, and Saxena [29] who focus on transferring trajectories.

Congruent to learning joint embeddings, there has also been work in robotic grasping on learning how objects and actions interact with each other, and what effects they cause. Bayesian networks have been used to explore these effects by Montesano et al. [16], who use discretized high-level features and learn the structure of the network as part of a more generalized architecture. Other work with Bayesian networks include Song et al. [28], who explore how large input spaces can be discretized efficiently for use in grasping tasks.

Our approach differs significantly from these works in the scale of data, application, and network structures. Further, with respect to Bayesian networks, we work with continuous data without any need for discretization.

II-C Deep learning and robotic grasping

The majority of work in DL and robotic grasping has focused on the use of parallel-plate effectors with few degrees of freedom. Both Lenz [9], and Pinto [20] formulate grasping as a detection problem, and train classifiers to predict the most likely grasps through supervised learning. By posing grasping as a detection problem, different areas of the image could correspond to many different grasps and fit with the multimodality of grasping; yet, to obtain multiple grasps for the same image patch, some form of stochastic components or a priori knowledge of the types of grasps is required.

Mahler et al. [14] approach the problem of grasping through the use of deep, multi-view CNNs to index prior knowledge of grasping an object from an experience database. Levine et al. [11] work towards the full motion of grasping by linking the prediction of motor commands for moving a robotic arm with the probability that a grasp at a given pose will succeed. Other work on full-motion robotic grasping includes Levine [10] and Finn [2] who learn visuomotor skills using deep reinforcement learning.

DL and robotic grasping have also recently extended to the domain of multi-fingered hands. Kappler et al. [6] used DL to train a classifier to predict grasp stability of the Barrett hand under different quality metrics. In this work, rather than treating grasping as a classification problem, we propose to predict where to grasp an object with a multi-fingered hand, through a gripper-agnostic representation of available contact positions and contact normals.

III Grasp motor imagery

Although the field is far from a consensus on the best uses and structures for DL and robotic grasping, most work appears to use deep architectures for processing visual information. At the core of our approach is the autoencoder structure. We briefly review the principles of this method before reviewing the probabilistic models considered herein.

III-A Autoencoders

An autoencoder (AE) is an unsupervised deep learning algorithm that attempts to dissociate latent factors in data using a combination of encoding and decoding networks (Figure 1). In the encoding stage, an input $\mathbf{x}$ is mapped through a series of (typically) constricting nonlinear hidden layers to some low-dimensional latent representation of the input $f(\mathbf{x})$. In autoencoders that have an odd number of layers, this layer is often denoted by $\mathbf{z}$. The decoding phase forces the learned representation to be meaningful by mapping it to some reconstruction $g(f(\mathbf{x}))$ of the original input. Training proceeds by iteratively encoding and decoding a datapoint, measuring the difference between the reconstruction and original input, and backpropagating the error to update the weights of both the encoder and decoder.

While the AE is capable of untangling the latent factors of complex data one limitation of this architecture is that autoencoders are inherently deterministic in nature; a single input will always produce the exact same latent representation and reconstruction. For tasks such as robotic grasping, which is inherently a many-to-many mapping (i.e. many different representations may map to many different grasps), deterministic networks such as the AE are unsuitable for implementing such a mapping in a generative sense.

III-B Variational autoencoders

The variational autoencoder (VAE) [7, 22] is a directed graphical model composed of generator and recognition networks (Figure 2). The goal of the recognition network is to learn an approximation to the intractable posterior $p_{\theta}(\mathbf{z|x})$ by using an approximate inference network $q_{\phi}(\mathbf{z|x})$ through some nonlinear mapping, typically parametrized as a feedforward neural network. The generator network takes an estimate of $\mathbf{z}$, and learns how to generate samples from $p_{\theta}(\mathbf{x|z})$, such that $p_{\theta}(\mathbf{x})=\sum_{\mathbf{z}}p(\mathbf{x}|\mathbf{z})p(\mathbf{z})$ approximates the data distribution $p(\mathbf{x})$.

These networks can be composed in a fashion similar to the classical AE, where the recognition network forms a kind of encoder, and generator network constitutes the decoder. This is, in fact the proposed method of training them within the SGVB framework [7], which adds only the complexity of sampling to classical methods, and still performs updates using the backpropagation algorithm.

The objective of the VAE shares some similarities to the classical AE; that is, in the case of continuous data the optimization of a squared error objective along with an additional regularizing term. This objective is denoted as the variational lower bound on the marginal likelihood:

[TABLE]

where $D_{KL}$ is the KL-Divergence between the encoding and prior distribution, analagous to a regularization term in a standard autoencoder. The form of the $p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z})$  will depend on the nature of the data, but typically is Bernoulli or Gaussian. Note that the VAE formulation only permits continuous latent variables.

Commonly, both the prior $p_{\theta}(\mathbf{z})$ and encoding distributions are chosen to be multivariate Gaussians with diagonal covariance matrices $\mathcal{N}(\mathbf{\mu,\sigma^{2}I})$, in which case the recognition network learns to encode $\mathbf{\mu}$ and $\mathbf{\sigma}$ for each latent variable. This simple parameterization allows for the KL-divergence to be computed analytically without the need for sampling. In order to compute the expectation, the reparameterization trick introduced by Kingma and Welling [7] reparameterizes (a non-differentiable) $\mathbf{z}$ through some differentiable function $g_{\phi}(\mathbf{x,\epsilon})$:

[TABLE]

where $\epsilon$ is sampled from the noise distribution $p(\epsilon)=\mathcal{N}(0,\mathbf{I})$, and $\mathbf{\mu}$, $\mathbf{\sigma}$ are the mean and standard deviation of the encoding distribution respectively. Thus, an estimate of the lower bound can be computed according to:

[TABLE]

where $\mathbf{z}$ is reparameterized according to Equation 2, and $L$ is the number of samples drawn from the prior distribution for computing the expectation111In many cases (such as large minibatch sizes), only a single sample needs to be drawn..

Somewhat abstracted from view, but fundamental to the decision to pursue them in this work is that VAEs are not only probabilistic models, but that the stochasticity of $\mathbf{z}$ allows for modeling multiple modes within the data distribution. That is, sampling different $\mathbf{z}$’s may localize the network’s predictions to different high-probability regions of the reconstructed output space. In our setup, we only condition on visual input and sample grasps, not vice-versa. Therefore we do not explicitly treat the many-to-many mapping. However, VAEs may also permit the joint modeling of grasp, vision, and other perceptual inputs, such as tactile sensors. This is out of application scope and reserved for future work.

III-C Conditional variational autoencoders

The conditional variational autoencoder (CVAE) [27], is an extension of the VAE architecture to settings with more than one source of information. Given an input $\mathbf{x}$, output $\mathbf{y}$, and latent variables $\mathbf{z}$, conditional distributions are used for the recognition network $q_{\phi}(\mathbf{z|x,y})$, generator network $p_{\theta}(\mathbf{y|x,z})$, and for a prior network $p_{\theta}(\mathbf{z|x})$222The prior network is a technique for modulating the latent variables via some input data, and is used in place of the prior specified for the VAE.. In this work, $\mathbf{x}$ represents input visual information (i.e. images), and $\mathbf{y}$ is the output grasp configuration as shown in Figure 3.

The lower bound of the CVAE is similar to the VAE, except for the addition of a second source of given information:

[TABLE]

The structure of each of these networks (recognition, generator, and prior) is a design choice, but following [27], we design both our recognition and generator network to have a convolutional (CNN-based) pathway from the image for its numerous benefits, among them, reducing the number of free parameters. The typical way to evaluate such models is achieved through estimates of the conditional log-likelihood (CLL), using either Monte Carlo (Equation 5) or importance sampling (Equation 6), the latter typically requiring a fewer number of samples.

[TABLE]

III-D Grasp motor image

Consider, for example, transferring prior motor experience to novel objects, where the selection of an initial grasp was influenced by internal and external perceptual object properties. One approach to transferring this knowledge could be to directly use past experience – existing in some high-dimensional space – to initialize priors within a system. A different approach, e.g. in MI, could be to access some latent, low-dimensional representation of past actions which are shared among a variety of different situations.

Instead of learning a direct, image-to-grasp mapping through neural networks, we instead learn an (image-and-grasp)-to-grasp mapping. Our approach is intuitive: based on perceptual information about an object, and an idea of how an object was previously grasped, we index a shared structure of object-grasp pairs to synthesize new grasps to apply to an object. As shown in Section III-C this scheme exists as a single module and is trainable in a fully end-to-end manner. Further, the structure of the CVAE model allows us to model and generate grasps belonging to not only one, but possibly many different modes.

Building the grasp motor image only requires that object properties are captured in some meaningful way. We use a single data-modality (i.e. visual information) which exploits CNN modules for efficient and effective visual representation. Our continued interest lies in investigating how the GMI can be gradually learned through multiple modalities; specifically, those that capture internal properties and require object interaction (such as force and tactile data).

IV Experimental Setup

There are a few different directions that reasoning about grasp synthesis using GMI affords. Due to abstractions at both the object and action level, we hypothesize that the model should require fewer number of samples to run, and evaluate this by restricting the amount of training data available to the model. We also evaluate synthesized grasps on objects similar to those seen at training time, along with families of objects the model has never seen before.

IV-A Dataset

We collected a dataset of successful, cylindrical precision robotic grasps using the V-REP simulator [23], and object files provided by Kleinhans et al. [8] on a simulated “picking” task. While there are several three-fingered grippers being employed for commercial and research applications (e.g. the ReFlex hand http://labs.righthandrobotics.com), a number of recent works have adopted the Barrett hand [6, 8]333Note that the Barrett hand is an underactuated, three-fingered gripper parameterized through a total of 8 joints and 4 degrees of freedom.. In this work, we favour this gripper for its ability to capture a variety of different grasps dependent on the shape of an object.

The object files comprise a set of various everyday household items, ranging in size from small objects such as tongs, to larger objects such as towels and vases. Each of these object classes has a unique ancestral template, and perturbations of these templates were performed to yield a number of meshes with different shape characteristics. During data collection we assumed that all objects are non-deformable, share the same friction value, and share the same mass of $1\text{\,}\mathrm{kg}$. In a simulated setting, these assumptions allow us to direct our attention towards the effects of visual information on the GMI and forgo properties that are better captured through e.g. tactile sensory systems.

From each object class, we remove 1 object and place it into a test set. The training set is comprised of 161 objects from the 20 most populated classes, containing around 47,000 successful image/grasp pairs. From this set, we randomly select 10% to be used for validation during training. In order to study the effects of generating grasps for novel objects, we partition this test set into two distinct groups: objects that are similar to those of the training set (i.e. belong to the same class of object but are not the same instance), and the other set comprised of object classes never encountered during training (different). The final dataset statistics are reported in Table I.

IV-A1 Visual information

Recorded for each successful grasp trial are RGB and Depth images (size 64$\times$64 pixels), as well as a Binary segmentation mask of equal size, indicating the object’s spatial location in the image. Each image collected uses a simulated Kinect camera with primary axis pointing towards the object (along the negative z-direction of the manipulator’s palm), and the y-axis pointing upwards. This configuration means that the same image could correspond to a number of different grasps, and allows us to capture multimodality that may exist within the grasp space.

IV-A2 Definition of grasps

Our experiments leveraged the three-fingered Barrett hand, and defines a grasp as the 18-dimensional vector $[\vec{p_{1}},\vec{p_{2}},\vec{p_{3}},\vec{n_{1}},\vec{n_{2}},\vec{n_{3}}]$, where the subscript denotes the finger, and the contact positions $p$ and normals $n$ each have $(x,y,z)$ Cartesian components. While the contact positions specify where to place the fingertips of a gripper, the purpose of the contact normals is to describe the relative orientation. Note that with this parameterization, the unique characteristics of a manipulator (i.e. number of joints or degrees of freedom) have been abstracted into the number of available fingertips and renders the representation as being gripper-agnostic.

We encode all grasps into the object’s reference frame {O}, which is obtained by applying PCA on the binary segmentation mask. We assign the directional vectors $\vec{O_{z}}$ and $\vec{O_{y}}$ coincident with the first and second principal components respectively, and ensure that $\vec{O_{x}}$ always points into the object. We define the object’s centroid as being the mean $x_{p}$ and $y_{p}$ pixel coordinates of the object.

IV-B Learning

We build our networks within the Lasagne framework [1], using the Adam optimizer, a learning rate of 0.001, and a minibatch size of 100. Due to class imbalance in the dataset (complicated by some objects being easier to grasp than others), we train with class-balanced minibatches. We standardize all data to have zero mean and unit-variance.

For the recognition network we use a 5-layer CNN, applying max pooling every 2 layers and using filter sizes of [7, 5, 5, 5, 3] and number of filters [16, 32, 32, 64, 64] respectively. The output of the convolution operations feeds into a network-in-network structure with average pooling [12] for reducing dimensionality of the system, and is subsequently followed by a 64-neuron hidden layer. The outputs from processing the images and grasps are fused into a shared representation, then followed with another 64 neuron hidden layer before encoding the latent representation. As a regularizer, we inject white noise at all network inputs with $\sigma=0.05$ and apply weight normalization [24].

As mentioned, in order to compute $p_{\theta}(\mathbf{y|x,z})$, we employ a prediction network for making an initial guess (i.e. $p_{\theta}(\mathbf{y|x})$), and add it to a prediction of $p_{\theta}(\mathbf{y|z})$ from the generator network using a sampled $\mathbf{z}$ (Figure 3(a)). Our prior network follows a similar structure to the recognition network; in the CVAE model, we drop the input $\mathbf{y}$, and only process the visual stream. In the recurrent CVAE (R-CVAE), we take an initial guess made by the prediction network, and feed it back into the corresponding input in the prior network.

Two other models were tested. The Gaussian stochastic neural network (GSNN) [27], which is derived by setting the recognition and prior network equal (i.e. $q_{\theta}(\mathbf{z|x,y})=p_{\theta}(\mathbf{z|x})$), and a baseline convolutional neural network (CNN) that takes input images and tries to predict a grasp configuration directly. For evaluating the CLL, we found that 100 samples for importance sampling (R-CVAE and CVAE), and 1000 samples for Monte-Carlo estimates with the GSNN were sufficient to obtain consistent estimates.

V Results

To emphasize the fact that the networks are learning distributions over the data instead of deterministic predictions, Figure 4 presents a sample histogram of the latent and output variable space for a single object-grasp instance.

V-A Conditional log-likelihood

Table II presents the estimated negative CLL for each of the model types, split between the two experimental paradigms: test sets composed of similar or different objects relative to the training set. The CLL scores for the CVAE and R-CVAE show that they significantly outperform the GSNN and CNN, indicating a tighter lower-bound and better approximation to $\log p(\mathbf{x})$. This result could be due to the parameterization of the prior distribution; in the CVAE and R-CVAE models, the prior was modulated by the use of a prior network, allowing the predicted means flexibility in shifting their inputs. The GSNN, on the other hand, uses a recognition network that only has information about input visual information $\mathbf{x}$, and is unable to shift the prior mean based on the grasps $\mathbf{y}$.

V-B Simulator

To evaluate how the network predictions transfer back to the “picking” task, we evaluate a single prediction (again using the distribution means for the stochastic networks) in the simulator. Given that the task is to pick up an object, we define a grasp to be successful if the object is still held within the gripper at the height of the picking operation. If the object is no longer held by the fingertips, contacting other components, or the gripper failed to contact the object during initial finger placement, the grasp is deemed a failure.

In order to position the manipulator for each initial grasp, our grasp planner consists of calculating an optimal initial wrist placement by minimizing the distance of each of the manipulator’s fingers to the predicted target positions:

[TABLE]

where $\alpha,\beta,\gamma$ are the yaw, pitch, and roll rotational components, while $T_{x}$, $T_{y}$, $T_{z}$ are the $x$, $y$, and $z$ translational components. In this optimization, $N$ is the number of fingertips, and $C_{i},Y_{i}$ are the ground-truth and predicted fingertip positions relative to a common frame. The results for similar and different objects can be seen in Table III.

Two key results can be seen from this table. First, grasp predictions made by the baseline CNN appear to be unable to match that of the generative CVAE and R-CVAE models. It is likely that this result stems from learning a discriminative model with a multimodal setting. Partial success for the CNN’s predictions may also be due in part to a grasp planner that succeeds under fairly weak predictions.

Second, with respect to data efficiency, the relative gains for the baseline CNN model (between the 16k and 42k training set sizes) appears to be much greater then the generative CVAE and R-CVAE models. The literature has reported advances in supervised learning that have been able to leverage very large labeled training datasets, but the gains for unsupervised learning methods are less documented.

V-C Multimodal grasp distributions

In Figures 6 & 6, we demonstrate the learned multimodality of our networks by sampling different grasps from a single grasp instance444The plotted grasp configuration is the result of solving for that grasp that maximizes: $\mathbf{y^{*}}=\operatorname*{arg\,max}_{y}\frac{1}{L}\sum_{l=1}^{L}p_{\theta}(\mathbf{y|x,z}^{(l)})$, using $\mathbf{z}^{(l)}\sim p_{\theta}(\mathbf{z|x})$ and $L$ = 50.. In these figures, we present the input RGB image (left), the plotted grasp configuration & object (middle), as well as a t-SNE [13] plot of the learned grasp space (right). t-SNE is a method for visualizing high-dimensional data by projecting it down to a lower-dimensional space. In these plots, one can clearly see distributions with multiple modes, which in turn appear to be reflected in the plotted grasp space.

VI Discussion

Further inspection of the grasp space in Figures 6 & 6 appears to show that many of the grasps share a similar vertical distribution. We believe this may be a result of the data collection process, where reachability constraints prevent certain grasps from being executed (e.g. on the lower part of objects due to contact with the table).

We have identified a few potential limitations of our current approach. First, our method still requires tens of thousands of examples to train, which is expensive to collect in the real world. Second, our evaluation strategy has only focused on objects with fixed intrinsic properties, which is a simplification of real-world characteristics. Compared to feed-forward networks, a sampling-based approach is more computationally expensive and there may be alternate ways of simplifying its computational requirements. Finally, there are also other practical considerations that could be taken during data-collection, such as optimizing the closing strategy of the manipulator for e.g., more reactive grasping.

VII Conclusion

In this work we presented a conceptual framework for robotic grasping, the grasp motor image, which integrates perceptual information and grasp configurations using deep generative models. Applying our method to a simulated grasping task, we demonstrated the capacity of these models to transfer learned knowledge to novel objects under varying amounts of available training data, as well as their strength in capturing multimodal data distributions.

Our primary interest moving forward is in investigating how objects with variable intrinsic properties can be learned with the GMI, specifically, by introducing additional sensory modalities into the system. We are also interested in investigating how continuous contact with an object contributes to the formation of the GMI. We find work within the cognitive sciences on the effects of somatosensory input on motor imagery (c.f. [15]) to be an interesting starting point.