# Visual Manipulation Relationship Network

## TL;DR

This paper introduces VMRN, a CNN architecture that detects objects and predicts manipulation relationships in real time, facilitating robotic grasping tasks involving multiple objects.

## Contribution

The paper proposes VMRN with OP2L for end-to-end real-time detection of objects and manipulation relationships, and provides a new dataset VMRD for training.

## Key findings

- VMRN achieves real-time detection and relationship prediction.
- The dataset VMRD contains over 17,000 object instances.
- Experimental results validate the effectiveness of VMRN in robotic tasks.

## Abstract

Robotic grasping detection is one of the most important fields in robotics, in which great progress has been made recent years with the help of convolutional neural network (CNN). However, including multiple objects in one scene can invalidate the existing CNN-based grasping detection algorithms, because manipulation relationships among objects are not considered, which are required to guide the robot to grasp things in the right order. This paper presents a new CNN architecture called Visual Manipulation Relationship Network (VMRN) to help robot detect targets and predict the manipulation relationships in real time. To implement end-to-end training and meet real-time requirements in robot tasks, we propose the Object Pairing Pooling Layer (OP2L) to help to predict all manipulation relationships in one forward process. Moreover, in order to train VMRN, we collect a dataset named Visual Manipulation Relationship Dataset (VMRD) consisting of 5185 images with more than 17000 object instances and the manipulation relationships between all possible pairs of objects in every image, which is labeled by the manipulation relationship tree. The experimental results show that the new network architecture can detect objects and predict manipulation relationships simultaneously and meet the real-time requirements in robot tasks.