SEBVS: Synthetic Event-based Visual Servoing for Robot Navigation and Manipulation

TL;DR

This paper introduces SEBVS, an open-source simulation tool for event-based vision in robotics, demonstrating its effectiveness in navigation and manipulation tasks with transformer-based policies trained via behavior cloning.

Contribution

It presents a novel ROS package for generating synthetic event streams from RGB feeds in Gazebo, enabling evaluation of event-based robotic policies in simulation.

Findings

Event-guided policies outperform RGB-based methods in various conditions.

Transformer-based ERPs achieve competitive performance in navigation and manipulation.

The simulation framework facilitates future research in event-driven robotic perception.

Abstract

Event cameras offer microsecond latency, high dynamic range, and low power consumption, making them ideal for real-time robotic perception under challenging conditions such as motion blur, occlusion, and illumination changes. However, despite their advantages, synthetic event-based vision remains largely unexplored in mainstream robotics simulators. This lack of simulation setup hinders the evaluation of event-driven approaches for robotic manipulation and navigation tasks. This work presents an open-source, user-friendly v2e robotics operating system (ROS) package for Gazebo simulation that enables seamless event stream generation from RGB camera feeds. The package is used to investigate event-based robotic policies (ERP) for real-time navigation and manipulation. Two representative scenarios are evaluated: (1) object following with a mobile robot and (2) object detection and grasping with a robotic manipulator. Transformer-based ERPs are trained by behavior cloning and compared to RGB-based counterparts under various operating conditions. Experimental results show that event-guided policies consistently deliver competitive advantages. The results highlight the potential of event-driven perception to improve real-time robotic navigation and manipulation, providing a foundation for broader integration of event cameras into robotic policy learning. The GitHub repo for the dataset and code: https://eventbasedvision.github.io/SEBVS/