# Visual end-effector tracking using a 3D model-aided particle filter for humanoid robot platforms

## TL;DR

This paper presents a novel 3D model-aided particle filter approach for markerless visual tracking of a robot's end-effector, enabling robust, closed-loop control using only visual data from cameras.

## Contribution

It introduces a 3D model-aided particle filter with HOG features for end-effector tracking, integrating CAD models and rendering for improved robustness and accuracy.

## Key findings

- Robust end-effector tracking in cluttered environments
- Effective compensation for kinematic errors
- Successful closed-loop arm control using vision

## Abstract

This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.