Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments

TL;DR

This paper introduces Deep-3DMTS, a deep learning-based visual servoing method that enhances robotic crop harvesting in occluded and unstructured environments by creating a single-perspective approach.

Contribution

It develops and validates a CNN-based method that matches standard 3DMTS performance, improving visual guidance in complex agricultural settings.

Findings

Deep-3DMTS achieves end effector positioning within 11.4 mm of the baseline.

It increases fruit size in images by a factor of 17.8.

Performance is validated via simulation against standard 3DMTS.

Abstract

3D Move To See (3DMTS) is a mutli-perspective visual servoing method for unstructured and occluded environments, like that encountered in robotic crop harvesting. This paper presents a deep learning method, Deep-3DMTS for creating a single-perspective approach for 3DMTS through the use of a Convolutional Neural Network (CNN). The novel method is developed and validated via simulation against the standard 3DMTS approach. The Deep-3DMTS approach is shown to have performance equivalent to the standard 3DMTS baseline in guiding the end effector of a robotic arm to improve the view of occluded fruit (sweet peppers): end effector final position within 11.4 mm of the baseline; and an increase in fruit size in the image by a factor of 17.8 compared to the baseline of 16.8 (avg.).