Object segmentation in the wild with foundation models: application to vision assisted neuro-prostheses for upper limbs

TL;DR

This paper explores using foundation models for semantic object segmentation in cluttered, real-world scenes to assist upper limb neuroprostheses, demonstrating improved segmentation accuracy through gaze-based prompts and fine-tuning.

Contribution

It introduces a gaze-guided prompt generation method for the Segment Anything Model and fine-tunes it for egocentric visual data in complex environments.

Findings

Segmentation IoU improved by up to 0.51 points on real-world data.

Proposed method effectively guides segmentation in cluttered scenes.

Fine-tuning enhances model performance for neuroprosthetic applications.

Abstract

In this work, we address the problem of semantic object segmentation using foundation models. We investigate whether foundation models, trained on a large number and variety of objects, can perform object segmentation without fine-tuning on specific images containing everyday objects, but in highly cluttered visual scenes. The ''in the wild'' context is driven by the target application of vision guided upper limb neuroprostheses. We propose a method for generating prompts based on gaze fixations to guide the Segment Anything Model (SAM) in our segmentation scenario, and fine-tune it on egocentric visual data. Evaluation results of our approach show an improvement of the IoU segmentation quality metric by up to 0.51 points on real-world challenging data of Grasping-in-the-Wild corpus which is made available on the RoboFlow Platform (https://universe.roboflow.com/iwrist/grasping-in-the-wild)