HideAndSeg: an AI-based tool with automated prompting for octopus segmentation in natural habitats

TL;DR

This paper presents HideAndSeg, an innovative AI tool that automates octopus segmentation in natural habitats, overcoming challenges like camouflage and occlusion, and establishing a baseline for future research in underwater animal analysis.

Contribution

It introduces a minimally supervised, fully automated pipeline combining SAM2 and YOLOv11 for octopus segmentation, with novel metrics for quality evaluation without ground-truth data.

Findings

Achieves satisfactory segmentation performance in natural environments.

Reduces manual effort and noise in segmentation results.

Successfully re-identifies octopuses after occlusion periods.

Abstract

Analyzing octopuses in their natural habitats is challenging due to their camouflage capability, rapid changes in skin texture and color, non-rigid body deformations, and frequent occlusions, all of which are compounded by variable underwater lighting and turbidity. Addressing the lack of large-scale annotated datasets, this paper introduces HideAndSeg, a novel, minimally supervised AI-based tool for segmenting videos of octopuses. It establishes a quantitative baseline for this task. HideAndSeg integrates SAM2 with a custom-trained YOLOv11 object detector. First, the user provides point coordinates to generate the initial segmentation masks with SAM2. These masks serve as training data for the YOLO model. After that, our approach fully automates the pipeline by providing a bounding box prompt to SAM2, eliminating the need for further manual intervention. We introduce two unsupervised metrics - temporal consistency $DICE_t$ and new component count $NC_t$ - to quantitatively evaluate segmentation quality and guide mask refinement in the absence of ground-truth data, i.e., real-world information that serves to train, validate, and test AI models. Results show that HideAndSeg achieves satisfactory performance, reducing segmentation noise compared to the manually prompted approach. Our method can re-identify and segment the octopus even after periods of complete occlusion in natural environments, a scenario in which the manually prompted model fails. By reducing the need for manual analysis in real-world scenarios, this work provides a practical tool that paves the way for more efficient behavioral studies of wild cephalopods.