Orchestrating segment anything models to accelerate segmentation annotation on agricultural image datasets

TL;DR

A new tool called ARAMSAM uses advanced AI models to speed up the labeling of agricultural images, making it much faster for experts to annotate.

Contribution

ARAMSAM is a novel user interface that combines SAM models with annotation tools to accelerate segmentation annotation in agriculture.

Findings

SAM 2's F2-score improved from 0.05 to 0.74 after hyperparameter optimization of its AMG.

User interaction time was reduced to 1.6 s/mask for SAM 2 on image sequences compared to polygon drawing.

ARAMSAM will be released as open-source software under the AGPL-3.0 license.

Abstract

Increasingly many applications of machine vision and artificial intelligence (AI) can be observed in agriculture. Yet, high-quality training data remains a bottleneck in the development of many AI solutions, particularly for image segmentation. Therefore, ARAMSAM (agricultural rapid annotation module based on segment anything models) was developed, a user interface that orchestrates the pre-labelling capabilities of both the segment anything models (SAM 1, SAM 2) and conventional annotation tools. One in silico experiment on zero-shot performance of SAM 1 and SAM 2 on three unseen agricultural datasets and another experiment on hyperparameter optimization of the automatic mask generators (AMG) were conducted. In a user experiment, 14 agricultural experts applied ARAMSAM to quantify the reduction of annotation times. SAM 2 benefited greatly from hyperparameter optimization of its AMG. Based on ground-truth masks matched with predicted masks, the F2-score of SAM 2 improved from 0.05 to 0.74, while that of SAM 1 was improved from 0.87 to 0.93. The user interaction time could be reduced to 2.1 s/mask on single images (SAM 1) and to 1.6 s/mask on image sequences (SAM 2) compared to polygon drawing (9.7 s/mask). This study demonstrates the potential of segment anything models as incorporated into ARAMSAM to significantly accelerate the process of segmentation mask annotation in agriculture and other fields. ARAMSAM will be released as open-source software (AGPL-3.0 license) at https://github.com/DerOehmer/ARAMSAM.

Figures