Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE

TL;DR

This study compares an AI model called ODYSSEE with human annotators in identifying live oysters in field images, finding that while the AI is faster, it is less accurate than both experts and non-experts.

Contribution

The study evaluates the ODYSSEE AI model's performance against human annotators for identifying live oysters and identifies factors affecting its accuracy.

Findings

ODYSSEE is faster than both expert and non-expert annotators in identifying live oysters.

The model overpredicts live oysters and achieves lower accuracy (63%) compared to human annotators (74-75%).

Image quality significantly affects model and human performance, with better images improving human accuracy but worsening model accuracy.

Abstract

Oysters are ecologically and commercially important species that require frequent monitoring to track population demographics (e.g., abundance, growth, mortality). Current methods of monitoring oyster reefs often require destructive sampling methods and extensive manual effort. However, these methods are destructive and are suboptimal for small-scale or sensitive environments. A recent alternative, the ODYSSEE model, was developed to use deep learning techniques to identify live oysters using video or images taken in the field of oyster reefs to assess abundance. The validity of this model in identifying live oysters on a reef was compared to expert and non-expert annotators. In addition, we identified potential sources of prediction error. Although the model can make inferences significantly faster than expert and non-expert annotators (39.6 s,
2.34±0.61
h,
4.50±1.46
h, respectively), the model overpredicted the number of live oysters, achieving lower accuracy (63%) in identifying live oysters compared to experts (74%) and non-experts (75%) alike. Image quality was an important factor in determining the accuracy of the model and annotator. Better quality images improved human accuracy and worsened model accuracy. Although ODYSSEE was not sufficiently accurate, we anticipate that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes will greatly improve the model’s predictive power based on the results of this analysis. Future research should address methods that improve the detection of living vs dead oysters.

Figures