No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking

TL;DR

This paper evaluates the effectiveness of foundation models in animal tracking annotation, emphasizing that careful quality control and combined manual-automated approaches improve tracking robustness and accuracy.

Contribution

It provides an objective analysis of foundation models for animal tracking annotation, highlighting the importance of quality over automation and proposing a combined annotation strategy.

Findings

Automated annotations alone can reduce tracking accuracy.

Combining manual and automated annotations improves IDF1 score.

Careful oversight is essential to prevent noise from automated annotations.

Abstract

We analyze the capabilities of foundation models addressing the tedious task of generating annotations for animal tracking. Annotating a large amount of data is vital and can be a make-or-break factor for the robustness of a tracking model. Robustness is particularly crucial in animal tracking, as accurate tracking over long time horizons is essential for capturing the behavior of animals. However, generating additional annotations using foundation models can be counterproductive, as the quality of the annotations is just as important. Poorly annotated data can introduce noise and inaccuracies, ultimately compromising the performance and accuracy of the trained model. Over-reliance on automated annotations without ensuring precision can lead to diminished results, making careful oversight and quality control essential in the annotation process. Ultimately, we demonstrate that a thoughtful combination of automated annotations and manually annotated data is a valuable strategy, yielding an IDF1 score of 80.8 against blind usage of SAM2 video with an IDF1 score of 65.6.