Designing A Sustainable Marine Debris Clean-up Framework without Human Labels

TL;DR

This paper presents a label-free, drone-based framework using computer vision for detecting, classifying, and mapping marine debris, enabling efficient ecological monitoring and cleanup without manual labeling.

Contribution

The study introduces a novel zero-shot detection and classification framework for marine debris using aerial imagery, eliminating the need for labeled training data.

Findings

Achieved 0.69 mean IoU in detection accuracy

Attained 0.74 F1 score in classification across seven debris classes

Comparable performance to supervised methods without labeled data

Abstract

Marine debris poses a significant ecological threat to birds, fish, and other animal life. Traditional methods for assessing debris accumulation involve labor-intensive and costly manual surveys. This study introduces a framework that utilizes aerial imagery captured by drones to conduct remote trash surveys. Leveraging computer vision techniques, our approach detects, classifies, and maps marine debris distributions. The framework uses Grounding DINO, a transformer-based zero-shot object detector, and CLIP, a vision-language model for zero-shot object classification, enabling the detection and classification of debris objects based on material type without the need for training labels. To mitigate over-counting due to different views of the same object, Scale-Invariant Feature Transform (SIFT) is employed for duplicate matching using local object features. Additionally, we have developed a user-friendly web application that facilitates end-to-end analysis of drone images, including object detection, classification, and visualization on a map to support cleanup efforts. Our method achieves competitive performance in detection (0.69 mean IoU) and classification (0.74 F1 score) across seven debris object classes without labeled data, comparable to state-of-the-art supervised methods. This framework has the potential to streamline automated trash sampling surveys, fostering efficient and sustainable community-led cleanup initiatives.