Absolute 3D Pose Estimation and Length Measurement of Severely Deformed Fish from Monocular Videos in Longline Fishing

TL;DR

This paper introduces a novel monocular video-based method for estimating the absolute 3D pose and length of deformed fish in longline fishing, eliminating the need for multi-view data or 3D ground truth.

Contribution

It proposes a frame-based approach using a relative 3D fish template and a closed-form solution for accurate 3D pose and length estimation from a single 2D segmentation mask.

Findings

Accurately estimates 3D fish pose and length from monocular videos.

Outperforms state-of-the-art multi-view methods.

Refines length measurement through statistical temporal inference.

Abstract

Monocular absolute 3D fish pose estimation allows for efficient fish length measurement in the longline fisheries, where fishes are under severe deformation during the catching process. This task is challenging since it requires locating absolute 3D fish keypoints based on a short monocular video clip. Unlike related works, which either require expensive 3D ground-truth data and/or multiple-view images to provide depth information, or are limited to rigid objects, we propose a novel frame-based method to estimate the absolute 3D fish pose and fish length from a single-view 2D segmentation mask. We first introduce a relative 3D fish template. By minimizing an objective function, our method systematically estimates the relative 3D pose of the target fish and fish 2D keypoints in the image. Finally, with a closed-form solution, the relative 3D fish pose can help locate absolute 3D keypoints, resulting in the frame-based absolute fish length measurement, which is further refined based on the statistical temporal inference for the optimal fish length measurement from the video clip. Our experiments show that this method can accurately estimate the absolute 3D fish pose and further measure the absolute length, even outperforming the state-of-the-art multi-view method.