Adaptive Navigation Scheme for Optimal Deep-Sea Localization Using Multimodal Perception Cues

TL;DR

This paper introduces an adaptive, multimodal underwater navigation scheme that enhances deep-sea localization accuracy by combining dense point cloud generation, plane registration, and adaptive perception cue selection, validated through simulations and field trials.

Contribution

It presents a novel two-stage adaptive navigation approach that integrates multimodal perception cues for improved underwater localization performance.

Findings

Enhanced localization accuracy in deep-sea scenarios.

Effective noise filtering using probabilistic maps.

Validated approach through simulation and field trials.

Abstract

Underwater robot interventions require a high level of safety and reliability. A major challenge to address is a robust and accurate acquisition of localization estimates, as it is a prerequisite to enable more complex tasks, e.g. floating manipulation and mapping. State-of-the-art navigation in commercial operations, such as oil & gas production (OGP), rely on costly instrumentation. These can be partially replaced or assisted by visual navigation methods, especially in deep-sea scenarios where equipment deployment has high costs and risks. Our work presents a multimodal approach that adapts state-of-the-art methods from on-land robotics, i.e., dense point cloud generation in combination with plane representation and registration, to boost underwater localization performance. A two-stage navigation scheme is proposed that initially generates a coarse probabilistic map of the workspace, which is used to filter noise from computed point clouds and planes in the second stage. Furthermore, an adaptive decision-making approach is introduced that determines which perception cues to incorporate into the localization filter to optimize accuracy and computation performance. Our approach is investigated first in simulation and then validated with data from field trials in OGP monitoring and maintenance scenarios.

Tables

Table 1. Table I: Dense map, plane extraction and orientation measures on simulated stereo sequences

Method	Accuracy	Coverage	Planes	Holes	Error ${[}^{\circ }]$
RGB-D	1.0	1.0	1620	456	$11.7\pm 3.3$
ELAS	0.781	0.579	2708	713	$16.2\pm 7.3$
Dispnet	0.713	0.943	1987	123	$19.4\pm 8.5$
ELAS+Filter	0.854	0.468	2061	204	$12.1\pm 6.4$
Dispnet+Filter	0.798	0.833	1254	18	$09.3\pm 2.1$

Table 2. Table II: Image quality based navigation performance

	VO-ORB	VO-planes
CPU $[\%]$	3.2	6.8
GPU $[\%]$	0.1	17.6
Time $\left[s\right]$	0.145	3.151

Table 3. Table IV: Tests 𝒯 L ​ i subscript 𝒯 𝐿 𝑖 \mathcal{T}_{Li} measure results for position/orientation error and trajectory autocorrelation

	${𝒯}_{L1}$	${𝒯}_{L2}$	${𝒯}_{L3}$
${\overline{m}}_{M,F}\left({𝒯}_{Li}⟨\overline{𝐩}⟩\right)\left[m\right]$	0.65 $\pm$ 0.58	0.31 $\pm$ 0.11	0.85 $\pm$ 0.22
${\overline{m}}_{M,F}\left({𝒯}_{Li}⟨\overline{𝐪}⟩\right)\left[\mathrm{deg}\right]$	14.65 $\pm$ 8.42	7.21 $\pm$ 2.10	11.89 $\pm$ 4.55
$m_A\left({𝒯}_{Li}\right)$	0.88	0.94	0.91

Equations

$${\mathbf{T}}_P^O={\mathbf{T}}_R^O{\mathbf{T}}_C^R{\mathbf{T}}_M^C{\mathbf{T}}_P^M$$

$${\mathbf{T}}_R^O={\mathbf{T}}_P^O{\mathbf{T}}_M^P{\mathbf{T}}_C^M{\mathbf{T}}_R^C$$

$${\mathbf{C}}_R^O=\mathrm{diag}({\sigma }_{{\mathbf{p}}_x}^2,{\sigma }_{{\mathbf{p}}_y}^2,{\sigma }_{{\mathbf{p}}_z}^2,{\sigma }_{{\mathbf{q}}_{\varphi }}^2,{\sigma }_{{\mathbf{q}}_{\theta }}^2,{\sigma }_{{\mathbf{q}}_{\psi }}^2).$$

$$\mathbf{A}=\left(\begin{array}{ccc}{\Gamma }_n(x,x)& {\Gamma }_n(x,y)& {\Gamma }_n(x,z)\\ {\Gamma }_n(y,x)& {\Gamma }_n(y,y)& {\Gamma }_n(y,z)\\ {\Gamma }_n(z,x)& {\Gamma }_n(z,y)& {\Gamma }_n(z,z)\end{array}\right)$$

$${\Gamma }_{l+1}(\alpha ,\beta )=$$

$$-m_{\alpha }(l+1)(\sum _{j=1}^{l+1}{p}_j+m_{\alpha }(l+1))$$

$$+m_{\alpha }(l)(\sum _{j=1}^l{p}_j+m_{\alpha }(l))$$

$$|log{|}^1{H}_i{|}_{+}-log{|}^2{H}_i{|}_{+}|<{\bar{L}}_{det}$$

$${}^1{\hat{n}}_{i_1}^{\top }{}^1{\hat{n}}_{i_2}\approx {}^2{\hat{n}}_{j_1}^{\top }{}^2{\hat{n}}_{j_2}$$

$$\underset{{}_2^{1}R}{\max }{\zeta }_r=constant+\sum _{i=1}^S{\omega }_i{}^1{\hat{n}}_i\cdot ({}_2^{1}R{}^2{\hat{n}}_i)$$

$$\underset{{}_2^1\hat{q}}{\max }{\zeta }_r$$

$${}_2^1{\mathbf{H}}_{\hat{q}\hat{q}}=2(K-{\mu }_{max}(K)I)$$

$${}_2^1{\mathbf{C}}_{\hat{q}\hat{q}}=-{{}_2^1{\mathbf{H}}_{\hat{q}\hat{q}}}^{+}$$

$$m(\mathcal{E})=$$

$$=$$

Full text

I Introduction

The marine environment is challenging for automation technologies. Yet, oceans are one of the main forces driving commerce, employment, economic revenue and natural resources exploitation, which in turn triggers profound interest in the development of new technologies to facilitate intervention tasks, e.g., in oil&gas production (OGP). Remote Operated Vehicles (ROVs) are the current work-horse used for these tasks, which include inspection of ships, submerged structures and valves manipulation.

In particular, manipulation tasks are extremely challenging without stable and accurate robot localization. In general, a global positioning based navigation is desirable to correct measurements from inertial navigation systems (INS) in a tightly-coupled approach [1]. However, such data has to be transmitted acoustically through ultra-short/long baseline (USBL/LBL) beacons that have low bandwidth [2], signal delays and deployment constraints. Additional sensors, i.e., Doppler velocity logs (DVLs) and digital compasses, can improve the localization accuracy but still not at the required standards to perform floating-base manipulation.

We present a navigation scheme that uses visual odometry (VO) methods based on stereo camera imagery and an initial probabilistic map of the working space to boost localization accuracy in challenging conditions. The application scenario is the monitoring and dexterous manipulation of a OGP panel (Fig. 1) within the EU-project DexROV [3, 4]

To address the challenges of underwater vision, we combine plane registration and feature tracking methods. 3D planes are extracted from dense point cloud (DPC) generators, which produce complete disparity maps at the cost of depth accuracy; density being the key factor to find reliable 3D planes. This is particularly useful in structured or man-made environments, which predominantly contain planar surfaces, like the installations used in OGP. Furthermore, a decision-making strategy based on image quality is introduced: it allows to select the visual odometry method to be used in order to obtain reliable measurements and improve computation times. In summary our contributions are:

• •

Development of different visual odometry (VO) modalities based on: knowledge-enabled landmarks, 3D plane registration and feature tracking.

• •

Integration of the multimodal VO into an underwater localization filter that adapts its inputs based on image quality assessments.

• •

A two-step navigation scheme for structured environments. Initial suboptimal localization measurements are used to compute a coarse probabilistic 3D map of the workspace. In turn, this map is used to filter noise and optimize the integration of further measurements.

• •

Validation of the presented scheme in realistic field conditions with data from sea trails off-shore the coast of Marseille, France.

II Related Work

A great number of theoretical approaches on localization filters for marine robotics have been proposed in the literature. In recent years, this also includes increasing efforts to address practical issues such as multi-rate sampling, sensor glitches and dynamic environmental conditions. In [5], a review of the state-of-the-art in underwater localization is presented and classified into three main classes: inertial/dead reckoning, acoustic and geophysical.

The surveyed methods show a clear shift from technologies like USBL/LBL positioning systems [6] towards two research areas. First, dynamic multiagent systems which include a surface vehicle that complements the underwater vehicles position with GPS data [7]; and secondly, the integration of visual navigation techniques, i.e., visual odometry [8] and SLAM [9], into marine systems. We also integrate inertial data from DVL and IMU with vision-based techniques using standard 2D features and in addition 3D plane registration. The work in [10] shows that the combination of standard visual features with geometric visual primitives increases odometry robustness in low texture regions, highly frequent in underwater scenes.

Three methods are commonly used for plane primitive extraction: RANSAC, Hough transform and Region Growing (RG). State of the art methods [11, 12, 13] often use RG because it exploits the connectivity information of the 3D points and, thus, have more consistent results in the presence of noise. These are better suited for our application since the input point cloud for the plane extraction algorithm is not directly generated from an RGB-D camera but from a stereo image processing pipeline. We compare some of these stereo pipelines to investigate their impact on the overall localization accuracy (see Sec. IV-A).

Finally, to test the complete framework, we used the continuous system integration and validation (CSI) architecture proposed in our previous work [14]. With this architecture, parts of the developmental stack can be synchronized with real-world data from field trials to close the discrepancy between simulation and the real world; this process is inspired by the simulation in the loop (SIL) methodology [3].

Based on this, we first compute the accuracy of our approach in an optimized simulation environment reflecting similar light conditions as observed in underwater trials. Then, its effectiveness is validated on field trial data featuring real-world environmental conditions.

III Methodology

Figure 2 shows the proposed two-stage navigation scheme:

First stage-Workspace definition with loose localization

1.1.

Approach the target (oil&gas panel) until its global 3D pose its confidently determined based on a priori knowledge; see Sec. III-A, Fig. 2(a).

1.2.

Navigate close to the target using odometry from navigation sensors and visual landmarks (baseline localization); see Sec. III-A2 and Fig. 2(a)(b).

1.3.

Compute a probabilistic map from stereo input of the target object/area while navigating based on the odometry uncertainty; see examples in Fig. 2(c).

Second stage-Optimized localization

2.1

Evaluate the reliability of the visual input, i.e., stereo image quality (Sec. III-D, Fig. 2(h)), and determine which of the next VO modalities to use:

2.2.a

Extract planes (Sec. III-B2) from dense pointclouds (Sec. III-B1), filtered using the probabilistic map computed in the first stage to prevent large drifts and noise artifacts. See Fig. 2(g)and Fig. 3.

2.2.b

Extract and track robust 2D features from imagery; see Sec. III-C2, Fig. 2(f).

2.3.

Compute visual odometry either from plane registration or feature tracking (Sec. III-C1, III-C2) depending on the image quality assessment (IQA) and integrate the results into the localization filter.

The objective of the first stage is to compute a probabilistic map (octomap [15]) of the expected ROV workspace area. A coarse 3D representation of the scene can be obtained given few samples. Figure 2(c) illustrates this by comparing the octomap generated with a simulated RGB-D camera (reference) and the one generated by pointclouds from stereo imagery. High precision is not crucial here since mapping is not the final goal, but to filter spurious 3D pointclouds, e.g., from dynamic objects like fish or from processing artifacts, as shown in Fig. 3. We will now describe in detail each component from the second stage, optimized localization.

III-A Knowledge-enabled localization

Underwater missions are cost-intensive and high-risk, thus prior knowledge about the mission reduces the risk of failures and increases safety. Especially in visual inspection or manipulation tasks of man-made structures, the incorporation of prior knowledge can be exploited. Therefore, we built a knowledge base that contains properties of task-related objects. Along with offline information, like CAD models and kinematic descriptions of the robot and OGP panel, the knowledge base is updated based on current information gathered over the execution of a task, e.g. panel valve poses.

III-A1 Panel Pose Estimation

The panel pose estimation is the basis for projecting the panel model and its kinematic properties into the world-model. This further enables reliable task benchmarking in simulation and real operations, i.e., manipulation of valves and handles. Our approach incorporates offline knowledge such as the panel CAD model and visual markers placed at predefined locations, see Fig. 2(a). Based on this augmentation of the panel with markers, we exploit the panel as a fixed landmark and infer the robot pose whenever a visual marker is in the camera view as described in the next Sec. III-A2. The panel pose in odometry frame \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle P\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle P\hskip 3.61111pt}}} can be reliably estimated using the detected marker pose w.r.t. the camera frame \,\mathchoice{{\hskip 7.55415pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle C\hskip 7.22223pt}_{\scriptstyle M\hskip 7.22223pt}}}{{\hskip 7.55415pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle C\hskip 7.22223pt}_{\scriptstyle M\hskip 7.22223pt}}}{{\hskip 5.39581pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle C\hskip 5.05556pt}_{\scriptscriptstyle M\hskip 5.05556pt}}}{{\hskip 5.39581pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle C\hskip 3.61111pt}_{\scriptscriptstyle M\hskip 3.61111pt}}}, the camera pose on the robot frame \,\mathchoice{{\hskip 5.50374pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle R\hskip 7.22223pt}_{\scriptstyle C\hskip 7.22223pt}}}{{\hskip 5.50374pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle R\hskip 7.22223pt}_{\scriptstyle C\hskip 7.22223pt}}}{{\hskip 3.93124pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle R\hskip 5.05556pt}_{\scriptscriptstyle C\hskip 5.05556pt}}}{{\hskip 3.93124pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle R\hskip 3.61111pt}_{\scriptscriptstyle C\hskip 3.61111pt}}}, the panel pose in marker frame \,\mathchoice{{\hskip 7.55415pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle M\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 7.55415pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle M\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 5.39581pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle M\hskip 5.05556pt}_{\scriptscriptstyle P\hskip 5.05556pt}}}{{\hskip 5.39581pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle M\hskip 3.61111pt}_{\scriptscriptstyle P\hskip 3.61111pt}}}, and the current robot pose in odometry frame \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}, see Fig. 2(e):

[TABLE]

When $n$ different markers are constantly detected during $k$ image frames $I$, $n$ pose estimates \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle P\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle P\hskip 3.61111pt}}} are extracted. These are used to compute the mean position and orientation, determined by spherical linear interpolation (Slerp).

III-A2 Visual Landmark-Based Odometry

Once the panel pose has been estimated and fixed, the robot pose can be inferred every time there is a visual marker observation and used as an Extended Kalman Filter input modality.

Fig. 2(e) shows a sample pose estimate of a visual marker; note that the panel is partially observed but the marker is used to infer the panel pose through the space transformations. Since the panel pose is fixed, the robot pose \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}} can be estimated as follows:

[TABLE]

where \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle P\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle P\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle P\hskip 3.61111pt}}} is the panel pose in odometry frame, \,\mathchoice{{\hskip 7.55415pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle P\hskip 7.22223pt}_{\scriptstyle M\hskip 7.22223pt}}}{{\hskip 7.55415pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle P\hskip 7.22223pt}_{\scriptstyle M\hskip 7.22223pt}}}{{\hskip 5.39581pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle P\hskip 5.05556pt}_{\scriptscriptstyle M\hskip 5.05556pt}}}{{\hskip 5.39581pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle P\hskip 3.61111pt}_{\scriptscriptstyle M\hskip 3.61111pt}}} is one marker pose in panel frame, \,\mathchoice{{\hskip 7.55415pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle M\hskip 7.22223pt}_{\scriptstyle C\hskip 7.22223pt}}}{{\hskip 7.55415pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle M\hskip 7.22223pt}_{\scriptstyle C\hskip 7.22223pt}}}{{\hskip 5.39581pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle M\hskip 5.05556pt}_{\scriptscriptstyle C\hskip 5.05556pt}}}{{\hskip 5.39581pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle M\hskip 3.61111pt}_{\scriptscriptstyle C\hskip 3.61111pt}}} is the camera pose w.r.t. the marker and \,\mathchoice{{\hskip 5.50374pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle C\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.50374pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle C\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.93124pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle C\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.93124pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle C\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}} is the robot fixed pose w.r.t. the camera. Further on, the means of robot position \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}} and orientation \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}} w.r.t. the odometry frame are estimated from multiple marker detections using Slerp. In addition, a covariance matrix \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{C}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{C}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{C}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{C}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}} for the robot pose is computed:

[TABLE]

The full robot pose estimate \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}=\langle\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}},\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle R\hskip 5.00002pt}}}\rangle along with the respective covariance matrix \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{C}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{C}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{C}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{C}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}} is then taken as an input for the localization filter in the final setup. Alternatively, it can be used as a ground truth value to optimize parameters, i.e., sensor biases and associated covariances, since it is difficult to acquire absolute global ground truth underwater. For more details refer to our work in [3].

III-B Dense depth mapping and plane extraction

III-B1 Depth map computation

This is a preprocessing step for the plane-based odometry computation (Sec. III-C1). We consider two state-of-the-art real-time dense point cloud (DPC) generators: Efficient Large-Scale Stereo Matching (ELAS) [16] and Dispnet [17]. ELAS is geometry based and relies on a probabilistic model built from sparse keypoints matches and a triangulated mesh, which is used to compute remaining ambiguous disparities. Since the probabilistic prior is piece wise linear, the method is robust against uniform textured areas. Dispnet is a data driven approach that uses a convolutional neural network for disparity regression.

The reason to compare these methods is due to their underlying distinct approaches, which in turn offer different advantages and disadvantages. For example, ELAS has faster processing times in CPU and outputs more precise but incomplete maps when specular reflections or very large textureless areas occur. Dispnet produces a complete map even in the presence of image distortions, but smooths the disparity changes around object boundaries.

On top of the depth estimation, it is important to include techniques that reduce noise-induced artifacts commonly present when performing outdoor stereo reconstruction. Fig. 3 shows an example where object borders produce comet-like streaks fading away from the camera position. We also encountered this artifact when running the system with Dispnet during sea field trials. It was observed that when the GPU RAM memory gets overloaded some layers of the Dispnet neural network produce spurious results.

In order to mitigate these artifacts, the incoming point cloud is filtered by rejecting points which do not intersect with the octomap [15] representing the workspace obtained from the first navigation stage (loose localization). For an efficient nearest-neighbor search between point cloud and octomap voxels, a kd-tree is applied to represent the workspace octomap. Consequently, a substantial amount of points is neglected (Fig. 2(g) – depth map point cloud generation) and also reduces computation cost and memory.

III-B2 Plane extraction

Due to the noisy nature of stereo generated pointclouds, we used a region growing based technique for plane extraction [11]. It also outputs a covariance matrix that describes the planarity of the found planes, which can then be integrated for a better estimation of the localization uncertainty (Fig. 2(g) – plane extraction-based visual odometry). Moreover, it efficiently represents not only planes but found holes as polygons, which allows to reason about the quality of the data.

In summary, point clouds are segmented into several planes $\mathbb{\varPi}=\{\pi_{i}|i=1,...,N\}$. Initially, an arbitrary point $p_{0}$ and its nearest neighbor $p_{1}$ form an initial set $\mathbb{P}_{i}$ representing the plane $\pi_{i}$. Then the set $\mathbb{P}_{i}$ grows by finding its nearest neighbors $p_{j}$ and adding $p_{j}$ to $\mathbb{P}_{i}$ if $p_{j}$ belongs to $\pi_{i}$ (plane test), and stops when no more points can be added. Afterwards, the process continues until all the points are distributed into a plane set $\mathbb{P}$ or considered as noise.

III-C Visual odometry

The visual markers attached to the panel (Sec. III-A2) are not always observable. Therefore, further methods are beneficial to aid navigation. Here we adapt plane-based and featured-based odometry methods to our scenario to exploit structures and features found in the environment.

III-C1 Odometry from plane registration

After plane segmentation (Sec. III-B2), the plane normal equals to the eigen vector with smallest eigen value of Matrix $\mathbf{A}$:

[TABLE]

where $\Gamma_{n}(\alpha,\beta)=\sum_{j}^{n}(\alpha_{j}-m_{\alpha})(\beta_{j}-m_{\beta}),\alpha,\beta\in\{x,y,z\}$ and $m$ is the mass center of the points in plane set $\mathbb{P}_{i}$. To update the matrix $\mathbf{A}$ and hence the planes normal representation as fast as possible when new points are considered, the sum of orthogonal distances $\Gamma_{l}(\alpha,\beta)$ is updated with a new point $p_{l+1}$ as follows:

[TABLE]

Then the relative pose ${{}^{C}_{R}\mathbf{T}_{rel}}$ in camera frame at time $t$ and $t+1$ can be calculated by the extracted planes. Here we exploit the plane registration method in [18] to estimate rotation only. As shown in Sec. IV-A experiment, in our deep-sea scenario we commonly encountered between 3 to 5 planes per frame, and at least 4 plane correspondences are necessary to estimate translation.

Suppose planes extracted at time $t$ and $t+1$ be $\varPi_{1}=\{{{}^{1}\pi_{i}}|i=1,...,M\}$ and $\varPi_{2}=\{{{}^{2}\pi_{j}}|j=1,...,N\}$ respectively, then the $M\times N$ candidate pairs $({{}^{1}\pi_{i}},{{}^{2}\pi_{j}})$ are filtered and registered by the following tests from [18], which are adapted to our deep-sea setup:

• •

Size-Similarity Test: The Hessian matrix $H$ of the plane parameters derived from plane extraction is proportional to the number of points in the plane $\pi$. Thus, the Hessian of two matched planes should be similar, i.e.,

[TABLE]

where $\lvert H_{i}\rvert_{+}$ is the product of the singular values of $H_{i}$ and $\bar{L}_{det}$ is the similarity threshold.

• •

Cross-Angle Test: The angle between two planes $({{}^{1}\pi_{i_{1}}},{{}^{1}\pi_{i_{2}}})$ in frame ${{}^{1}\mathcal{F}}$ should be approximately the same as the angle to the correspondent two planes $({{}^{2}\pi_{j_{1}}},{{}^{2}\pi_{j_{2}}})$ in frame ${{}^{2}\mathcal{F}}$, described as

[TABLE]

where $\hat{n}_{k}$ is the normal to the plane $\pi_{k}$.

• •

Parallel Consistency Test: Two plane pairs $({{}^{1}\pi_{i_{1}}},{{}^{2}\pi_{j_{1}}})$ from the frames ${{}^{1}\mathcal{F}}$ and ${{}^{2}\mathcal{F}}$ are defined as parallel if their normals meet ${{}^{1}\hat{n}^{\top}_{i_{1}}}{{}^{1}\hat{n}_{i_{2}}}\approx 1$ and ${{}^{2}\hat{n}^{\top}_{j_{1}}}{{}^{2}\hat{n}_{j_{2}}}\approx 1$, or ${{}^{1}\hat{n}^{\top}_{i_{1}}}{{}^{1}\hat{n}_{i_{2}}}\approx-1$ and ${{}^{2}\hat{n}^{\top}_{j_{1}}}{{}^{2}\hat{n}_{j_{2}}}\approx-1$.

If only one plane is extracted from the current frame, it is tested only by the Size-Similarity test because others require at least two plane correspondences. Then the filtered plane pairs are used to calculate the rotation ${{}^{1}_{2}R}$ between frame ${{}^{1}\mathcal{F}}$ and ${{}^{2}\mathcal{F}}$ by the equation:

[TABLE]

which can be solved by Davenport’s q-method and where $w_{i}$ are weights inversely proportional to the rotational uncertainty. If the rotation ${{}^{1}_{2}R}$ is represented as quaternion ${{}^{1}_{2}\hat{q}}$, Eq. 8 can be written as:

[TABLE]

Then the covariance ${{}^{1}_{2}\mathbf{C}_{\hat{q}\hat{q}}}$ of quaternion ${{}^{1}_{2}\hat{q}}$ is

[TABLE]

where $\mu_{max}(K)$ is the maximum eigen value of $K$, derived from Davenport’s q-method. The covariance ${{}^{1}_{2}\mathbf{C}_{\hat{q}\hat{q}}}$ and rotation ${{}^{1}_{2}\hat{q}}$ are used as input for the our navigation filter.

III-C2 Feature-based tracking

Whenever there are distinctive and sufficient 2D texture features in the environment, related methods provide a reliable and fast way to compute odometry. Here, ORB-SLAM2 [19] is used. It consists of three main threads: tracking, local mapping, and loop closing. Considering our application, we briefly describe the tracking process. When a new stereo image is input, the initial guess of the pose ${{}^{C}_{R}\mathbf{T}^{\prime}}$ is estimated through the tracked features from the last received image. Afterwards, the pose ${{}^{C}_{R}\mathbf{T}}$ can be improved by conducting bundle adjustment on a memorized local map $\mathbf{M}_{i}$. Moreover, the tracking thread also decides whether the stereo image should be an image keyframe or not. When tracking fails, new images are matched with stored keyframes to re-localize the robot.

ORB-SLAM2 was chosen because direct VO methods (DSO,LSD-SLAM) assume brightness constancy throughout image regions [20], which seldom happens in underwater due to light backscatter. Likewsie, visual-inertial SLAM methods (VINS,OKVIS) require precise synchronization between camera and IMU [21], but by hardware design all sensors are loose-coupled in our application.

III-D Adaptive image quality based navigation

At the end of the localization pipeline the EKF can integrate all the inputs based on their measurement confidence, i.e., covariance matrix. For efficiency, it is preferable to filter out low confidence odometry values before using them for the EKF. This could be done by examining the covariance matrix after the vision processing. But computation time is an important factor in our real-time application. We hence use decision criteria on the sensor (image) quality to omit visual odometry computations, which are likely to generate low confidence results.

We introduced multiple visual odometry modalities in Sec. III-C; see Fig. 2(e)(f)(g). The visual marker-based odometry, as part of the baseline inputs, is not filtered out due to its high reliability and precision. Feature tracking ORB-SLAM localization is highly dependent on image quality; textureless regions and low-contrast significantly reduce its accuracy. In contrast, plane-based odometry copes well with textureless environments given that there is an underlying structure. But it is very computationally demanding due to dense depth estimation and plane extraction (Sec. III-B).

Based on this, we propose an image quality assessment (IQA) to reason about which visual cues to use in the localization pipeline. We aggregate a non-reference image quality measure based on Minkowski Distance (MDM) [22] and the number of tracked ORB features between consecutive frames. The MDM provides three values in the $\left[0,1\right]$ range describing the contrast distortion in the image; thus, the number of ORB features is normalized based on the predefined maximum number of features to track. If each of these IQA values is defined as $m_{I}(t)$, the final measurement for each timestamp $t$ is their average $\overline{m}_{I}(t)$. Experiments in Sec. IV-B show the performance of these IQA measurements.

IV Experimental Results

The first two experiments are performed in simulation to analyze their algorithmic behavior. The simulator ambient lighting parameters are adjusted to match conditions from the sea trials. The simulated ROV navigates around while keeping \approx$$1.5\text{\,}\mathrm{m} from the OGP panel since it was found to be an optimal distance for our stereo camera baseline of $30\text{\,}\mathrm{c}\mathrm{m}$; also a constant $z$-axis value (depth) is kept.

First, we evaluate the impact of the dense pointcloud generators, ELAS and Dispnet, on the plane extraction, registration and orientation computation (Sec. III-B2, III-C1). Furthermore, we study how the filter based on the probabilistic map generated from the first stage of our navigation scheme improves performance. The second experiment assesses the accuracy of the VO approaches, i.e., our plane registration and feature tracking (ORB-SLAM2), with different types of imagery. The last experiment tests our complete pipeline with real-world data from DexROV field trials in the sea of Marseille, France (see Fig. 1). The cameras used are Point Grey Grasshoppers2 which have a resolution of $688\times 516$ pixels and $10\text{\,}\mathrm{H}\mathrm{z}$ rate; both are in underwater housings with flat-panel that allows for image rectification using the PinAx model [23].

IV-A Plane segmentation from dense depth maps

In this first experiment, we investigate how the plane extraction and registration algorithms perform with different dense point cloud generators. Table I shows the experiment results. We use as a baseline the simulated RGB-D camera available in the Gazebo simulation engine, which provides ground truth depth/disparity maps. To measure the accuracy of the stereo disparities (second column) the same principle as the 2015 KITTI stereo benchmark was followed [24], all disparity differences greater than 3 pixels or $5\%$ from the ground truth are considered erroneous. The coverage score (third column) counts how many image pixels have a valid associated disparity value; textureless regions reduce this value. Furthermore, we also count the number of extracted planes and holes within them (fourth and fifth column) using the method from Sec. III-B2. Finally, the last column of Table I shows the orientation error computed from the plane registration, Sec. III-C1.

It is important to note that the tests are performed one more time using the probabilistic map generated from the first stage of our methodology (Fig. 2(c)) as a filter. We draw the next conclusions from this experiment:

ELAS depth maps have more accurate 3D information at the cost of incomplete maps, which produce higher number of planes and holes due to the inability to connect regions corresponding to the same planes. 2. 2)

Since these redundant planes are still accurate space representations, they produce better orientation estimations than the complete but inaccurate Dispnet maps. 3. 3)

Filtering point clouds with the probabilistic map boosts accuracy and reduces the coverage of both methods, which validates the efficiency of our two-stage navigation scheme. 4. 4)

Dispnet$+$Filter orientation accuracy is even greater than the one based on the simulated RGB-D camera. From our observations, Dispnet$+$Filter generates less planes than the RGB-D depth maps; RGB-D very high accuracy produces planes for small objects such as the panel’s valves which add ambiguity. 5. 5)

Thus, highly accurate point clouds (overfitting) negatively affects plane registration, i.e., the likelihood of incorrectly registering nearby small planes increases. 6. 6)

Based on the 367 analyzed image frames, the mean number of planes generated with Dispet+Filter per frame is $3$ or $4$.

IV-B Image quality based navigation performance

Based on the previous experiment, we choose Dispnet as dense point cloud generator. To analyze the strengths and weaknesses of the VO methods, the ROV circles the panel while acquiring very diverse stereo imagery. The panel has three sides with distinct purposes: valve manipulation (side 1), dextereous grasping (side 2) and textureless side (side 3), see Fig. 4. This last panel side helps evaluating how the methods work with scarce image features, and how our image quality measure $\overline{m}_{I}(t)$ from Sec. III-D evaluates this.

The expected error pose is defined as the difference between the ground-truth robot pose in simulation \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{S}} and the robot pose determined from visual odometry \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{VO}}:

[TABLE]

where d\left(\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{}_{S}},\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\bar{\mathbf{p}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\bar{\mathbf{p}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{}_{VO}}\right) is the Euclidean distance between positions and d\left(\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{}_{S}},\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 5.53387pt\hbox{\textstyle{\mathbf{\bar{q}}}}^{\scriptstyle O\hskip 5.00002pt}_{\scriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{\mathbf{\bar{q}}}}^{\scriptscriptstyle O\hskip 5.00002pt}_{\scriptscriptstyle P\hskip 5.00002pt}}}{{}_{VO}}\right) is the minimal geodesic distance between orientations. For our experiments, we also compute the lag-one autocorrelation m_{A}=\sum_{t}\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{F}}(t)\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{F}}(t-1) on the EKF filter predicted poses \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{F}}; $m_{A}$ is a measure of trajectory smoothness, important to prevent the robot from performing sudden jumps.

IV-B1 Visual odometry accuracy

First, we evaluate the accuracy of the proposed VO methods. Fig. 5(top) shows the results, the time axis has been normalized and the error is logarithmically scaled for better readability. The orange horizontal lines indicate the time when no visual marker is detected, and the vertical red lines show when the ROV is transitioning into another side of the panel.

As expected, the orientation derived from the markers is the most accurate but it also presents the outliers with the largest errors, e.g., close to $0.1\text{\,}\mathrm{s}$ and $0.7\text{\,}\mathrm{s}$ in Fig. 5. The feature tracking ORB-SLAM2 method presents the greatest error but with the least variance. When there are very scarce features to track, such as in panel’s side 3, the error abruptly increases until a memory-saved keyframe from the panel’s side 1 is seen again; close to time $1.0\text{\,}\mathrm{s}$.

The plane registration method has similar accuracy as the visual marker-based odometry and with scarce outliers only when the ROV transitions between panel sides. During these periods the corner of the panel is seen, which is not a planar but cylindrical surface. Then, depending on the viewpoint it will be represented with different various planes.

These results are complemented by Table II(a), which shows the higher computational costs of the plane-based VO. During field trials, the overall ROV perception$+$manipulation control system and its graphical interface have reported peaks of $92\%$ GPU RAM usage; hence, using Dispnet can lead to GPU overuse and spurious 3D maps. Likewise, the slow update times of plane-based VO might limit the ROV velocity. For these reasons, VO based on feature tracking is given preference when the image quality is good.

IV-B2 Image quality assessment

In this experiment, we validate that the proposed image quality measure $\overline{m}_{I}(t)$ detects when an image has low contrast and/or large uniform texture regions. This is shown in Fig. 5(bottom); $\overline{m}_{I}(t)$ is the lowest when panel side 3 is in view and when the ROV navigates around the corners. As it can be seen, $\overline{m}_{I}(t)$ mostly shows an inverse behavior than the VO accuracy with ORB-SLAM2.

Based on this simulations, we set a threshold $(\approx 0.45)$ for $\overline{m}_{I}(t)$ to only trigger the computationally expensive plane-based VO when the image quality is poor. When using the IQA to decide which VO inputs to integrate into the localization filter (EKF-adaptive), we reduce the pose error and increase the smoothness of the followed trajectory, see Table II(b). Simply integrating all odometry inputs (EKF-all) does not boost performance as the Kalman filter does not reason about the quality of the sensor data except for examining the inputs covariance matrix.

IV-C Field trials localization

In the following experiments, we use the visual landmarks (markers) pose estimates \,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{M}} as ground truth since they are quite accurate and the robot global pose can be obtained from them (Sec. III-A2). We perform three different tests $\mathcal{T}_{Li}$ explained in Table III; the corresponding results are shown in Table IV and Fig. 6. In these tests we compute the measure m_{M,F}(\mathcal{T}_{Li})=d\left(\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{M}},\,\mathchoice{{\hskip 5.53387pt\hbox{\displaystyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 5.53387pt\hbox{\textstyle{{\mathbf{T}}}}^{\scriptstyle O\hskip 7.22223pt}_{\scriptstyle R\hskip 7.22223pt}}}{{\hskip 3.95276pt\hbox{\scriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 5.05556pt}_{\scriptscriptstyle R\hskip 5.05556pt}}}{{\hskip 3.95276pt\hbox{\scriptscriptstyle{{\mathbf{T}}}}^{\scriptscriptstyle O\hskip 3.61111pt}_{\scriptscriptstyle R\hskip 3.61111pt}}}{{}_{F}}\right) as defined in equation 11, plus the lag-one autocorrelation $m_{A}(\mathcal{T}_{Li})$.

The use of visual landmarks has shown to substantially improve the localization filter accuracy [3] compared to using only navigation sensors. With data from DexROV sea trials, we first evaluate this method in $\mathcal{T}_{L1}$ and use it as reference. Fig. 6(a) shows that the majority of the largest errors occur when the robot is closer to the panel’s corners because markers are observed from highly skewed perspectives. Or when markers are not in view for a long period of time, e.g., in reference point 1 in Fig. 6(a). Of course, there can be other sources of error like spurious DVL measurements that affect the overall accuracy.

In test $\mathcal{T}_{L2}$, we use our navigation scheme based on IQA. Table IV and Fig. 6(b) show great reduction in the pose/orientation error $(\approx\%50)$ and an increase in the autocorrelation measure, i.e., smoother trajectories. Moreover, errors at the panel’s corners decrease, e.g. reference point 1. However, the largest errors still happen at these locations; after all, less features are observable and cylindrical corners (see Fig. 1) are imperfectly modeled by planes.

Finally in test $\mathcal{T}_{L3}$, we analyze the performance of our method without the use of visual landmarks. The objective is to strive towards a more general localization filter that can function without fiducial landmarks. Table IV shows that although the position and orientation error increase, they are not far from $\mathcal{T}_{L1}$ results. Furthermore, the error variance is significantly less; in Fig 6(c) the circles representing the pose errors have a more uniform size. This is more suitable for control algorithms, i.e., waypoint navigation and manipulation, which need a certain response time to converge into desired states. Highly variable measures may cause controllers to not converge. The same advantage can be said about the high autocorrelation values from $\mathcal{T}_{L2}$ and $\mathcal{T}_{L3}$. In contrast, $\mathcal{T}_{L1}$ variances are more than $50\%$ of the mean error.

V Conclusion

Underwater operations are harsh due to the dynamic environment and the limited access to the system. However, the commercial demand to develop these technologies increases every year. One of the many challenges to tackle, and commonly the first in the work pipeline, is the achievement of robust, reliable and precise localization. For this reason, we investigate the use of visual odometry in underwater structured scenarios, especially a plane-based method adapted for underwater use with stereo processing and a standard feature based method. Furthermore, an image quality assessment is introduced that allows decision making to exclude computationally expensive visual processing, which is likely to lead to results with high uncertainty. The approach is validated in simulation and especially also in challenging field trials.