Streaming Scene Maps for Co-Robotic Exploration in Bandwidth Limited Environments

TL;DR

This paper introduces a bandwidth tunable, real-time probabilistic scene mapping technique for co-robotic exploration in communication-limited environments, demonstrated with underwater robots and adaptable to various bandwidth constraints.

Contribution

It presents a novel, adjustable scene modeling approach that balances scientific utility and communication bandwidth in co-robotic exploration.

Findings

The method effectively adapts scene maps to different bandwidth constraints.

Preliminary experiments validate the approach in simulated underwater environments.

Quantifies how model parameters affect both utility and bandwidth requirements.

Abstract

This paper proposes a bandwidth tunable technique for real-time probabilistic scene modeling and mapping to enable co-robotic exploration in communication constrained environments such as the deep sea. The parameters of the system enable the user to characterize the scene complexity represented by the map, which in turn determines the bandwidth requirements. The approach is demonstrated using an underwater robot that learns an unsupervised scene model of the environment and then uses this scene model to communicate the spatial distribution of various high-level semantic scene constructs to a human operator. Preliminary experiments in an artificially constructed tank environment as well as simulated missions over a 10m$\times$10m coral reef using real data show the tunability of the maps to different bandwidth constraints and science interests. To our knowledge this is the first paper to quantify how the free parameters of the unsupervised scene model impact both the scientific utility of and bandwidth required to communicate the resulting scene model.

Equations

$$(x,y,d)\sim N((x^{\prime },y^{\prime },d^{\prime }),\Sigma ),$$

$${\theta }_t\sim {\text{CRP}}_{x,y,d}(\gamma ,\alpha ),$$

$$z\sim \text{Categorical}({\theta }_t).$$

$${\varphi }_k\sim \text{Dirichlet}(\beta ),$$

$$w\sim \text{Categorical}({\varphi }_k).$$

$${\pi }^{\ast }=({\alpha }^{\ast },{\beta }^{\ast },{\gamma }^{\ast })$$

$$\mathbf{I}(Z_{\pi },A)$$

Full text

©2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

The challenges of exploration in remote and extreme environments such as the deep seas [1, 2], cave systems [3], outer space [4] and during or after a natural disaster [5, 6] have much in common. It is expensive and inherently dangerous for humans to explore such locations directly; hence, the use of mobile robots is desirable. However, if communication bottlenecks exist in the environment, prohibiting live streaming of video or other sensor data, then direct control of the robots is generally not possible. This paper describes a novel approach to co-robotic exploration in communication starved environments, and presents a system implementation of an under-sea exploration robot for co-robotic exploration of marine environments.

Although physically controlling a robot can be achieved over relatively low bandwidth, it is difficult to transmit the scene information necessary for an operator or scientist to make high level navigational decisions. We propose a spatially correlated Chinese Restaurant Process (CRP)-based [7] scene understanding model, that can be tuned to operate with the available bandwidth, for characterizing the environment in a manner useful for describing the operator’s scientific interest.

The most common approach to underwater exploration today is to either use a tethered vehicle or an autonomous underwater vehicle (AUV) to traverse a pre-planned path while collecting sensor data, which is reviewed once the robot returns to a location (the surface or a docking station) where high speed communication is possible [8]. Such missions are useful for collecting population statistics for dense and stationary phenomena, but have limited utility when collecting data on spatially patchy or transient phenomena, especially in new or poorly mapped locations.

II Background

Scientific robots have been deployed to study a variety of extreme environments, necessitating flexible models that represent scientifically relevant detail and structure in the environment. Simultaneous localization and mapping (SLAM) techniques have been used to address the challenge of robot localization and to construct occupancy and geometric maps of terrestrial caves and mines [9, 10], as well as in marine environments [11]. However, the occupancy maps used in SLAM are often not sufficient for modeling complex scientific phenomenon.

Gaussian processes (GPs) [12] and Gaussian mixture models have seen widespread application for modeling smooth, continuous environmental phenomena, such as chemical plumes [13] and plankton concentrations [14], in autonomous science applications. Other work has sought to explicitly model discrete, causal structure in scientific environments using Bayesian Networks (BN). Bayesian network models are suitable when generative model of the phenomena to be explored is known, but has its own set of challenges. Exact inference in BN can be difficult in all but very small problems. Arora et al. [15] have used BN’s to model geological phenomena; their formulation relies on a significant amount of prior domain knowledge specified by scientific experts in the form of conditional probability tables. Other works have used fixed thresholds on scalar sensor data set manually by domain scientists to dictate autonomous behavior [16] or rely on coarse prior maps of the environment, which are available when planning a mission [17]. While relying on prior scientific expertise or survey data is sometimes realistic, in environments such as the deep sea or space domain scientists may not have enough a priori knowledge to formulate a scientific model and hypotheses. The unsupervised terrain model used in this work does not require prior scientific expertise, training data, or environmental maps; instead, a terrain model is learned directly from discrete visual word observations in a camera stream.

Bayesian nonparametric models have been used extensively for unsupervised scene understanding due to their ability to characterize complex scenes with unknown complexity in terms of number of objects. Furthermore, strong Bayesian priors on the scene structure can enable realtime in-situ learning, even with high dimensional sensor data streams [18]. Sudderth et al. [19] have proposed the use of thresholded global positioning system (GPS) for modeling spatial distribution of scene constructs. Joho et al. [20] have used a Dirichlet Process in combination with a Beta process to represent indoor scene structure in terms of objects and their spatial configuration. Steinberg et al. [21] employ a Dirichlet process clustering technique to learning underwater terrain models. In our prior work [22] we proposed heirarchical Dirichlet process realtime online spatiotemporal topics (HDP-ROST), which uses a spatially coupled CRP to model the spatial distribution of observed substrate types. This unsupervised model allows for structure learning directly from visual observations, without the need for prior scientific knowledge about an environment or smoothness constraints. This paper combines HDP-ROST based scene understanding approach with a physical platform for bandwidth-limited robot autonomy to present a cohesive system for human-robot co-robotic exploration of marine environments.

From a hardware perspective, platforms for co-robotic exploration often seek to overcome the challenges of extreme environments using innovative physical designs. Ocean exploration using a tethered Remotely Operated Vehicles (ROV) is perhaps the most common approach to co-robotic scientific exploration [23, 24, 25], as it enables high speed real-time relay of all sensor data to a remote scientist, and enable low-level control of the vehicle. However some ocean environments such as under the Arctic ice have unique challenges that prevent the deployment of a regular ROV. The moving ice can make it difficult for a ship to keep station and deploy ROVs. The Nereid Under Ice [26] vehicle uses a novel approach to co-robotic exploration under-ice by maintaining a high-speed communication link through a hair-thin fiber-topic cable. This cable is unspooled from both the ship and the robot, and the robot maintains the capability to return to the open ocean to be recovered in case the tether breaks.

Conversely, long duration ocean observing platforms such as underwater gliders relay subsets of water column data over low speed satellite links when at the surface to direct future sampling efforts [27, 28]. Remotely gathered data is also fed back into increasingly sophisticated environmental models which are both used to understand the environment and to plan future trajectories of in-situ assets [29, 30]. Such approaches, however, are only suitable for co-robotic operation when the quantity of interest is a slowly varying low-dimensional field.

Our proposed approach to co-robotic exploration is as follows. The robot, unsupervised and equipped with a multitude of sensors, constructs a scene model to describe its environment in a concise but meaningful way. This scene model can be tuned to fit the available bandwidth budget, and describe the world at the level of abstraction suitable for describing the phenomena of interest. This scene model is constantly updated with new observation data, and transmitted regularly to the scientist. Given the scene model, the scientist then defines a utility function over various types of scene labels and other high level constraints, which is then broadcast to one or more underwater robots. In the case of multiple robots, it is possible to learn a shared representation through regular message passing between them [31].

The underwater robots can then use the utility function to plan an informative path plan, using either a greedy approach [32] or more spatially aware planning [33, 34], collecting more information that is then used to update the scene model.

This entire process continues in a loop (Fig. 2), enabling scientists to continually define high level goals for the robots, even in unknown environments, while letting robots perform low level path planning and sensing autonomously.

The focus of this paper is on 1) demonstrating a bandwidth-tunable scene understanding technique that can operate in unknown environments and be made to describe the scene at different levels of abstraction by changing only a handful of parameters; and 2) developing the hardware and software infrastructure to enable such an underwater co-robotic exploration mission.

III Scene Modeling

In the context of co-robotic exploration, we require a scene model that can operate in geographic coordinate space, and produce high-level and concise scene descriptions, which are transmitted over a low-bandwidth communication channel such as underwater acoustic communications. In this work we extend HDP-ROST [22] to operate on a stream of image observation data with 3D geographic coordinates.

III-A Generative Process

The generative model for the observed data is described as follows. At time $t$, we consider generating a set of categorical observations. Every one of these observations $w$ has an associated 3D observed position $(x,y,d)\in\mathbb{R}^{3}$ that is noisy, and modeled as a random sample from a Gaussian centered around the true position $(x^{\prime},y^{\prime},d^{\prime})$ with some measurement noise $\Sigma$:

[TABLE]

where $d$ is depth. Additionally, each visual observation is associated with a latent scene label $z$, drawn from a location-specific mixture over scene labels $\theta_{t}$:

[TABLE]

[TABLE]

We model the distribution of scene labels $z$ using a spatially correlated CRP. We discretize the world into cells, where each cell is represented by one pixel of the transmitted scene map. Each cell has a Chinese Restaurant (topic model) with an infinite number of tables, each corresponding to different topic labels. We assign a new customer (observation) to one of the $K$ occupied tables with probability proportional to $(n+\alpha)$, where $n$ is the sum of the number of customers sitting at the table in that restaurant and at corresponding tables in the neighboring restaurants, defined using the Von Neumann neighborhood of the cell. The hyperparameter $\alpha$ prioritizes labels used elsewhere in the scene. The customer sits at a new table with probability proportional to $\gamma$. Thus, the scene label $z$ will take a value $k\in[1,K+1]$, where $K$ is the number of distinct scene labels that we have observed thus far. The advantage of using a CRP is that we do not need to explicitly specify the number of scene labels a priori, and it is allowed to grow automatically with the size and complexity of the observation data.

Some random samples from the above generative process are shown in Fig. 3. We see that varying $\alpha$ and $\gamma$ gives us random maps with different spatial characteristics in terms of number of different types of patches, and size of the patches.

Finally, each scene label (topic) is associated with image descriptors (words), which we model using a categorical distribution over a set of $V$ features with the symmetric Dirichlet prior:

[TABLE]

[TABLE]

The use of Dirichlet priors biases the generative model towards learning sparse distribution of features used to describe a scene construct.

III-B Hyperparameter Selection

The above generative model can be used to characterize a scene at different levels of abstraction. For example, for a given set of hyperparameters it is possible that we learn labels that correspond to different parts of a coral, or individual coral species, or a colony of different types of corals that often co-exist. On a scientific exploration mission we would like the robot to characterize the scene at the same level of abstraction that is of interest to the scientist. Our proposed solution to enable this behavior is to use an annotated dataset from a location similar to what will be explored, with labels that describe the scene at the desired level of abstraction. Given these annotations $A$, we then find the hyperparameters $\pi=(\alpha,\beta,\gamma)$ for the generative model that maximizes the mutual information $\boldsymbol{I}$ between the latent scene labels $Z=\{z_{i}\}$ and the corresponding annotations $A=\{a_{i}\}$ using grid search. Assuming that $z=k\in[1,K]$ and $a=j\in[1,J]$:

[TABLE]

Note that compared with the standard approach for supervised learning, where the annotations are used to learn all the parameters of the model, we propose to use the annotations only for identifying 3 hyperparameters, while still learning the other parameters of the model in an unsupervised manner. Hence the scene model can still be used to characterize completely unknown environments.

IV System Design

Our system design is motivated by the vision that a team of underwater and surface robots with various sensing payloads would be interactively controlled by a scientist in an unknown environment to characterize phenomena observable by the sensing payload. Our current robot system (Fig. 4) is a step towards this vision, and consists of 1) a surface platform that is used as an air-to-sea communication and localization system to link the scientist to the robots; 2) an AUV equipped with acoustic and optical imaging capabilities, and sufficient computing power to generate concise scene maps from complex high dimensional sensor data streams; 3) a Robot Operating system (ROS) based software architecture to enable a compute graph that spans multiple computers, and optionally multiple robots.

IV-A Surface Vehicle

The autonomous surface vehicle (ASV) is primarily responsible for acting as a communication and localization relay for the AUV. It is equipped with an Ultra Short Baseline (USBL) acoustic positioning and communication system (shown in Fig. 4 bottom-right) to communicate with the underwater robot, and a high speed RF radio to communicate with the scientist on-board the ship or on land. The surface vehicle estimates its distance to the AUV, and fuses a GPS location with the USBL estimate to provide 3D global localization information to the AUV. Furthermore, the surface vehicle is capable of autonomously following the underwater robot to stay directly above it, maximizing bandwidth and localization accuracy. Peak throughput of the acoustic communication system is about 10kbit/sec over 1km range, using the 18-34KHz band.

IV-B Underwater Vehicle

The underwater vehicle (Fig. 4) is based on the BlueROV2 platform111The base platform details are at http://docs.bluerobotics.com/brov2/, with the addition of stereo cameras and side scan sonars for high resolution optical and acoustic imaging, an echosounder for altitude measurement, an acoustic modem/USBL transponder, and two Jetson TX2 computers for on-board processing of sensor data stream. For in-tank localization and testing, AprilTags [35] are employed. The BlueROV2 has open source hardware and software, and is low cost, which makes it suitable for scaling the robot system to multiple robots, and for easily adding new sensing capabilities.

IV-C Software Architecture

The unsupervised approach to exploration relies upon online robot-side scene modelling to construct a low-bandwidth representation of its surroundings. This representation takes the form of a time-varying scene map constructed from mapping 3D geographic coordinates to their most likely scene label. This can be visualized by the scientist as a colored 3D point cloud, with an injective mapping between colors and scene labels.

The process of constructing the scene map from the depth image produced by the stereo cameras is given in Algorithm 1, and implemented using ROS nodes. The scene map is defined at the resolution of cells, which are sparsely-packed N-dimensional boxes of uniform size, saved in a spatial hash map. Since new cells are added continuously, memory efficient N-D trees are not suitable as they require rebuild after each insertion, compared to a spatial hash map with $O(1)$ insertion and access time. The cell size is set at initialization to the minimum resolution required to resolve the phenomena of interest.

Since the cell dimensions are upper bounded by the scale of the smallest phenomena of interest, relatively large expanses of uniform terrain (such as sand bars or rocky outcrops) over which the robot may travel will need to be covered by many cells assigned the same topic. This presents an opportunity to reduce the size of the map through compression. Our approach to compression requires first discarding the z-component of the location of each cell, so that the map can be flattened into two dimensions. Then, a 2D map image is created such that each pixel represents the area of a single flattened cell and the image represents the entire region that has been explored. The value of each pixel in the image is set to the positive integer representing the corresponding cell’s topic label, or 0 if the cell is empty. Finally, this image is losslessly compressed using the portable network graphic [36] algorithm to minimize bandwidth, and then transmitted.

Having been presented with the scene map, the operator can learn what a label represents by retrieving representative imagery for that label. This capability will eventually be used to allow the operator to evaluate the relative importance of a label so that the underwater robot can plan an trajectory that maximizes the operator’s data collection objectives.

V Experiments and Demonstrations

We evaluate the proposed scene understanding system, as outlined in Algorithm 1, using a coral reef dataset and then demonstrate the scene modeling system running on our underwater robot in a controlled tank environment with an artificially created scene. In both cases, the robot starts with no prior knowledge of the environment.

V-A Simulated Coral Reef Exploration

We evaluated the ability of the proposed scene modeling approach to characterize a coral reef environment, given the bandwidth constraints. We used the HAW-2016-48 dataset [37] from the 100 Island Challenge222https://scripps.ucsd.edu/labs/sandin/research/100-island-challenge/ project to simulate observations

Figure 5(a) shows the mutual information with the directly observed imagery (Fig. 5(c)) and size of the generated scene map as we vary the scene model hyperparameters $\alpha=\{1.0,0.1,0.01,0.001\},\beta=\{10.0,1.0,0.1\},\gamma=\{10^{-3},10^{-4},10^{-5}\}$. Figure 5(b) shows the same scene maps, but with mutual information computed w.r.t. coral reef species annotation (Fig. 5(f)).

V-B Tank Demonstration

We demonstrated the scene modelling approach in a physical controlled environment that contained sufficient visual cues, such as artificial seaweed, stones, sand, grass, and a crab, and attempted to build a global topic map from streamed data.

The robot was equipped with outward-facing AprilTags, and an external camera was mounted facing into the tank to provide global localization information that mimicked the position information provided by the surface vehicle via GPS and USBL. The AUV was then autonomously commanded to follow a lawn-mower pattern in the tank. The downward facing Zed stereo camera provided an RGB-labelled point cloud which were then streamed into the topic modelling system. Old data points are updated with new topic information even as new data points are processed, as shown in Fig. 6.

VI Conclusion

We have proposed a novel approach to co-robotic exploration in low bandwidth environments that enables 3D or 2D vision-based robotic exploration while keeping a scientist in the loop. Our approach uses a nonparametric Bayesian scene model to characterize the explored regions of the space in terms of high level, scientifically relevant scene constructs (such as substrate type and coral species), which are then transmitted to the scientist in the form of a scene map. Such scene maps can be used to concisely define utility functions that could be used by robots to perform scientifically relevant informative path planning. Preliminary experiments have demonstrated the ability of the proposed approach to build scene maps that can be transmitted with very low bandwidth requirements, such as while using acoustic communications underwater. We also presented the design of a low cost underwater co-exploration robotic system, and demonstrated the proposed approach on this platform.