Reinforcement Learning with Non-uniform State Representations for Adaptive Search

TL;DR

This paper introduces a reinforcement learning-based search algorithm that optimizes spatial exploration by non-uniform sampling and state aggregation, enabling rapid target detection in search and rescue missions.

Contribution

It presents a novel non-uniform state aggregation method for policy search in reinforcement learning applied to robotic search tasks.

Findings

Efficient trajectories reduce search time for lost targets.

The algorithm adapts to new probability distributions using learned parameters.

Non-uniform sampling improves coverage of high-probability regions.

Abstract

Efficient spatial exploration is a key aspect of search and rescue. In this paper, we present a search algorithm that generates efficient trajectories that optimize the rate at which probability mass is covered by a searcher. This should allow an autonomous vehicle find one or more lost targets as rapidly as possible. We do this by performing non-uniform sampling of the search region. The path generated minimizes the expected time to locate the missing target by visiting high probability regions using non-myopic path generation based on reinforcement learning. We model the target probability distribution using a classic mixture of Gaussians model with means and mixture coefficients tuned according to the location and time of sightings of the lost target. Key features of our search algorithm are the ability to employ a very general non-deterministic action model and the ability to generate action plans for any new probability distribution using the parameters learned on other similar looking distributions. One of the key contributions of this paper is the use of non-uniform state aggregation for policy search in the context of robotics.

Figures

Equations

$$\underset{\theta }{\max }J(\theta )=\underset{\theta }{\max }{\mathbb{E}}_{{\tau }_{\theta }}[\exp (-T/c)],$$

$${\theta }_{k+1}={\theta }_k+\eta {\nabla }_{\theta }{J}_{\theta },$$

$${\nabla }_{\theta }{J}_{\theta }={\int }_{\tau }{\nabla }_{\theta }{p}_{\theta }(\tau )R(\tau )d\tau$$

$$\begin{gathered} {\nabla }_{\theta }{J}_{\theta }=\frac{1}{m}\sum _{i=1}^m\sum _{t=0}^{H-1}{\nabla }_{\theta }\log {\pi }_{\theta }(a_t^{(i)}|s_t^{(i)}) \\ \left(\sum _{j=t}^{H-1}r(s_j^{(i)},a_j^{(i)})-b(s_t^{(i)})\right). \end{gathered}$$

$$\pi (\mathbf{s},\mathbf{a})=\frac{e^{{\theta }^T{\varphi }_{sa}}}{{\sum }_b{e}^{{\theta }^T{\varphi }_{sb}}},\forall s\in S;a,b\in A,$$

$${\mathbb{E}}_{{\tau }_{\theta }}\left[\sum _{t=0}^{H-1}{\gamma }^{t}r({\mathrm{s}}_t,{\mathrm{a}}_t)\right]={\mathbb{E}}_{{\tau }_{\theta },\mathrm{y}}[R(\tau )]$$

$${\mathbb{E}}_{\mathrm{y}}R(\tau )$$

$$={\mathbb{E}}_{\mathrm{y}}\left[\sum _{t=0}^{H-1}{\gamma }^t\delta ({\mathrm{x}}_t,\mathrm{y})\right]$$

$$=\sum _{t=0}^{H-1}{\gamma }^t{\mathbb{E}}_{\mathrm{y}}[\delta ({\mathrm{x}}_t,\mathrm{y})]=\sum _{t=0}^{H-1}{\gamma }^{t}r({\mathrm{s}}_t,{\mathrm{a}}_t),$$

$$\begin{gathered} \mathrm{Var}\sum _{t=0}^{H-1}{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t)\sum _{j=t}^{H-1}{\gamma }^{j}r({\mathrm{s}}_j,{\mathrm{a}}_j) \\ \le \mathrm{Var}{\mathbb{E}}_X\sum _{t=0}^{H-1}{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t)\sum _{j=t}^{H-1}{\gamma }^j\delta ({\mathrm{x}}_j,\mathrm{y}) \end{gathered}$$

$${\mathbb{E}}_{\tau }\left[{\left(\sum _{j=0}^{H-1}{\gamma }^j{r}_j\sum _{t=0}^j{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t)\right)}^2\right]$$

$$-{\mathbb{E}}_{\tau }{\left[\sum _{j=0}^{H-1}{\gamma }^j{r}_j\sum _{t=0}^j{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t)\right]}^2$$

$$\le$$

$$-{\mathbb{E}}_{\tau }{\left[{\mathbb{E}}_X\sum _{j=0}^{H-1}{\gamma }^j\delta ({\mathrm{x}}_j,\mathrm{y})\sum _{t=0}^j{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t)\right]}^2.$$

$${\mathbb{E}}_{\tau }\left[\sum _{j=0}^{H-1}\sum _{i=0}^{H-1}{\gamma }^{i+j}\left(r_j{r}_i-{\mathbb{E}}_{\mathrm{y}}\left[\delta ({\mathrm{x}}_i,\mathrm{y})\delta ({\mathrm{x}}_j,\mathrm{y})\right]\right)S_j{S}_i\right]\le 0$$

$$S_j=\sum _{t=0}^j{\nabla }_{\theta }\log {\pi }_{\theta }({\mathrm{a}}_t|{\mathrm{s}}_t),$$

$${\mathbb{E}}_{\tau }\left[\sum _{j=0}^{H-1}\sum _{i=0}^{H-1}{\gamma }^{i+j}\left(r_j{r}_i-\delta (i,j)r_i\right)S_j{S}_i\right]\le 0.$$

$${\mathbb{E}}_{\tau }\left[\sum _{j=0}^{H-1}\sum _{i=0}^{H-1}{\gamma }^{i+j}\left(r_j{r}_i-\delta (i,j)r_i\right)S_j{S}_i\right]$$

$$=\sum _{i=0}^{H-1}\sum _{j=0}^{H-1}{\gamma }^{i+j}{r}_i{r}_j{S}_i{S}_j-\sum _{i=0}^{H-1}{r}_i{\gamma }^{2i}{S}_i^2$$

$$\le \left(2-\sum _{k=0}^{H-1}{r}_k\right)\sum _{i=0}^{H-1}\sum _{j=0}^{H-1}{\gamma }^{i+j}{r}_i{r}_j{S}_i{S}_j-\sum _{i=0}^{H-1}{r}_i{\gamma }^{2i}{S}_i^2$$

$$=-{\mathbb{E}}_{\tau }\left[\sum _{i=0}^{H-1}{r}_i{\left({\gamma }^i{S}_i-\sum _{j=0}^{H-1}{\gamma }^j{r}_j{S}_j\right)}^2\right].$$

Full text

We compare the paths generated by our algorithm with other accepted spatial coverage techniques such as distribution independent boustrophedonic coverage and model dependent spiral search. We present a proof showing that rewarding for clearing probability mass instead of locating the target does not bias the objective function. The experiments show that the learned policy outperforms several well-known baselines even in scenarios different from the one it has been trained on.

I INTRODUCTION

This paper addresses the discovery and synthesis of efficient search trajectories based on learned domain-specific policies and probabilistic inference. We also explore the use of non-uniform state space representations to enhance the performance of policy search methods.

Searching for a lost target or a person in the wilderness can be tedious, labor-intensive, imprecise, and very expensive. In some cases, the manual search and rescue operations are also unsafe for the humans involved. These reasons motivate the use of autonomous vehicles for such missions. In many search and rescue problems, timeliness is crucial. As time increases, the probability of success decreases considerably [1], and every hour the effective search radius increases by approximately $3$ km [2]. We present an active sampling algorithm that generates an efficient path in real-time to search for a lost target, thus making an autonomous search and rescue mission realistic and more beneficial.

Given an approximate region of interest, the first task is to decide where to search. Within the region of interest, some areas have higher probability of containing the target than the others. Data shows that more than $90\%$ of searches based on a probability distribution over the region have been resolved successfully within a short duration [3]. We model the spatial field of search using classic Gaussian mixture models based on sightings of the lost target. The modeled probability distribution map over the search region is used as an input for planning the search paths. The traditional way to assign probabilities to search regions is with a distance ring model: a simple ‘bulls-eye’ formed by the $25\%$, $50\%$, $75\%$, and $95\%$ probability circles [3]. There are more advanced methods that propose a Bayesian model that utilizes publicly available terrain data to help model lost person behaviors enabling domain experts to encode uncertainty in their prior estimations [4]. Since modeling of the search region probability distribution is not the focus of this paper, we use a simpler approach depending only on location and time of the target sightings. Our search algorithm would perform efficiently independent of the underlying probability distribution model.

Once the probability distribution of the search region is mapped, the task is to search this region efficiently such that the regions with high probability of finding the target (hotspots) are searched in the early phase of the exploration. Many complete coverage approaches use line-sweep based techniques [5, 6, 7] which are not suitable for search and rescue tasks because of their non-timeliness. Cooper et al. discuss the relationship between area coverage and search effort density. They claim that any solution to the optimal search density problem is also a solution to the optimal coverage problem [8]. We present a proof showing that coverage of the search region to reduce probability mass is equivalent to searching for the target. We have previously demonstrated an algorithm to selectively cover a region based on an underlying reward distribution [9, 10]. This technique, however, does not scale up smoothly for larger regions because of the computational complexity.

In this paper, we present an adaptive sampling technique that generates efficient paths to search the missing target as fast as possible by performing non-uniform sampling of the search region. The path generated minimizes the expected time to locate the missing target by visiting high search probability regions using non-myopic path planning based on reinforcement learning. A non-myopic plan is one that accounts for possible observations that can be made in the future [11]. Fig. 1 presents an overview of the whole path generation system. Our algorithm gets trained with a generic model of the probability distribution, which can be a map generated by a Gaussian mixture. These learnt parameters are used on the generated lost target probability distribution map to come up with an action plan (policy $\pi$). For a given state, an action is chosen according to the probability distribution $\pi$ and thus a path is generated for the searcher robot. Training the system with discounted rewards helps the planner to achieve paths that cover hotspots at the earlier stages of the search task. A major contribution of this paper is the use of non-uniform state aggregation for policy search in the context of robotics.

The key feature of our search algorithm is the ability to generate action plans for any new probability distribution using the parameters learnt on other similar looking distributions, i.e. an action planner trained on generic search and rescue distribution models is capable of planning high rewarding paths with a new probability distribution map without being trained again.

II PROBLEM FORMULATION

The search region is a continuous two-dimensional area of interest $\mathcal{E}\subset\mathbb{R}^{2}$ with user-defined boundaries. The spatial search region is discretized into uniform grid cells, such that the robot’s position $\boldsymbol{\mathrm{x}}$ and the target’s position $\boldsymbol{\mathrm{y}}$ can be represented by two pairs of integers $\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{y}}\in\mathbb{Z}^{2}$. Each grid-cell $(i,j)$ is assigned a prior probability value $\operatorname{q}(i,j)$ of the target being present in that cell.

The aim is to find the target in as short a time as possible. Formally, we specify this objective as maximizing $\mathbb{E}[\exp(-T/c)]$, where $T$ is the time elapsed until the target is found and $c$ is a condition-dependent constant. This objective reflects the assumption that the probability of the target’s condition becoming bad is constant, and the aim is to locate the target in good condition.

We will specify the robot’s behavior using a parametrized policy. This is a conditional probability distribution $\pi_{\boldsymbol{\mathrm{\theta}}}(\boldsymbol{\mathrm{s}},\boldsymbol{\mathrm{a}})=p(\boldsymbol{\mathrm{a}}|\boldsymbol{\mathrm{s}};\boldsymbol{\mathrm{\theta}})$ that maps a description of the current state $\boldsymbol{\mathrm{s}}$ of the search to a distribution over possible actions $\boldsymbol{\mathrm{a}}$. Our aim will be to automatically find good parameters $\boldsymbol{\mathrm{\theta}}$, after which the policy can be deployed without additional training on new problems. The maximization objective then becomes:

[TABLE]

where the expectation is taken with respect to trajectories $\tau_{\boldsymbol{\mathrm{\theta}}}=(s_{0},a_{0},s_{1},a_{1},...)$ generated by the policy $\pi_{\boldsymbol{\mathrm{\theta}}}$.

III MODELING OF THE SEARCH AREA

We model the spatial field of search using a classic Gaussians mixture model. The map $\operatorname{q}$ is initialized according to these generated prior probabilities. An example map is illustrated in Fig. 2. Our search algorithm can take any map $\operatorname{q}$ in accordance to the prior belief over where the target is. In this work, we will test our method on randomly generated Gaussian mixtures, but the algorithm could be trained on any type of distribution.

IV POLICY GRADIENT BASED SEARCH

Finding a sequence of actions that maximizes a long-term objective could be done using dynamic programming. However, in our formulation the system state is described using a map containing the per-cell probability of the target being present and this map changes as areas are visited by the agent. The result is an extremely large state space where dynamic programming is impracticable - especially if the time to solve each particular problem is limited.

Instead, we turn to methods that directly optimize the policy parameters $\boldsymbol{\mathrm{\theta}}$ based on (simulated) experience. To apply these methods, we will first formalize the search problem as a Markov Decision Process (MDP).

IV-A Formalizing search as MDP

A Markov Decision Process is a formal definition of a decision problem that is represented as a tuple $(S,A,T(\boldsymbol{\mathrm{s}}_{t+1}|\boldsymbol{\mathrm{s}}_{t},\boldsymbol{\mathrm{a_{t}}}),r(\boldsymbol{\mathrm{s}}_{t},\boldsymbol{\mathrm{a}}_{t}),\gamma)$, where $S$ and $A$ are the state and action space, $T$ models transition dynamics based on the current state and action, and $r$ defines the reward for the current state-action pair. $\gamma$ is a discount factor that reduces the desirability of obtaining a reward $t$ time-steps from now rather than in the present by $\gamma^{t}$. The objective is then to optimize the expected discounted cumulative reward $J=\mathbb{E}_{\tau}[\sum_{t=0}^{H}\gamma^{t}r_{t}(s_{t},a_{t})]$, where $H$ is the optimization horizon.

In the proposed approach, we take the state $\boldsymbol{\mathrm{s}}$ to include the position of the robot $\boldsymbol{\mathrm{x}}$ as well as the map $\operatorname{q}$ containing the per-location presence probability for the target, $\boldsymbol{\mathrm{s}}=(\boldsymbol{\mathrm{x}},\operatorname{q})$. The actions we consider are for the robot to move to and scan the cell North, East, South or West of its current location. Transitions deterministically move the agent in the desired direction. When scanning does not reveal the target, the probability mass $\operatorname{q}(i,j)$ of the current cell $(i,j)$ is then reduced to [math]111More complex models specify a probability of detection (POD) given that robot and target are in the same area [8]. For now, our work assumes a probability of detection (POD) of 1. A more realistic POD could easily be included in our approach by updating the probability mass in the cell accordingly. Furthermore, note that as probability mass is cleared, the numbers in $\operatorname{q}$ no longer sum up to 1, so $\operatorname{q}$ is an unnormalized probability distribution..

The most intuitive definition of the reward function corresponding to (Eq. 1) would give the reward of $1$ for recovering the target, coupled with a discount factor $\gamma=\exp(-1/c)$. However, this reward function has a high variance, as with the exact same search strategy the target could be found quickly, slowly, or not at all due to chance. Instead, we reward the robot for scanning cells with a high probability of containing the target. This does not introduce bias in the policy optimization, while reducing statistical variance222Proofs are given in the Appendix.. A lower statistical variance typically allows optimal policies to be learned using fewer sampled trajectories.

IV-B Policy Gradient Approach

Policy gradient methods use gradient ascent for maximizing the expected return $J_{\theta}$. The gradient of the expected return ($\nabla_{\theta}J_{\theta}$) guides the direction of the parameter ($\theta$) update. The policy gradient update is given by,

[TABLE]

where $\eta$ is the learning rate. The likelihood ratio policy gradient [12] is given by,

[TABLE]

However, this expression depends on the correlation between actions and previous rewards. These terms are [math] in expectation, but cause additional variance. Ignoring these terms yields lower-variance updates, which are used in the Policy Gradient Theorem (PGT) algorithm and the GPOMDP algorithm [13, 14, 15]. Accordingly, the policy gradient is given by,

[TABLE]

In this equation, the gradient is based on $m$ sampled trajectories from the system, with $\boldsymbol{\mathrm{s}}_{j}^{(i)}$ the state at the $j^{\text{th}}$ time-step of the $i^{\text{th}}$ sampled roll-outs. Furthermore, $b$ is a variance-reducing baseline. In our experiments, we set it to the observed average reward.

IV-C Policy design

We consider a policy that is Gibbs distribution in a linear combination of features given by

[TABLE]

where $\phi_{sa}$ is an $l$-dimensional feature vector characterizing state-action pair ($s,a$) and $\theta$ is an $l$-dimensional parameter vector. This is a commonly used policy in reinforcement learning approaches [14]. The final feature vector $\phi_{sa}$ is formed by concatenating a vector $\phi^{\prime}_{s}\delta_{aa^{\prime}}$ for every action $a^{\prime}\in\{North,East,South,West\}$, where $\phi^{\prime}_{s}\subset\mathbb{R}^{k}$ is a feature representation of the state space, and $\delta_{aa^{\prime}}$ is the Kronecker delta. Thus, the final feature vector has $4\times k$ entries, $75\%$ of which corresponding to non-chosen actions will be [math] at any one time step.

We consider two types of feature representations ($\phi^{\prime}_{s}$) for our approach. The first feature construction is to consider a vector that represents all rewards in $\operatorname{q}$ robot-centric, as illustrated in Fig. 3(a). This feature vector grows in length as the size of the search region increases, resulting in higher computation times for bigger regions. In the second design we consider a multi-resolution feature aggregation resulting in a fixed number ($24$) of features irrespective of the size of the search region. In this case, features corresponding to larger cells are assigned a value equal to the average of values of $\operatorname{q}(i,j)$ that fall in that cell. In multi-resolution aggregation, the feature cells grow in size along with the distance from the robot location as depicted in Fig. 3(b). Thus, areas close to the robot are represented with high resolution and areas further from the robot are represented in lower resolution. The intuition behind this feature design is that the location of nearby target probabilities is important to know exactly, while the location of faraway target probabilities can be represented more coarsely. The multi-resolution feature design is also suitable for bigger worlds as it scales logarithmically to the size of the world.

Both these feature designs were tested with the policy gradient based searching algorithm and we found that the multi-resolution aggregated features produce higher accumulated rewards and better discounted rewards (Fig. 3(c), 3(d)). Also, the computation gets quadratically expensive as the size of the grid world increases (Fig. 3(e)). Based on these results, a non-uniform, multi-resolution, robot-centric feature design is more beneficial for efficient searching. These results further strengthen our belief that the immediate actions are influenced by the nearer rewards and the farther low-resolution features enhance non-myopic planning of the whole path. Theoretically, replacing the individual cell values by their averages causes some information loss. However, to plan for far-away target probabilities, a coarse location is enough. In fact, the results in Fig. 3(c) and Fig. 3(d) show that, perhaps due to the regulating effect of averaging, multi-resolution grids perform better than the uniform representation even after extensive training.

Further in this paper we will only consider the multi-resolution robot-centric feature design in our policy gradient search algorithm. The aggregated feature design is only used to achieve better policy search, but the robot action is still defined at the grid-cell level.

V EXPERIMENTAL RESULTS AND DISCUSSION

In this section, we will introduce the experimental set-up and baseline methods, before presenting and discussing the experimental results.

V-A Setup

We generated a generic training scenario with probability distributions for a lost target using Gaussian mixtures. We model the search space of a simulated aerial search vehicle as a $30\times 30$ grid world with each grid-cell spanning $100m\times 100m$. We used 20 roll-outs in every iteration of the training phase. A discount factor of $0.9$ was used in these experiments. During the test phase, an action with maximum probability is chosen at a given state. Fig. 2 illustrates the probability distribution grid used for training our policy based searcher. However, as mentioned in Section III, the searcher algorithm could be trained on any other type of distribution too. Two significantly different test-scenarios are presented to evaluate and compare the search algorithms. The first test-scenario (Fig. 4(a)) comprises of two Gaussians imitating the probability distribution of a lost target. The second test-scenario (Fig. 4(b)) cannot be represented as a mixture of a few Gaussians.

V-B Baselines

Traditionally, exhaustive sampling of a partially observable, obstacle-free region employs a boustrophedon path [16]. The boustrophedon or lawnmower path is the approach a farmer takes when using an ox to plow a field, making back and forth straight passes over the region in alternating directions until the area is fully covered. Another efficient search pattern reported in the robotic search literature is spiral, which minimizes the time to find a single stationary lost target in several impressive idealized scenarios [17, 18].

We compare our search algorithm with these two search techniques. Fig. 5(a)-5(f) illustrate the paths generated by the three coverage algorithms overlaid on top of the test scenario probabilities. The paths shown are generated by running the search algorithms for $300$ time-steps.

V-C Results and discussion

Spiral search can generate efficient, or even optimal, paths under suitable conditions such as the first test scenario (Fig. 5(b)). Its satisfactory performance is only assured for a restricted class of unimodal distributions (as opposed to that in Fig. 5(e)). We compare these algorithms based on the rewards collected by removing the probability mass in the search region, corresponding to the (discounted) probability of finding the target. Our goal is to reduce the probability mass as fast as possible by visiting regions with high probability mass (hotspots). We use discounted rewards as a metric to measure the timeliness of an algorithm.

The plots in Fig. 5(g) illustrate how our proposed method is the fastest to cover the target probability mass on both test scenarios, even though all methods eventually visit all states of potential interest. Spiral search performs on par with our algorithm in terms of total rewards accumulated. Nonetheless, policy based searcher out performs both, spiral and boustrophedon search techniques in total discounted rewards on the second test scenario by a significant margin. Thus the policy based searcher exhibit a higher order of timeliness, which is a key requirement for any search algorithm to be applicable in search and rescue missions.

It is important to note that despite the dissimilarity between the training scenario and the test scenario, the policy-based search algorithm achieves better performance than the other methods.

VI CONCLUSIONS

We presented an optimization algorithm that results in a policy that can generate non-myopic search plans for novel environments in real time. Our policy gradient based search algorithm is well suited for applications in search and rescue because of its ability to come up with a search plan on-the-go, based on the new probability distribution of the lost target, with no time wasted on retraining. Other time-critical applications like aerial surveys after a calamity, water surface surveys to monitor and contain algal blooms, and searching for remains under ocean after an accident, can use the presented algorithm to explore the region of interest.

In the near future, we would like to enhance the performance of the search algorithm by exploring different feature aggregation techniques and designing an adaptive aggregation that can combine features in real-time according to the observations made during the survey.

ACKNOWLEDGMENT

We would like to thank the the Natural Sciences and Engineering Research Council (NSERC) for their support through the Canadian Field Robotics Network (NCFRN).

First, we want to show that giving a reward for clearing probability mass instead of locating the target does not bias the objective function.

Proposition 1

[TABLE]

where the discount factor $\gamma=\exp(-1/c)$, $\boldsymbol{\mathrm{y}}$ is the target’s location, and the proxy reward $r(\boldsymbol{\mathrm{s}}_{t},\boldsymbol{\mathrm{a}}_{t})$ is equal to the probability of the target being at the robot’s location $\boldsymbol{\mathrm{x}}$ according to $\operatorname{q}$333The reward map $\operatorname{q}$ is not normalized, yet after clearing a fraction $g$ or probability mass, the probability that the target has not been found yet is $1-g$. If the target were not found yet, the normalized probability that the target is in cell $(i,j)$ is $q(i,j)/(1-g)$. So the probability of finding the target in $(i,j)$ indeed equals $(1-g)q(i,j)/(1-g)=q(i,j)$.. $R(\tau)=\exp(-T/c)$ with $T$ is the time until the target is found if the target is found within $H$ time steps, or [math] otherwise444The proof here assumes a static target for notational simplicity, but can be generalized to dynamic targets by making both $r$ and $\boldsymbol{\mathrm{y}}$ time-step dependent, and calculating expected values over all $\boldsymbol{\mathrm{y}}_{1},\ldots,\boldsymbol{\mathrm{y}}_{H}$ jointly..

Proof:

For any trajectory,

[TABLE]

where $\delta$ is the Kronecker delta. Since this equality holds for any trajectory, it must hold for a linear combination of trajectories. ∎

Now, we want to show that the variance of gradient estimates using the proxy reward for clearing probability mass is lower than that of the gradient estimates using the original objectives.

Proposition 2

[TABLE]

Proof:

We use the definition of the variance and rearrange terms as in the GPOMDP method [13] to reformulate the proposition. We introduce the shorthand $r_{j}=r(\boldsymbol{\mathrm{s}}_{j},\boldsymbol{\mathrm{a}}_{j})$ make the equations more readable, and obtain

[TABLE]

Note that (6) and (8) are expectations of unbiased estimates of the respective objective, and so equal the gradient of the expected value of the respective objective. By Proposition 4, these expected values are the same, so (6) and (8) cancel each other out. Writing out the squares in (5) and (7) and combining like terms yields

[TABLE]

where we introduced the shorthand

[TABLE]

and used the fact that $S_{j}$, $S_{i}$ are independent of the target location $\boldsymbol{\mathrm{y}}$. Note that $r_{i}$ is just the probability that $\delta(\boldsymbol{\mathrm{x}}_{i},\boldsymbol{\mathrm{y}})=1$, and since the agent cannot find the target twice, $\delta(\boldsymbol{\mathrm{x}}_{i},\boldsymbol{\mathrm{y}})\delta(\boldsymbol{\mathrm{x}}_{j},\boldsymbol{\mathrm{y}})$ is non-zero only if $i=j$. Thus, the proposition is equivalent to

[TABLE]

The left-hand side of this inequality can be expressed and upper-bounded as follows:

[TABLE]

The inequality is due to the sum of rewards always being smaller than 1 (as the rewards denote probabilities of finding the target, and would sum up to 1 if and only if the robot visits all cells where probability mass was initially present in this trajectory). Note that the expected value in (11) is non-negative: it is a weighted sum of squared terms, where all weights are non-negative (again, due to their interpretation as probabilities). Thus, (11) is non-positive, so (10) must $\leq 0$, confirming (9). ∎