Can a mobile robot learn from a pedestrian model to prevent the sidewalk salsa?

TL;DR

This paper explores how a reinforcement learning agent can learn to interact safely and effectively with pedestrians by leveraging a pedestrian behavior model, reducing perceived risk during sidewalk interactions.

Contribution

It demonstrates a novel approach where RL agents use a pedestrian behavior model to improve interaction safety and communication in sidewalk scenarios.

Findings

Basic RL agent learned to interact with pedestrian model.

Risk-averse RL agent communicated intentions effectively.

Perceived risk was substantially lowered with the risk-aware approach.

Abstract

Pedestrians approaching each other on a sidewalk sometimes end up in an awkward interaction known as the "sidewalk salsa": they both (repeatedly) deviate to the same side to avoid a collision. This provides an interesting use case to study interactions between pedestrians and mobile robots because, in the vast majority of cases, this phenomenon is avoided through a negotiation based on implicit communication. Understanding how it goes wrong and how pedestrians end up in the sidewalk salsa will therefore provide insight into the implicit communication. This understanding can be used to design safe and acceptable robotic behaviour. In a previous attempt to gain this understanding, a model of pedestrian behaviour based on the Communication-Enabled Interaction (CEI) framework was developed that can replicate the sidewalk salsa. However, it is unclear how to leverage this model in robotic planning and decision-making since it violates the assumptions of game theory, a much-used framework in planning and decision-making. Here, we present a proof-of-concept for an approach where a Reinforcement Learning (RL) agent leverages the model to learn how to interact with pedestrians. The results show that a basic RL agent successfully learned to interact with the CEI model. Furthermore, a risk-averse RL agent that had access to the perceived risk of the CEI model learned how to effectively communicate its intention through its motion and thereby substantially lowered the perceived risk, and displayed effort by the modelled pedestrian. These results show this is a promising approach and encourage further exploration.

Full text

I Introduction

Suppose you’re walking down a sidewalk and another pedestrian comes around a corner and approaches you head-on (Figure 1-A). You move to one side to avoid them, but they simultaneously move to the same side. When you try to move to the other side of the sidewalk, they do the same. The third attempt to move works, and after an awkward interaction, you can both continue.

Although you will probably not experience this every day, it is a well-known phenomenon known under many names [1]. Here, we refer to it as the ”Sidewalk Salsa”. This phenomenon provides an interesting use case to study interactions between pedestrians and mobile robots since it encompasses a negotiation based on implicit communication to come to a safe solution (i.e., not colliding). In the vast majority of human-human interactions, this negotiation leads to a successful and mutually accepted outcome. Yet, in the case of a sidewalk salsa, this negotiation fails; A miscommunication happens. If we understand how this failure occurs, we understand how we can prevent it. This knowledge can potentially help design communicative automated behaviour for robots operating in environments where they interact with individual pedestrians; for example, mobile parcel delivery robots (e.g., [2, 3]) or indoor mobile robots (e.g., [4, 5]) used in healthcare or industry [6, 7].

One way to gain a deeper understanding of human behaviour in these interactions is by making a computational model describing it [8]. One modelling framework specifically aims to describe human-human interactions where (implicit) communication plays an important role: the Communication-Enabled Interaction (CEI) framework [9]. In this framework, pedestrians (or other traffic participants) are assumed to have a deterministic plan for their own future positions and a probabilistic belief about others’ future positions (Figure 1-C). Combined, these result in a perceived risk (of colliding); when this risk exceeds a personal risk threshold, pedestrians are assumed to change their plan to lower the risk. Communication is what links one pedestrian’s plan to another pedestrian’s belief, i.e., pedestrians form a belief based on observed communication. Models based on this framework have successfully captured driving behaviour in merging interactions [10, 11], and reproduced the sidewalk salsa [1].

However, although the process of computational modelling can result in fundamental knowledge about human behaviour [8], and this knowledge has the potential to inform robotic decision-making and path planning, it is unclear how to directly leverage such a model in robotic planning and decision-making algorithms. In automotive research, human behaviour models (i.e., driver models) have been used to improve autonomous behaviour. For example, by using models in benchmark tests for verification and validation [12, 13]. This approach allows for an indirect effect on robotic behaviour: the control algorithms can be (manually) improved after validation. Robotic behaviour changes through a design iteration of the algorithm; the model has no direct effect on the planning and decision-making.

Some approaches use models that directly inform autonomous planning; mostly in the context of game theory (e.g. [14, 15]). Although useful from a control perspective, game theory makes strong assumptions about human behaviour that are questionable in the context of traffic interactions [9]. Most importantly, game theory assumes that there is no communication between ”players” since it is not in their best interest to inform others about their plans when aiming to win a game. How to use models that do not assume human rationality and allow for communication directly in robotic planning and discussion making is thus an under-explored topic.

One possibility to directly leverage a behaviour model for robotic planning and decision-making is to train a Reinforcement Learning (RL) agent to control the robot while interacting with the model during training. Provided that the model is sensitive to the same communicative cues human pedestrians use while interacting, the RL agent might learn how to express these cues to safely and acceptably interact with pedestrians. An added benefit of this approach is that many models try to capture the internal representation humans use in their decision-making, i.e., models aim to capture the underlying mechanisms that drive human behaviour. When an RL agent has access to this internal state during training, it can potentially learn to influence how it is perceived in a positive way. For example, in the case of a CEI model, the RL agent could be trained to minimise the risk perceived by pedestrians it interacts with.

Here, we present an initial exploration of this approach, showing a proof-of-concept for training an RL agent in interaction with the CEI model of pedestrians [1]. We compare the RL agent with a social forces model [16] and show that the agent successfully learns to express useful communicative cues. Furthermore, we train a second RL agent to minimise the perceived risk. This risk-averse agent learns a slightly different behaviour compared to the basic RL agent, yet its actions are more communicative towards the CEI model. With this proof-of-concept, we show a promising first step towards a new approach for leveraging human behaviour models in planning and decision-making for mobile robots.

II Methods

We used the simulation environment111https://github.com/tud-hri/sidewalk-simulation developed for the CEI model of the sidewalk salsa as a baseline for the reinforcement learning environment. The CEI model as described in [1] controlled one of the pedestrians in the simulation. These pedestrians are modelled with a dynamic pedestrian model [17], which allows steering inputs (as in a bicycle model), yet also allows sidestepping in the form of lateral velocity and acceleration. The sidewalk in the simulation was $15~m$ long and $2.5~m$ wide. The modelled pedestrians interact with a simulated robot. We implemented the original social forces model as described in [16] as a baseline robot. We used two-dimensional point mass dynamics with the same parameters as in the original model, but we excluded the noise, visual view angle, and attraction forces.

Furthermore, we trained two Reinforcement Learning agents in interaction with the CEI agent to control the robot. Both RL agents used the same architecture, training method, and reward function, except that for the risk-averse agent, we included a minor penalty in the reward function based on the perceived risk of the CEI agent. The policy network was a multilayer perceptron with three hidden layers of 256 neurons each, using LayerNorm and LeakyReLU activations. The network outputs the mean and standard deviation for normalised input for a dynamic bicycle model: acceleration and steering ([-1, 1]). During training, actions were sampled; during evaluation, we used the mean. The agents observed the full state of the robot and pedestrian (positions, velocities, headings, angular velocities) and the previous action taken by the robot.

Training was performed using the REINFORCE algorithm (AdamW optimiser, learning rate $1e-3$, discount factor $0.99$), in batches of 20 episodes, for a total of 12,000 episodes. We used curriculum learning, where in the first 1000 episodes, there was no pedestrian present, in the second 1000, the pedestrian was stationary, and after that, the pedestrian behaved according to the CEI model. The reward function provided small positive rewards for progress along the sidewalk and small penalties for deviation from the desired velocity and heading. Furthermore, small penalties were given for cumulative inputs and the input derivative. Collisions and going beyond track bounds were heavily penalised, and a large reward was given when the robot reached the opposite end of the sidewalk.

During training, the CEI agent was assigned a random risk threshold, sampled from a uniform distribution $[0.6,0.9]$. Both agents were placed at the centre of the sidewalk on opposite ends with a random offset sampled from a uniform distribution $[-0.4~m,0.4~m]$ in $x$ and $y$ direction.

To evaluate the robotic agents, we simulated $200$ interactions with a simulated pedestrian for each robot (social forces, basic RL agent, and risk-averse RL agent). In this validation, the CEI model was assigned a random risk threshold sampled from a uniform distribution $[0.6,0.7]$, consistent with the values used in [1]. Initial positions were appended with a random offset from a uniform distribution $[-0.1~m,0.1~m]$, to increase the conflict with respect to training.

III Results

All $600$ simulations ended with both the simulated pedestrian and robot reaching the opposite side of the sidewalk, avoiding collisions and going out of bounds. Both RL agents thus successfully learned to interact with the CEI model. The position traces (Figure 2) show substantial differences between the simulated robots. While the social forces model follows the centre of the sidewalk until it enters the influence of the other pedestrians, the RL agents learned to pick a side early on. This clearly signals to the pedestrian on which side it will pass, thus the RL agent learned how to communicate its intention through interactions with the model. The risk-averse agent shows a subtle difference in behaviour that has a substantial effect on the simulated pedestrian.

The basic RL agent learned to pick a side to pass the pedestrian as quickly as possible. While this is already a good strategy to avoid collisions, the risk-averse agent learned that the CEI model uses heading angle as an important cue in communication [1]. Therefore, the risk-averse agent maintains its position at the sidewalk centre initially, until it reaches the range of influence of the pedestrian. The CEI model uses a mechanism that increases uncertainty for belief points in the future; thus, the influence of communication only increases with a certain look-ahead time [1]. Within this range (roughly between the $7$ and $12~m$ markers in Figure 2), the robot steers clear of the pedestrian with a heading pointing away from the pedestrian (instead of parallel to the sidewalk in the basic RL agents case). This successfully communicates to the pedestrian that it is planning to pass on that side.

This successful communication is reflected in the risk perceived by the simulated pedestrian (Figure 3). While almost all interactions with the social forces model result in the CEI agent’s perceived risk reaching its threshold, the basic RL agent substantially lowers the perceived risk. The risk-averse agent shows an even lower maximum perceived risk; in this case, almost none of the CEI agents reach their risk threshold. The risk-averse agent successfully learned how to change its communicative behaviour to lower the perceived risk of a pedestrian.

These lower risks in turn lead to lower effort, or input actions, taken by the pedestrian to mitigate a collision (Figure 4). In interaction with the social forces model, the CEI model shows the highest median cumulative steering and side-stepping inputs. The basic RL agent causes the CEI model to use higher side-stepping actions in some simulations, yet both median inputs are lower compared to the social forces model. In the case of the risk-averse agent, the CEI model hardly uses any inputs at all. While the position traces (Figure 2) of the two RL agents show only minor differences, this minor behavioural difference has a large effect on how the behaviour of the robot is understood by the modelled pedestrian and thus on how much action they take to lower their perceived risk.

IV Discussion

Although promising as a first step, the results of this work need to be interpreted with care. Our work illustrates the potential of our proposed approach, but without the rigorous work needed to conclude whether using a CEI model to train an RL robot is beneficial in real-world settings. The first major limitation in this work is that the CEI model used is not (yet) validated on pedestrian behaviour. So, although we can conclude that the RL agents have learned to successfully communicate their intention through their movement to the CEI model, this does not necessarily generalise to interaction with human pedestrians.

A second major limitation is that we have used a heavily simplified simulation of interactions on a sidewalk. Both agents are represented by simple dynamical models and are assumed to have full control without (output) noise. The robot has access to full observations without noise. And the RL agents are only trained on a sidewalk with fixed dimensions, interacting with a single other pedestrian at the same velocity in every trial. Yet, even under these simplified conditions, the robot shows behaviour that might not be desirable in a real robot. Both RL agents show some swerving behaviour before they enter the influence of the pedestrian, and they tend to speed up when getting close to the opposite side of the sidewalk (to reach the goal and get their reward quicker). These issues should be addressed before the approach can be evaluated for use in real-world settings.

Yet, despite these limitations, the results show potential and encourage further exploration of this approach.

V Conclusion

In this work, we provided a proof-of-concept for leveraging a human behaviour model in the planning and decision-making of a mobile robot. We trained a reinforcement learning algorithm to control a mobile robot in interactions with a Communication-Enabled Interaction model of a pedestrian. The results show that a basic RL agent successfully learned to interact with the CEI model. Furthermore, a risk-averse RL agent, that had access to the perceived risk of the CEI model during training, learned how to more effectively communicate its intention through its motion and thereby significantly lowered the risk perceived by the CEI model, and the effort the simulated pedestrian had to use to avoid the robot in a safe and acceptable manner.