Passivity-Based Control of Human-Robotic Networks with Inter-Robot Communication Delays and Experimental Verification

TL;DR

This paper develops and experimentally verifies a passivity-based control framework for human-robotic networks with communication delays, ensuring synchronized motion and stable interaction between robots and humans.

Contribution

It introduces a novel passivity-based control approach using scattering transformation to handle communication delays in human-robot networks, validated through real-world experiments.

Findings

Communication delays affect human operator behavior.

Passivity-based control maintains stability despite delays.

Experimental setup demonstrates effective synchronization.

Abstract

In this paper, we present experimental studies on a cooperative control system for human-robotic networks with inter-robot communication delays. We first design a cooperative controller to be implemented on each robot so that their motion are synchronized to a reference motion desired by a human operator, and then point out that each robot motion ensures passivity. Inter-robot communication channels are then designed via so-called scattering transformation which is a technique to passify the delayed channel. The resulting robotic network is then connected with human operator based on passivity theory. In order to demonstrate the present control architecture, we build an experimental testbed consisting of multiple robots and a tablet. In particular, we analyze the effects of the communication delays on the human operator's behavior.

Equations

$${\dot{q}}_i=u_i\forall i\in \mathcal{V},$$

$$\underset{t\to \infty }{\lim }\Vert q_i-q_r\Vert =0\forall i\in \mathcal{V}.$$

$$u_i=u_{r,i}+{\delta }_i{u}_{h}i\in \mathcal{V}.$$

$${\dot{\xi }}_i$$

$$u_{r,i}$$

$$z:=\frac{1}{m}\sum _{i\in {\mathcal{V}}_h}{q}_i,$$

$$\underset{t\to \infty }{\lim }\Vert {\dot{q}}_i-v_r\Vert =0,\underset{t\to \infty }{\lim }\Vert q_i-q_j\Vert =0\forall i,j\in \mathcal{V},$$

$${\dot{\xi }}_i$$

$${\dot{q}}_i$$

$$\displaystyle p_{ij}:=M_{ij}\left(\left[\begin{array}[]{cc}\!r^{q}_{ij}\!\\ \!r^{\xi}_{ij}\!\end{array}\right]-\left[\begin{array}[]{cc}\!q_{i}\!\\ \!\xi_{i}\!\end{array}\right]\right)=M_{ij}\left[\begin{array}[]{cc}\!r^{q}_{ij}-q_{i}\!\\ \!r^{\xi}_{ij}-\xi_{i}\!\end{array}\right],$$

$$\displaystyle\left[\begin{array}[]{c}\dot{q}_{i}\\ \dot{\xi}_{i}\end{array}\right]=\mu_{i}+\left[\begin{array}[]{c}\delta_{i}\\ 0\end{array}\right]u_{h}$$

$$S_i:=\frac{1}{2}\Vert q_i{\Vert }^2+\frac{1}{2}\Vert {\xi }_i{\Vert }^2,$$

$${\dot{S}}_i=$$

$$=$$

$$s_{ij}^{+}=$$

$$s_{ji}^{+}=$$

$$s_{ji}^{+}(t)=s_{ij}^{+}(t-T),s_{ij}^{-}(t)=s_{ji}^{-}(t-T),$$

$$S_{ij}^c:=$$

$${\dot{S}}_{ij}^c=$$

$$-\Vert -p_{ij}-\sigma {\bar{r}}_{ij}{\Vert }^2)=-p_{ij}^T{\bar{r}}_{ij}-p_{ji}^T{\bar{r}}_{ji}.$$

$${\dot{\bar{q}}}_i$$

$${\dot{\xi }}_i$$

$$\displaystyle\left[\begin{array}[]{c}\dot{\bar{q}}_{i}\\ \dot{\xi}_{i}\end{array}\right]=\mu_{i}+\left[\begin{array}[]{c}\delta_{i}\\ 0\end{array}\right]u_{h}.$$

$${\bar{S}}_i:=\frac{1}{2}\Vert {\bar{q}}_i{\Vert }^2+\frac{1}{2}\Vert {\xi }_i{\Vert }^2,i\in \mathcal{V}.$$

$${\dot{\bar{S}}}_i=$$

$$=$$

$$S:=\frac{1}{m}\sum _{i\in \mathcal{V}}{\bar{S}}_i+\frac{1}{m}\sum _{(i,j)\in \mathcal{E}}{S}_{ij}^c.$$

$$\dot{S}=$$

$$=$$

$${\dot{S}}_h^{\Phi }\le (q_r-z{)}^T{u}_h-ϵ\Vert q_r-z{\Vert }^2.$$

$$U:=S+{\int }_0^t\left(-\bar{z}(\tau )u_h(\tau )-ϵ\Vert \bar{z}(\tau ){\Vert }^2\right)d\tau +\beta ,$$

$$\dot{U}$$

$$=-\frac{1}{m}\sum _{i\in \mathcal{V}}\sum _{j\in {\mathcal{N}}_i}{a}_{ij}\Vert q_i-r_{ij}^q{\Vert }^2-ϵ\Vert \bar{z}{\Vert }^2\le 0.$$

$$\underset{t\to \infty }{\lim }(r_{ij}^q-q_i)=0(i,j)\in \mathcal{E}$$

$$\underset{t\to \infty }{\lim }\bar{z}=0$$

Full text

I Introduction

Complex robotic coordination tasks in highly uncertain environments require the system designers to take advantage of human operator’s strengths, high-level decision-making and flexibility. Motivated by these needs, semi-autonomous operation of robots is gaining increasing research interests [1].

One of the most promising design tools for such semi-autonomous systems is passivity, as confirmed in the history of a traditional semi-autonomous robot control problem, bilateral teleoperation [2]-[12]. In this research field, the human operator has been treated as a passive component and then control architectures have been established while ensuring closed-loop stability based upon this assumption. This paradigm has also been taken in teleoperation of multiple networked robots [2]–[4]. Rodriguez-Seda et al. [2] present an architecture such that the robotic network is controlled by operating a leader robot so that the robots forms a specified formation while avoiding collisions. Liu [3] extends motion synchronization on the joint space to the task space under time-varying delays. Fully distributed control algorithms are presented by Franchi et al. [4], where stability is ensured in the presence of split and join events. Other interesting extensions of bilateral teleoperation are presented in [5] and [6]. Varnell and Zhang [5] employ a non-classical human-robot interfaces, namely a tablet, and discuss the stability while assuming a human pointing model. Saeidi et al. [6] introduce the notion of robot-to-human trust and mixed initiative control scheme in order to improve the performance and reduce workload of the operator.

While the above papers consider robot dynamics together with force feedback, the authors [13, 14] studied another problem formulation focusing on interactions between a human operator and kinematic robots as in [15, 16]. The schematic of the intended system is illustrated in Fig. 1. Then, we investigated motion synchronization of robots to human references under distributed information exchange between the human and robots, and among robots. To this end, we presented a novel passivity-based control architecture and proved synchronization under explicit bilateral connections between the human and robotic network. For implementation on real robotic systems, both the human-robotic network and inter-robot interactions must be implemented using appropriate communication technology. In this case, the communication channels may suffer from delays in the transmission of information, which is indeed one of the most important issues in bilateral teleoperation. However, [13, 14] did not address this issue explicitly.

Our objective is thus to extend the results of [13, 14] to the case with time delays and its experimental studies. Although both the human-robotic network and inter-robot communications may have delays, this paper investigates the latter since this issue emerges in the one-human-multiple-robot interactions while the former has been in-depth studied in bilateral teleoperation. Delays in inter-robot communications have been extensively studied in the field of cooperative control (See [7] and references therein). Among many approaches, this paper focuses on the scheme presented in [7], and synchronization is guaranteed despite the delays. In this paper, the control architecture in [7] is shown to be successfully integrated with the cooperative control for human-robotic networks presented in [13, 14]. However, the integration is not straightforward and requires novel technical extensions in the formulation and this is presented in this paper. Furthermore, as pointed out in [17], real deployments of robotic network is important from the viewpoint of difficulties to simulate each intended task and situation faithfully. Therefore, we investigate our proposed architecture in a testbed and show the influences of communication delays not only on a robotic network but also on a real human’s behaviour.

The organization of this paper is as follows: Section II presents the intended problem and briefly reviews the results of [13, 14]. In Section III, we design a novel control architecture based on the scattering transformation [7], and then, show passivity of the resulted system. We also show position synchronization to the reference values in Section IV. We demonstrate our results through experiments in Section V, and finally, summarize the obtained results in Section VI.

II Problem Setting

In this section, we start by reviewing the problem formulation and results presented in [13, 14]. Please refer to [13, 14] for more details.

II-A System Configuration and Objectives

Let us consider a system with a human operator and $n$ mobile robots $\mathcal{V}=\{1,\cdots,n\}$ located on a 2-D plane as shown in Fig. 1. We suppose that the motion of every $i$-th robot is described by the kinematic model

[TABLE]

where $q_{i}\in\mathbb{R}^{2}$ and $u_{i}\in\mathbb{R}^{2}$ are the position and velocity input of $i$-th robot, respectively.

Every robot can interact with neighboring robots and the inter-robot information exchanges are modeled by a graph $G=(\mathcal{V},\mathcal{E}),\ \mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$. Then, robot $i$ has access to information of the robots belonging to the set $\mathcal{N}_{i}=\{j\in\mathcal{V}|\ (i,j)\in\mathcal{E}\}$. Throughout this paper, we assume the communication graph is fixed, undirected and connected. Differently from [13], we assume that the inter-robot communication suffers from time delays, which will be discussed in Section II.

In addition to the total robots set $\mathcal{V}$, we define accessible robots set $\mathcal{V}_{h}$ which is a subset of $\mathcal{V}$ (Fig. 2). Accessibility means that the operator can send a signal to and receive feedback information from only the robots in $\mathcal{V}_{h}$. Here, we assume that the human operator determines a command signal $u_{h}$ based on certain information $y_{h}$ visually fed back from $\mathcal{V}_{h}$ through a monitor in front of the operator. The command $u_{h}$ is then sent back to all the accessible robots. We also introduce the notation $\delta_{i}$ such that $\delta_{i}=1$ if $i\in\mathcal{V}_{h}$ and $\delta_{i}=0$ otherwise.

In this paper, we address position synchronization. Let us assume that the human operator has a desired position denoted by $q_{r}$. Then, the goal of the position synchronization is defined by

[TABLE]

The objective here is to design the robot controller $u_{i}$ and the information $y_{h}$ displayed on the monitor so as to guarantee the above control goal.

II-B Control Architecture without Time Delays

The authors [13, 14] take the control input $u_{i}$ as

[TABLE]

Then, the signal $u_{r,i}$ to achieve inter-agent motion synchronization is designed as

[TABLE]

where $a_{ij}=a_{ji},b_{ij}=b_{ji}$ $\forall{i,j}\in\mathcal{V}$, $a_{ij}>0,b_{ij}>0$ if $(i,j)\in\mathcal{E}$ and $a_{ij}=b_{ij}=0$ otherwise.

Then, the collective dynamics (1), (3) and (5) for all $i$ is passive from $u_{h}$ to $z$, where $z$ is defined as

[TABLE]

and $m$ is the number of elements of $\mathcal{V}_{h}$. Based on this passivity, we let the feedback information $y_{h}$ for the human operator be $y_{h}=z$. Namely, the human operator can obtain the average position of the accessible robots $z$. In [13], position synchronization is theoretically proved in the absence of delays under a passivity assumption on the human’s decision process together with some additional assumptions. However, inter-robot communication delays may destabilize the above control system. This is why we present a system architecture with robustness against inter-robot communication delays.

Remark 1

The system architecture (4), (5) can achieve not only position synchronization but also synchronization of velocity at the same time, i.e.,

[TABLE]

where $v_{r}$ is a constant reference velocity. However, we omit this part because of space limitations.

III Cooperative Control Architecture under Inter-Robot Communication Delays

III-A Passivity of Robot Dynamics

In this section, we redesign the cooperative controller generating $u_{r,i}$ in order to guarantee stability and synchronization despite the inter-robot communication delays.

In the presence of the delays, $i$-th robot cannot obtain $q_{j}(t)$ and $\xi_{j}(t)$ at time $t$. We thus replace these signals by $r^{q}_{ij},r^{\xi}_{ij}$, respectively. Then, $i$-th robot’s dynamics is described by

[TABLE]

Let us now define $p_{ij}\in\mathbb{R}^{4}$ as

[TABLE]

where M_{ij}:=\left[\begin{array}[]{cc}\!a_{ij}I_{2}\!&\!-b_{ij}I_{2}\!\\ \!b_{ij}I_{2}\!&\!0\!\end{array}\right] for all $j\in\mathcal{N}_{i}$ and $i$. It is easy to see from the definition that $M_{ij}$ is a passive map.

Then, (8) and (9) is described as a feedback system consisting of

[TABLE]

and the operation $\mu_{i}:=\sum_{j\in\mathcal{N}_{i}}p_{ij}$. Then, if there is no interaction with a human operator, i.e., $u_{h}\equiv 0$, and we define a storage function as

[TABLE]

the system (21) is passive from $\mu_{i}$ to $[q_{i}^{T}\ \xi^{T}_{i}]^{T}$.

From the above discussions, the system (8) and (9) can be reduced to a feedback interconnection of passive systems and a collection of passive maps $M_{ij}\ j\in{\mathcal{N}}_{i}$. Accordingly, we can obtain the following inequality for each robot dynamics,

[TABLE]

The above inequality implies that each robot dynamics is passive from the stacked vector of $r_{ij}$ to the stacked vector of $p_{ij}$ from all neighbor robots with respect to the storage function (22).

III-B Scattering Transformation and Passivity of Communication Block

In this subsection, we present a novel control architecture based on [7]. Hereafter, similarly to [13, 14], we assume that reference $q_{r}$ is constant, which means that we are assuming that these signals are varying so slowly that the analyses under the constant reference are applicable to the actual system. Time varying reference is also addressed in [14], however its extension to the case with delays is left as a future work.

Following the architecture of [7], we let each agent exchange the variable $p_{ij}$ through scattering transformation instead of directly sending $q_{i}$ and $\xi_{i}$ as illustrated in Fig. 3. In the present case, the scattering variables are defined as

[TABLE]

where $r_{ij}=[(r^{q}_{ij})^{T}\ (r^{\xi}_{ij})^{T}]^{T}$ and $\sigma>0$ is constant. The operation is illustrated in Fig. 4. It is immediate to see from this figure that

[TABLE]

where $T>0$ is a delay in communication channel from $i$-th robot to $j$-th robot and it is assumed to be constant. Hereafter, we suppose that both of $s^{+}_{ji}$ and $s^{-}_{ij}$ are equal to zero during the negative time $t<0$ for all $i,j$.

We define $s_{q}:=[(\sqrt{\sigma/2})q_{r}^{T}\ 0^{T}]^{T}\in\mathbb{R}^{4}$ and the storage function for the communication channel as

[TABLE]

Thus, if we define $\bar{r}^{q}_{ij}:=r^{q}_{ij}-q_{r}$ and $\bar{r}_{ij}:=[(\bar{r}^{q}_{ij})^{T}\ (r^{\xi}_{ij})^{T}]^{T}$, the time derivative of $S^{c}_{ij}$ is given by

[TABLE]

Thus, if we interpret $[-p_{ij}^{T}\ -p_{ji}^{T}]^{T}$ and $[\bar{r}_{ij}^{T}\ \bar{r}_{ji}^{T}]^{T}$ as the input and output respectively, the communication channel becomes passive with respect to $S^{c}_{ij}$.

III-C Passivity of Robotic Network

In this subsection, we show passivity of the robotic network discussed above. First, we define the signals $\bar{q}_{i}:=q_{i}-q_{r}$, for all $i$ and $\bar{z}:=z-q_{r}$. Then, the error dynamics on (8) and (9) is given by

[TABLE]

where (28) is hold because $q_{r}$ is constant. From (28) and (29), (21) is rewritten as

[TABLE]

Then, we define the storage function for the error dynamics as

[TABLE]

From (23), the time derivative of $\bar{S}_{i}$ along the trajectory of (34) is given by

[TABLE]

Hence, we define a total storage function for the robotic network as

[TABLE]

From (36) and (III-B), the time derivative of $S$ is given by

[TABLE]

Thus, the robotic network is passive from human input $u_{h}$ to the average position of accessible robots $\bar{z}$ with respect to (37). In the next section, we close the robotic network with a human operator based on this passive property.

IV Interconnected system of Robotic Network and Passive Human Operator

In this section, we prove position synchronization (2) by utilizing passivity.

IV-A Synchronization to Human’s Desired Position

First, we start by the description of a human operator. Similarly to [13, 14], we assume that the operator determines the command $u_{h}$ based on the error $q_{r}-y_{h}$ between the reference $q_{r}$ and the feedback information $y_{h}=z$. Then, the entire system is illustrated as Fig. 5, where the human decision process is denoted by $\Phi$. Then, we assume that a human operator’s decision process $\Phi$ is input strictly passive from $q_{r}-z$ to $u_{h}$, i.e., there exists a storage function $S_{h}^{\Phi}$ and $\epsilon>0$ such that

[TABLE]

The validity of this assumption is examined in [13, 14] using experimental data obtained from a human-in-the-loop simulator, where it is confirmed that the statement is true over a prescribed frequency domain. We need to analyze the effect of the communication delays on the human property but this exceeds the scope of this paper and we leave the issue as a future work.

From this human passivity assumption, we have the following result.

Proposition 1

Consider the system (28) and (29), and the scattering transformation (24) and (25) for all $j\in\mathcal{N}_{i}$ and all $i$. Then, the feedback system achieves the condition (2) if the communication graph is fixed, undirected and connected, and the human decision process $\Phi$ is input strictly passive.

Proof:

First, we define an energy function $U$ as

[TABLE]

where $\beta$ is a positive constant. From the passivity of robotic network shown in Section III and human operator’s passivity assumed above, we obtain

[TABLE]

Thus, it is guaranteed that the all states are bounded in spite of communication delays.

Let us define $x^{i}_{t}$ such that $x^{i}_{t}(\theta)=x_{i}(t+\theta)$ for $\theta\in[-T,0]$. Then, the LaSalle’s invariance principle for time delay systems [18] is applicable and any solution $x_{t}$ of the system converges to the largest invariant set in the set of functions satisfying $\dot{U}\equiv 0$. Thus, $\dot{U}=0$ means $q_{i}\equiv r^{q}_{ij}\ (i,j)\in\mathcal{E}$ and $\bar{z}\equiv 0$. Hence, we can conclude as follows.

[TABLE]

Next, in order to analyze further, we need to follow the behavior of $\bar{r}_{ij}$. First, we define $e_{ij}$ and $e_{ji}$ as

[TABLE]

Then, each element of (24)–(26) can be given as

[TABLE]

Furthermore, subtracting (44) at time $t-T$ from (43) yields

[TABLE]

Utilizing the equations in (46), we have $r^{\xi}_{ij}=r^{\xi}_{ij}(t-2T)+e_{ij}+e_{ji}(t-T)$. Now, according to (41), the signals $e_{ij}$ and $e_{ji}$ converge to [math], i.e.,

[TABLE]

holds. Thus, taking the limit of (47) yields

[TABLE]

The same equation holds for $k\in\mathcal{N}_{i}$ as

[TABLE]

On the other hand, because the signal $r_{ij}^{q}$ for all $j\in\mathcal{N}_{i}$ converges to $q_{i}$ from (41), we obtain

[TABLE]

Subtracting (50) from (49) and using (51), we have

[TABLE]

Because of (41), equation (52) implies $\lim_{t\to\infty}(q_{j}-q_{k})=0$, which holds for $(j,k)\in\mathcal{E}$. Then, we have $\lim_{t\to\infty}(q_{i}-q_{j})=0$ for all $i,j\in\mathcal{V}$. Furthermore, from (42), we have

[TABLE]

Therefore, we can conclude that $\lim_{t\to\infty}(q_{i}-q_{r})=0$ for all $i\in\mathcal{V}$. This completes the proof. ∎

Remark 2

As in the same way, the velocity synchronization (7) is achieved with the same control architecture. However, we omit the result because of space limitations.

V Experiment

In this section, we show the experiment results of the discussed control system. In this paper, we focus on the influence of the communication delay on the human operator’s behavior. For that purpose, we conduct two types of experiments, without inter-robot communication delay and with inter-robot communication delay. Then, we investigate the differences by comparing the robots’ trajectories and human input. Although it is also important to investigate the human passivity assumption, as already mentioned in Section IV, we leave this as a future work.

V-A Experimental Setting

In this subsection, we introduce our experiment system and the intended scenario. The architecture of experiment system is illustrated in Fig. 7. We use 6 omnidirectional robots named TDO48 from TOSADENSHI Inc. and obtain image data by using a ceiling camera named Firefly MV from Point Grey. From image processing by C++, we obtain all robots positions. Even though the control architecture is implementable by each robot processor, we artificially implement all cooperative controller in a computer for simplicity. The inter-robot communication delays are also generated in Simulink and the delays set 0.5s. Simulink is running on dSPACE for the real time implementation. On the other hand, we show the average position of accessible robots and given reference position via GUI programmed by C++. Then, the participant inputs velocity command through mouse. Finally, the velocity inputs are converted to motor angular velocities, and then, transmitted to each robot via Bluetooth.

For the scenario, we ask the participant to control robotic network to a given position, as shown by red dot in Fig. 7. The reference position is $q_{r}=[0.55\ 0.60]^{T}$. Furthermore, we add the obstacles which the participant have to avoid collision with them. In order to avoid the collisions between each robot, we utilize the biases $d_{i}\ \forall{i}\in\mathcal{V}$ and denote real position $\eta_{i}$ as $\eta_{i}=q_{i}+d_{i}\ \forall{i}\in\mathcal{V}$. The biases are given as $d_{1}=[0.35\ 0.175]^{T},d_{2}=[0\ 0.175]^{T},d_{3}=[-0.35\ 0.175]^{T},d_{4}=[-0.35\ -0.175]^{T},d_{5}=[0\ -0.175]^{T},d_{6}=[0.35\ -0.175]^{T}$. During this section, we show all figures using the biased positions, rather than the real positions $\eta_{i}$. The communication graph of the robotic network is shown in Fig. 7. The initial positions of all robots are $q_{i}(0)=[2.6\ 1.6]^{T}$ and $\xi_{i}(0)=0\ \forall{i}\in\mathcal{V}$. The parameters are set as $a_{ij}=0.2,\ b_{ij}=0.05$ and $\sigma=1.0$.

V-B Experimental Results

First, the result without inter-robot communication delay is shown in Fig. 8. The human operator successfully control the accessible robots, and all robots reached the target position. Note that non-accessible robots could close the distances with the accessible robots around 30s while the accessible robots are still moving. This shows one of the advantage in exchanging the integral value, $\xi_{i}$, of $-\sum_{j\in\mathcal{N}_{i}}b_{ij}(q_{i}-q_{j})$. The human input is shown in the bottom part of Fig. 8. The result shows that the human operator smoothly controlled the robotic network.

Next, we show the result with inter-robot communication delay and the scattering transformation in Fig. 9. In the same way as the delay free case, all robots reached the target position. However, comparing the trajectories with the result without delay, non-accessible robots couldn’t close the distances with the accessible robots. As a result, the accessible robots moved back and forth around the reference position. The arrival time is delayed by about 70s. The human input is shown in the bottom of Fig. 9. After 50s, the human operator repeated adjustments because non-accessible robots followed late. Comparing to the result of delay free case, the human input looks more fluctuated, which can be interpreted as the deterioration of human’s operability.

In summary, position synchronization by the proposed architecture is verified through this experiment. In addition to this result, since we observed performance degradation of human, thorough investigation is needed regarding human performance. To this end, we need a criteria to measure the performance qualitatively.

VI Conclusions

In this paper, we have investigated a cooperative control architecture of human-robotic networks in the presence of inter-robot communication delays, where the objective is to guarantee position synchronization to human’s desired position. First, we proposed control architecture based on [13, 14] and the scattering transformation. Then, we showed passivity of the robotic network. Next, by using human passivity assumption, we showed that the feedback system achieves position synchronization. Finally, we demonstrated the efficiency of the proposed architecture through experiments, which managed to move all robots to target position in the same way as delay free case. Furthermore, we investigated the influences of inter-robot communication delays on human’s operability.