Embodied robots driven by self-organized environmental feedback

TL;DR

This paper explores how snake-like robots can develop complex behaviors through self-organizing principles driven by environmental feedback, without external sensors, leading to emergent locomotion and interaction behaviors.

Contribution

It demonstrates that mechanical coupling and sensorimotor loops can produce self-organized locomotive behaviors in passive, segmented robots without external sensing.

Findings

Robots exhibit complex behaviors like pushing blocks and climbing slopes.

Locomotion arises from self-organized limit cycles in sensorimotor loops.

Behavioral transitions are characterized by bifurcations in dynamical systems.

Abstract

Which kind of complex behavior may arise from self-organizing principles? We investigate this question for the case of snake-like robots composed of passively coupled segments, with every segment containing two wheels actuated separately by a single neuron. The robot is self organized both on the level of the individual wheels and with respect to inter-wheel coordination, which arises exclusively from the mechanical coupling of the individual wheels and segments. For the individual wheel, the generating principle proposed results in locomotive states that correspond to self-organized limit cycles of the sensorimotor loop. Our robot interacts with the environment by monitoring the state of its actuators, that is via propriosensation. External sensors are absent. In a structured environment the robot shows complex emergent behavior that includes pushing movable blocks around, reversing direction when hitting a wall and turning when climbing a slope. On flat grounds the robot wiggles in a snake-like manner, when moving at higher velocities. We also investigate the emergence of motor primitives, viz the route to locomotion, which is characterized by a series of local and global bifurcations in terms of dynamical system theory.

Equations

$$\tau \dot{x}=\cos \phi -x,y=\tanh (ax),$$

$$F_{k}=k\big{(}x^{(t)}-x^{(a)}\big{)},\qquad\quad x^{(t)}=y,\qquad\quad x^{(a)}=\cos(\varphi)\,,$$

$$\tau \dot{x}=\cos \phi -x,\dot{\phi }=\omega ,I\dot{\omega }=R(F_k\sin \phi -f\omega ),$$

$$\sin \phi =0\text{mbox}ory(x)=\cos \phi ,$$

$$J(\varphi)\ =\ \left(\begin{array}[]{ccc}-1/\tau&\,-\sin\varphi/\tau&0\\[2.15277pt] 0&0&1\\[2.15277pt] ka(1-y^{2})\sin\varphi&A&-f\end{array}\right)\,,$$

$${\lambda }_0(0,\pi )=-\frac{1}{\tau },{\lambda }_{\pm }(0,\pi )=-\frac{f}{2}\pm \frac{1}{2}\sqrt{f^2+A(0,\pi )},$$

$$J(\varphi=\pm\pi/2)\ =\ \left(\begin{array}[]{ccc}-1/\tau&\,\mp 1/\tau&0\\[2.15277pt] 0&0&1\\[2.15277pt] \pm ka&k&-f\end{array}\right)\,$$

$$\framebox{$\phantom{|}\omega\to 0\phantom{|}$}\ \ \xrightarrow{\mbox{\small HB}}\ \ \framebox{$\phantom{|}\omega\neq 0,\,\overline{\omega}=0\phantom{|}$}\ \ \xrightarrow{\mbox{\small SHE}}\ \ \framebox{$\phantom{|}\omega\to 0\phantom{|}$}\ \ \xrightarrow{\mbox{\small HE}}\ \ \framebox{$\phantom{|}\omega\neq 0,\,\overline{\omega}\neq 0\phantom{|}$}$$

Full text

Our robot interacts with the environment by monitoring the state of its actuators, that is via propriosensation. External sensors are absent. In a structured environment the robot shows complex emergent behavior that includes pushing movable blocks around, reversing direction when hitting a wall and turning when climbing a slope. On flat grounds the robot wiggles in a snake-like manner, when moving at higher velocities. We also investigate the emergence of motor primitives, viz the route to locomotion, which is characterized by a series of local and global bifurcations in terms of dynamical system theory.

1 Introduction

Wheeled snake-like robots [1] are a class of hypermobile robots [2] that are able to navigate flexibly through rough terrains and restricted geometries. Movements may be generated either via central pattern generators [3], or via top-down commands [4], with the latter being a challenging task when a large number of actuators is involved. An alternative is autonomous decentralized control, which has been studied for the case of serpentine robots in terms of a chain of locally coupled oscillators [5, 6], and neurally-inspired generating schemes able of sensorless pathfinding [7].

A key rationale for developing biologically inspired robots is the drive for robust and highly adaptive designs [8, 9]. Similarly, this is most of the time also the motive for studying how adaptive locomotion can be realized [10], f.i. with soft robots [11]. Abstracting from the direct engineering benefit, it is of particular interest to study generating mechanisms of locomotion in general, an approach taken here. Our focus is on compliant locomotion generated by self-organizing dynamical systems, which may take the form of either limit-cycle [12] or playful behavior [13].

Compliance, which denotes the ability of a robot to react elastically to environmental feedback [14], may be achieved in several distinct ways, which include from the engineering perspective suitably designed actuators [15] and control algorithms [16]. Compliant behavior can emerge on the other side also through the reciprocal dynamical coupling of control, body and environment [17], the sensorimotor loop. A particularly interesting limit is here, from the perspective of complex system theory, the limit of a fully reactive and hence embodied controller. In this limit, the controller is inactive in absence of environmental feedback, with the consequence that the sensorimotor feedback has not only a modulating effect on locomotion, becoming instead essential. Locomotion then arises via limit cycles and chaotic attractors that emerge within the sensorimotor loop, the telltale sign of self-organized locomotion. It is hence important to ask, as we will do in this study, how locomotion is generated in terms of a dynamical systems bifurcation diagram.

Decomposing complex behavior into a series (or into a superposition) of basic reusable building blocks, the motor primitives [18], is a well studied approach for reducing the control problem of complex robots. Movement primitives may be modeled by nonlinear dynamical systems [19] using, e.g., Gaussian mixture models [20], where the parameters of the dynamical system are either uniquely defined or drawn from a suitable distribution [21, 22]. Motor primitives can emerge also from embodied dynamics in terms of chaotic itinerancy [23], or, alternatively, as self-organized attracting states in the sensorimotor loop [24], that is within the state space comprising the controller, the body of the robot and the environment. Here we propose a new type of self-organized controller for wheeled robots that leads to multiple fixpoint and limit-cycle attractor states and hence to self-organized motor primitives in the sensorimotor loop. With the behavior of the robot being self organized on the level of the individual wheels and with respect to inter-wheel coordination, the resulting dynamics reflects its affordances [25] when placed in simple but structured environments.

1.1 Control frameworks and the sensorimotor loop

Several in part non-exclusive routes for the generation of locomotion in robots and animats do exist in generic terms. Standard top-down control, as illustrated in Fig. 1, consists of a central processor generating motor commands either reactively, in response to sensory inputs, or deliberately on its own [26]. The actuator may be in turn stiff, as for industrial robots, or compliant, as for the muscles and tendons of animals, reacting either passively or actively to external forces [15]. For the latter case, as sketched in Fig. 1, the actuator changes its stiffness upon sensing its own state. Compliance arises then in response to propriosensation.

We are interested in locomotion that arises through the interaction of the degrees of freedom of the robot, including both internal variables and the body, with environmental feedback. The combined variables of the resulting sensorimotor loop constitute then the phase space for dynamical attracting states, fixed points, limit cycles and chaotic attractors, that correspond to self-organized behavioral primitives. The locomotion generated in this way is highly compliant in the sense that the attracting states in the sensorimotor loop respond elastically to additional top-down commands changing internal parameters.

2 Locomotive principles

Studies of real-world and simulated robots may focus either on performance, and its improvement, or on the generative capabilities of locomotive principles. The latter approach is gaining in importance in view of a recent study of the neural coding of leg dynamics in flies, which showed that the dynamics of the leg becomes dysfunction once the feedback loop between leg proprioception and motor commands is cut [27]. These findings imply that self-organization plays a commanding role in fly locomotion. Distributed computations has been found to be of relevance for the nematode C. elegans [28]. Here we concentrate on generative principles that are time reversal symmetric in the sense that a given set of internal parameters allows the robot to move both forwards and backwards. The direction selected by the robot then depends on the initial state, like a small positive initial velocity or force.

2.1 Locomotion via time reversal symmetry breaking

Locomotion is parametrized typically by a velocity vector $\mathbf{v}=d\mathbf{r}/dt$ that incorporates both the direction and the magnitude of the movement. Reversing time $t\leftrightarrow(-t)$ reverses then also the velocity vector. Here we are interested in self-organized robots that break time reversal symmetry spontaneously, which in our case implies that the attracting states in the sensorimotor loop come in pairs that are related via time reversal symmetry. Whether the robot moves for- or backwards depends then only on the initial conditions. For this purpose we use the one-neuron controller illustrated in Fig. 2.

A wheel with a rotational angle $\varphi$ is regulated individually via

[TABLE]

where $x$ is the membrane potential of the controlling neuron, $y=\tanh(ax)$ the neural activity and $\tau$ the membrane time constant. The motor command is proportional to the spring force

[TABLE]

where $k$ is a spring constant and $x^{(a)}$ and $x^{(t)}$ respectively the actual and the target position of the wheel in terms of a projection to the ground [29]. Note that the angle $\varphi$, which enters the right-hand side of Eqs. (1) and (2) as $\cos(\varphi)$, is the measured, the actual angle of the wheel. All forces, gravitational and mechanical, impact the controller hence exclusively via their influence on the angle $\varphi$.

The controller simulates the transmission rod of a classical steam engine, as sketched in Fig. 2, as it translates the bounded forth and back motion of the neural activity $y(t)$ into a rotational motion. Alternatively, instead of using the angle $\varphi$ as the determining variable, one could postulate a discrete map $\omega^{(t)}=\tanh(a\omega^{(a)})$ between the actual and a target angular velocity [30], $\omega^{(a)}$ and $\omega^{(a)}$. This is not a problem for simulated robots, for which $\omega$ is a directly accessible variable. To obtain a reliable estimate of the instantaneous angular velocity for real-world robots working with duty cycles of the order of $20\,\mathrm{Hz}$ would however be a challenge [29].

A controller enabling locomotive limit cycles to emerge in the sensorimotor loop [12], as described here by (1) and (2), differs qualitatively from controlling schemes employing local phase oscillators [31], for which a spontaneous reversal of the direction of motion would not be possible.

2.2 Isolated wheel

The individual wheels of the simulated robots are controlled exclusively by (1) and (2). There is no explicit inter-wheel coupling present. It is illustrative to model, for comparison, an idealized isolated wheel with moment of inertia $I$, radius $R$, angle $\varphi$ and angular velocity $\omega$. The force $F_{k}$ generated by the simulated transmission rod then enters the equations of motion as a torque $RF_{k}\sin(\varphi)$,

[TABLE]

where $f>0$ is a friction coefficient. Eq. (3) is manifestly invariant under $\omega\leftrightarrow(-\omega)$, $\varphi\leftrightarrow(-\varphi)$ and $x\leftrightarrow x$, which implies time-reversal symmetry in terms of an invariance with respect to reversing the direction of motion. We will investigate (3) further in Sect. 3.5, noting here that symmetry breaking may occur also in embodied robots that incorporate forward world models [32].

3 Results

We used the LPZRobots physics simulation package [33] for the simulation of robots composed of chains of 1-5 two-wheeled cars linked passively through hinge joints, which are equipped with passively damped torsion springs. In the absence of motor commands or external forces the equilibrium position of the hinge joints induces a straight alignment of the connected body segments. During locomotion the joints can store, on the other hand, potential energy when bent.

Shown in Fig. 3 is the trajectory of the train of cars climbing up a slope that is intersected orthogonally by two other slopes. One observes wiggling and straight locomotion together with direction reversal and large turns. In order to develop an understanding we start by investigating the velocity profile of a robot on an extended slope, concentrating on the dependence of the self-regulated steady state velocity on the spring constant $k$ of the actuator and on the inclination of the slope. We note that the simulation cycle times of the LPZRobots simulation package, which is based on the Open Dynamics Engine [34], are of the order of 50 ms.

3.1 Moving up and down an infinite slope

In Fig. 4 we present the velocity profile for a 5-segmented robot moving on a slope parallel to the gradient, that is straight up and down. The downward velocity decreases in magnitude with decreasing slope and spring constant $k$, as expected.

For the robot moving on a horizontal plane there exists a critical $k_{c}\approx 0.54$, such that the limit-cycles corresponding to regular forward or backward movement disappear for $k<k_{c}$. For a spring constant of $k=0.2$, which is below $k_{c}$, the robot moves therefore only when the slope has a finite downward inclination, as shown in Fig. 4.

The torque exerted on the wheels is directly proportional to the spring constant, being generated otherwise through the sensorimotor feedback. Moving upward the slope the torque $RF_{tan}$, and hence also the sensorimotor feedback, needs to counter the gravitational downhill force $F_{G}$. For an engine producing a constant torque, the balancing of the tangential and the gravitational force would lead to an uphill velocity $v_{slope}\propto F_{tan}-F_{G}$ that vanishes linearly and hence continuously at a critical inclination. This is however not the case when the motion is generated through sensorimotor feedback, as evident from the data presented in Fig. 4. The sensorimotor feedback involves a self-consistency condition that breaks down discontinuously, at finite values of the uphill velocity, when the inclination of the slope becomes too large. It remains however the case that larger spring constants $k$ allow the robot to move up steeper slopes.

3.2 Autonomous direction reversal

The trajectory of the robot presented in Fig. 3 hits twice an intersecting slope. A robot equipped with actuators producing constant torques would move segment by segment onto the intersecting slope, up to the point where the gravitational downward pull of the increasing number of segments cancels with the locomotive force. At this point the robot would remain in place.

The locomotive force of the snake-like robot presented here is however highly compliant, being generated within the sensorimotor loop. The velocity profile presented in Fig. 4 implies that an equilibrium position resulting from the balance of an upward locomotive force and the downhill gravitational pull is not possible, namely that the limit-cycle attractors present in the sensorimotor loop allow the robot to move only up- or downhill. We note that the stable fixpoint attractors corresponding to a non-moving state, that exist in conjunction with the limit-cycle locomotion for an isolated wheel, as discussed in Sect. 3.5, possesses only a vanishing small basin of attraction for the case of a train of cars. A robot hitting a slope that is too steep is therefore likely to reverse direction, as observed in Fig. 3, instead of being pulled into a fixpoint attractor and coming to a stop.

3.3 Straight, meandering and chaotic modes

The last leg of the trajectory shown in Fig. 3 shows growing left- and rightward swings. This is a typical behavior at larger velocities, here due to the slight downhill direction, that results from a transversal mechanical instability of the connected segments. The joints are elastic and therefore capable to store a certain amount of energy, akin to what happens when a string starts to vibrate. The final upturn of the robot may occur at an angle, interestingly, that puts the robot below criticality. The robot will then stop moving and reverses direction.

In Fig. 5 we present the angular frequency $\omega(t)$ for one of the two wheels of the first and the last car, respectively, of a five-segmented snake-like robot moving on a flat ground. Shown are the timelines for two regular and for a chaotic mode. In the first limit-cycle mode the train of cars moves straight. The ten wheels are in this case synchronized in the sense that the small modulation of the respective angular velocities, which occur because the controller (1) is not rotationally invariant, appear all at exactly the same time. Their respective amplitudes vary however from car to car, which implies that the pitch angles of the individual cars oscillates, even though only slightly.

For larger average velocities, the limit-cycle straight mode tends to be become unstable, making way to meandering and chaotic modes, as illustrated in Fig. 5. We did not investigate the exact nature of the respective transition to chaos, which may be due to a cascade of period-doubling within the space of meandering modes [35]. We also note that our classification of the highly irregular mode as chaotic relies here only on a visual inspection, as we did not apply a formal test for the presence of deterministic chaos [36].

3.4 Interacting with a structured environment

The robots exhibit interesting behavioral patterns when situated in a structured environment. As a first example we show in Fig. 6 the interaction of a 5-segmented robot with a wall. Before hitting the wall the robot possesses, due to time-reversal symmetry, both a forward- and a backward-moving limit cycle. Approaching the wall the sensorimotor state of the robot is in the forward limit cycle, which becomes however destroyed upon hitting the wall. The flow in phase space, that is the evolution of the membrane potentials $x$ of the individual wheels, is then attracted by the remaining limit cycle, which is the one corresponding to moving backward. The robot hence reverses direction.

The observed direction reversal occurs autonomously in the absence of top-down control signals. As the robot possesses only sensors measuring the angles of the wheels there are furthermore no external sensors present that would inform the robot about the distance to the obstacle. One observes, as shown in Fig. 6, that the angle at which the robot bounces varies considerably as a consequence of the wiggling of the initial forward motion. In a slow-velocity non-wiggling mode the bouncing occurs exactly perpendicularly.

We also studied the interaction of the multi-segmented robot with movable boxes, as illustrated in Fig. 7. Due to the mechanical feedback of the passive but elastic hinge joints, the body of the train of cars bends, such that the robot continues to push the box in ever smaller circles. During the push, the robot slows down continuously, until a critical velocity is reached and the forward limit cycle disappears. At this point the robot stops, reversing direction autonomously.

The robot disposes of a single controlling neuron per actuator, which obtains in turn information about the external world only indirectly, namely via the measured angle of the respective wheel. The behavior illustrated in Fig. 7, such as pushing around boxes, is hence emergent and an example that embodied controlling frameworks may lead robustly to novel behavioral patterns. Functionally, the ability of the snake-like robot to follow spiral-shaped trajectories, when pushing a box, is a direct consequence of the highly elastic working regime of the actuators, viz of compliance. By themselves, the wheels on the left-hand and the right-hand side of the body would acquire, being subject to identical controllers, identical angular velocities. The fact, however, that the turning speed of the wheels is not determined by a top-down signal, but by the sensorimotor feedback, allows the emerging limit cycles to adapt autonomously to environmental forces. The wheels on opposite sides of the body react as a consequence distinctly when the box exerts a non-symmetric resistance onto the robot. The respective angular velocities then differ.

3.5 Theory for an isolated single wheel

The fixpoints of an isolated and non-moving single wheel, as described by Eq. (3), are determined by $\omega=0$, $x=\cos\varphi$ and

[TABLE]

compare Eqs. (3) and (2). The Jacobian $J(\varphi)$ is in general

[TABLE]

where $A=ky\cos\varphi+k(\sin^{2}\varphi-\cos^{2}\varphi)$. We have set $R=1$, measuring in addition $k$ and $f$ relative to $I$.

• –

For the first fixpoint $\sin\varphi=0$, that is for $(x,\omega,\varphi)=(1,0,0)$ and $(-1,0,\pi)$, one has $A(0,\pi)=k(\pm y(\pm 1)-1)=k(\tanh(a)-1)$, which is always negative. The eigenvalues of the Jacobian are then

[TABLE]

Since $A(0,\pi)<0$, the fixpoints at $\varphi=0$ and $\varphi=\pi$ are always stable.

• •

For the second set of fixpoints one has to solve the self-consistency condition $y(x)=\tanh(ax)=x$, with $x=\cos\varphi$. The trivial solution $x=0$, that is $(x,\omega,\varphi)=(0,0,\pm\pi/2)$ splits at $a_{\mathrm{c}}=1$ via a pitchfork transition, allowing for three coexisting fixpoints for $a>a_{c}$. For small $k$ the trivial fixpoint $x=0=\cos\varphi=y$ with the Jacobian

[TABLE]

is a saddle for $0<a\leq 1$, having two negative and one positive eigenvalues, being stable however for $a>1$. For larger values of the spring constant $k$, the $x=0$ solution undergoes a Hopf bifurcation leading to limit-cycle oscillations.

3.6 Routes to locomotion

It is interesting to study how limit-cycle locomotion arises from a configuration of individual fixpoints upon increasing the force acting on the wheel, that is the spring constant $k$.

In Fig. 8 we illustrate the case $a<1$, for which the saddles at $\varphi=\pm\pi/2$ do not undergo a pitchfork bifurcation yet. Locomotion arises in this case via a one-step heteroclinic transition which allows for the generation of limit cycles of finite amplitudes. A pair of stable and unstable limit cycles is eventually produced when increasing the spring constant $k$, with the stable limit cycle corresponding to a locomotive behavioral primitive. Note that the phase space of (3) is three dimensional and that the flow shown in Fig. 8 corresponds to a projection onto the $(\varphi,\omega)$-plane. Trajectories may hence intersect.

In Fig. 9 a multi-step route to locomotion for $a>1$ is presented. One observes first a Hopf-bifurcation (HB) at $\varphi=\pm\pi/2$, which leads to a first intermediate phase characterized by a closed limit-cycle in the $(\varphi,\omega)$-plane. This limit cycle, corresponding to small amplitude forth-and-back periodic motion, is destroyed when hitting in a symmetric heteroclinic transition (SHE) the two saddles present additionally for $a>a_{\mathrm{c}}=1$. Motion ceases in the subsequent second intermediate phase, for which the unstable trajectories emerging from $\varphi=\pm\pi/2$ lead to the fixpoints $\varphi=0,\pi$. A stable limit-cycle emerges however from a second heteroclinic transition (HE) when increasing the spring constant $k$ further, namely when the unstable manifold of one of the additional saddles hits another saddle.

For the parameters used for Fig. 9 there are hence for $a>1$ two phases without locomotion, viz for which the angular frequency $\omega$ decays to zero. The average angular frequency $\overline{\omega}$ vanishing for forth-and-back motion, but not for limit-cycle locomotion:

[TABLE]

With the motor command being proportional to the spring constant $k$, it is somewhat intuitive that one needs a critical $k$ for locomotion to emerge. Relatively large spring constants have been used in Fig. 8 for illustrative purposes.

4 Conclusions

The gait of an animal corresponds to a coordinated pattern of limb movements that repeats with a certain frequency. Gaits are generated typically by a central pattern generator [3], that is by a central processing unit that produces coordinated motor signals. We have examined here an alternative framework for which the actuators of an animat are self active, with the dynamics of the individual actuators resulting from the presence of limit-cycle attractors within the sensorimotor loop. The actuators, in our case the wheels of a snake-like robot, are coupled in this framework only via the mechanics of the body and by the reaction of the environment. The gaits of the animat result in our framework therefore from self-organizing principles. For a simulated wheeled snake-like animat, we find that the robot interacts autonomously with the environment, e.g. by turning on its own on a slope. The robot will also push a movable box around for a while when colliding with one.

We have tested in addition that locomotion modes also arise for heterogeneous (not identical) body segments, e.g. when the controllers have different spring constants $k_{i}$, or different wheel sizes. The multi-segmented robot is capable of generating locomotion in particular when several actuators are subcritical, i.e. with wheels which on their own would not maintain oscillatory dynamics. Embodiment leads in our study robustly to emergent locomotion.

The here employed dynamical-system type approach to robotic locomotion allows in addition to characterize the motion primitives in terms of self-organized attractors formed in the extended phase space of the robot and environment. Incorporating objects of the environment into the overarching dynamical system allows in consequence dynamical system approaches also to classify computational models of affordance [37]. In this sense self-organizing attractors play an important role in the generation of useful behavior for the discovery of dynamic object affordances [38].

Acknowledgments

The support of the German Science Foundation (DFG) is acknowledged.