Model-Free and Learning-Free Proprioceptive Humanoid Movement Control

TL;DR

This paper introduces a model-free, learning-free proprioceptive control method for humanoid robots that relies solely on sensory outputs, enabling stable quasi-static movements without prior model knowledge.

Contribution

It presents a novel proprioceptive framework that does not require robot models or simulation training, using an optimization-based approach for stable humanoid movement control.

Findings

Successfully generates stable motions on NAO robot

Operates without prior kinematic or inertial models

Demonstrates effectiveness in real-world experiments

Abstract

This paper presents a novel model-free method for humanoid-robot quasi-static movement control. Traditional model-based methods often require precise robot model parameters. Additionally, existing learning-based frameworks often train the policy in simulation environments, thereby indirectly relying on a model. In contrast, we propose a proprioceptive framework based only on sensory outputs. It does not require prior knowledge of a robot's kinematic model or inertial parameters. Our method consists of three steps: 1. Planning different pairs of center of pressure (CoP) and foot position objectives within a single cycle. 2. Searching around the current configuration by slightly moving the robot's leg joints back and forth while recording the sensor measurements of its CoP and foot positions. 3. Updating the robot motion with an optimization algorithm until all objectives are achieved. We demonstrate our approach on a NAO humanoid robot platform. Experiment results show that it can successfully generate stable robot motions.

Tables

Table 1. TABLE I: Desired cost design

Index	Gait Transition	Desired Terms
		$𝐂_d^{}{}_{}{}^T$	${𝐏}_d{\left[l\right]}^T$	${𝐏}_d{\left[r\right]}^T$
1	a $\to$ b	$\left[\mathrm{0\hspace{0.33em}0}\right]$	$\left[0W\mathrm{\hspace{0.33em}0}\right]$	$\left[\mathrm{0\hspace{0.33em}0\hspace{0.33em}0}\right]$
2	b $\to$ c		$\left[0Wh\right]$
3	c $\to$ d	$\left[s^{\prime }\mathrm{\hspace{0.33em}0}\right]$	$\left[sW\mathrm{\hspace{0.33em}0}\right]$
4	d $\to$ e	$\left[s^{\prime }w\right]$
5	e $\to$ f	$-\left[s^{\prime }\mathrm{\hspace{0.33em}0}\right]$	$\left[\mathrm{0\; 0\; 0}\right]$	$-\left[sW\mathrm{\hspace{0.33em}0}\right]$

Equations

$$f=\frac{1}{2}(||\mathbf{C}-{\mathbf{C}}_d|{|}_{{\mathbf{w}}_{\mathbf{c}}}^2+\sum ||\mathbf{P}[j]-{\mathbf{P}}_d[j]|{|}_{{\mathbf{w}}_{\mathbf{p}}}^2)$$

$$\mathbf{C}=\sum _{i=1}^4{f}_i{\mathbf{p}}_i/\sum _{i=1}^4{f}_i.$$

$$\displaystyle{{\bf T}_{0}^{j}}={{\bf T}_{0}}^{-1}{\bf T}_{j}=\left[\begin{array}[]{cc}{\text{R}_{0}^{j}}&{\bf p}_{0}^{j}\\ \bf{0}^{T}&1\end{array}\right]$$

$$\displaystyle{{\bf T}_{0}^{s}}=\left[\begin{array}[]{cc}{\text{R${}_{z}$}}(\gamma_{0}^{r})&{\bf p}_{0}^{r}\\ \bf{0}^{T}&1\end{array}\right]$$

$$\displaystyle{{\bf T}_{s}^{l}}={{\bf T}_{0}^{s}}^{-1}{\bf T}_{0}^{l}=\left[\begin{array}[]{cc}{\text{R}_{s}^{l}}&{\bf p}_{s}^{l}\\ \bf{0}^{T}&1\end{array}\right]$$

$$\mathbf{P}[l]=[{{\mathbf{p}}_s^l}^T{\alpha }_s^l{\beta }_s^l{\gamma }_s^l{]}^T$$

$${\mathbf{q}}_{t+1}={\mathbf{q}}_t-\eta \nabla f({\mathbf{q}}_t)$$

$$f=aS+b,$$

$$\underset{\zeta }{\mathrm{argmin}}J=\sum _{k=1}^N(|n[k]-n_r{|}_{{\mathbf{w}}_{\mathbf{n}}}^2+$$

$$||\mathbf{c}[k]-{\mathbf{c}}_r[k]|{|}_{{\mathbf{w}}_{\mathbf{c}}}^2+||\mathbf{f}[k]-{\mathbf{f}}_r[k]|{|}_{{\mathbf{w}}_{\mathbf{f}}}^2),$$

$$\left[\begin{array}{c}{n}_r\\ {\mathbf{c}}_r\end{array}\right]\approx \left[\begin{array}{c}{\sum }_{i=1}^8(a_{0}S[i]+b_0)\\ {\sum }_{i=1}^8(a_{0}S[i]+b_0)\mathbf{p}[i]/{\sum }_{i=1}^8(a_{0}S[i]+b_0)\end{array}\right].$$

$$\left[\begin{array}{c}⋮\\ {\mathbf{c}}_r[k]n_r\\ ⋮\end{array}\right]\approx \left[\begin{array}{cc}⋮& ⋮\\ {\sum }_{i=1}^{8}S[k][i]\mathbf{p}[k][i]& {\sum }_{i=1}^8\mathbf{p}[k][i]\\ ⋮& ⋮\end{array}\right]\left[\begin{array}{c}{a}_0\\ b_0\end{array}\right]$$

$$e_n=(\sum _{k=1}^N|n[k]-n_r[k]|)/N,$$

$$e_c=(\sum _{k=1}^N||\mathbf{c}[k]-{\mathbf{c}}_r[k]||)/N,$$

Full text

I Introduction

Many works have addressed the topic of humanoid robot movement control over the last few decades. Great challenges in the planning and control of biped robots include high degrees of freedom (DoFs), nonlinear dynamics, and balancing with one foot on the ground. In the existing literature, model-based walking methods use either a simplified robot model [1] or a whole-body dynamic modeling approach [2, 3, 4]. Both rely on the accuracy of the robot’s inertial parameters, such as mass, center of mass (CoM), and link lengths. Complex control methods are also needed to compensate for modeling errors. The remaining literature on walking focuses primarily on reinforcement learning (RL). However, these RL methods often require a robot’s kinematic model and inertial parameters to build simulation environments, or a pre-planned walking trajectory is used as a reference signal [5, 6, 7]. And certain robustness adjustments, such as domain randomization, are also needed for sim-to-real transfer [8].

A widely used model-based method is proposed by Shuuji et al. [9, 10]. They facilitate the sophisticated nonlinear biped dynamics system into a simple 3D Linear Inverted Pendulum Mode (LIPM). A preview controller is used to track zero-moment point (ZMP) trajectories. Because the walking pattern is produced from a specified ZMP trajectory, this method is also called ZMP-based walking pattern generation [11]. Different from using the CoM of the whole robot, the authors in [12] utilize a five-mass distribution model to plan whole-body motion. The momentum vector has a linear connection to the robot joint speed vector. Thus, other research uses a reference linear and angular momentum to generate whole body motion, called Resolved Momentum Control [13, 14]. Angular momentum is also used in Capture Point (CP) to prevent a legged robot from falling by taking N steps before stopping [15, 16, 17]. The CP is calculated by extending the LIPM with a flywheel body. George et al. [18] combine CP and a passivity-based controller for a humanoid robot to walk over compliant terrain. Hybrid Zero Dynamics (HZD)[19, 20, 21] is a full-order model-based method. It depends on linear input and output, which can obtain more natural robot walking dynamics. However, these methods rely on the high accuracy of dynamic modeling, which may not be precise, and hence need external controllers. They also require high-performance CPUs and are computationally time-consuming. Additionally, these algorithms have high hardware requirements for robots. Thus, they may not be appropriate for small-sized and low-cost humanoid robots.

RL-based methods also show great potential in robot locomotion without detailed model-based planning and control in real-time. Instead, they rely heavily on offline training. However, directly training the RL policy on a real-world robot can be risky and may cause damage. Therefore, researchers typically train in simulation environments. To bridge the gap between simulation and the real world, complex robustness regulations are required. In [22], the authors demonstrate that a 3D biped can learn locomotion skills in simulation with limited prior model knowledge using a two-level hierarchical deep RL structure. Both levels use a uniform style of the actor-critic algorithm. Zhaoming et al. [23] use a multi-layer neural network with policy-gradient to learn a feedback controller for robot walking in simulation. However, the method requires a pre-defined joint angle trajectory as a reference signal and acquires the robot’s full state space information. But in reality, the states must be measured with noisy sensors. Combining with HZD, an RL framework is built to achieve variable speed controller learning for 3D robot walking in [24]. Nevertheless, the method is only validated in a customized simulation environment and not on a real robot. Other works aim to train stable policies for robots walking in a complex environment. In [25], a gradient-free RL algorithm is used to achieve robust biped walking on varying sloped terrain in simulation. Diego et al. [26] train a push-recovery whole-body motion planning policy in simulation with an RL framework. Although the works cited above obtain stable gait policies in simulation, the physics engines in the simulation environments still require robot dynamics models. That is, the above approaches cannot be generalized to robots without kinematic models or inertial parameters.

In this paper, a model-free framework for humanoid robot movement control is introduced. As it is an initial effort in this area, only quasi-static situation is addressed here. This proprioceptive framework depends only on sensor data, which does not need any knowledge of robot parameters, such as inertial parameters and kinematic models. Thus, the difficulties induced by model uncertainty can be resolved. The approach can be directly implemented on real robots without pre-training in simulation or reference trajectories. Therefore, it has the potential to be extended to explore unknown environments. Compared to other frameworks, our method is much simpler, requiring only low computing power, data storage capacity, and graphics card performance. Furthermore, the loss function can converge in only dozens of updates, indicating that a stable motion trajectory can be obtained in a short amount of time, without prior knowledge of a robot.

II Methodology

In order to evaluate the effectiveness of our method, we conduct two experiments, one focus on walking and another on self-calibration of shoe sensors. In this section, we provide a detailed explanation of the walking experiment as an example to illustrate our approach.

To generate the motion trajectory for one walking cycle, several pairs of the CoP and foot positions are pre-designed. These pairs are treated as objectives for the robot to reach during its walking II-A. We then define a cost function based on the Cartesian distances from the measured CoP and foot positions to their corresponding objectives II-B. Then, the robot slightly moves each of its leg joints forward and backward, while the robot’s CoP and foot positions are collected by sensors at each time II-C. After this, the local derivatives can be approximated using the finite difference of sensor outputs without an explicit function. Finally, an optimization algorithm is used to update the robot’s motion till all the objectives are achieved. II-D

II-A CoP and Foot Objectives Design

This part introduces details of our objective CoP and foot position design. No prior knowledge of a robot’s kinematic model or inertial parameters is used in this procedure.

The ZMP reference trajectory is designed by the humanoid robot’s footsteps to stay in balance. The robot’s slow movement can be approximated to quasi-static [27], in which case the ZMP and COP are equivalent. A simplified CoP path is used as shown in Fig.1. To begin, the robot is given the target of moving its CoP horizontally along the $y$-axis to the middle of the right foot while keeping both feet at the origin (Fig. 1a $\rightarrow$ b). When the CoP is in the center of the right foot, the robot can lift its left foot vertically without falling, as Fig. 1b $\rightarrow$ c. After the left foot is lifted to a certain height, the robot can step forward for a certain distance $s$ as Fig. 1c $\rightarrow$ d. In the meantime, the CoP of the robot should also move forward for half of $s$, which makes sense that the CoP should stay close to the center of the support polygon [28]. Then, the robot moves its CoP horizontally along the $y$-axis back to the center as Fig. 1d $\rightarrow$ e. The solved joint trajectories are stored and the motion of the other leg can just mirror the stored configuration trajectories due to the symmetry of biped robots. Therefore, setup methods in transition Fig. 1e $\rightarrow$ f $\rightarrow$ g are the same as above with the inverse of left and right.

II-B Cost Function Design

The general formula of the cost function for each posture transition is

[TABLE]

where ${\bf C}=[x,y]^{T}$ is the measured CoP of the robot, ${\bf C}_{d}$ is the desired CoP, $j=l,r$ means left and right foot, ${\bf P}=[x,y,z,\alpha,\beta,\gamma]^{T}$ is the current position and Euler angle of the robot’s foot, and ${\bf P}_{d}$ is the desired foot posture, $w_{c}$ and $w_{f}$ are the weights for the cost terms. Obviously, $f$ is a function of the robot joint angles, $\bf q$. As a result, one can find the derivatives of $f(\bf q)$ and minimize the cost. But in the present method, knowledge of $f({\bf q})$ as an analytical function is not required. The details will be shown in III-B. According to Fig. 1, the desired terms in the cost for each walking pattern transition are in Table I. Because the desired rotation of the robot’s feet is always zero, only 3-D positions of feet are listed.

II-C CoP and Foot Positions Measurement

The robot-sensor overview is shown in Fig. 2a, which includes a Nao robot H25 V6, a pair of force-sensing shoes, three April tags, and a camera. The force-sensing shoes are the same as [29]. We further attach an April tag to the bottom of the robot’s foot to track foot positions (Fig. 2b). April tags attached to the feet are called foot frames, and the one on the test bench is called the world frame.

II-C1 CoP Sensing Principle

Each force-sensing shoe has 4 load cells on the four corners of the shoe and measures the CoP. Through multiplying the force $f_{i}$ measured by each sensor by its corresponding 2D position ${\bf p}_{i}=[p_{ix}\;p_{iy}]^{T}$, and dividing by the sum of all the forces, the CoP, $\bf C$ in Eq. 1 is

[TABLE]

Using all eight sensors, the same sensing principle that works for single support can also be used for double support.

II-C2 Foot Positions Measurement Principle

There are four reference frames in the experiment: camera frame, world frame, foot frame, and floating base (Fig. 1). Taking transition a $\rightarrow$ b as an example, to calculate ${\bf P}[j]$ in Eq. 1, one needs to obtain the poses of the foot frames with respect to the floating base. From the camera and three April tags, one can directly get the transformation of the world frame and foot frames relative to the camera frame as ${\bf T}_{0}$, ${\bf T}_{j}$, where $j=l,r$ means left and right foot. Transfer the ${\bf T}_{j}$ into the world frame ${\bf T}_{0}$:

[TABLE]

where {\text{R}_{0}^{j}}={\text{R{}{z}}}(\gamma_{0}^{j}){\text{R{}{y}}}(\beta_{0}^{j}){\text{R{}_{x}}}(\alpha_{0}^{j})

From Fig. 1, the floating base is attached on the shoe (Fig. 2a ). It is the same as the right foot frame except that it is always parallel to the test bench. That is, it will not rotate around $x$ or $y$-axis of the world frame with $\beta_{0}^{r}=0$, $\alpha_{0}^{r}=0$. Therefore, from Eq. 5, the pose of the floating base with respect to the world frame is

[TABLE]

The pose of left foot frame with respect to the floating base is

[TABLE]

where {\text{R}_{s}^{l}}={\text{R{}{z}}}(\gamma_{s}^{l}){\text{R{}{y}}}(\beta_{s}^{l}){\text{R{}_{x}}}(\alpha_{s}^{l})

As a result, the ${\bf P}[l]$ in Eq. 1 is as

[TABLE]

The pose of the right foot frame relative to the floating base and the sensors’ positions can be derived by the same process.

II-D Optimization Algorithms

This section describes the optimization algorithm used to converge the robot’s measured CoP and foot positions to their objectives. Simple gradient-based methods is used.

Gradient Descent (GD) Given a multi-variable function $f({\bf q})$, the iterative law of the GD [30] is

[TABLE]

where $\nabla{f}=\frac{\partial\bf f}{\partial{\bf q}}$is the gradient of the objective function and $\eta$ is a step size that can be determined in different ways like Armijo rule [31].

To approximate the derivatives in the optimization algorithm formula above, finite difference [32] is used. In the literature, the finite difference method is often used due to its speed. Here it is used because there is not an explicit function whose derivative can be computed analytically. The function is empirical, resulting directly from sensor measurements and how their values are perturbed as the robot configuration is slightly varied.

III Experiment i

III-A NAO Robot

In this paper, the NAO V6 robot is used to test the general method. The mass is 5.3$kg$ and the height is 57$cm$. The robot has 25 degrees of freedom (DoFs). In this model-free biped walking framework, the five joints for each leg are considered in Fig. 3.

III-B Model-Free Mapping of CoP Trajectory and Foot Placement to Joint Angles

In the common humanoid robot, Pitch joints (Fig. 3a) are rotations around the $y$-axis, and Roll joints (Fig. 3b) are around the $x$-axis. Due to this structure, the robot can move the body’s CoP horizontally along the $y$-axis by simply rotating four Roll joints of both legs, such as Fig. 1a $\rightarrow$ b and d $\rightarrow$ e $\rightarrow$ f. For the same reason, it can lift one of its feet and step forward with the rotation of Pitch joints as Fig. 1b $\rightarrow$ c $\rightarrow$ d.

Taking transition Fig. 1a $\rightarrow$ b as an example: assuming the joint vector in Fig. 1a is ${\bf q}_{0}=[q_{1}\;q_{2}\;q_{3}\;q_{4}]^{T}$, NAO will rotate one of its joint forward for a certain angle as $f(q_{1}+\Delta q_{1},q_{2},q_{3},q_{4})$. This value can be used to calculate the gradient of $f({\bf q}_{0})$. Each time the robot turns one or more joints and reaches a pose where the cost of calculating the derivatives can be found, it is called a search. According to finite difference in II-D, it takes 8 searches to find the gradient of $f({\bf q}_{0})$ for an update. Then the joints are updated to the next state $\bf{q}_{01}$ by the optimization methods in II-D. The robot continues this cycle until the value of the cost is less than a certain value or the difference between the costs of two consecutive updates is small enough. The same procedure is also taken in other transitions in Fig. 1. The only difference is that the rotation joints are Pitch joints (Fig. 3a) in transition Fig. 1b $\rightarrow$ c $\rightarrow$ d.

III-C Demonstrations and Results

To test the effectiveness of the method, we directly implemented it on a real robot without simulation. GD with a fixed step size is used to minimize the cost function in real-world experiments to present our method.

Transition examples are displayed in Fig. 4. Because the transitions d $\rightarrow$ e $\rightarrow$ f are the same as a $\rightarrow$ b with moving CoP horizontally along the $y$-axis, only examples of a $\rightarrow$ b $\rightarrow$ d are listed.

There are two problems for NAO hardware when it supports its entire weight on a single leg: 1. The HipRoll joint is unable to match the reference position. This is because the single support requires very high torque from this joint. As a result, the swinging leg does not leave the ground properly and will touch the ground prematurely, which destabilizes the robot [33]. 2. For small-sized humanoid robots with under-powered motors, the single support generates excessive torque. This results in the NAO robot platform suffering from severe leg joint overheating issues [27]. To solve the first problem, the authors in [33] design a trapezoid function feedback controller based on pre-planned joint trajectories. However, their method is model-based, which is not suitable for our model-free framework. As for the second problem, the authors in [27] choose double support rather than single support for safety concerns. Therefore, we directly do the transition Fig. 1b $\rightarrow$ d, without posture c. And a ${\bf C}_{d}$ closer to the center of double support is also chosen to reduce the joint torque.

The results of generating gait trajectories with our model-free framework are displayed in Fig. 4. Fig. 4a shows that the CoP is getting closer to the target location and the cost function converges within a dozen of updates. Next, the robot can step forward with several times of updates Fig. 4b. There is no collision problem in the experiments. This is because the robot’s joints only turn a small angle at each search. And the robot’s desired left and right foot positions in the cost function have a certain distance between each other. The overall results demonstrate that our approach can generate a stable gait trajectory without the robot’s inertial parameters, a kinematic model, reference joint signals, or a training process.

IV Experiment ii

In our previous work [27], we utilized a robot model to plan whole-body trajectories and relied on modeled CoP and ground reaction force (GRF) as reference data to enable the robot to self-calibrate its shoe sensors. However, this model-based approach may not always be practical, especially when the robot is repaired or retrofitted with additional equipment, which may result in the loss of the robot model. To overcome this limitation, we integrated our model-free framework into the self-calibration process. This model-free self-calibration (MFSC) approach allows the robot to self-calibrate its shoe sensors without the need for a pre-existing model.

IV-A Sensor Parameter Identification

Our proposed method comprises two parts: 1. The robot follows planned CoP paths using our model-free method II when the shoe sensors are accurate. The measured CoP and GRF are saved as reference ground truth data. 2. When the sensors lose accuracy, the robot replays the previous trajectories, and the sensor parameters are optimized by reducing the difference between the reference data and the current measured data collected during the robot’s movement.

The load cell sensing principle is

[TABLE]

where $S$ is the voltage output of the load cell, $a$ and $b$ are the constant coefficients.

To determine the load cells’ parameters, we use nonlinear least squares (NLS) to minimize the error between the sensors’ current raw measurements and their corresponding reference GRF and CoP values Fig. 5 a and b. The optimization is formulated as:

[TABLE]

where ${\bf\zeta}=[(a_{1},b_{1}),\ldots,(a_{8},b_{8})]$ are the optimization variables; $k$ is the number of training points; ${n}$ and ${\bf c}$ represent current GRF and CoP measurements; $n_{r}$ and ${\bf c_{r}}$ are their reference data; ${\bf f}$ and ${\bf f}_{r}$ represent the measured force and reference force of each load cell. The weights for the cost terms are denoted as $w_{n}$, $w_{c}$, and $w_{f}$.

IV-B Initial Guess of Sensor Parameters

A reasonable initial guess is required for the NLS process. Given that all sensors in the shoes are of the same type, we use an equal initial estimation of $[a_{0},b_{0}]$ for all sensors’ parameters. We assume that the GRF and CoP measurements using this initial guess are approximately equivalent to their corresponding reference values,

[TABLE]

$i$ represents the sensor index, and ${\bf p}$ is the sensor position obtained from the camera. To include more training samples, the equations can be rearranged as

[TABLE]

where $k$ is the number of the training samples. The initial guess $(a_{0},b_{0})$ can therefore be resolved using least squares regression.

IV-C Performance Evaluation

To assess the accuracy of the shoe measurements, mean absolute error (MAE) is used. The MAE of the GRF, $e_{n}$, is given by:

[TABLE]

The MAE of CoP, $e_{C}$, is defined as:

[TABLE]

IV-D Demonstrations and Results

To verify the efficacy of our MFSC approach, we compare the shoe measurements using the initial guess parameters with those using model-free self-calibrated parameters, as shown in Fig. 5a and b. In this case, the camera only serves as an unknown mass, making the robot model uncertain. The corresponding MAE values for the two datasets are displayed in Fig. 5c. The results indicate that the GRF (3.5 N MAE) and CoP (4.5 mm MAE) measurements through the initial estimation parameters are far from the ground truth(Fig. 5c). On the contrary, the MAE of GRF and CoP measured by our MFSC approach are around 0.09 N and 0.8 mm respectively, which is much closer to the ground truth. Additionally, the performance of the training dataset (Fig. 5c, white background) and testing dataset (grey background) does not differ significantly, indicating that there is no over-fitting. Overall, the results suggest that our model-free self-calibration method can effectively recover sensor measurements without relying on robot models, manual intervention or initial sensor data.

V Conclusion and Future work

This research presents a novel model-free framework for quasi-static humanoid robot walking and self-calibrating foot force-sensing modules based only on sensor outputs. The proposed approach utilizes gradient descent to optimize the objective function that includes the measured CoP and positions of the robot’s feet. We have directly implemented this framework on a real robot. The results show that it is capable of planning a steady walking trajectory without prior knowledge of the robot’s inertial parameters or kinematic model. Although demonstrated on the NAO, this proprioceptive framework can theoretically be generalized to other robots. This method simply needs initial planning to generate one motion that can be saved for future usage. The framework utilizes rich sensor information, which can be integrated with other model-based techniques to explore unknown environments. Future work will involve adding filters to reduce sensor noise and implementing stochastic-based algorithms on various robots. With fast computation, we can also do dynamic proprioceptive motion by including the velocity and acceleration space in the search process.

VI Acknowledgement

This work was supported by NUS Startup grants A-0009059-02-00, A-0009059-03-00, CDE Board account E-465-00-0009-01, Tier 2 grant REBOT A-8000424-00-00, and National Research Foundation, Singapore, under its Medium Sized Centre Programme - Centre for Advanced Robotics Technology Innovation (CARTIN), sub award A-0009428-08-00.