Material Recognition CNNs and Hierarchical Planning for Biped Robot Locomotion on Slippery Terrain

TL;DR

This paper presents a method for visually estimating surface friction using CNNs and integrating this information into hierarchical biped robot planning to enable autonomous movement on slippery terrains.

Contribution

It introduces a novel approach combining visual friction prediction with hierarchical planning using chance constraints for improved robot locomotion.

Findings

Effective friction prediction from real outdoor images.

Successful autonomous navigation on slippery surfaces.

Robust integration of friction uncertainty into planning.

Abstract

In this paper we tackle the problem of visually predicting surface friction for environments with diverse surfaces, and integrating this knowledge into biped robot locomotion planning. The problem is essential for autonomous robot locomotion since diverse surfaces with varying friction abound in the real world, from wood to ceramic tiles, grass or ice, which may cause difficulties or huge energy costs for robot locomotion if not considered. We propose to estimate friction and its uncertainty from visual estimation of material classes using convolutional neural networks, together with probability distribution functions of friction associated with each material. We then robustly integrate the friction predictions into a hierarchical (footstep and full-body) planning method using chance constraints, and optimize the same trajectory costs at both levels of the planning method for consistency. Our solution achieves fully autonomous perception and locomotion on slippery terrain, which considers not only friction and its uncertainty, but also collision, stability and trajectory cost. We show promising friction prediction results in real pictures of outdoor scenarios, and planning experiments on a real robot facing surfaces with different friction.

Tables

Table 1. TABLE I: Normal distribution parameters of each material’s coefficient of friction, measured manually on the robot foot

Material	${\mu }_i$	${\sigma }_i$
Asphalt	0.74	0.12
Grass	0.53	0.10
Rock	0.80	0.08
Carpet	0.82	0.02
Ceramic	0.97	0.05
Marble	0.83	0.15
Metal	0.80	0.15
Wood	0.88	0.12
Sand, Sky, Snow, Water, Cloth, Paper*	0	0

Equations

$$p(\mu |\theta ,I)=\sum _{i=1}^{m}p(\mu |l_i)P(x_k=l_i|\theta ,I).$$

$$\begin{array}{rl}c(s,s^{\prime })=& \underset{p}{\min }{\hat{f}}_{\mathrm{cost}}(s,s^{\prime },p)\\ & \text{subject to}\\ & P({\hat{f}}_{\mathrm{RCOF}}(s,s^{\prime },p)<{\mu }^{(k)})\ge \eta \\ & k=1,...,K,\end{array}$$

$$F_{{\mu }^{(k)}|\theta ,I}({\hat{f}}_{\mathrm{RCOF}}(s,s^{\prime },p))\le 1-\eta .$$

$${\hat{f}}_{\mathrm{RCOF}}(s,s^{\prime },p)\le Q_{1-\eta }^{(k)},$$

$$\begin{array}{rl}h(s)=& d_{xy}(s,s^{\mathrm{goal}}).\underset{s,s^{\prime },p}{\min }\frac{{\hat{f}}_{\mathrm{cost}}(s,s^{\prime },p)}{d_{xy}(s,s^{\prime })},\end{array}$$

$$\underset{q_1,...,q_T}{\mathrm{minimize}}$$

$$f_{\mathrm{stance}}(q_t,s_t)=0{\forall }_{t\in 1,...,T}$$

$$f_{xy}(q_t)\in {\mathcal{P}}_t{\forall }_{t\in 1,...,T}$$

$$f_{\mathrm{roll}}(q_t)=0{\forall }_{t\in 1,...,T}$$

$$A_t{q}_t\le b_t{\forall }_{t\in 1,...,T},$$

Full text

I INTRODUCTION

Legged and humanoid robot locomotion planning is an important problem for disaster response and service robots. One of the difficulties of this problem is the complexity of general environments and the need to consider several factors such as collision, energy consumption, surface geometry and friction. In this paper we deal with the specific problem of humanoid robot locomotion when environment friction is considered. Our claim is that friction and its uncertainty can be estimated from vision and robustly integrated into algorithms for motion planning with contact. We argue that even if the precise coefficient of friction cannot be predicted from vision before touching a surface, priors and accumulated experience associated with surface material or condition (think coefficient of friction tables) can provide a probability distribution of friction. Motion planning with contact can also become prohibitively expensive once multiple factors are considered, such as locomotion cost, collision and friction. In this paper we propose a hierarchical approach to the problem, where a footstep planner optimizes the same cost function as a full-body motion planner by use of an oracle, and considers collision and friction by using simple bounding box collision checks and an “extended footstep planning” [1] approach.

The contributions of this paper are the following:

• •

We propose a solution to the friction from vision problem using a state-of-the-art deep Convolutional Neural Network (CNN) architecture to predict broad material classes from images, together with known (or learned) distributions of material friction;

• •

We propose a hierarchical planning architecture for biped robots that optimizes the same objective at both levels, and deals with friction, stability, collision and cost to produce full-body trajectories;

• •

We show empirical friction prediction results, as well as planning experiments which show the usefulness and applicability of the approach in complex environments with varying friction.

II RELATED WORK

Recent full-body motion planning [2, 3] and control [2, 4, 5] algorithms for legged robots have started to consider friction by using friction cones in optimization problems. Such methods rely on the fundamental assumption that the coefficient of friction can be predicted in advance. While in itself a challenging problem, partial evidence from human visual perception motivate such an approach to the problem. For example, humans are known to use visual cues to estimate friction, related to surface texture [6], shine [7] and detection of materials or contaminants (e.g. water) [8]. Furthermore, in the human gait literature there is evidence that humans use accumulated previous experience to predict friction and adapt walking style before touching slippery ground [9]. In this paper, such “accumulated previous experience” is implemented as probability distributions of friction associated with material classes. The term “material” is used in a broad sense to refer to visually classifiable classes related to material, condition and context (e.g. “dry metal”, “wet asphalt road”). The use of material classes for prediction here is motivated by a recent study [10] which identifies material as one of the most predictive features of both coefficient of friction values and human judgements of friction.

Friction estimation work related to this paper includes that of Angelova et. al [11], which predicts the percentage of slip (i.e. lack of locomotion progress) of a rover from terrain classification and slope. Predictions are made based on non-linear regression of data gathered on a learning stage. Compared to [11], we estimate the coefficients of friction instead of slip, we use open segmentation datasets [12, 13] to train material classification, and we decouple the problem from the physical robot. Importantly, our approach allows for sharing material friction data among different robots as long as they have similar foot soles.

Several approaches exist to the friction-constrained motion planning problem for legged robots. One approach is the non-hierarchical, full-scale trajectory optimization formulation with implicit contact constraints of [14, 15]. While technically elegant and showing promising results, these can still be computationally expensive for online planning. In order to make the problem tractable, full-body motion can be planned after contact (or footstep) planning [16, 17], in what is called the contact before motion approach. One common issue with such methods is that contact planners do not take the same friction or trajectory criteria into account as the subsequent full-body planners. One exception to this lack of consistency between planning levels is the extended footstep planning work of [1], in which learned models are used at the footstep planning level that predict full-body feasibility. The approach can also account for friction constraints by using timing variables and learned slippage models at the footstep planning level. Still, in [1] the costs optimized at the footstep planning level are not further optimized at the full-body level. In this paper we improve the method by using an oracle at the footstep planning level which predicts the costs obtained by a full-body trajectory optimizer, thus increasing consistency across planning levels.

III FRICTION FROM VISION

In this paper we propose to estimate friction of surfaces from visual input by classifying surface material at each image pixel and assuming known (or learned) probability distributions of friction for each material. For convenience we will use the term “friction of a material” to refer to the coefficient of friction between the robot foot sole and a second surface of a given material.

We consider a pixel-wise labelling algorithm that, given an input image $I$ with $n$ pixels, provides a probability distribution $P(X|\theta,I)$, where $X=\{x_{1},...,x_{n}\}$ are the pixel labels and $\theta$ are internal parameters of the algorithm. Each pixel can take one of $m$ possible labels, such that $x_{k}\in\mathcal{L}=\{l_{1},...,l_{m}\}$. Furthermore, let each label be a material associated with a probability distribution function (p.d.f.) of a coefficient of friction $p(\mu|l_{i})$. Then at pixel $k$, the conditional p.d.f. of $\mu$ is

[TABLE]

For the results shown in this paper we estimated the friction distributions $p(\mu|l_{i})$ experimentally, by measuring maximum friction force of the robot foot on several surfaces for each material. We describe the procedure in more detail in Section V.

We use a deep convolutional neural network (CNN) to obtain pixel-wise material predictions $P(x_{p}=l_{i}|\theta,I)$. In particular we use the encoder-decoder architecture of [18], which achieves good results in image segmentation applications and is characterized by a low number of parameters. Its low number of parameters leads to fast inference, which is crucial for robotics. The architecture consists of an encoder network of 13 convolutional layers as in VGG16 [19], followed by a decoder network of 13 layers and a final softmax layer. The output of the last layer of the network (a softmax classifier) is at each pixel a vector of probabilities for each class, that is, the probabilities $P(x_{p}=l_{i}|\theta,I)$ used in equation (1).

IV HIERARCHICAL PLANNING

In this paper we plan full-body robot motion using a contact before motion approach. A footstep planner first searches a stance graph using transition costs provided by an oracle. The stances are then used as constraints in a full-body trajectory optimizer that considers full-body trajectory costs, collisions, joint limits and static stability. The obtained trajectory is finally interpolated and locally adapted for dynamic stability using a ZMP-based method. The oracle basically takes each stance transition and predicts the costs obtained at the end of the whole planning pipeline. This leads to footstep plans which optimize the same criteria as the full-body planner. See Figure 1 for a visual representation of the architecture.

IV-A Extended footstep planning with an oracle

The footstep planner searches a graph of stances to find a feasible path between the start and goal stance. Each node in the graph is a stance $s$, which is defined by a set of contacts with the environment. A contact is a tuple (link, position, rotation). A neighbor stance $s^{\prime}$ either adds or removes a contact with respect to $s$. In this paper we deal with biped walking only, and hence stances simply transition from double-support to left-foot-contact, to double-support, to right-foot-contact, back to double-support, etc. The advantage of this representation instead of, for example, double-support stances only, is that the swept-volume between consecutive stances can be used by the optimizer to guide a swing leg out of collision. Such an approach is also used by other works focusing on collision detection [20].

In this paper we use the extended footstep planning framework of [1]. The footstep planner is “extended” because extra parameters associated with stance transitions (e.g. step timing) are computed from the transition itself by a function learned offline. Here we call this function an oracle because it predicts the costs that will be obtained by a subsequent full-body trajectory optimizer.

We now briefly describe the footstep planning algorithm. We first discretize the search space by constraining contact positions to a point cloud, and rotations by aligning contact normals with the environment and constraining the links’ yaw orientation to a discrete set of values in the global coordinate frame. Then we use an A* variant, ARA* [21], to search the stance graph based on oracle costs. At each state (i.e. stance) of the graph $s$, a contact is either added or removed to generate successor stances. Contact removal generates one new stance. On the other hand, adding a new contact consists of doing a range search of points in a radius around the foot in contact. For each of those points, footsteps are placed at all yaw angles and checked for feasibility. We implement feasibility as empirical stance distance limits, as well as foot-foot and COM-environment collision checking using bounding boxes for the feet and trunk. The feasible stances are added as successors of $s$.

To find an optimal path to the goal state, A* search requires a state transition cost function $c(s,s^{\prime})$ and a heuristic cost-to-go function $h(s)$. Similarly to [1], we define the cost as

[TABLE]

where $k$ is an index of the contacts of $s$ and $s^{\prime}$, $\mu^{(k)}$ is the friction at these contacts, and $p$ are state transition parameters. Since in this paper we use two full-body posture waypoints per stance at the trajectory optimization level (Section IV), we set transition parameters $p=(\Delta t,\Delta t^{\prime})$. These are the time spent from the second waypoint of $s$ until the first waypoint of $s^{\prime}$, and the time spent from the first waypoint of $s^{\prime}$ to the second, respectively. The main difference in (2) with respect to [1] is that we consider uncertainty in the coefficient of friction variable by using chance constraints. RCOF stands for required coefficient of friction and corresponds to the maximum tangential-to-normal force ratio exerted over the whole trajectory [1]. Therefore, the constraints $P(\hat{f}_{\mathrm{RCOF}}(s,s^{\prime},p)<\mu^{(k)})\geq\eta$ in (2) implement robust Coulomb friction at each contact by forcing the inequalities to hold with at least probability $\eta$. The constraints can also be rewritten using the cumulative distribution function of (1) denoted by $F_{\mu^{(k)}|\theta,I}$,

[TABLE]

Since each $\mu^{(k)}$ is one-dimensional then $F$ can be inverted and the constraints rewritten in deterministic form

[TABLE]

where $Q_{1-\eta}^{(k)}$ is the $(1-\eta)$-quantile of $F_{\mu^{(k)}|\theta,I}$, which can be computed by an integral of (1) over $\mu$.

Regarding the heuristic cost-to-go function of A* search, as in [1], we set it to

[TABLE]

where the function $d_{xy}(.,.)$ computes Euclidean distance on the horizontal plane between two stances (i.e. the distance between left feet and right feet summed). The heuristic (5) is a lower bound on the cost-of-transport times distance, which guarantees that $h(s)$ does not overestimate the total cost to the final stance $s^{\mathrm{goal}}$ (i.e. is admissible, a necessary condition for A* optimality).

In this paper, the functions $\hat{f}_{\mathrm{cost}}$ and $\hat{f}_{\mathrm{RCOF}}$ are given by an oracle which predicts the value of $f_{\mathrm{cost}}$ and $f_{\mathrm{RCOF}}$ obtained at the end of the whole planning pipeline. Notice that in this paper, contrary to [1], $f_{\mathrm{cost}}$ is the function that will be optimized at the full-body trajectory optimization level. We implement $\hat{f}_{\mathrm{cost}}$ and $\hat{f}_{\mathrm{RCOF}}$ as hash tables. The tables are filled offline, by feeding the whole planning pipeline (i.e. trajectory optimization, interpolation, dynamic stabilization) with uniformly distributed samples of $(s,s^{\prime},p)$ as shown in Figure 1. The discrete optimization problems in (2), (5) are then solved for a large number of discretized stances and coefficient of friction quantiles and stored in new hash tables for fast access to costs and heuristics during search.

IV-B Full-body trajectory optimization

The full-body trajectory optimizer takes a footstep plan with $N$ stances and produces a full-body trajectory, parameterized by $T$ discrete-time waypoints. Waypoints are full-body robot configurations $q_{t}\in\mathbb{R}^{D}$, $t=1,...,T$, where $D$ is the number of degrees-of-freedom consisting of the joints’ angle values and the pose of the robot base. Each stance is associated with 2 full-body postures (at start and midstance) and so $T=2N$. For convenience we use $s_{t}$ to refer to the stance associated to $q_{t}$.

Our optimizer solves the problem

[TABLE]

where $q_{1},...,q_{T}$ are the optimization variables, $\alpha$ is a penalty constant and:

• •

The function $f_{\mathrm{cost}}$ computes the sum of the squared static torques of all joints at all waypoints, as implemented in the trajopt library [22]

• •

The function $f_{\mathrm{collision}}$ is a collision cost as proposed by [22]. It is the sum of a discrete collision cost computed by the signed distance between each link and all other geometries, and a continuous collision cost computed by the signed distance between the swept volume of each link with the environment

• •

The function $f_{\mathrm{stance}}(q_{t},s_{t})$ computes the pose error of all links in contact as a $6C$-dimensional vector where C is the number of active contacts in $s_{t}$. This is computed as the translation and axis-angle error between the target link pose (given by $s_{t}$) and the current link pose (given by $q_{t}$)

• •

The function $f_{xy}(q_{t})$ computes the (x,y) coordinates of the COM, and $\mathcal{P}_{t}$ is the support polygon of $s_{t}$. The constraint thus enforces approximate static stability. The support polygon of $s_{t}$ is computed by the convex hull of the horizontal projection of links in contact and does not include contacts removed in $s_{t+1}$

• •

The function $f_{\mathrm{roll}}(q_{t})$ computes the rotation around the X axis for the waist link, with respect to the global reference frame. This constraint is necessary as “zero roll” is an assumption of the subsequent dynamic stabilization method (Section IV-C)

• •

$A_{t},b_{t}$ enforce joint angle and velocity limits.

We solve problem (6) using the Sequential Quadratic Programming method of [22] as implemented in the trajopt library111URL: http://rll.berkeley.edu/trajopt.

IV-C Interpolation and stabilization

To obtain a densely-sampled trajectory for execution on the robot, we interpolate trajectory waypoints using Hermite cubic splines with derivatives set to zero for smooth contact transitions. The time between two consecutive waypoints $q_{t}$ is given by the oracle, as we describe in Section IV-A.

Since the obtained trajectory is not dynamically stable, we then apply an FFT-based ZMP trajectory compensation scheme [23]. The method considers the rigid-body dynamics of the full body and locally adapts COM motion on the horizontal plane using analytic inverse kinematics to iteratively reduce the error between the real and reference ZMP trajectory. We set the reference ZMP trajectory to the interpolated $f_{\mathrm{xy}}(q_{t})$, which were used in the optimization problem (6) and are inside the support polygon at each waypoint. Furthermore, our implementation of the analytic inverse kinematics of the robot WABIAN-2 assumes zero roll angle of the waist link with respect to the world reference frame. We include this constraint in the optimization problem (6) for consistency.

V RESULTS

V-A Material segmentation results

To train the CNN we first collected 7,791 annotated images from publicly available semantic-segmentation datasets: 5,216 from the VOC2010 Context dataset [13] and 2,575 from the OpenSurfaces dataset [12]. We selected all images in the datasets with at least one of the following labels: asphalt, concrete, road, grass, rock, sand, sky, snow, water, carpet, rug, mat, ceramic, tile, cloth, fabric, marble, metal, paper, tissue, cardboard, wood. Due to similarity between some classes at the image and semantic level we joined the labels (asphalt, concrete, road), (carpet, rug, mat), (ceramic, tile), (cloth, fabric) and (paper, tissue, cardboard). The total number of considered classes in the output CNN layer was 14. Sky was only included to avoid classifying it as any of the other materials on outdoor pictures.

We used stochastic gradient descent with 0.1 learning rate and 0.9 momentum as in the original SegNet publication [18], and trained the network on an Amazon Elastic Cloud node with a 4GB NVIDIA GPU. We ran a total of 90,000 iterations with a mini-batch size of 5 (maximum allowed by the GPU). Training was done on 60% of the images, while the other 40% were used as the test set.

We obtained a global classification accuracy of 0.7929 and class-average accuracy of 0.4776 on the test set. See Figure 2 for examples of the (highest probability) material predictions given by the CNN on the test set. The global accuracy is comparable to state-of-the-art performance in semantic segmentation (e.g. [18, 24]), and the class-average accuracy is slightly below state-of-the-art (which is around 0.60 [18]). We believe one important way to improve classification accuracy is to improve the dataset itself since, for instance, there is moderate visual similarity between some of the materials such as marble and ceramic, and some materials are lowly sampled (e.g. the lowest sampled materials are snow and sand, present in 173 and 46 images respectively).

The material segmentation results in Figure 2 show an overall good accuracy of the CNN, particularly on wood, grass and sky labels. The figure also shows typical misclassifications such as white walls recognized as sky or metal (picture 6), hard snow as rock (picture 7), and some overlap between asphalt/road, ceramic and marble. These are arguably understandable since material labels themselves semantically overlap. However, our approach to the visual friction estimation problem is such that if there is uncertainty in the material label, then this uncertainty can be used to weight the friction of the surface through material and friction probability distributions (Section III).

V-B Friction prediction results

We empirically measured the coefficient of friction associated with each material label using a force gauge and the robot foot loaded with a 1.5kg mass. The foot is rigid and its sole is covered with a high stiffness soft material for shock absorption and an anti-slippage sheet. We checked whether surfaces were horizontal with a level, then placed the foot and measured maximum friction force with the force gauge. See Figure 3 for an illustration of the procedure. We took 5 friction measurements on each surface, and used at least 3 surfaces of each material. We fitted a normal distribution to the measurements, obtaining separate parameters $\mu_{i}$ and $\sigma_{i}^{2}$ for each material, where $\mu_{i}$ is the mean friction of material $i$, and $\sigma_{i}^{2}$ the variance. The materials sand, snow, water, cloth, paper were an exception, and since our robot is currently not capable of walking on them (i.e. fall or damage risk is too high) we directly set them to $\mu_{i}=0$, $\sigma_{i}^{2}=0$. We similarly set sky’s friction to zero as well. See Table I for the parameters of the friction p.d.f. of each material.

In Figure 2 we show the test-set’s highest probability material predictions along with the $(1-\eta)$-quantile of the coefficient of friction which is used in equation (4). We set a typical value of $\eta=0.95$. The friction images are darker where friction is higher ($\mu=1$ would be black). Note that ceramic-like surfaces have high predicted friction (pictures 1, 5, 6); beds and jackets have very low friction (pictures 3, 4, 14); grass patches have lower friction than roads (pictures 2, 9, 11, 12, 13); and that water is mostly white - zero friction - (pictures 10, 12).

The figure also shows the advantage of using the whole probability distribution of materials (instead of using the highest probability material) to estimate friction. For example in picture 14, the jacket on the ground is classified as cloth and rock depending on the region, but friction is low on most of the object’s area since the cloth label still has high probability.

V-C Planning results

We prepared a mock-up scenario in the laboratory which demonstrates the capabilities of our planner. The scenario consists of a floor with two areas of different materials. One is made of wood ($\mu=0.84$) and the other is a high-friction flooring resembling ceramic tiles both in appearance and coefficient of friction ($\mu=1.00$). The perception-planning algorithms were run on this scenario, and then a piece of cloth (T-shirt) was laid flat on one of the surfaces to provoke changes in friction and force a different plan. See Figure 4 for the scenario, segmentation and friction as seen from the robot’s camera at the initial condition. Once again, the figures show the advantage of using the full probability distribution of materials given by the CNN. While cloth is the highest-ranking material only in part of the object region, friction is low on a larger region which is highly consistent with object borders.

The robot starts in double-support, with one foot on each surface. The goal stance is one meter ahead, also with a foot on each surface. After the robot is placed at the initial state, the perception and planning algorithms run without any human input except the push of a button to execute the planned full-body trajectory open-loop. Trajectory optimization parameters are the collision penalty weight $\alpha$ of equation (6), which is set to 50, and the distance at which the collision penalty starts being applied (for all links except those in contact), which we set to 2.5cm. The obtained full-body trajectory is tracked by position control at the joint level.

For these experiments we used the human-sized humanoid robot WABIAN-2 [25] customized with a Carnegie Robotics Multisense SL sensor-head. The perception pipeline predicts pixel-wise material label distributions and pixel-wise friction using SegNet [18] and equation (1), and combines them with the stereo depth maps computed onboard by the Multisense. It produces friction-annotated point clouds at 2Hz. For collision checking, the point cloud is converted into a mesh using the fast surface reconstruction algorithm of [26] as implemented in PCL [27]. All perception and planning computation ran on an external PC with network connection to the robot’s onboard PC, and we used ROS [28] for communication.

We show the results of the perception-planning experiments in Figure 5. From left to right, we show the material and friction point clouds, the footstep plan, the collision-checking bounding boxes used by the footstep planner and the final planned full-body trajectory after optimization and stabilization.

In the first situation there are only wood and ceramic surfaces, but the predicted lower bound of friction of the wood surface is lower than that of the tiles ($Q_{1-0.95}=$0.1 vs 0.4). The footstep planner returns a sequence of stances that reduces the amount of times wood is stepped on. This behavior comes naturally from the extended footstep planning approach [1], since walking on low friction ground requires higher stance times (slower motion) and thus more energy cost. Furthermore, note that the trajectory optimization uses all degrees-of-freedom to satisfy the constraints (e.g. trunk roll use is clear in the image sequence, important mainly for the stability constraints), and that the knees are relatively stretched in order to reduce torque consumption but still satisfy stability constraints. Also note that swing leg clearance happens automatically due to the use of collision costs.

In the second situation we laid a flat piece of cloth on a ceramic spot used by the previous trajectory. The cloth was correctly classified and its friction was practically zero. The footstep planner returned a trajectory around the cloth and on the wood surface, which led to a slightly longer time and energy cost of the full-body trajectory (63 vs 60 seconds, 5% longer than on the first situation). Note that while the times are long they correspond to 25 stances because of step length limits, and thus the average time per stance is approximately 2.5 seconds.

Footstep planning took approximately 20 seconds in the first situation and 10 in the second. The reason for the difference is clear from the scenario: while in the first situation stances on both surfaces are expanded by A* in order to guarantee optimality, in the second situation no stances are expanded on the surface with cloth since friction zero has infinite cost. Full-body trajectory optimization took approximately 40 seconds and dynamic stabilization 2 seconds. Note that these are for 25-stance, 60 second trajectories, and therefore they should be considerably faster in case planning is done one or two steps at a time.

VI CONCLUSIONS AND DISCUSSION

In this paper we proposed a complete solution to the problem of biped robot locomotion on slippery terrain. We developed both a visual friction estimation algorithm and an objective-consistent hierarchical planning method which considers trajectory costs, collision, stability and friction.

We empirically showed that friction estimates in our algorithm are more consistent with object/material borders than the highest-probability material label segmentation, which shows a good integration of segmentation uncertainty into friction estimation. We also showed that the algorithms work for varied terrain and are applicable to planning on a real robot. The algorithms are relevant since not only obstacles but also different terrain types abound in the real world, and locomotion choices should take them into account - whether for safety or energetic considerations.

Regarding the perception problem, we opted to decouple it into (broad sense) material segmentation and per-material friction distributions. Even though we obtained the material friction distributions manually, these could also be learned over time with locomotion experience. Alternatively, friction could also be learned from images directly by end-to-end training, for example by initialization of a CNN with the parameters obtained with our architecture.

For the context of this paper all surfaces were dry. Wet surfaces could also be included, although from our experience they should be treated as separate material labels (e.g. “dry metal” and “wet metal”) so that the distribution $\mu|l_{i}$ does not become bimodal. Thus, one important detail in this work is the notion of material, which should be taken in a broad sense, as a visually distinguishable terrain class.

Importantly, one problem with the proposed perception and planning approach is that wrong material classifications can lead to there being no solution to the footstep planning problem. An example of such a situation is when a material the robot cannot walk on, such as water in our case, is mistakenly given very high confidence. Our view is that the solution could be semi-supervision where a teleoperator can correct a segmented region’s material label. However, we believe that humans should not directly annotate COF, since despite their relative ability to adapt gait to slippery ground humans have difficulties in estimating coefficient of friction values [6, 10].

Finally, full-body trajectories in this paper were interpolated after trajectory optimization at waypoints. While we did this for implementation simplicity, one possible direction of improvement could be to use the spline representation directly in the optimization problem, using constraints at collocation points.