Bayesian Nonparametric Feature and Policy Learning for Decision-Making

TL;DR

This paper introduces a Bayesian nonparametric model for learning decision-making behaviors from demonstrations, focusing on latent features and policies to understand and predict actions in complex environments.

Contribution

It proposes a novel generative model that infers the number of features, their structure, and policies, enabling analysis and prediction of observed behaviors.

Findings

Effective in learning latent features from demonstrations

Accurately predicts actions in new states

Demonstrates real-world applicability in driver behavior modeling

Abstract

Learning from demonstrations has gained increasing interest in the recent past, enabling an agent to learn how to make decisions by observing an experienced teacher. While many approaches have been proposed to solve this problem, there is only little work that focuses on reasoning about the observed behavior. We assume that, in many practical problems, an agent makes its decision based on latent features, indicating a certain action. Therefore, we propose a generative model for the states and actions. Inference reveals the number of features, the features, and the policies, allowing us to learn and to analyze the underlying structure of the observed behavior. Further, our approach enables prediction of actions for new states. Simulations are used to assess the performance of the algorithm based upon this model. Moreover, the problem of learning a driver's behavior is investigated, demonstrating the performance of the proposed model in a real-world scenario.

Tables

Table 1. Table 1: Parameter settings for the algorithm used in the simulation experiments

Parameter	Value	Meaning
$h_{{\alpha }_{\sigma }}^{\left(1\right)}$	1000	​​​​​
$h_{{\alpha }_{\sigma }}^{\left(2\right)}$	1
$h_{{\beta }_{\sigma }}^{\left(1\right)}$	1
$h_{{\beta }_{\sigma }}^{\left(2\right)}$	1
$h_{{\alpha }_A}^{\left(1\right)},h_{{\alpha }_A}^{\left(2\right)}$	1	hyperparameters for ${\alpha }_a$
$h_{{\beta }_A}^{\left(1\right)}$	1	hyperparameter for ${\beta }_a$
$h_{{\beta }_A}^{\left(2\right)}$	10	hyperparameter for ${\beta }_a$
${\alpha }_{\gamma },{\beta }_{\gamma }$	1	hyperparameters for $𝐖$
$h_{{\varphi }_A}^{\left(1\right)},h_{{\varphi }_A}^{\left(2\right)}$	1	hyperparameters for $𝚽$
${\alpha }_s,{\beta }_s$	1	hyperparameters for $𝐒$
$P^{+}$	0.01	probability of accepting $K^{+}=1$ features
$T_{\text{corr}}$	0.9	threshold for merging similar features
$N_{\text{iter}}$	10000	number of iterations of the Gibbs Sampler
$L$	100	size of $𝒮$
$N_t$	1000	number of samples of the policy indicators

Table 2. Table 2: Confusion matrix for the hold-out data set of Scene 11. The overall accuracy is 74.32 % percent 74.32 74.32\% , where 31 features have been inferred.

	Prediction
Ground truth	Const. speed	Deceleration	Acceleration
Const. speed	12	2	4
Deceleration	1	15	6
Acceleration	5	1	28

Table 3. Table 3: Confusion matrix for the hold-out data set of Scene 20. The overall accuracy is 73.33 % percent 73.33 73.33\% , where 17 features have been inferred.

	Prediction
Ground truth	Acceleration	Lane change (right)	Const. speed
Acceleration	3	3	7
Lane change (right)	0	29	0
Const. speed	0	6	12

Equations

$${\mathrm{z}}_n={\mathrm{s}}_n\mathrm{F}+{ϵ}_n,$$

$$\mathrm{F}=\mathrm{A}\odot \mathrm{W}.$$

$$P(u|\mathrm{s},\mathrm{\Phi })\propto \sum _{k=1}^K{s}_{k}P(u|{\varphi }_k),$$

$$p(\mathrm{Z}|\mathrm{W},\mathrm{A},\mathrm{S},{\sigma }_{\text{z}}^2)=\prod _{n=1}^{N_{\text{z}}}{\mathcal{N}}_{{\mathrm{z}}_n}\left({\mathrm{s}}_n\left(\mathrm{A}\odot \mathrm{W}\right),{\sigma }_{\text{z}}^2\mathrm{I}\right).$$

$$p(\mathrm{W}|{\gamma }_{\text{w}})=\prod _{k=1}^K\prod _{d=1}^D{\text{Exp}}_{w_{k,d}}\left({\gamma }_{\text{w}}\right).$$

$$\begin{array}{rl}P(\mathrm{A}|{\alpha }_a,{\beta }_a)& :=\underset{K^{\star }\to \infty }{\lim }P({\mathrm{A}}^{\star }|{\alpha }_a,{\beta }_a)\\ & =\frac{({\alpha }_a{\beta }_a{)}^K}{{\prod }_{\mathrm{h}\in \{0,1{\}}^D\backslash 0}{K}_h!}\exp \{-\mathrm{bar}K\}\prod _{k=1}^K\text{B}\left(m_k,D-m_k+{\beta }_a\right),\end{array}$$

$$P(\mathrm{S})$$

$$\displaystyle\hskip 56.9055pt+P(s_{n,k}\neq 0\,|\,{\theta_{s=0}})(1-\delta\!\left(s_{n,k}\right)\,)\bigg{)}$$

$$\times p(\theta |{{\alpha }_s}_{=0},{{\alpha }_s}_{\ne 0})\mathrm{d}{\theta }_{s=0}$$

$$\propto \prod _{k=1}^K{\text{BetaBin}}_{{\mathrm{s}}_k}\left(m_{s=0,k}+{{\alpha }_s}_{=0},m_{s\ne 0,k}+{{\alpha }_s}_{\ne 0}\right),$$

$$P(u|{\varphi }_k)={\text{Cat}}_u\left({\varphi }_k\right),$$

$$P(u|\mathrm{\Phi },\mathrm{s})=\frac{1}{Z_u}\sum _{k=1}^K{s}_{k}P(u|{\varphi }_k),$$

$$p(\mathrm{\Phi })=\prod _{k=1}^K{\text{Dir}}_{{\varphi }_k}\left({\alpha }_{\varphi },\dots ,{\alpha }_{\varphi }\right),$$

$$P(\mathrm{u}|\mathrm{\Phi },\mathrm{S})=\prod _{n=1}^{N_{\text{z}}}\frac{1}{Z_{u_n}}\sum _{k=1}^K{s}_{k}P(u_n|{\varphi }_k).$$

$$\begin{array}{rl}p(\mathrm{W},\mathrm{A},& \mathrm{S},\mathrm{\Phi },{\sigma }_{\text{z}}^2,{\gamma }_{\text{w}},{\alpha }_a,{\beta }_a,{\alpha }_{\sigma },{\beta }_{\sigma },{\alpha }_{\varphi }|\mathrm{Z},\mathrm{u})\propto \\ & p(\mathrm{Z}|\mathrm{W},\mathrm{A},\mathrm{S},{\sigma }_{\text{z}}^2)P(\mathrm{u}|\mathrm{S},\mathrm{\Phi })p({\sigma }_{\text{z}}^2|{\alpha }_{\sigma },{\beta }_{\sigma })\\ & \times p(\mathrm{S})p(\mathrm{W}|{\gamma }_{\text{w}})P(\mathrm{A}|{\alpha }_a,{\beta }_a)p(\mathrm{\Phi }|{\alpha }_{\varphi })\\ & \times p({\gamma }_{\text{w}}|{\alpha }_{\gamma },{\beta }_{\gamma })p({\alpha }_{\sigma })p({\beta }_{\sigma })p({\alpha }_a)p({\beta }_a)p({\alpha }_{\varphi }).\end{array}$$

$$\Omega =\{\mathrm{W},\mathrm{A},\mathrm{S},\mathrm{\Phi },{\sigma }_{\text{z}}^2,{\gamma }_{\text{w}},{\alpha }_a,{\beta }_a,{\alpha }_{\sigma },{\beta }_{\sigma },{\alpha }_{\varphi }\}\in \mathbf{\Omega },$$

$$p({\sigma }_{\text{z}}^2|-)$$

$$+\frac{1}{2}\sum _{n=1}^{N_{\text{z}}}\sum _{d=1}^D{\left(z_{n,d}-\sum _{k=1}^K{s}_{n,k}{a}_{k,d}{w}_{k,d}\right)}^2).$$

$$\begin{array}{rl}P(s_{n,k}|-)& \propto p({\mathrm{z}}_n|\mathrm{W},\mathrm{A},{\mathrm{s}}_n,{\sigma }_{\text{z}}^2)P(u_n|{\mathrm{s}}_n,\mathrm{\Phi })P(s_{n,k}|\mathrm{S}\backslash s_{n,k}),\end{array}$$

$$P(s_{n,k}|\mathrm{S}\backslash s_{n,k})\propto \{\begin{array}{l}{m}_{s=0}+{\alpha }_{s=0}\text{for sn,k=0}\\ m_{s\ne 0}+{\alpha }_{s\ne 0}\text{for sn,k=0}\end{array},$$

$$p({\mathrm{w}}_k|-)$$

$$\propto {\mathcal{T}\mathcal{N}}_{{\mathrm{w}}_k}\left({\mu }_{{\mathrm{w}}_k},{\mathrm{\Sigma }}_{{\mathrm{w}}_k}\right),$$

$${\mathrm{\Sigma }}_{{\mathrm{w}}_k}^{-1}$$

$$\displaystyle\begin{split}\boldsymbol{\mathrm{\mu}}_{\boldsymbol{\mathrm{w}}_{k}}&=\boldsymbol{\mathrm{\Sigma}}_{\boldsymbol{\mathrm{w}}_{k}}^{-1}\Big{(}\frac{1}{\sigma_{\text{z}}^{2}}\sum_{n=1}^{N_{\text{z}}}s_{n,k}\Big{(}\boldsymbol{\mathrm{z}}_{n}-\sum_{\begin{subarray}{c}k^{\prime}=1\\ k^{\prime}\neq k\end{subarray}}^{K}s_{n,k^{\prime}}\left(\boldsymbol{\mathrm{a}}_{k^{\prime}}\odot\boldsymbol{\mathrm{w}}_{k^{\prime}}\right)\Big{)}\odot\boldsymbol{\mathrm{a}}_{k}-\boldsymbol{\mathrm{1}}_{D}\frac{1}{{\gamma_{\text{w}}}}\Big{)}.\end{split}$$

$$p({\gamma }_{\text{w}}|-)\propto {\text{IGa}}_{{\gamma }_{\text{w}}}\left({\alpha }_{\gamma }+\frac{KD}{2},{\beta }_{\gamma }+\frac{1}{2}\sum _{k=1}^K\sum _{d=1}^D{w}_{k,d}\right).$$

$$P(a_{k,d}|-)\propto p({\mathrm{z}}_d|\mathrm{S}{\mathrm{f}}_d,{\sigma }_{\text{z}}^2)P(a_{k,d}|{\mathrm{a}}_{k\backslash d}),$$

$$P(a_{k,d}=1|{\mathrm{a}}_{k\backslash d})=\frac{m_{k\backslash d}}{D+{\beta }_a-1},$$

$$q({\theta }^{+}|\theta )=q({\theta }^{+})=P(K^{+}|-)p(\mathrm{W}|{\gamma }_{\text{w}})P(\mathrm{S})p(\mathrm{\Phi }|{\alpha }_{\varphi })$$

$$P(K^{+}|-)\sim {\text{Poisson}}_{K^{+}}\left(\frac{{\alpha }_a{\beta }_a}{{\beta }_a+D-1}\right).$$

$$r=\frac{P(\mathrm{Z}|{\theta }^{+})}{P(\mathrm{Z}|\theta )}.$$

$$P(t_n=k|u_n,\mathrm{S},\varphi )\propto s_{k}P(u_n|{\varphi }_k).$$

$$p({\varphi }_k|\mathrm{u},\mathrm{t})={\text{DirMult}}_{{\varphi }_k}\left(m_{{\varphi }_k,1}+{\alpha }_{\varphi },\dots ,m_{{\varphi }_k,N_{\text{u}}}+{\alpha }_{\varphi }\right)$$

$$\begin{array}{rl}P(u^{\star }|& {\mathrm{z}}^{\star },\mathcal{D})={\int }_{\mathcal{S}}{\int }_{\mathbf{\Omega }}P(u^{\star },{\mathrm{s}}^{\star }|{\mathrm{z}}^{\star },\Omega )p(\Omega |{\mathrm{z}}^{\star },\mathcal{D})\mathrm{d}\Omega \mathrm{d}{\mathrm{s}}^{\star },\end{array}$$

Full text

Key words: Bayesian nonparametrics, decision-making, learning from demonstrations, feature learning, imitation learning

1 Introduction

Decision-making plays a crucial role in many applications, such as robot learning, driver assistance systems, and recommender systems [40]. A fundamental question in decision-making is how an agent can learn to make optimal decisions. Learning from an experienced teacher provides a natural means to solve this problem, without the need of explicitly defining rules for the desired behavior. Further, observing a teacher may provide a deeper understanding of the decision-making process. Therefore, Learning From Demonstrations (LFD) [1] has gained a lot of interest in the recent past.

According to [1], approaches for LFD can be grouped into (i) reward-based models and (ii) imitation learning. In reward-based models, it is assumed that the agent makes its decision based on a reward which is, in the context of LFD, learned from observations (as in Inverse Reinforcement Learning (IRL) [30]). In imitation learning, it is assumed that an experienced teacher can be observed. Thus, the policy, telling the agent how to act in a given situation, can be learned by understanding the direct relation between the teacher’s states and actions. Especially in reward-based approaches, problems defined on infinite spaces, where the state of an agent can take continuous values, are mostly intractable to solve and efficient approximations are needed. Only in the case of finite state and action spaces and known rewards, learning optimal policies has been solved [3]. A typical approach to facilitate this problem is the representation of the state space in a feature domain, c.f. [33, 5].

However, decision-making has been viewed only from a limited feature-based perspective, where features are usually designed by experts and mainly serve the purpose of reducing the size of the state space. We argue that, in many practical systems, the agent makes its decision based on a compact representation of the observed data, which can be considered as a projection onto a feature space of the decision-making problem. As an example, consider a person driving a car. The driver’s observations consists of, i.a., the location, speed, and acceleration of his vehicle and the surrounding vehicles, the type of the road and the weather conditions. However, the driver makes his decision based on a subset of the available information, e.g., on the time-to-reach between his and the other road users’ cars. This idea has also been investigated from a psychological point of view by the concept of discovering latent causes in human behavior, which is related to learning state space representations [12].

Assuming that a certain structure underlies the observations, we aim at inferring the latent features and build a feature-based representation of the states, yielding the following two advantages: first, the states can be represented compactly, rendering the decision-making problem much more efficient as compared to working in the original domain. Second, the features can be regarded as causes for the observed decisions, allowing for a deeper understanding of the observed behavior. In particular, we consider a LFD problem and propose a Bayesian nonparametric framework for feature learning in LFD. We assume that the policies depend on the features of the observed demonstrations. With this model, we are able to (i) significantly reduce the state space by (ii) learning the features as well as the number of features and (iii) provide a better understanding of what caused the teacher to take the observed actions.

In the next parts of this section, we formulate the problem and review the state-of-the-art for feature learning in LFD. In Section 2, we motivate and present the model for feature-based decision-making. The model is detailed in Section 3 and we explain how the number of latent features can be inferred from the data. Section 4 presents the inference scheme for the latent variables. The proposed algorithm is empirically evaluated with simulation experiments in Section 5, demonstrating its performance. Real data experiments are conducted in Section 6. We discuss our findings in Section 7 and end with a short conclusion in Section 8.

1.1 Problem formulation

The goal of this work is to introduce a feature-based LFD framework. While this model can be used for teaching an agent given observations of the desired behavior, the focus of this work lies on the analysis of the observed behavior by means of the features. Since we consider a decision-making task, the investigated problem can be modeled by means of a Partially Observable Markov Decision Process (POMDP) [21, 42], which is defined by

• •

a set of observations, $\mathcal{Z}$,

• •

a set of states, $\mathcal{X}$,

• •

a finite set of ${N_{\text{u}}}$ actions, $\mathcal{U}$,

• •

a transition model, which describes the probability of entering a state after taking an action in the current state,

• •

an observation model which explains how the observations are generated from the states,

• •

a discount factor, which penalizes long-term rewards,

• •

and a reward function, $R$.

As the main goal of this work is to provide a means to understand observed behavior, we consider an imitation learning approach and learn the relation between states and actions from the observations directly, where the reward and, hence, the discount factor, are not considered.

We assume that we (or an agent) have access to ${N_{\text{z}}}$ noisy observations, $\boldsymbol{\mathrm{z}}_{n}\in\mathcal{Z}$, of the states, $\boldsymbol{\mathrm{x}}_{n}\in\mathcal{X}$, with $n=1,\ldots,{N_{\text{z}}}$. In particular, we consider Gaussian noise, i.e., $\boldsymbol{\mathrm{z}}_{n}$, describes observations of the states, $\boldsymbol{\mathrm{x}}_{n}$, with additive Gaussian noise. Further, we assume that the actions, $u_{n}\in\mathcal{U}$, taken in the corresponding states, can be observed. The observations are assumed to be optimal in the sense that they represent the behavior of the agent seeking its goal, i.e., without any exploratory steps.

A simple approach to the considered problem would be the use of a feature extraction technique such as Principal Component Analysis (PCA) [20] or Non-negative Matrix Factorization (NMF) [23] to learn features from the observed states. Following this solution, features cannot be jointly learned and shared between different actions. Moreover, learning discriminative features is not guaranteed. Therefore, we argue that the features and policies need to be learned jointly such that a trade-off between feature sharing, promoting a compact model, and discrimination capability is found based on the observations.

1.2 Relation to existing work

In the following, we provide an overview about the current state of the art of feature learning for LFD. As IRL usually requires to solve an Markov Decision Process (MDP), which is usually done by means of a Reinforcement Learning (RL) algorithm, we start with an introduction in feature learning for RL.

1.2.1 Reinforcement Learning

Large state and action spaces are especially problematic for value-based RL algorithms such as value-iteration [3] or Q-learning [48] since the value function, representing the expected accumulated reward, needs to be approximated. In early approaches, a set of basis functions, often referred to as features, is linearly weighted to represent the this function [5]. However, there exists only little work on learning these basis functions. Riedmiller [38] has proposed the Q-fitted value iteration where the value function is approximated by means of a neural network, where the features are learned in the layers of the network. In [29], this approach has been extended by replacing the neural network by a deep-layered counterpart. A different concept to employ features is proposed by Hutter with the Feature Reinforcement Learning framework [18]. In this framework, the goal is to learn a feature mapping from the agent’s history (comprised of actions, states, and rewards) to a MDP state, enabling decision learning for infinite state spaces [8]. An alternative framework for learning the latent structure of the state space is proposed in [9] which is based on Factored Markov Decision Processs [4]. In the FMDP framework, it is assumed that the observed states can be represented compactly by exploiting the structure within the states, enabling efficient learning. This approach has been extended in [31] to an online approach, where the features are selected from a large set by means of Group LASSO. In these approaches, the inferred features mainly serve the purpose of dimensionality reduction. Therefore, the features do not necessarily possess a meaning that can be easily interpreted.

1.2.2 Inverse Reinforcement Learning

IRL is concerned with the problem of learning the reward function from observed behavior [30]. As in RL, features are often used to linearly parameterize the reward function, e.g., in [30, 17]. Recent attempts have been made to consider a Deep Learning (DL) architecture [49] for IRL, providing means for nonlinear, hierarchical feature learning.

A Bayesian nonparametric approach is proposed in [6], utilizing an Indian Buffet Process (IBP) to model feature activations. As the features of the reward function are assumed to be known, this approach can be understood rather as a feature selection than feature learning for IRL, where the number of features is inferred by the IBP. Different results on Bayesian nonparametrics for IRL, which is indirectly related to feature learning, are given in [27], where a partitioning of the state space is sought for, or [45], where complex behavior is decomposed into several, simpler behaviors that can be easily learned.

1.2.3 Imitation Learning

Instead of estimating the reward as in IRL, imitation learning aims at inferring the underlying policy directly [2, 44]. Usually, handcrafted features are used, e.g., in [37, 39]. Attempts to introduce new features are made in [36] as an extension of the maximum margin planning algorithm which is proposed in [37]. As explained in [1], imitation learning can be considered as a supervised learning task. Thus, feature selection and learning techniques developed for classification and regression can also be used in imitation learning. An excellent overview is given in [16]. Although these models work well in practice, they might not be able to provide a deeper understanding of the observed behavior as they do not explicitly model the states and actions.

2 Choice of the Model

In the first part of this section, we propose a feature model for LFD. In the second part, we explain the relevance of the transition model to the proposed framework. Since we assume a mixture of policies in this framework, we briefly discuss the intuition behind this assumption in the third part. Alternative models for feature learning for LFD are discussed in the forth part.

2.1 Feature model for learning from demonstrations

We assume a linear latent feature model, similar to NMF [23] and PCA [20]. Thus, the noisy observations, $\boldsymbol{\mathrm{z}}_{n}\in\mathbb{R}^{1\times D},n=1,\ldots,{N_{\text{z}}},$ are assumed to be composed of the latent features, ${\boldsymbol{\mathrm{F}}}\in\mathcal{F}^{K\times D}$, and the feature coefficients, $\boldsymbol{\mathrm{s}}_{n}\in\mathcal{S}^{1\times K}$,

[TABLE]

where $\boldsymbol{\mathrm{\epsilon}}_{n}$ represents Gaussian i.i.d. noise with variance $\sigma_{\text{z}}^{2}$ and the states are given as $\boldsymbol{\mathrm{x}}_{n}=\boldsymbol{\mathrm{s}}_{n}{\boldsymbol{\mathrm{F}}}$. The number of features is $K$ and the dimension of the observations is $D$. Clearly, the feature space, $\mathcal{F}$, depends on the application. In the following, we assume that the features are positive-valued, i.e., $\mathcal{F}=\mathbb{R}_{+}$. Following [22], the feature matrix is composed of a binary activation matrix, ${\boldsymbol{\mathrm{A}}}\in\{0,1\}^{K\times D}$, and a weighting matrix, ${\boldsymbol{\mathrm{W}}}\in\mathcal{F}^{K\times D},$ where the relation is given by the Hadamard product,

[TABLE]

The feature model in Eq. (2) can be easily extended to an infinite feature model by placing an IBP [13, 15] prior over ${\boldsymbol{\mathrm{A}}}$. The IBP assumes an infinite number of features, while the observed data can be explained by a finite number. This gives rise to a nonparametric model, where the number of features is implicitly modeled by means of the IBP. This is detailed in Section 3.

A fundamental difference to most existing work on decision-making using feature representations is that we assume that the agent makes its decision based on features, where each feature attracts the agent to take a specific action. We consider a (latent) linear feature model where the feature coefficients depend on the observed state and determine the actions. For this reason, we refer to the feature coefficients also as substates.

2.2 Transition model

In decision-making problems, the transition model explains which actions the agent can take in each state. Thus, the transition model acts as a constraint on the possible actions. As we focus on inferring the latent causes for the observed actions by means of features, the transition model is less relevant as it only provides additional information. Besides, the transition model is rarely known in practice. As we are given observations, we can, theoretically, infer the transition model. For this, we could either employ a parametric or a nonparametric model. Defining a parametric model for the transitions is not trivial, as the dynamics in the latent space may be highly nonlinear. Alternatively, a nonparametric model can be assumed. This, however, requires a large amount of observations for the estimation of the parameters, which is often not available. Assuming that the noise in the observations is low, we argue that we can reliably infer the substates from the corresponding observations, eliminating the need for a transition model.

2.3 Feature-based policy

As each feature imposes its on own policy, the probability of the agent taking an action, $u$, in a substate, $\boldsymbol{\mathrm{s}}$, is a mixture of the feature policies, $P(u\,|\,\boldsymbol{\mathrm{\phi}}_{k})$,

[TABLE]

where $\boldsymbol{\mathrm{\phi}}_{k}$, $k=1,\ldots,K$, are the parameters of the feature policies and ${\boldsymbol{\mathrm{\Phi}}}=[\boldsymbol{\mathrm{\phi}}_{1},\ldots,\boldsymbol{\mathrm{\phi}}_{K}]$. The mixture of policies can be interpreted either as a stochastic or a deterministic policy. In the first case, the action to be taken should be sampled according to Eq. (3). In the second case, simply the most probable action is taken. Mixture polices have been investigated in multi-objective problems, where an agent aims at reaching several objectives, some of which can even be conflicting [32, 46, 41]. A stochastic policy is needed in this framework to ensure that all goals can be satisfied. Similar problems occur if the problem at hand is described by a POMDP, where the true state of the agent is unknown. The uncertainty about the state of the agent is expressed by beliefs over states. Acting according to a stochastic policy maximizes the expected return [21].

In case of single agents, it has been shown that deterministic policies are optimal solutions for MDPs [34]. As explained in Section 2.1, we argue that we can reliably infer the substates. Thus, we assume a deterministic policy in the following, where the probability in Eq. (3) expresses our confidence about the actions given the states.

Note that we also assume that each feature imposes a deterministic policy such that we expect a strongly peaked distribution for the feature policies, expressing the confidence of the chosen action.

2.4 Alternative Feature-based Models

Of course, there are other possibilities to incorporate feature learning in LFD. We briefly discuss two alternatives with their potential advantages and drawbacks. For completeness, as mentioned in the introduction, in the most trivial setup, one could simply cluster the states according to the observed actions and then learn the features. This, however, has the significant drawback that the features cannot be shared between the clusters.

2.4.1 Unique coefficient model

Instead of assuming, as in our model, that the features determine the behavior of the agent, the feature coefficients can be clustered, where the clusters indicate the optimal actions given the coefficients. Thus, the clusters can be interpreted as latent substates. Supposing that the elements of the feature coefficients are binary-valued, we can convert each feature coefficient vector to a unique identifier, representing the cluster. Thus, instead of employing expensive clustering, a fast deterministic mapping from the binary coefficients to the cluster identifier can be used. However, as the identifiers must be unique for the cluster assignments, we can have at maximum $2^{K}$ different clusters in this setup, i.e., the number of clusters (and, hence, possible actions) is strictly limited by the number of features. An advantage of this model is that the relation between substates and policies can be nonlinear. However, a significant drawback is that this model suffers severely from errors in the inference of the substates. Consider the case, where, for instance, due to noise, one element of the substate is incorrectly set. This substate is then assigned to a new cluster, for which the policy must be inferred from potentially little data, yielding highly varying policy estimates. As in our approach, an IBP prior can be placed over the feature activations to infer the number of latent features.

2.4.2 Clustering-based approach

A clustering-based approach assumes that similar states can be grouped and result in the same behavior. This can also be understood as a single feature model, in which we assume that the observations can be described by a single feature, representing the clusters. Thus, the substates reduce to cluster indicators, i.e., they indicate which feature best represents the observed state. The number of latent states is then equal to the number of features. One possibility to infer this number is to utilize a Chinese Restaurant Process (CRP), giving rise to a Bayesian nonparametric model [27]. A similar model, where the state space is clustered according to the played actions, is proposed in [43].

2.4.3 Relation to the proposed model

Our model has the advantage, as opposed to the clustering-based approach and the unique coefficient model, that the features provide a means to understand the observed behavior. In contrast to the unique coefficient model, we neither require binary coefficients nor a clustering step. Compared to the clustering-based approach, our model is able to significantly reduce the number of latent features and policies, as the features can be shared by different states. However, our model suffers from the assumption of a linear relation between features, substates, and policies. This problem is alleviated in the other approaches, as the features are decoupled from the policies. Note that our model becomes similar to the clustering-based model at the price of higher computational costs, if each substate is represented by only one feature.

3 Bayesian Nonparametric Model for Feature Learning

In this section, we provide a general framework for Bayesian nonparametric feature learning for decision-making based on the model proposed in Section 2.1. In order to learn the structure, we assume that we are given a set of observations consisting of state-action pairs, $\mathcal{D}=\{(\boldsymbol{\mathrm{z}}_{1},u_{1}),\ldots,(\boldsymbol{\mathrm{z}}_{N_{\text{z}}},u_{N_{\text{z}}})\}$.

3.1 Observation likelihood and noise variance

The observations, $\boldsymbol{\mathrm{z}}_{n},n=1,\ldots,{N_{\text{z}}}$, are assumed to be conditionally independent. As we assume Gaussian noise in Eq. (1), the state likelihood can be expressed as

[TABLE]

with ${\boldsymbol{\mathrm{Z}}}={\begin{bmatrix}{\boldsymbol{\mathrm{z}}_{1}}^{\text{T}}&\ldots&{\boldsymbol{\mathrm{z}}_{N_{\text{z}}}}^{\text{T}}\end{bmatrix}}^{\text{T}}$ and ${\boldsymbol{\mathrm{S}}}={\begin{bmatrix}{\boldsymbol{\mathrm{s}}_{1}}^{\text{T}}&\ldots&{\boldsymbol{\mathrm{s}}_{N_{\text{z}}}}^{\text{T}}\end{bmatrix}}^{\text{T}}$. The variance of the noise, $\sigma_{\text{z}}^{2}$, is assumed to be Inverse-Gamma distributed with hyperparameters ${\alpha_{\sigma}},{\beta_{\sigma}}$. Further, we place hyperpriors on ${\alpha_{\sigma}}$ and ${\beta_{\sigma}}$, following Gamma distributions with hyperparameters ${h^{(1)}_{\alpha_{\sigma}}},{h^{(2)}_{\alpha_{\sigma}}},{h^{(1)}_{\beta_{\sigma}}}$, and ${h^{(2)}_{\beta_{\sigma}}}$.

3.2 Prior for the feature weights

As explained above, the prior probability of the feature weights, ${\boldsymbol{\mathrm{W}}}$, depends on the problem at hand. We consider i.i.d. positive-valued feature weights, $w_{k,d}$ with $k=1,\ldots,K$ and $d=1,\ldots,D$, and assume an Exponential prior,

[TABLE]

The scaling factor, ${\gamma_{\text{w}}}$, is assumed to be Inverse-Gamma distributed with hyperparameters ${\alpha_{\gamma}}$ and ${\beta_{\gamma}}$.

If the features are assumed to be real-valued, the prior can be modeled with a Gaussian distribution with straightforward modifications.

3.3 Prior for the feature activations

The feature activations are modeled by means of an IBP [13, 15], assuming an infinite number of features. In the following, we consider the two-parameter generalization [14] which allows to sample sparse as well as dense matrices.

The IBP is derived as follows. For a finite number of features, $K^{\star}$, the sums over the rows of the feature activation matrix, ${\boldsymbol{\mathrm{A}}^{\star}}\in\{0,1\}^{K^{\star}\times D}$, are assumed to follow i.i.d. Binomial distributions. Placing a Beta prior with hyperparameters $\frac{{\alpha_{a}}{\beta_{a}}}{K^{\star}}$ and ${\beta_{a}}$ over the parameter of the Binomial distribution and marginalizing over this parameter yields a Beta-Binomial distribution [13, 14]. Since we are interested in sampling from an infinite number of features, we consider the limit for $K^{\star}\rightarrow\infty$, resulting in the distribution of the activation matrix ${\boldsymbol{\mathrm{A}}}$ [13, 14],

[TABLE]

where the number of occurrences of the binary vector $\boldsymbol{\mathrm{h}}\in\{0,1\}^{D}$ is denoted by $K_{h}$ and B is the Beta function. Note that ${\boldsymbol{\mathrm{A}}}$ is a matrix with infinitely many rows. However, due to the sparsity assumption, only a finite number of the rows of the realizations contain active elements, denoted by $K$. Thus, we need to store only rows with active elements in memory, which can be understood as realizations of $K$ features, where the average number of active rows is given by $\mathrm{bar}{K}={\alpha_{a}}\sum_{d=1}^{D}\frac{{\beta_{a}}}{{\beta_{a}}+d-1}$ [13, 10]. The hyperparameters ${\alpha_{a}}$ and ${\beta_{a}}$ reflect our prior belief about the number of features and the sparsity of the matrix [14]. The probability of activating an element increases with both hyperparameters. As the number of expected features grows linearly with ${\alpha_{a}}$, ${\alpha_{a}}$ can be used to control the number of generated features. Considering the limits for ${\beta_{a}}\rightarrow 0$ and ${\beta_{a}}\rightarrow\infty$ shows that the expected number of features is limited to ${\alpha_{a}}D$. As the probability of an element of ${\boldsymbol{\mathrm{A}}}$ being active increases, ${\beta_{a}}$ can be understood as a means to control the sparsity of the realizations. We infer ${\alpha_{a}}$ and ${\beta_{a}}$ from the observations, assuming that they follow Gamma distributions, i.e., ${\alpha_{a}}\sim\text{Ga}_{{\alpha_{a}}}\!\left({h^{(1)}_{\alpha_{A}}},{h^{(2)}_{\alpha_{A}}}\right)\,$ and ${\beta_{a}}\sim\text{Ga}_{{\beta_{a}}}\!\left({h^{(1)}_{\beta_{A}}},{h^{(2)}_{\beta_{A}}}\right)\,$ [22].

3.4 Prior for the substates

As formulated in Eq. (1), we assume that the observations are composed of a mixture of features weighted by the substates. Similar to a FMDP333A fundamental difference between our model and a FMDP is that the substates in our model are, theoretically, not restricted to be finite as we do not need to enumerate over them., we restrict the domain of the substate elements, $s_{n,k}$, with $n=1,\ldots,{N_{\text{z}}}$ and $k=1,\ldots,K$, to take values from a finite set, $\mathcal{S}=\{\breve{s}_{1},\ldots\breve{s}_{L}\}$, where $L$ denotes the number of elements, to simplify inference. For convenience, we assume equidistant elements in $\mathcal{S}$ and set $\breve{s}_{1}=0$ and $\breve{s}_{L}=1$. Note that the limited range does not restrict the model, as the features can be scaled to fit the observations.

Further, we assume that the substates are sparse, meaning that each observation consists of only a few features such that only few polices determine the observed action. In particular, we consider a sparsity-promoting mixture prior on the substates, similar to a spike and slab model [28, 19]. The components are given by a Categorical distribution, where $P(s=0\,|\,\theta_{s=0})$ is the sparsity component and $P(s\neq 0\,|\,\theta_{s\neq 0})$ is the weight component. Note that all categories, except $\breve{s}_{1}=0$, have equal probability. We place a Beta prior, $p({\boldsymbol{\mathrm{\theta}}_{s}}\,|\,\frac{{\alpha_{s}}_{=0}}{2},\frac{{\alpha_{s}}_{\neq 0}}{2})$, with hyperparameters ${\alpha_{s}}_{=0}$ and ${\alpha_{s}}_{\neq 0}$, over the mixture weights ${\boldsymbol{\mathrm{\theta}}_{s}}=\{{\theta_{s=0}},{\theta_{s\neq 0}}\}$ with ${\theta_{s\neq 0}}=1-{\theta_{s=0}}$. Marginalizing over ${\theta_{s=0}}$ yields,

[TABLE]

where $\delta\!\left(s\right)\,$ returns one if $s=0$ and zero otherwise, $m_{s=0,k}$ counts the zero elements in $\boldsymbol{\mathrm{s}}_{k}$, and $m_{s\neq 0,k}={N_{\text{z}}}-m_{s=0,k}$.

3.5 Mixture of policies and action likelihood

Since we assume a finite set of actions, we consider a Categorical distribution for each mixture component,

[TABLE]

such that the mixture model can be written as

[TABLE]

with normalization constant $Z_{u}=\sum_{u\in\mathcal{U}}\sum_{k=1}^{K}s_{k}P(u\,|\,\boldsymbol{\mathrm{\phi}}_{k})$. The parameters of the policy of each mixture component follow Dirichlet distributions with identical hyperparameters, ${\alpha_{\phi}}$,

[TABLE]

where we assume independent policies. We consider the hyperparameter, ${\alpha_{\phi}}$, as a Gamma distributed variable with parameters ${h^{(1)}_{\phi_{A}}}{}$ and ${h^{(2)}_{\phi_{A}}}$.

As explained, we require a data set containing observed actions, $u_{n}$, $n=1,\ldots,{N_{\text{z}}}$. Assuming that the observed actions are independent, the likelihood is given as

[TABLE]

3.6 Joint posterior distribution

The full joint posterior distribution can be factorized as

[TABLE]

The conditional independences in this model are exploited in Section 4, where inference based on this model is explained. The structure of the posterior is illustrated as a graphical model in Fig. 1.

4 Inference

We consider the problem of learning the latent variables and the prediction of optimal actions for new observations as a Bayesian inference problem. Thus, we are interested in the joint posterior distribution in Eq. (7). Since the joint posterior is not directly tractable, we represent it by samples generated by means of Gibbs sampling. Therefore, in the first part of this section, we derive the conditional distributions of the variables. After explaining the sampling scheme in the second part, in the third part, we detail how to predict actions for new observations using this model.

For convenience, we use the bar symbol ($-$) in what follows to denote the set of conditional variables, i.e., all variables except the one that shall be sampled. The set of all latent variables is denoted by

[TABLE]

where $\mathbf{\Omega}$ denotes the joint space of the latent variables.

4.1 Conditional distributions

4.1.1 Sampling the noise variance

The hyperpriors of ${\alpha_{\sigma}}$ and ${\beta_{\sigma}}$ are conjugate to the prior of $\sigma_{\text{z}}^{2}$. Hence, the conditional $p(\sigma_{\text{z}}^{2}\,|\,-)$ is also Inverse-Gamma distributed,

[TABLE]

We use a Metropolis-Hastings algorithm with a Gamma proposal distribution to generate samples of the hyperparameters, ${\alpha_{\sigma}}$ and ${\beta_{\sigma}}$.

4.1.2 Sampling the substates

Sampling the substates, ${\boldsymbol{\mathrm{S}}}$, is simple thanks to the assumption of a finite space of the elements. The conditional consists of the state and action likelihoods as wells as the (conditional) prior,

[TABLE]

for all $s_{n,k}\in\mathcal{S}$, where ${\boldsymbol{\mathrm{S}}}\backslash s_{n,k}$ denotes the elements of ${\boldsymbol{\mathrm{S}}}$ without $s_{n,k}$. As the prior for the substates follows a Beta-Binomial distribution, the conditional is simply given as

[TABLE]

where $m_{s=0}$ and $m_{s\neq 0}$ are defined in Section 3.4.

4.1.3 Sampling the feature weights

Since the likelihood is Gaussian and the weights are i.i.d. following an Exponential distribution, the conditional of the $k$th row of the weight matrix, $\boldsymbol{\mathrm{w}}_{k}$, is a truncated Gaussian distribution,

[TABLE]

with $\mathcal{TN}_{\boldsymbol{\mathrm{w}}_{k}}\!\left(\boldsymbol{\mathrm{\mu}}_{\boldsymbol{\mathrm{w}}_{k}},\boldsymbol{\mathrm{\Sigma}}_{\boldsymbol{\mathrm{w}}_{k}}\right)\,$ denoting a truncated Gaussian, where the elements of $\boldsymbol{\mathrm{w}}_{k}$ are constraint to be positive-valued. The mean, $\boldsymbol{\mathrm{\mu}}_{\boldsymbol{\mathrm{w}}_{k}}$, and the covariance, $\boldsymbol{\mathrm{\Sigma}}_{\boldsymbol{\mathrm{w}}_{k}}$, are given as

[TABLE]

We use the algorithm presented in [7] to sample from a truncated Gaussian distribution. Note that we neglect the dependency on the activations in Eq. (9) since the covariance would be infinite for $a_{k,d}=0$. This does not affect the sampling scheme, as the corresponding weight will be ignored due to $a_{k,d}=0$ anyway. Sampling the hyperparameter, ${\gamma_{\text{w}}}$, is fairly easy since the conditional of ${\gamma_{\text{w}}}$ is an Inverse-Gamma distribution,

[TABLE]

4.1.4 Sampling the feature activations

Sampling from the IBP consists of two steps: For each row, (i) the active columns are updated and then (ii) new features are proposed. In the first step, an element of the activation matrix, $a_{k,d}$, is set active with probability

[TABLE]

with $\boldsymbol{\mathrm{a}}_{k\backslash d}$ denoting the $k$th row of ${\boldsymbol{\mathrm{A}}}$ without the $d$th element and $\boldsymbol{\mathrm{f}}_{d}$ the $d$th column of ${\boldsymbol{\mathrm{F}}}$. As ${\boldsymbol{\mathrm{A}}}$ is assumed to follow an IBP, the conditional in Eq. (11) is [13, 14]

[TABLE]

where $m_{k\backslash d}$ is the sum over $\boldsymbol{\mathrm{a}}_{k\backslash d}$.

In the second step, $K^{+}$ new features are proposed in a Metropolis step [10, 22]. The proposal distribution, $q(\theta^{+}\,|\,\theta)$, is independent of the previous sample, $\theta$, as it consists of the priors for the substates, the feature weights, and the policies of the features,

[TABLE]

with $\theta=\{{\boldsymbol{\mathrm{W}}},{\boldsymbol{\mathrm{S}}},{\boldsymbol{\mathrm{A}}},{\boldsymbol{\mathrm{\Phi}}}\}$ and $\theta^{+}=\{{\boldsymbol{\mathrm{W}}}^{+},{\boldsymbol{\mathrm{S}}}^{+},{\boldsymbol{\mathrm{A}}}^{+},{\boldsymbol{\mathrm{\Phi}}}^{+}\}$, where ${\boldsymbol{\mathrm{W}}}^{+},{\boldsymbol{\mathrm{S}}}^{+},{\boldsymbol{\mathrm{A}}}^{+}$, and ${\boldsymbol{\mathrm{\Phi}}}^{+}$ denote the proposed feature weights, activations, coefficients, and policies. The probability of adding $K^{+}$ features is given as [13, 14]

[TABLE]

As the proposal distribution is independent of the previous sample, the acceptance ratio, $r$, is equal to the likelihood ratio between the new and existing features [26],

[TABLE]

Since the IBP tends to mix slowly, we augment this ratio with probability $P^{+}$ of accepting a single new feature, resulting in a modified acceptance ratio which is derived in [22]. This increases the probability of proposing new features, leading to a faster convergence to the stationary distribution. The hyperparameters ${\alpha_{a}}$ and ${\beta_{a}}$ are sampled as described in [22].

4.1.5 Sampling the policies

Sampling from the conditionals for the policies directly is difficult and would be computationally expensive due to the mixture model (Eq. (6)). An efficient approach is to introduce auxiliary variables, $t_{n}$, $n=1,\ldots,{N_{\text{z}}}$, for each observation, indicating from which policy the observed action, $u_{n}$, has been generated [25]. Given the indicators, the mixture components $\boldsymbol{\mathrm{\phi}}_{k}$, $k=1\ldots,K$, in Eq. (6) become conditionally independent of the mixture weights, $\boldsymbol{\mathrm{s}}_{n}$, $n=1,\ldots,{N_{\text{z}}}$, which makes sampling the components straightforward. The sampling algorithm thus consists of two steps.

First, the indicators are sampled according to

[TABLE]

In order to approximate the conditional for the policies, we draw $N_{t}$ indicator samples from Eq. (13). Drawing the samples is easy, since the indicators follow Categorical distributions with $t_{n}\in\{1,\ldots,K\}$.

Second, given the indicators, the parameters of the $k$th feature policy, $\boldsymbol{\mathrm{\phi}}_{k}$, is sampled from a Dirichlet-Multinomial distribution,

[TABLE]

where $m_{\phi_{k},i}$, $i=1,\ldots,{N_{\text{u}}}$, counts the co-occurrences between the policy indicators, $\boldsymbol{\mathrm{t}}$, and the actions, $u_{n}\in\mathcal{U}$.

The hyperparameter, ${\alpha_{\phi}}$, can be sampled in a Metropolis step, using a Gamma proposal distribution.

4.2 Sampling algorithm

The Gibbs sampler is initialized with only one feature and the variables are sampled from their prior distributions. After several iterations of the Gibbs sampler, samples from the target distribution are generated.

Note that there is the chance to generate new features in each iteration of the Gibbs sampler. Especially in scenarios with strong noise, different rows of the feature matrix may converge to similar realizations, increasing the number of features unnecessarily. We propose to merge features reducing the number of features, if they show a similarity larger than a prefixed threshold, $T_{\text{corr}}$, where we keep the activations of both features and average the policies. The similarity is measured by means of the estimated correlation between the feature samples.

A maximum-a-posteriori (MAP) estimator can be utilized, if we are interested in an estimate of the latent variables, $\Omega_{\text{MAP}}$, containing, i.a., estimates of the features, $\hat{{\boldsymbol{\mathrm{F}}}}_{\text{MAP}}$ [35], and the policies, $\hat{{\boldsymbol{\mathrm{\Phi}}}}_{\text{MAP}}$. The MAP estimator can be approximated by choosing the sample with the highest posterior probability. The posterior probability is calculated according to Eq. (7). For the prediction of actions for new observations, we can also use a Minimum Mean Squared Error (MMSE) estimator which provides better generalization capabilities. This is detailed in the next section.

4.3 Prediction of actions

The proposed model can be used to learn the structure of the observed states as well as for the prediction of actions, given new observed states. For the prediction of an optimal action, $u^{\star}$, given a new observation, $\boldsymbol{\mathrm{z}}^{\star}$, we can evaluate the posterior predictive distribution, giving rise to a MMSE estimator,

[TABLE]

where we exploit that $u^{\star}$ is independent of the data set containing the observations, $\mathcal{D}$, given $\Omega$. Eq. (14) shows that $\Omega$ depends on the new observations, $\boldsymbol{\mathrm{z}}^{\star}$. Thus, all variables would need to be inferred for each prediction, which would be computationally expensive. To remedy this issue, we assume that the observed data in $\mathcal{D}$ sufficiently represents the conditional distribution for $\Omega$, such that we can ignore the dependency and simply infer $\Omega$ based on $\mathcal{D}$, leaving only $u^{\star}$ and $\boldsymbol{\mathrm{s}}^{\star}$ to be inferred during prediction. The marginalization over $\Omega$ is approximated by Monte Carlo integration. For this, we need to draw samples of $\boldsymbol{\mathrm{s}}^{\star}$ given the samples of $\Omega$. The samples can be generated by drawing from the conditional in Eq. (8). Alternatively, we obtain a MAP estimate of $u^{\star}$ by maximizing the posterior predictive distribution with respect to $\boldsymbol{\mathrm{s}}^{\star}$ given $\Omega_{\text{MAP}}$.

Since, as justified in Section 2.3, we assume a deterministic policy, the optimal action, $u^{\star}_{\text{opt}}$, is the action that maximizes $P(u^{\star}\,|\,\boldsymbol{\mathrm{z}}^{\star},\mathcal{D})$. Thus, the posterior predictive distribution expresses our confidence about the actions and can be considered as the policy of the agent.

Note that, especially when we are interested in the prediction of actions, a modification on the model can help to increase the prediction accuracy dramatically. Especially with high-dimensional observation, the observation likelihood determines the posterior, where the effect of the action likelihood nearly vanishes, leading in the worst case to a neglect of the actions. A remedy consists in considering an action variable for each entry of the observation. However, we assume that these actions variables are identical. This increases the influence of the action log-likelihood by factor $D$, increasing the weight on the posterior significantly. The necessary modification in the inference algorithm are simple: First, for the conditional of Eq. (8), the action log-likelihood is multiplied by $D$. Second, the same modification is applied to the joint posterior distribution in Eq. (7). Sampling the policies is not affected as the actions for each substate are identical. We apply this modification on the model for the real data experiments in Section 6, as the observations in the experiments are high-dimensional.

5 Experimental Results

In order to demonstrate the performance of the proposed method, we consider simulations as well as real data experiments. In this section, we evaluate the performance of the inference algorithm by simulating observations with different SNRs and different latent numbers of features. For this, we simulate the ground truth values by drawing samples from distributions which are similar to the prior distributions of the variables. This is detailed in the following. The true hyperparameters of the features weights are set to ${\check{h}_{\alpha_{\gamma}}}={\check{h}_{\beta_{\gamma}}}=100$. Thus, the true hyperparameter for the weights, $\check{\gamma}_{\text{w}}$, as well as the true weights, $\boldsymbol{\mathrm{\check{W}}},$ are sampled from their prior distributions. An element of the true feature activation matrix, $\check{{\boldsymbol{\mathrm{A}}}}$, is activated with probability $P(\check{a}_{k,d}=1)=0.5$. The true substates, $\check{\boldsymbol{\mathrm{s}}}_{n}$, are simulated by drawing samples from a Dirichlet distribution with parameters $\frac{1}{K}\boldsymbol{\mathrm{1}}_{K}$, resulting in a peaked distribution. The true noise variance, ${\check{\sigma}_{\text{z}}}^{2}$, is determined by the chosen SNR, where the signal power is estimated from $\boldsymbol{\mathrm{\check{X}}}=\check{\boldsymbol{\mathrm{S}}}\boldsymbol{\mathrm{\check{F}}}$. The parameters of the ground truth policies, $\check{\boldsymbol{\mathrm{\Phi}}}$, are chosen such that one action has high probability mass, reflecting deterministic policies, as justified in Section 2.3. In all simulations, we consider ${N_{\text{z}}}=100$ observations of dimensionality $D=30$ with ${N_{\text{u}}}=4$ different actions. In order to evaluate the prediction performance of the model, we split the data set into a training and a test data set, leaving 80 observations for training and 20 for testing. We sweep the SNR from $10\text{\,}\mathrm{d}\mathrm{B}$ to $30\text{\,}\mathrm{d}\mathrm{B}$ with a step size of $5\text{\,}\mathrm{d}\mathrm{B}$ and vary the number of features $K$ within the set of $\{5,7,9,12,15,18\}$. We set the hyperparameters of the algorithm according to Tab. 1.

We organize the evaluation in three parts. First, we investigate the performance of the estimation of the features, the feature coefficients, and the reconstruction of the states. Second, we compare the estimated policies to the true policies and evaluate the prediction in terms of the accuracy, $A_{\text{u}}$, which is the average rate of correctly predicted actions of the test data set. These estimates are obtained utilizing a MAP estimator. As for the prediction of actions we can also use the MMSE estimator, as detailed in Section 4.3, we also discuss the results obtained by this estimator. Third, we provide results for the inferred number of features and compare it with the true value.

5.1 Estimation of the features

The quality of the MAP estimates is evaluated in terms of the Mean Squared Error (MSE) for the elements of the features, ${\boldsymbol{\mathrm{F}}}_{\text{MAP}}$, the substates, ${\boldsymbol{\mathrm{S}}}_{\text{MAP}}$, and the reconstructions, $\boldsymbol{\mathrm{X}}_{\text{MAP}}={\boldsymbol{\mathrm{S}}}_{\text{MAP}}{\boldsymbol{\mathrm{F}}}_{\text{MAP}}$. Further, we compute the RMSEs over 20 Monte Carlo runs, yielding RMSE measures for each estimated variable, $\epsilon_{\boldsymbol{\mathrm{F}}}$, $\epsilon_{\boldsymbol{\mathrm{S}}}$, and $\epsilon_{\boldsymbol{\mathrm{X}}{}}$. The results are shown in Fig. 2(a)–(c).

We obtain good results with low errors especially for a small latent number of features, almost independent of the SNR. With increasing $K$, we observe a strong growth of the error. The error becomes even more significant in case of strong noise. It is remarkable that the error of the estimated feature values as well as the feature coefficients behave similarly. Due to the linear relation between the features and the substates, the errors of the reconstructions also grows with the number of features.

5.2 Action prediction

We evaluate the correctness of the estimated policies in terms of the RMSE of the Mean Absolute Deviation (MAD) over 20 Monte Carlo runs. As shown in Fig. 2(d), the error slightly grows with an increasing number of features. Again, the SNR has, compared to the number of features, only little effect on the accuracy of the policies.

The accuracy of the action prediction using the MAP and MMSE estimators are depicted in Fig. 2(e). In case of few features ($K<15$) and low noise ($\text{SNR}>$20\text{,}\mathrm{d}\mathrm{B}$$), we obtain highly accurate predictions with over $90\text{\,}\%$ accuracy. Only for strong noise and many latent features the accuracy drops slightly below $70\text{\,}\%$. This observation can be explained by the relation between the action and substates, which are challenging to infer in case of many features.

Since we assume in the simulation experiments that there is a fixed set of parameters that explains the observations, the MMSE estimator yields only minor improvements over the MAP estimator concerning the accuracy of predicted actions, as indicated by the dotted lines in Fig. 2(e).

5.3 Estimation of the number of features

Assuming that the observations have been generated by a fixed number of features, we evaluate how accurately the algorithm is able to infer this number. The results for the simulations are depicted in Fig. 2(f), showing the RMSE of the MAD over the 20 Monte Carlo runs. As can be observed, the MAP estimator is able to infer the correct number of features reliably, especially in case of few features. On the one hand, if the noise in the observations increases and the observations are based on many latent features, the error grows significantly. On the other hand, if the SNR is reasonably high, i.e., $\text{SNR}>$15\text{,}\mathrm{d}\mathrm{B}$$, the results deviate on average by only four from the true number in case of 18 latent features. The error in estimating the number of features is probably due to the fact that some simulation examples can be explained by less features than used for their generation.

6 Real Data Experiments

We consider the problem of analyzing a driver’s behavior, which is an important task for user-adaptive driver assistance systems [24, 47], in order to demonstrate the performance of the proposed model in a real-world scenario. For this, we observe the surrounding of the vehicle and the actions taken by the driver, aiming at learning what caused the driver to make the observed decisions. Using the proposed model, we can also predict which action the driver is likely to take given a certain situation. Thus, we also investigate the predictive performance of our approach by randomly creating training and test data sets. For this, we consider real data provided by the KITTI Vision Benchmark Suite [11] containing several challenges in urban driving. We use the data for the tracking challenge as it contains time-sequential LIDAR measurements of different situations in public road traffic. A schematic plot of the setup is illustrated in Fig. 3. We consider Scene 11 and 20 of the benchmark suite, which are detailed below in Section 6.1 and Section 6.2. Before running the proposed inference algorithm, we pre-process the data as follows. First, we apply a thresholding on the height values of the measurements such that we keep only samples above ground level, which is roughly $1.5\text{\,}\mathrm{m}$ below the LIDAR. Afterwards, we discretize the measurements to obtain positive-valued occupancy grids, where the values of the grid elements refer to the maximum measured height.

As observation in the $n$th time frame, we consider the joint occupancy grid of the current and the previous frame. This is required to implicitly include velocity information, which enables to decide, e.g., if the host vehicle is faster or slower than the vehicle in front. Thus, the minimum speed, $v_{\min}$, between the host vehicle, $H$, and an obstacle, $O$, that can be resolved depends on the resolution, $R$, of the grid as follows,

[TABLE]

where $d_{n}$ denotes the distance between the host vehicle and the obstacle in the $n$th time frame. The time stamp is denoted by $t_{\text{F},n}$ and ${f_{\text{s}}}$ is the frequency at which the measurements are sampled (in the KITTI data set, ${f_{\text{s}}}=$10\text{,}\mathrm{Hz}$$).

We consider the environment of the vehicle in the range from $-3\text{\,}\mathrm{m}$ to $3\text{\,}\mathrm{m}$ in the lateral direction. This covers the lane of the vehicle plus parts of (if existent) neighboring lanes. In Scene 11, the longitudinal range is limited between $-10\text{\,}\mathrm{m}$ and $30\text{\,}\mathrm{m}$ and for Scene 20 between $-40\text{\,}\mathrm{m}$ and $40\text{\,}\mathrm{m}$. The range in longitudinal direction gives a time window of more than $2\text{\,}\mathrm{s}$ to react to the observed situation when driving at $50\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$, which is the speed limit in German cities.

For both scenes, we choose a grid size of $21\times 65$ pixels, resulting in a spatial resolution in lateral direction of $R_{\text{lat}}\approx$0.3\text{,}\mathrm{m} and in longitudinal direction of $R_{\text{long}}\approx$0.6\text{\,}\mathrm{m} for Scene 11 (and $R_{\text{long}}\approx$1.2\text{,}\mathrm{m} for Scene 20). Thus, the minimum velocities that can be detected are $v_{\text{lat},\min}>$3\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1} in the lateral direction and $v_{\text{long},\min}>$6\text{,}\mathrm{m}\text{,}{\mathrm{s}}^{-1} in the longitudinal direction (and $v_{\text{long},\min}>$12\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1} for Scene 20).

We labeled the scenes to obtain the observed actions by means of the measured acceleration of the host vehicle, obtained from the onboard inertial measurement unit. The ground truth for Scene 11 is depicted in Fig. 4(a) and for Scene 20 in Fig. 4(b). The set of actions, $\mathcal{U}$, we consider in this experiment is comprised of acceleration (A), deceleration (D), lane change (right) (R), and moving at constant speed (C). However, due to the short duration of the sequences, usually fewer actions are observed in each scene.

We use the same settings for the hyperparameters as in Tab. 1. Only one of the two hyperparameter for the IBP, ${h^{(2)}_{\alpha_{A}}}$, is modified to reduce the number of expected features, ${h^{(2)}_{\alpha_{A}}}=10$. Further, we perform 500 iterations of the Gibbs sampler. In both data sets, after a mere 100 iterations, the sampler converges to the stationary distribution and produces samples from the target distribution.

6.1 Scene 11 - Traffic jam

Scene 11 of the KITTI data set shows an urban scenario, in which the driver follows another vehicle. The vehicle in front cannot be overtaken due to a single-lane road. As shown in Fig. 4(a), the driver first accelerates. After a few seconds, the driver has to decelerate until the car stops due to a halt of the preceding car. When the vehicle in front starts moving again, the host vehicle accelerates.

Applying the proposed algorithm to the observations results in 31 features using the MAP estimator. For the analysis of the features, we only visualize three of the most relevant features, which are selected according to the confidence in the corresponding feature policy.

Considering the current and the previous frames as inputs, we obtain feature estimates of consecutive frames. Observing the differences between the frames can be difficult as the features are likely to be highly similar due to the low temporal difference. Therefore, we show the feature of the current time frame and the difference of the features. A difference plot can be interpreted as the temporal gradient of two consecutive frames. Thus, from the depicted patterns we can draw conclusions about the relative speed of the obstacles. For instance, if the difference plot shows negative values left of positive values (red-blue pattern), the obstacle is faster than the host vehicle. In contrast, a blue-red pattern indicates a slower vehicle. The width of the bars reflect the magnitude of the relative velocities.

The features are illustrated in Fig. 5. In the figures, the $x$-axis corresponds to the longitudinal and the $y$-axis to the lateral direction. Shown is the top-view on the host vehicle, which is indicated by the red cross. The intensity reflects the measured heights at each pixel. As explained, the difference features are obtained by subtracting the features of the previous from the current time frame. Features A1 and A2 clearly show that the driver accelerates as long as the preceding car is significantly faster. A3, however, indicates acceleration though the preceding car decelerates. Since the car seems to be sufficiently far away, the driver has not yet decided to reduce the speed. Features D1 and D2 explain the deceleration of the driver with a slower car in front. In contrast, D3 shows that the preceding car accelerates. However, due to the low distance between both cars, the driver decides to decelerate. Features C1 and C2 show a vehicle in front of the driver’s car. The difference images reveal only little differences between the velocities of both vehicles, such that the driver maintains the current velocity. C3 represents an empty road, where, again, the driver does not need to adapt the speed of his car.

In order to evaluate the prediction performance, as in the simulations, 20% of the observations are used for testing while the rest is used for inferring the structure. Using the MAP estimator results in an accuracy of $74.32$ %. The confusion matrix in Tab. 2 shows that, most notably, in some cases moving at constant speed and acceleration are confused. Further, deceleration is in few cases misclassified as acceleration.

Quantifying the results of the state reconstruction is difficult, as a ground truth is not available. To provide at least an intuition about the quality of the reconstructions, we compute the MSE between the reconstructed states, based on the estimated substates and features, and the observations. This results in a MSE of approx. $0.0329$. As the values of the observations are in a similar range as in the simulations, this result indicates low errors, resembling the results of the simulations. Still, the number has to be taken with care, as we compare the denoised reconstruction with noisy observation.

6.2 Scene 20 - Lane change

Scene 20 shows a lane change maneuver on a two-lane road. As can be observed from the acceleration signals depicted in Fig. 4(b), the driver accelerates three times, depending on the current traffic situation. The lane change takes place from time frame 158 to 220.

Applying the proposed algorithm reveals 17 features. Using these features to predict the actions of the test data set yields an accuracy of $73.33\%$. The confusion matrix in Tab. 3 shows that lane changes are reliably predicted. Acceleration maneuvers are in some cases misinterpreted as moving at constant speed or as a lane change. A third of the moving at constant speed observations are misclassified as lane changes. Comparing the reconstructed states with the noisy observation yields a MSE of $0.0027$.

For illustration of the inferred features, Fig. 6 shows three out of the 17 features, each indicating a different action. The first feature (a) shows a lane change (right). As the driver is already performing the maneuver, the scene is rotated. The dark areas in the lower left corner represent vehicles behind the host car. The second feature (b) contains a vehicle behind the driver’s car and another vehicle on the left lane. Due to the high traffic density, the driver does not accelerate. In contrast, the third feature (c) shows a completely empty road, motivating the driver to accelerate the vehicle.

7 Discussion

As shown in the simulation experiments, the algorithm based on the proposed model is able to reliably infer the number of features, the features and the policies. In case of strong noise, the algorithm finds several, almost equally probable explanations for the observations, resulting in variations of the MAP estimate. Especially when the number of features is high compared to the dimensionality of the observations, inferring the correct number of features can be challenging.

The real data experiments show that the model is able to provide deeper insights into the observations which may yield new conclusions about the observed behavior. As explained, in high-dimensional observations, the observation likelihood is likely to dominate the posterior leading to only little influence of the action likelihood and, hence, poor prediction performance. As proposed in Section 4.3, reweighting the action likelihood, assuming an action variable for each entry of the observation, yields a significant performance increase, while the observed states can still be reliably reconstructed.

An advantage of the proposed generative model is that it can be modified and extended easily. This is helpful especially for a different assumption on the feature weights. For example, if real-valued features are expected, the corresponding prior can be changed to a Gaussian distribution. Of course, inference has to be adapted accordingly. Further, one can easily extend the proposed model to a semi-supervised learning approach, in which we add variables for states where the actions are not observed. Thus, the state representations can be learned from even more data resulting in more accurate estimates.

8 Conclusion

We have presented a feature-based framework for learning from demonstrations that allows us to reason about the observed behavior and to predict actions for new states. A key assumption in the proposed model is that the observations are composed of latent features. Each feature imposes its own policy and contributes to the decision of the agent. To learn the structure of the behavior and to predict actions, we have considered a Bayesian nonparametric approach based on the Indian Buffet Process, which allows to infer the number of features and the features itself from the observed data. By means of this model, we are able to obtain a deeper understanding of the observed behavior as the features and their policies allow to reason about the observed decisions. The simulations show that the developed algorithm performs well. Only in scenarios with strong noise and many latent features, inference becomes challenging. Further, we have considered the task of learning a driver’s behavior. For this, we have applied our algorithm to real data obtained from the KITTI benchmark suite. The results reveal under which conditions the driver takes the observed actions. Further, prediction on a hold-out data set demonstrates that actions can be predicted reliably.