Time-Varying Interaction Estimation Using Ensemble Methods

TL;DR

This paper introduces an ensemble-based approach to improve the estimation of time-varying directed interactions in multivariate data, enhancing robustness and applicability to complex scenarios like crowded scenes.

Contribution

It proposes a novel ensemble method for adaptive directed information estimation, addressing parameter sensitivity and robustness in non-stationary interaction analysis.

Findings

Ensemble methods improve the robustness of interaction estimation.

Application to Stanford drone dataset demonstrates effectiveness.

Enhanced detection of dynamic interactions in crowded scenes.

Abstract

Directed information (DI) is a useful tool to explore time-directed interactions in multivariate data. However, as originally formulated DI is not well suited to interactions that change over time. In previous work, adaptive directed information was introduced to accommodate non-stationarity, while still preserving the utility of DI to discover complex dependencies between entities. There are many design decisions and parameters that are crucial to the effectiveness of ADI. Here, we apply ideas from ensemble learning in order to alleviate this issue, allowing for a more robust estimator for exploratory data analysis. We apply these techniques to interaction estimation in a crowded scene, utilizing the Stanford drone dataset as an example.

Figures

Equations

$$\text{DI}({\mathbf{X}}_{1:T}^i\to {\mathbf{X}}_{1:T}^j)=\sum _{t=1}^T\text{I}({\mathbf{X}}_{1:t}^i;X_t^j|{\mathbf{X}}_{1:t-1}^j),$$

$$\overline{\text{DI}}({\mathbf{X}}^i\to {\mathbf{X}}^j)=\underset{T\to \infty }{\lim }\frac{1}{T}\text{DI}({\mathbf{X}}_{1:T}^i\to {\mathbf{X}}_{1:T}^j).$$

$$\text{ADI}({\mathbf{X}}_{1:T}^i\to {\mathbf{X}}_{1:T}^j)=\sum _{t=1}^{T}g(t,T)\text{I}({\mathbf{X}}_{1:t}^i;X_t^j|{\mathbf{X}}_{1:t-1}^j),$$

$${\text{ADI}}_{1:t}^{i\to j}=\alpha \text{I}({\mathbf{X}}^i;X_t^j|{\mathbf{X}}_{1:t-1}^j)+(1-\alpha ){\text{ADI}}_{1:t-1}^{i\to j},$$

$$g^{\ast }(t,T)=\frac{{\sum }_{i=1}^{n_t}{w}_{i,t}{g}_i(t,T;t_0)}{{\sum }_{i=1}^{n_t}{w}_i},$$

$$v_{i,t}=w_{i,t-1}{e}^{-\gamma (y_{i,t}-i_t{)}^2},w_{i,t}=(1-\beta )v_{i,t}+\frac{\beta }{n_t}\sum _{i=1}^{n_t}{v}_{i,t},$$

$$\begin{gathered} \mathbb{E}\left[\sum _{t=1}^T(\overline{\text{ADI}}(t)-i_t{)}^2\right]\le \frac{m}{\gamma }\ln n_t \\ -\frac{1}{\gamma }\ln {\beta }^m(1-\beta {)}^{T-m}+\frac{\gamma }{8}T+m{\sigma }_{\ast }^2\ln \left(\frac{T}{e}\right), \end{gathered}$$

$$\mathbf{X}\sim \mathcal{N}(m_t,{\Sigma }_t),$$

$${\hat{m}}_t$$

$${\hat{\Sigma }}_t$$

$$\hat{\text{I}}({\mathbf{X}}_{1:t}^i;X_t^j|X_{t-1}^j,X_{t-1}^{[N]/\{i,j\}})=\frac{1}{2}\log \frac{\left|{\hat{\Sigma }}_{X_t^j|X_{t-1}^j,X_{t-1}^{[N]/\{i,j\}}}\right|}{\left|{\hat{\Sigma }}_{X_t^j|X_{t-1}^j,X_{t-1}^i,X_{t-1}^{[N]/\{i,j\}}}\right|}.$$

$$\text{AMI}({\mathbf{X}}_{1:T}^i,{\mathbf{X}}_{1:T}^j)=\sum _{t=1}^T{g}^{\ast }(t,T)\hat{\text{I}}(X_t^i;X_t^j|{\mathbf{X}}_{1:t-1}^i,{\mathbf{X}}_{1:t-1}^j),$$

$${\theta }_t^{i,j}=\arccos \left(\frac{{\mathbf{v}}_t^i\cdot {\mathbf{v}}_t^j}{v_t^i{v}_t^j}\right).$$

$$u^{\ast }(t)=\sum _{k=1}^m\sum _{t=1}^T\frac{1}{t-t_k+1}1(t_k\le t<t_{k+1})i_t,$$

$$R(u^{\ast }(T))=\sum _{t=1}^T(\overline{\text{ADI}}(t)-i_t{)}^2-\sum _{t=1}^T(u^{\ast }(t)-i_t{)}^2,$$

$$R(u^{\ast }(T))\le \frac{m}{\gamma }\ln n_t-\frac{1}{\gamma }\ln {\beta }^m(1-\beta {)}^{T-m}+\frac{\gamma }{8}T.$$

$$\mathbb{E}\left[(u^{\ast }(t)-i_t{)}^2\right]\le {\sigma }_t^2+\frac{1}{t-t_k+1}{\sigma }_{\ast }^2.$$

$$\mathbb{E}\left[(u^{\ast }(t)-i_t{)}^2\right]$$

$$=\mathbb{E}\left[{\left(\frac{1}{t-t_k+1}\sum _{i=t_k}^{t-1}{ϵ}_i-{ϵ}_t\right)}^2\right]$$

$$=\frac{1}{(t-t_k+1{)}^2}\left(\sum _{i=t_k}^{t-1}{ϵ}_i+(t-t_k{)}^2{ϵ}_t\right)$$

$$\le \frac{(t-t_k)}{(t-t_k+1{)}^2}{\sigma }_{\ast }^2+\frac{(t-t_k{)}^2}{(t-t_k+1{)}^2}{\sigma }_t^2$$

$$\le \frac{{\sigma }_{\ast }^2}{t-t_k+1}+{\sigma }_t^2.$$

$$\mathbb{E}\left[(\overline{\text{ADI}}(t)-i_t{)}^2\right]\ge \mathbb{E}\left[(\overline{\text{ADI}}(t)-{\theta }_t{)}^2\right]+{\sigma }_t^2.$$

$$(\overline{\text{ADI}(t)}-i_t{)}^2=(\overline{\text{ADI}}(t)-{\theta }_t{)}^2+2{ϵ}_t(\overline{\text{ADI}}(t)-{\theta }_t)+{ϵ}_t^2.$$

$$\mathbb{E}\left[2{ϵ}_t(\overline{\text{ADI}}(t)-{\theta }_t)\right]$$

$$=\mathbb{E}\left[2\sum _{j=1}^{n_t}{w}_{j,t-1}{g}_j(i,T;t_0)i_t{ϵ}_t\right]$$

$$=2{\sigma }_t^2\sum _{j=1}^{n_t}{w}_{j,t-1}{g}_j(i,T;t_0)\ge 0,$$

$$\begin{gathered} \sum _{t=1}^T\left(\mathbb{E}\left[(\overline{\text{ADI}}(t)-{\theta }_t{)}^2\right]+{\sigma }_t^2\right)- \\ \sum _{t=1}^T\left({\sigma }_t^2+\sum _{k=1}^m\frac{1}{t-t_k+1}1(t_k\le t<t_{k+1}){\sigma }_{\ast }^2\right)\le R(u^{\ast }(T)). \end{gathered}$$

$$\sum _{k=1}^m\sum _{t=1}^T\frac{1}{t-t_k+1}\le m\ln \left(\frac{T}{e}\right).$$

Full text

**Index Terms— ** directed information, adaptive directed information, temporal modeling, data exploration, interaction mining

1 INTRODUCTION

The study of interactions among entities of interest encompasses a broad array of applications and is crucial to understanding complex processes. Often times, we are interested in the directionality over time of these relationships. Examples include social influence estimation [19, 18, 21], entity interaction in video [5], and biological recording analysis, such as EEG [6, 20]. These interactions can also be used to summarize highly complex data topology, allow analysts to obtain a qualitative snapshot of the temporal interactions of the data, and make better informed decisions based on these simplified representations.

One tool that allows for the extraction of interactions is called directed information (DI). Originally created to analyze an information-theoretic channel with feedback, DI has been used in many contexts to estimate directed relationships between entities, including genetic data and social data. One deficiency of directed information is its inflexibility with respect to time-varying distributions [18, 17]. Adaptive directed information (ADI) was developed as an extension of directed information to better track changes in relationships over time.

In this paper, we address some of the issues associated with using ADI. Specifically, ADI requires a choice of filter and corresponding filter parameters, and the quality of the resulting interaction estimate is not generally robust to these choices. In addition, simple filters may have difficulty adapting to both abrupt changes in interaction, as well as slowly time-varying systems. An estimate that is able to accomplish both smoothing over time, as well as the ability to adapt to abrupt changes in interactivity quickly is desired.

In this paper, a form of ensemble learning is used to improve interaction estimation with ADI. Specifically, following [9, 23], we generate a filter that is a convex combination of simpler filters with different parameter specifications and whose weights are dependent on the data. In order to address the possibility of abrupt changes in the system, a growing ensemble of estimators is used to account for these changes in interactivity.

The proposed ADI estimator is applied to interaction estimation in a crowded scene, utilizing video from the Stanford drone dataset [22]. Utilizing a dynamic covariance model, the ADI is estimated and used to uncover interesting phenomena in specific scenes across the Stanford campus.

The paper is organized as follows: Sec. 2 discusses related work. Sec. 3 introduces the mathematical concepts of DI and ADI, and introduces our ensemble estimator. Sec. 4 introduces the dynamic covariance model used to estimate ADI. Sec. 5 discusses the results on the Stanford Video Dataset. Finally, Sec. 6 concludes the paper.

2 Related Work

Directed information has been studied in the context of theory and applications. Estimators for DI have been proposed for the case of a finite or countably infinite feature space [20, 11, 13]. Most, if not all, estimators use the stationary Markov assumption, including plugin estimators [17, 18]. Directed information has been used in many contexts, including EEG analysis [6], neural spike trains [20], and social influence analysis [17, 18]. Changepoint detection methods [2] is one approach to track time-varying data, and parametric as well as non-parametric methods exist. However, with few exceptions, e.g.,  [3] these methods are mostly univariate and often require a parametric model or use simple moment-based statistics that do not capture dependency.

Other methods of influence estimation have been studied, particularly in the context of i.i.d. observations; examples include glasso [8] and hub discovery-type methods [10]. In addition, semi-parametric extensions of these models have been created for non-Gaussian data [12]. The family of directed information measures and in particular ADI is concerned with directionality in time and with more complicated time-varying signals. In this paper, we assume a parametric multivariate Gaussian model, which is appropriate for the particular dataset.

The ensemble method used stems from the prediction with multiple experts, a popular problem in machine learning [4, 23, 9]. Here, we use these techniques for smoothing.

3 ADI and Ensemble Estimation

3.1 Definition of DI and ADI

We begin with some notation. We assume that we have $1,2,\ldots,N$ entities each with features $\mathbf{X}^{i}_{1:T}=\left[X^{i}_{1},X^{i}_{2},\ldots,X^{i}_{T}\right]$. In this paper, $X^{i}_{t}\in\mathbb{R}^{d}$. Directed information between $X^{i}$ and $X^{j}$ is defined as follows:

[TABLE]

where $\text{I}(X;Y|Z)$ is the Shannon conditional mutual information. Many interesting conservation properties have been derived for directed information, including a close connection to the standard Shannon mutual information; these will not be repeated here, but the reader is referred to papers [15, 16, 1]. When considering the asymptotic behavior of DI for stationary processes, one defines the directed information rate:

[TABLE]

If we assume that the entities form a $k$-Markov process, then $\text{I}(\mathbf{X}_{1:t}^{i};X^{j}_{t}|\mathbf{X}^{j}_{1:t-1})=\text{I}(\mathbf{X}_{1:t}^{i};X^{j}_{t}|\mathbf{X}^{j}_{t-k:t-1})$.

When stationarity cannot be assumed, then the traditional definition of $\overline{\text{DI}}$ is inapplicable. However, the instantaneous DI summand of (1) retains valuable information about temporal interactivity of the entities $i$ and $j$. In [18], we proposed to adaptively estimate this quantity using adaptive directed information (ADI), which is defined as follows:

[TABLE]

where $g(t,T)$ is a user-defined taper function. In past work [18], the focus has been on the exponential filter $g(t,T)=\alpha(1-\alpha)^{t-T},$ so that ADI obeys the recursive update:

[TABLE]

where $\text{ADI}_{1:t}^{i\rightarrow j}=\text{ADI}(\mathbf{X}_{1:t}^{i}\rightarrow\mathbf{X}_{1:t}^{j})$. However, the parameter $\alpha$ of the exponential filter must be tuned according to the specific application. The goal of the this paper is to improve the robustness of ADI when the underlying state is unknown and rapidly changing. In order to accomplish this, an ensemble filter is defined:

[TABLE]

where $g_{i}(t,T)$ are “base filters” with different parameter specifications. Implicitly, the weights $w_{i}$ are allowed to depend on past data. Further, the number of base filters included in the ensemble ($n_{t}$) is allowed to grow with $t$, and filter functions will be causal, i.e., $g(t,T;t_{0})=0$ for $t<t_{0}$.

3.2 Expanding Fixed Shares of Estimation

We apply an ensemble method based on the simple fixed shares algorithm [9], which was originally introduced in [23].

A set of base filter functions is defined, $G=\{g_{1},\ldots,g_{k}\}$ along with a parameter $\tau$ which defines the rate at which new filters are introduced into 2

At each time $t$, an estimate $\hat{\text{I}}(\mathbf{X}_{1:t}^{i};~{}X^{j}_{t}|\mathbf{X}^{j}_{1:t-1})$ is obtained and used to both update the weights $w_{i}$ and to update the ADI estimate.

The weights $w_{i}$ are updated in a similar manner to [23]:

[TABLE]

where $\beta\in[0,1]$ and $\gamma>0$ are user-defined hyperparameters. Theorem 3.1 provides a bound for the MSE, assuming that $\text{I}(\mathbf{X}_{1:t}^{i};~{}X^{j}_{t}|\mathbf{X}^{j}_{1:t-1})$ is piecewise constant, and the estimate has i.i.d. noise with bounded variance. We use the abbreviation $i_{t}=\text{I}(\mathbf{X}_{1:t}^{i};~{}X^{j}_{t}|\mathbf{X}^{j}_{1:t-1})$, and similarly $\hat{i_{t}}=\hat{\text{I}}(\mathbf{X}_{1:t}^{i};~{}X^{j}_{t}|\mathbf{X}^{j}_{1:t-1})$ for convenience.

Theorem 3.1**.**

Let $\hat{i}_{t}=i_{t}+\epsilon_{t}$, where $\epsilon_{t}$ is independent with mean 0 and variance $\sigma_{t}^{2}$, and $i_{t}$ is piecewise constant with $m$ transitions. Then the MSE of the ADI ensemble estimator is bounded by:

[TABLE]

where $\sigma_{*}^{2}=\max_{t}\sigma_{t}^{2}$.

The proof of Theorem 3.1 is given in Appendix A.

4 Spatial Interaction Estimation in a Scene

We illustrate ADI by applying it to discover salient time-varying interactions among actors in a scene. Here, the components $n=1,\ldots,N$ are actors moving around in space. For each sampled frame $t$ and actor $i$, define the position vector $X^{i}_{t}=[x^{i}_{t},y^{i}_{t}]$ on the plane.

4.1 Dynamic Covariance Model

We propose a dynamic Gaussian model, following the model in [7]. Assume that the combined feature matrix is distributed as:

[TABLE]

where $m_{t}$ is a mean vector and $\Sigma_{t}$ is a covariance matrix. We assume that $m_{t}$ and $\Sigma_{t}$ are slowly varying, and further use a kernel estimate of these quantities:

[TABLE]

where $K_{h}(t)$ is a kernel function. The conditional mutual information is a function of the covariance matrices under a Markovian Gaussian random process .

[TABLE]

5 Application to Stanford Drone Dataset

In this section, the proposed ensemble ADI estimator is applied to the Stanford Drone Dataset [22], which is a collection of 60 annotated videos across 8 scenes shot on the Stanford campus. These annotations allow for tracking the movement of pedestrians, cars, bicyclists and other moving actors in the scene. These estimated locations of actors are smoothed by a moving mean estimator in order to reduce artifacts introduced by the discretization of the annotations. These smoothed locations for each actor in the scene are then used to calculate the ADI.

For the analysis, an rbf kernel was used in (5) with parameter $h=5$, and the ADI ensemble parameters were set to $\tau=10,\beta=0.01,\gamma=1$, and $G=\{\text{exp}(0.1),\text{exp}(0.2),\text{unif}\}$. After calculating ADI, only interactions where the actors were within a certain distance (in pixels) from each other were considered - in this case, 100.

5.1 Interaction Example between Pedestrians

Fig. 1 shows one example of ADI and the corresponding interaction between two pedestrians. The pedestrians labeled 5 and 25 stop to chat briefly, with 25 actually reversing course for a small time to continue the conversation at frame 1280 to 1300 to continue the conversation. The estimated ADI is able to identify this interaction, and to identify that there is more influence from 5 to 25 than vice versa over this small window. This is compared with an adaptive version of mutual information:

[TABLE]

where the ensemble method outlined for ADI is applied to the estimated summand $\hat{\text{I}}(X_{t}^{i};X^{j}_{t}|\mathbf{X}^{i}_{1:t-1},\mathbf{X}^{j}_{1:t-1})$.

5.2 Visualization of Interactions based on ADI

We can use ADI as a tool to cluster and visualize many interactions in the dataset. First, the ADI for all interactions between actors in the bookstore scene from the Stanford Drone dataset across 5 different videos are collected, totaling $m=539$ interactions. Using symmetrized ADI, $\text{ADI}^{i,j}=\text{ADI}^{i\rightarrow j}+\text{ADI}^{j\rightarrow i}$, the maximal cross correlation between each interaction is found, and this correlation is used as an affinity measure $a_{k,l}$, with the corresponding affinity matrix $A=[a_{k,l}]_{k,l=1,\ldots m}$. Note that $a_{k,l}=a_{l,k}$, and so $A$ is symmetric. $A$ can then be used to apply a number of visualization and clustering techniques. Here, we use t-SNE dimension reduction and visualization method [14], by transforming $A$ to a distance matrix $D=[d_{i,j}]_{i,j=1,\ldots m}$, where d_{i,j}=\mathchoice{{\hbox{\displaystyle\sqrt{2(1-a_{i,j}),}}\lower 0.4pt\hbox{\vrule height=7.5pt,depth=-6.00003pt}}}{{\hbox{\textstyle\sqrt{2(1-a_{i,j}),}}\lower 0.4pt\hbox{\vrule height=7.5pt,depth=-6.00003pt}}}{{\hbox{\scriptstyle\sqrt{2(1-a_{i,j}),}}\lower 0.4pt\hbox{\vrule height=5.25pt,depth=-4.20003pt}}}{{\hbox{\scriptscriptstyle\sqrt{2(1-a_{i,j}),}}\lower 0.4pt\hbox{\vrule height=3.75pt,depth=-3.00002pt}}} and applying the method to this matrix. Fig. 2 shows the results. The colors correspond to different types of interactions, such as between pedestrians, or between a pedestrian and a bike, etc.

The visualization shows small clusterings of interactions. An example is circled in black, with representative traces shown in Fig. 3. More generally, we see that the pedestrian-biker interactions mostly cluster in the bottom-left portion of the plot, while the biker-biker and pedestrian-pedestrian interactions are less cohesive as a group, implying heterogeneity among these types of interactions. The small highlighted cluster of pedestrian interactions, for example, are characterized by long periods of low ADI combined with abrupt spikes. These are observed to correlate to pedestrians walking slowly in the same direction or standing still along with occasional changes in velocity or direction.

5.3 Relationship between ADI and Velocity

In this section we study the relationship between the velocity profile and ADI profile of particular types of interactions. For each interaction and each actor $i$ the instantaneous velocity vector $\mathbf{v}^{i}_{t}=[v^{i}_{t,x},v^{i}_{t,y}]$ is calculated, along with the corresponding instantaneous magnitude $v^{i}_{t}=\left\lVert\mathbf{v}^{i}_{t}\right\rVert$. Further, the instantaneous velocity angle between two actors $i$ and $j$ is calculated:

[TABLE]

Using the relative velocity angle, we can look for two specific types of interactions, and how their ADI profiles differ; those with high angle, so that the two actors are approaching from opposite directions, and low angle, where the two actors are moving in the same direction. Fig. 4 shows four representative interactions, two with low velocity angles and two with high velocity angles.

In general, interactions with high total velocity, defined as $v^{i}_{t}+v^{j}_{t}$, and low velocity angle see a stable and non-zero symmetrized ADI. In the low total velocity setting, the ADI is normally much smaller than its high velocity counterpart. Two examples of low-angle interactions are shown in the top row of Fig. 4. In the high angle case, ADI is less constant, and in many cases responds more to changes in total velocity, as shown in the bottom row of Fig. 4.

5.4 Average ADI between Different Types of Actors

Fig. 5 shows a graph of the average ADI between types of actors in the bookstore scene from the Stanford drone dataset across 5 different videos.

Skaters tend to have the lowest average ADI with other groups, followed by pedestrians, with bikers and carts having the largest interaction magnitudes. Interestingly, pedestrians influence bikers and carts more than the two groups influence pedestrians on average, possibly signifying that bikers and carts are more cautious and thus are more affected by pedestrians in the vicinity. As seen in Fig. 4, the velocity magnitudes in interactions can play a role, specifically that the magnitudes of velocity and ADI are positively correlated. With bikers being among the fastest moving actors in this graph, it makes sense that they have some of the largest interaction magnitudes.

6 Conclusion

In this paper, we introduced an ADI estimator that utilizes an ensemble technique in order to make ADI more robust to user-specified parameters. The estimator is applicable to real-world scenarios where directed information evolves as a function of time. We illustrated the power of the ensemble ADI estimator to detect latent interactions in a video using the Stanford drone dataset. In the future, ADI can be used as a data summarization and exploration tool or as a component in a larger system.

Appendix A Proof of Theorem 2.1

To aid in the proof, we prove two propositions and restate Theorem 2 in [23] as Lemma A.1. Since we assume that $i_{t}$ is piecewise constant with $m$ changes, we can define $[t_{1},t_{2},\ldots,t_{m}]$ as the (unknown) transition points, where $t_{1}=1$. We further define the “oracle mean estimator” $u^{*}(t)$:

[TABLE]

where $t_{m+1}=T+1$. The proof is based on the following result found in [23], restated in terms of ADI:

Lemma A.1**.**

The tracking regret of the ensemble ADI estimator in comparison with $u^{*}(t)$, defined as:

[TABLE]

is at most

[TABLE]

Proposition A.2**.**

[TABLE]

Proof.

[TABLE]

∎

Proposition A.3**.**

[TABLE]

Proof.

We first decompose the left side using the definition of $i_{t}$:

[TABLE]

The result follows from taking the expectation of both sides, along with the following observation:

[TABLE]

where the last inequality is due to the fact that $w_{j,t-1}$ and $g_{j}(i,T;t_{0})$ are non-negative, $\forall i,j,t$. ∎

Using the definition of $R(u^{*}(T))$ and Props. A.3, A.2, we obtain:

[TABLE]

Finally, note the following inequality:

[TABLE]

Combining this with Lemma A.1, and rearranging terms, achieves the desired bound.