What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention

TL;DR

This paper introduces a novel egocentric action anticipation model using rolling-unrolling LSTMs and a modality attention mechanism, achieving state-of-the-art results on large-scale benchmarks.

Contribution

The work presents a new architecture combining multi-scale LSTMs and an adaptive modality attention mechanism for egocentric action anticipation.

Findings

Outperforms prior methods by up to 7% on EPIC-Kitchens dataset

Generalizes well to other egocentric datasets like EGTEA Gaze+

Ranks first in the EPIC-Kitchens challenge 2019

Abstract

Egocentric action anticipation consists in understanding which objects the camera wearer will interact with in the near future and which actions they will perform. We tackle the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to 1) summarize the past, and 2) formulate predictions about the future. The input video is processed considering three complimentary modalities: appearance (RGB), motion (optical flow) and objects (object-based features). Modality-specific predictions are fused using a novel Modality ATTention (MATT) mechanism which learns to weigh modalities in an adaptive fashion. Extensive evaluations on two large-scale benchmark datasets show that our method outperforms prior art by up to +7% on the challenging EPIC-Kitchens dataset including more than 2500 actions, and generalizes to EGTEA Gaze+. Our approach is also shown to generalize to the tasks of early action recognition and action recognition. Our method is ranked first in the public leaderboard of the EPIC-Kitchens egocentric action anticipation challenge 2019. Please see our web pages for code and examples: http://iplab.dmi.unict.it/rulstm - https://github.com/fpv-iplab/rulstm.

Equations

$$(h_{m,t}^R,c_{m,t}^R)=LST{M}_{{\theta }_m^R}(f_{m,t},h_{m,t-1}^R,c_{m,t-1}^R)$$

$$(h_{m,j}^U,c_{m,j}^U)=LST{M}_{{\theta }_m^U}(f_{m,t},h_{m,j-1}^U,c_{m,j-1}^U)$$

$$(h_{m,j}^U,c_{m,j}^U)=LST{M}_{{\theta }_m^U}(f_{m,t+j-1},h_{m,j-1}^U,c_{m,j-1}^U)$$

$${\lambda }_t=D_{{\theta }^{MATT}}({\oplus }_{m=1}^M(h_{m,t}^R\oplus c_{m,t}^R))$$

Full text

1 Introduction

Anticipating the near future is a natural task for humans and a fundamental one for intelligent systems when it is necessary to react before an action is completed (e.g., to anticipate a pedestrian crossing the street from an autonomous vehicle [9]) or even before it starts (e.g., to notify a user who is performing the wrong action in a known workflow [53]). Additionally, tasks such as action anticipation [18, 32, 58] and early action recognition [2, 9, 39] pose a series of key challenges from a computational perspective. Indeed, methods addressing these tasks need to model the relationships between past, future events and incomplete observations. First Person (Egocentric) Vision [29], in particular, offers an interesting scenario to study anticipation problems. On one hand, whilst being a natural task for humans, anticipating the future from egocentric video is computationally challenging due to the ability of wearable cameras to acquire long videos of complex activities involving many objects and actions performed by a user from their unique point of view. On the other hand, investigating these tasks is fundamental for the construction of intelligent wearable systems able to anticipate the user’s goal and assist them [29].

In this paper, we address the problem of egocentric action anticipation.

As defined in [8] and illustrated in Figure 1, the task consists in recognizing an action starting at time $\tau_{s}$ by observing a video segment preceding the action starting at time $\tau_{s}-(\tau_{o}+\tau_{a})$ and ending at time $\tau_{s}-\tau_{a}$, where the “observation time” $\tau_{o}$ indicates the length of the observed segment, whereas the “anticipation time” $\tau_{a}$ denotes how many seconds in advance actions are to be anticipated. While action anticipation has been investigated in classic third person vision [1, 18, 25, 32, 58, 28], less attention has been devoted to the egocentric scenario [8, 16, 47].

We observe that egocentric action anticipation methods need to address two sub-tasks: 1) summarizing what has been observed in the past (e.g., “a container has been washed” in the observed segment in Figure 1), and 2) making hypotheses about what will happen in the future (e.g., “put-down container”, “close tap”, “take spoon” in Figure 1). While previous approaches attempted to address these two sub-tasks jointly [2, 8, 39, 58], our method disentangles them using two separate LSTMs. The “Rolling” LSTM (R-LSTM) is responsible for continuously encoding streaming observations to summarize the past. When the method is required to anticipate actions, the “Unrolling” LSTM (U-LSTM) takes over the current hidden and cell states of the R-LSTM and makes predictions about the future. Differently from previous works which considered a fixed anticipation time [8, 16, 58], the proposed method can anticipate an action at multiple anticipation times, e.g., $2s$, $1.5s$, $1s$ and $0.5s$ before it occurs. The network is pre-trained with a novel “Sequence Completion Pre-training” (SCP) technique, which encourages the disentanglement of the two sub-tasks. To take advantage of the complementary nature of different input modalities, the proposed Rolling-Unrolling LSTM (RU) processes spatial observations (RGB frames), motion (optical flow), as well as object-based features. Multimodal predictions are fused with a novel “Modality ATTention” mechanism (MATT), which adaptively estimates optimal fusion weights for each modality by considering the outputs of the modality-specific R-LSTM components. Experiments on two large-scale datasets of egocentric videos, EPIC-KTICHENS [8] and EGTEA Gaze+ [36], show that the proposed method outperforms several state-of-the-art approaches and baselines in the task of egocentric action anticipation and generalizes to the tasks of early action recognition and action recognition.

The contributions of this work are as follows: 1) we are the first to systematically investigate the problem of egocentric action anticipation within the framework of the challenge proposed in [8]; 2) our investigation benchmarks popular ideas and approaches to action anticipation and leads to the definition of RU, an architecture able to anticipate egocentric actions at multiple temporal scales; 3) the proposed model is shown to benefit from two techniques specific to the investigated problem, i.e., i) “Sequence Completion Pre-training” and ii) adaptive fusion of multi-modal predictions through Modality ATTention; 4) extensive evaluations highlight the limits of previous approaches and show significant improvements of the proposed method over the state of the art. To support future research, the code implementing the proposed method will be released upon publication.

2 Related Work

**Action Recognition  ** Our work is related to previous research on action recognition from third person vision [7, 13, 14, 30, 34, 35, 49, 57, 59, 60, 61, 67] and first person vision [11, 12, 36, 37, 38, 44, 48, 52, 54, 55, 56]. Specifically, we build on previous ideas investigated in the context of action recognition such as the use of multiple modalities for video analysis [49], the use of Temporal Segment Networks [61] as a principled way to train CNNs for action recognition, as well as the explicit encoding of object-based features [11, 38, 44, 51, 56] to analyze egocentric video. However, in contrast with the aforementioned works, we address the problem of egocentric action anticipation and show that approaches designed for action recognition, such as TSN [61] and early/late fusion to merge multi-modal predictions [49] are not directly applicable to the problem of egocentric action anticipation.

**Early Action Recognition in Third Person Vision  ** Early action recognition consists in recognizing an ongoing action as early as possible from partial observations [9]. The problem has been widely investigated in the domain of third person vision [2, 4, 6, 9, 19, 23, 24, 39, 46]. Differently from these works, we address the task of anticipating actions from egocentric video, i.e., the action should be recognized before it starts, hence it cannot even be partially observed at the time of prediction. Given the similarity between early recognition and anticipation, we consider and evaluate some ideas investigated in the context of early action recognition, such as the use of LSTMs to process streaming video [2, 19, 39], and the use of dedicated loss functions [39]. Moreover, we show that the proposed architecture also generalizes to the problem of early egocentric action recognition, achieving state-of-the-art performances.

**Action Anticipation in Third Person Vision  ** Action anticipation is the task of predicting an action before it occurs [18]. Previous works investigated different forms of action and activity anticipation from third person video [1, 15, 18, 25, 27, 28, 31, 32, 33, 40, 58, 64]. While we consider the problem of action anticipation from egocentric visual data, our work builds on some ideas explored in past works such as the use of LSTMs to anticipate actions [1, 18, 28], the use of the encoder-decoder framework to encode past observations and produce hypotheses of future actions [18], and the use of object specific features [40] to determine which objects are present in the scene. Additionally, we show that other approaches, such as the direct regression of future representations [18, 58], do not achieve satisfactory performance in the considered scenario.

**Anticipation in First Person Vision  ** Past works on anticipation from egocentric visual data have investigated different problems and considered different evaluation frameworks [5, 10, 17, 42, 45, 47, 50, 53, 65, 68]. Instead, we tackle the egocentric action anticipation challenge recently proposed in [8], which has been little investigated so far [16]. While a direct comparison of the proposed approach with the aforementioned works is unfeasible due to the lack of a common framework, our method incorporates some ideas from past approaches, such as the analysis of past actions [47] and the detection of the objects present in the scene to infer future actions [17].

3 Proposed Approach

**Processing Strategy   ** Figure 2 illustrates the processing strategy adopted by the proposed method. The video is processed in an on-line fashion, with a short video snippet $V_{t}$ consumed every $\alpha$ seconds, where $t$ indexes the current time-step. Specifically, an action occurring at time $\tau_{s}$ is anticipated by processing a video segment of length $l$ starting at time $\tau_{s}-l-\alpha$ and ending at time $\tau_{s}-\alpha$. The input video ends at time $\tau_{s}-\alpha$ as our method aims at anticipating actions at least $\alpha$ seconds before they occur. The processing is performed in two stages: an “encoding” stage, which is carried out for $S_{enc}$ time-steps, and an “anticipation” stage, which is carried out for $S_{ant}$ time-steps. In the encoding stage, the model summarizes the semantic content of the $S_{enc}$ input video snippets without producing any prediction, whereas in the anticipation stage the model continues to encode the semantics of the $S_{ant}$ input video snippets and outputs $S_{ant}$ action scores $s_{t}$ which can be used to perform action anticipation. This scheme effectively allows to formulate $S_{ant}$ predictions for a single action at multiple anticipation times. In our experiments, we set $\alpha=0.25s$, $S_{enc}=6$ and $S_{ant}=8$. In these settings, the model analyzes video segments of length $l=\alpha(S_{enc}+S_{ant})=3.5s$ and outputs $8$ predictions at the following anticipation times: $\tau_{a}\in\{2s,1.75s,1.5s,1.25s,1s,0.75s,0.5s,0.25s\}$. It should be noted that, since the predictions are produced while processing the video, at time step $t$ the effective observation time will be equal to $\alpha\cdot t$. Hence, the $8$ predictions are performed at the following effective observation times: $\tau_{o}\in\{1.75s,2s,2.25s,2.5s,2.75s,3s,3.25s,3.5s\}$. Our formulation generalizes the one proposed in [8], which is illustrated in Figure 1. For instance, at time-step $t=11$, our model will anticipate actions with an effective observation time equal to $\tau_{o}=\alpha\cdot t=2.75s$ and an anticipation time equal to $\tau_{a}=\alpha(S_{ant}+S_{enc}+1-t)=1s$, as in [8].

**Rolling-Unrolling LSTM  ** The proposed method uses two separate LSTMs to encode past observations and formulate predictions about the future. Following previous literature [49], we include multiple identical branches which analyze the video according to different modalities. Specifically, at each time-step $t$, the input video snippet $V_{t}$ is represented using different modality-specific representation functions $\varphi_{1},\ldots,\varphi_{M}$ depending on learnable parameters $\theta^{\varphi_{1}},\ldots,\theta^{\varphi_{M}}$. This process allows to obtain the modality-specific feature vectors $f_{1,t}=\varphi_{1}(V_{t}),\ldots,f_{M,t}=\varphi_{M}(V_{t})$, where $M$ is the total number of modalities (i.e., the total number of branches in our architecture), and $f_{m,t}$ is the feature vector computed at time-step $t$ for the modality $m$. The feature vector $f_{m,t}$ is fed to the $m^{th}$ branch of the architecture. While our model can easily incorporate different modalities, in this work we consider $M=3$ modalities, i.e., RGB frames (spatial branch), optical flow (motion branch) and object-based features (object branch).

Figure 3 illustrates the processing taking place in a single branch $m$ of the proposed RU model. For illustration purposes only, the figure shows an example in which $S_{enc}=1$ and $S_{ant}=3$. At time step $t$, the feature vector $f_{m,t}$ is fed to the Rolling LSTM (R-LSTM), which encodes its semantic content recursively, as follows:

[TABLE]

where $LSTM_{\theta_{m}^{R}}$ denotes the R-LSTM of branch $m$, depending on the learnable parameters $\theta_{m}^{R}$, whereas $h^{R}_{m,t}$ and $c^{R}_{m,t}$ are the hidden and cell states computed at time $t$ in the modality $m$. The initial hidden and cell states of the R-LSTM are initialized with zeros: $h^{R}_{m,0}=\textbf{0},\ c^{R}_{m,0}=\textbf{0}$.

In the anticipation stage, at time step $t$, the Unrolling LSTM (U-LSTM) is used to make predictions about the future. The U-LSTM takes over the hidden and cell vectors of the R-LSTM at the current time-step (i.e., $h_{m,t}^{R}$ and $c_{m,t}^{R}$) and iterates over the representation of the current video snippet $f_{m,t}$ for a number of times $n_{t}$ equal to the number of time-steps required to reach the beginning of the action, i.e., $n_{t}=S_{ant}+S_{enc}-t+1$. Hidden and cell states of the U-LSTM are computed as follows at the $j^{th}$ iteration:

[TABLE]

where $LSTM_{\theta_{m}^{U}}$ is the U-LSTM of branch $m$, depending on the learnable parameters $\theta_{m}^{U}$, and $h^{U}_{m,t}$, $c^{U}_{m,t}$ are the hidden and cell states computed at iteration $j$ for the modality $m$. The initial hidden and cell states of the U-LSTM are initialized from the current hidden and cell states computed by the R-LSTM: $h^{U}_{m,0}=h^{R}_{m,t},\ c^{U}_{m,0}=c^{R}_{m,t}$. Note that the input $f_{m,t}$ of the U-LSTM does not depend on $j$ (see eq. (2)) because it is fixed during the “unrolling” procedure. The main rationale of “unrolling” the U-LSTM for a different number of times at each time-step is to encourage it to differentiate predictions at different anticipation times.

Modality-specific action scores $s_{m,t}$ are computed at time-step $t$ by processing the last hidden vector of the U-LSTM with a linear transformation with learnable parameters $\theta^{W}_{m}$ and $\theta^{b}_{m}$: $s_{m,t}=\theta^{W}_{m}h^{U}_{m,n_{t}}+\theta^{b}_{m}.$

**Sequence Completion Pre-Training (SCP)  ** The two LSTMs composing the RU architecture are designed to address two specific sub-tasks: the R-LSTM is responsible for encoding past observations and summarizing what has happened up to a given time-step, whereas the U-LSTM focuses on anticipating future actions conditioned on the hidden and cell vectors of the R-LSTM. To encourage the two LSTMs to specialize on the two different sub-tasks, we propose to train the architecture using a novel Sequence Completion Pre-training (SCP) procedure. During SCP, the connections of the U-LSTM are modified to allow it to process future representations, rather than iterating on the current one. In practice, the U-LSTM hidden and cell states are computed as follows during SCP:

[TABLE]

where the input representations $f_{m,t+j-1}$ are sampled from future time-steps $t+j-1$. Figure 4 illustrates an example of the connection scheme used during SCP for time-step $t=2$. The main goal of pre-training the RU with SCP is to allow the R-LSTM to focus on summarizing past representations without trying to anticipate the future.

**Modality ATTention (MATT)  ** Coherently with past work on egocentric action anticipation [8], we found it sub-optimal to fuse multi-modal predictions with classic approaches such as early and late fusion. This is probably due to the fact that, when anticipating egocentric actions, one modality might be more useful than another (e.g., appearance over motion), depending on the processed sample. Inspired by previous work on attention [3, 62] and multi-modal fusion [41], we introduce a Modality ATTention (MATT) module which computes a set of attention scores indicating the relative importance of each modality for the final prediction. At a given time-step $t$, such scores are obtained by processing the concatenation of the hidden and cell vectors of the R-LSTM networks belonging to all branches $m=1,\ldots,M$ with a deep neural network $D$ depending on the learnable parameters $\theta^{MATT}$:

[TABLE]

where $\oplus$ denotes the concatenation operator and $\oplus_{m=1}^{M}(h_{m,t}^{R}\oplus c_{m,t}^{U})$ is the concatenation of the hidden and cell vectors produced by the R-LSTM at time-step $t$ across all modalities. Late fusion weights can be obtained normalizing the score vector $\lambda_{t}$ with the softmax function in order to make sure that fusion weights sum to one: $w_{m,t}=\frac{exp(\lambda_{t,m})}{\sum_{k}exp(\lambda_{t,k})}$, where $\lambda_{t,m}$ is the $m^{th}$ component of the score vector $\lambda_{t}$. The final set of fusion weights is obtained at time-step $t$ by merging the modality-specific predictions produced by the different branches with a linear combination as follows: $s_{t}=\sum_{m}w_{m,t}\cdot s_{m,t}$. Figure 5 illustrates an example of a complete RU with two modalities and the MATT fusion mechanism. For illustration purposes, the figure shows only three anticipation steps.

**Branches and Representation Functions  ** We instantiate the proposed architecture with $3$ branches: a spatial branch which processes RGB frames, a motion branch which processes optical flow, and an object branch which processes object-based features. Our architecture analyzes video snippets of $5$ frames $V_{t}=\{I_{t,1},I_{t,2},\ldots,I_{t,5}\}$, where $I_{t,i}$ is the $i^{th}$ frame of the video snippet. The representation function $\varphi_{1}$ of the spatial branch computes the feature vector $f_{1,t}$ by extracting features from the last frame $I_{t,5}$ of the video snippet using a Batch Normalized Inception CNN [26] trained for action recognition. The representation function $\varphi_{2}$ of the motion branch extracts optical flow from the $5$ frames of the current video snippet as proposed in [61]. The computed $x$ and $y$ optical flow form a tensor with $10$ channels, which is fed to a Batch Normalized Inception CNN trained for action recognition to obtain the feature vector $f_{2,t}$. Note that $\varphi_{1}$ and $\varphi_{2}$ allow to obtain “action-centric” representations of the input frame which can be used by the R-LSTM to summarize what has happened in the past. The representation function $\varphi_{3}$ of the object branch extracts objects from the last frame $I_{t,5}$ of the input snippet $V_{t}$ using an object detector. A fixed-length representation $f_{3,t}$ is obtained by accumulating the confidence scores of all bounding boxes predicted for each object class. Specifically, let $b_{t,i}$ be the $i^{th}$ bounding box detected in image $I_{t,5}$, let $b^{c}_{t,i}$ be its class and let $b^{s}_{t,i}$ be its detection confidence score. The $j^{th}$ component of the output representation vector $f_{3,t}$ is obtained by summing the confidence scores of all detected objects of class $j$, i.e., $f_{3,t,j}=\sum_{i}[b^{c}_{t,i}=j]b^{s}_{t,i}$, where $[\cdot]$ denotes the Iverson bracket. This representation only encodes the presence of an object in the scene, discarding its position in the frame, similarly to the representation proposed in [44] for egocentric activity recognition. We found this holistic representation to be sufficient in the case of egocentric action anticipation. Differently from $\varphi_{1}$ and $\varphi_{2}$, $\varphi_{3}$ produces object-centric features, which carry information on what objects are present in the scene and hence could be interacted next.111See supp. for details on implementation and training of our method.

4 Experimental Settings

**Datasets  ** We perform experiments on two large-scale datasets of egocentric videos: EPIC-Kitchens [8] and EGTEA Gaze+ [36]. EPIC-Kitchens contains $39,596$ action annotations, $125$ verbs, and $352$ nouns. We split the public training set of EPIC-Kitchens ($28,472$ action segments) into training ($23,493$ segments) and validation ($4,979$ segments) sets by randomly choosing $232$ videos for training and $40$ videos for validation. We considered all unique $(verb,noun)$ pairs in the public training set, thus obtaining $2,513$ unique actions. EGTEA Gaze+ contains $10,325$ action annotations, $19$ verbs, $51$ nouns and $106$ unique actions. Methods are evaluated on EGTEA Gaze+ reporting the average performance across the three splits provided by the authors of the dataset [36].

**Evaluation Measures  ** Methods are evaluated using Top-k evaluation measures, i.e., we deem a prediction correct if the ground truth action falls in the top-k predictions. As observed in previous works [16, 32], this evaluation scheme is appropriate given the uncertainty of future predictions (i.e., many plausible actions can follow an observation). Specifically, we use the Top-5 accuracy as a class-agnostic measure and Mean Top-5 Recall as a class aware metric. Top-5 recall [16] for a given class $c$ is defined as the fraction of samples of ground truth class $c$ for which the class $c$ is in the list of the top-5 anticipated actions. Mean Top-5 Recall averages Top-5 recall values over classes. When evaluating on EPIC-Kitchens, Top-5 Recall is averaged over the provided list of many-shot verbs, nouns and actions. Results on the EPIC-Kitchens official test set are reported using the suggested evaluation measures, i.e., Top-1 accuracy, Top-5 accuracy, Precision and Recall.

To assess the timeliness of anticipations, we design a new evaluation measure inspired by the AMOC curve [23]. Let $s_{t}$ be the score anticipated at time-step $t$ for an action of ground truth class $c$, let $\tau_{t}$ be the anticipation time at time-step $t$, and let $tk(s_{t})$ be the set of top-$k$ actions as ranked by the scores $s_{t}$. We define as “time to action” at rank $k$ the largest anticipation time (i.e., the time of earliest anticipation) in which a correct prediction has been made according to the top-$k$ criterion: $TtA(k)=max\{\tau(s_{t})|c\in tk(s_{t}),\ \forall t\}$. If an action is not correctly anticipated in any of the time-steps, we set $TtA(k)=0$. The mean time to action over the whole test set $mTtA(k)$ indicates how early, in average, a method can anticipate actions.

Performances are evaluated for verb, noun and action predictions on the EPIC-Kitchens dataset. We obtain verb and noun scores by marginalization over action scores for all methods except the one proposed in [8], which predicts verb and noun scores directly.

**Compared Methods  ** We compare the proposed method with respect to $7$ state-of-the approaches and baselines. Specifically, we consider the Deep Multimodal Regressor (DMR) proposed in [58], the Anticipation Temporal Segment Network (ATSN) of [8], the anticipation Temporal Segment Network trained with verb-noun Marginal Cross Entropy Loss (MCE) described in [16], and the Encoder-Decoder LSTM (ED) introduced in [18]. We also consider baselines obtained adapting approaches proposed for early action recognition to the problem of egocentric action anticipation: the Feedback Network LSTM (FN) proposed in [19], a single LSTM architecture (we use the same parameters of our R-LSTM) trained using the Ranking Loss on Detection Score proposed in [39] (RL), and an LSTM trained using the Exponential Anticipation Loss proposed in [28] (EL). These baselines adopt the video processing scheme illustrated in Figure 2 and are implemented as two stream networks with a spatial and a temporal branch whose predictions are fused by late fusion.222See supp. for implementation details of the considered methods.

5 Results

**Anticipation Results on EPIC-KITCHES  ** Table 1 compares RU with respect to the state-of-the-art on our EPIC-Kitchens validation set. The left part of the table reports Top-5 action accuracy for the $8$ anticipation times. Note that some methods [8, 16, 58] can anticipate actions only at a fixed anticipation time. The right part of the table breaks down Top-5 accuracy and Mean Top-5 Recall for verbs, nouns and actions, for anticipation time $\tau_{a}=1s$, and reports mean $TtA(5)$ scores. Best results are in bold, whereas second-best results are underlined. The last row reports the improvement of RU with respect to second best results.

The proposed approach outperforms all competitors by consistent margins according to all evaluation measures, reporting an average improvement over prior art of about $5\%$ on Top-5 action accuracy with respect to all anticipation times. Note that this margin is significant given the large number of $2,513$ action classes present in the dataset. Methods based on TSN (ATSN and MCE) generally achieve low performance, which suggests that simply adapting action recognition methods to the problem of anticipation is insufficient. Interestingly, DMR and ED, which are explicitly trained to anticipate future representations, achieve sub-optimal Top-5 action accuracy as compared to methods trained to predict future actions directly from input images (e.g., compare DMR with MCE, and ED with FN/RL/EL). This might be due to the fact that anticipating future representations is particularly challenging in the case of egocentric video, where the visual content changes continuously because of the mobility of the camera. RL consistently achieves second best results with respect to all anticipation times, except $\tau_{a}=0.25$, where it is outperformed by EL. RU is particularly strong on nouns, achieving a Top-5 noun accuracy of $51.79\%$ and a mean Top-5 noun recall of $49.90\%$, which improves over prior art by $+7.26\%$ and $+7.31\%$. The small drop in performance when passing from class-agnostic measures to class-aware measures (i.e., $51.79\%$ to $49.90\%$) suggests that our method does not over-fit to the distribution of nouns of the training set. It is worth noting that mean Top-5 Recall values are averaged over fairly large sets of $26$ many-shot verbs, $71$ many-shot nouns, and $819$ many-shot actions, as specified in [8]. Differently, all compared methods exhibit large drops in verb and action performance when passing from class-agnostic to class-aware measures. Our insight into this different pattern is that anticipating the next object which will be used (i.e., anticipating nouns) is much less ambiguous than anticipating the way in which the object will be used (i.e., anticipating verbs and actions). It is worth noting that second best Top-5 verb and noun accuracy are achieved by different methods (i.e., ATSN in the case of verbs and RL in the case of nouns), while both are outperformed by the proposed RU. Despite its low performance with respect to class-agnostic measures, ED systematically achieves second best results with respect to mean Top-5 recall and mean $TtA(5)$. This highlights that there is no clear second-best performing method. Finally, mean $TtA(k)$ highlights that the proposed method can anticipate verbs, nouns and actions $1.62$, $1.11$ and $0.76$ seconds in advance respectively.

Table 8 assesses the performance of the proposed method on the official test sets of EPIC-Kitchens dataset.333See supp. for precision and recall results. RU outperforms all competitors by consistent margins on both the “seen” test, which includes scenes appearing in the training set (S1) and the “unseen” test set, with scenes not appearing in the training set (S2). Also in this case, RU is strong on nouns, obtaining $+6.56\%$ and $+8.8\%$ in S1, as well as $+4.78\%$ and $+4.88$ in S2. Improvements in terms of actions are also significant: $+3.63\%$ and $+5.52\%$ in S1, as well as $+2.59\%$ and $+5.39\%$ on S2.

**Anticipation Results on EGTEA Gaze+  **

Table 3 reports Top-5 action accuracy scores achieved by the compared methods on EGTEA Gaze+ with respect to the $8$ considered anticipation times. The table also reports mean $TtA(5)$ action scores, denoted as $TtA$. The proposed method outperforms all competitors at all anticipation times. Note that the margins of improvement are smaller on EGTEA Gaze+ due to its smaller-scale ($106$ actions in vs $2,513$ actions in EPIC-KITCHENS). Differently from Table 1, EL systematically achieves second best performance on EGTEA Gaze+, which highlights again that there is no clear second best competitor to RU in our evaluations.

**Ablation Study on EPIC-Kitchens  ** To assess the role of rolling-unrolling, we consider a strong baseline composed of a single LSTM (same configuration as R-LSTM) and three branches (RGB, Flow, OBJ) with late fusion (BL). To tear apart the influence of rolling-unrolling and MATT, Table 4(a) compares this baseline with the proposed RU architecture, where MATT has been replaced with late fusion. The proposed RU approach brings systematic improvements over the baseline for all anticipation times, with larger improvements in the case of the object branch.

Table 4(b) reports the performances of the single branches of RU and compares MATT with respect to late fusion (i.e., averaging predictions) and early fusion (i.e., feeding the model with the concatenation of the modality-specific representations). MATT always outperforms late fusion, which consistently achieves second best results. Early fusion always leads to sub-optimal results. All fusion schemes always improve over the single branches. Figure 6 shows regression plots of modality attention weights computed on all samples of the validation set. RGB and OBJ weights are characterized by a strong and steep correlation. A similar pattern applies to Flow and OBJ weights, whereas Flow and RGB weights are characterized by a small positive correlation. This suggests that MATT gives more credit to OBJ when RGB and Flow are less informative, whereas the it relies on RGB and Flow when objects are not necessary.

Table 4(c) assesses the role of pre-training the proposed architecture with sequence completion. As can be noted, the proposed pre-training procedure brings small but consistent improvements for most anticipation times. Table 4(d) compares RU with the strong baseline of Table 4(a). The comparison highlights the cumulative effect of all the proposed procedures/component with respect to a strong baseline using three modalities. It is worth noting that the proposed architecture brings improvements for all anticipation times, ranging from $+1.53\%$ to $+4.08\%$.

**Qualitative Results  ** Figure 7 reports two examples of anticipations made by the proposed method at four anticipation times. Under each frame we report top-4 predictions, whereas modality weights computed by MATT are reported in percentage on the right. Green bounding boxes are shown around the detected objects and the optical flow is illustrated in orange. In the first example (top), the model can predict “close door” based on context and past actions (e.g., taking objects out of the cupboard), hence it assigns large weights to RGB and Flow and low weights to OBJ. In the second example (bottom), the model initially predicts “squeeze lime” at $\tau_{a}=2s$. Later, the prediction is corrected to “squeeze lemon” as soon as the lemon can be reliably detected. Note that in this case the network assigns larger weights to OBJ as compared to the previous example.444See supp. and https://iplab.dmi.unict.it/rulstm/ for additional examples and videos.

**Early Action Recognition  ** We also observe that the proposed method generalizes to the task of early action recognition. We adapt our processing scheme by sampling $8$ video snippets from each action segment uniformly in time and set $S_{enc}=0$, $S_{ant}=8$. The snippets are fed to the model, which produces predictions at each time-step, corresponding to the following observation rates: $12.5\%$, $25\%$, $37.5\%$, $50\%$, $62.5\%$, $75\%$, $87.5\%$, $100\%$. Branches are fused by late fusion in this case. We compare the proposed method with respect to FN, EL and RL.

Table 5 reports Top-1 early action recognition accuracy results obtained by the compared methods on our validation set of EPIC-Kitchens. The proposed method consistently outperforms the competitors at all observation rates. Interestingly, RU achieves an early action recognition accuracy of $33.09\%$ when observing only $75\%$ of the action, which is already comparable to the accuracy of $34.07\%$ achieved when the full action is observed. This indicates that RU can timely recognize actions before they are completed. RL achieves second best results up to observation rate $75\%$, whereas EL achieves second best results when more than $75\%$ of the action is observed, which confirms the lack of a clear second-best performer. Table 6 reports Top-1 accuracy results obtained on EGTEA Gaze+. The proposed RU outperforms the competitors for all observation rates by small but consistent margins. Coherently with Table 5, second best results are obtained by RL and EL.

**Egocentric Action Recognition Results  ** The proposed method can be used to perform egocentric action recognition by considering the predictions obtained for the observation rate of $100\%$. Table 9 compares the performance of the proposed method with other egocentric action recognition methods on the two test sets of EPIC-Kitchens.555See supp. for precision and recall results. Our RU outperforms all competitors in recognizing actions and nouns on both sets by significant margins, whereas it achieves second-best results in most cases for verb recognition. RU obtains an action recognition accuracy of $60.2\%$ on EGTEA Gaze+. Despite being designed for action anticipation, RU outperforms recent approaches, such as Li et al. [36] ($+6.9\%$ wrt $53.3\%$) and Zhang et al. [66] ($+3.19\%$ wrt $57.01\%$ - reported from [55]), and obtaining performances comparable to state-of-the-art approaches such as Sudhakaran and Lanz [56] ($-0.56\%$ wrt $60.76$) and Sudhakaran et al. [55] ($-1.66\%$ wrt $61.86\%$).666See supp. for the full table.

6 Conclusion

We presented RU-LSTM, a learning architecture which processes RGB, optical flow and object-based features using two LSTMs and a modality attention mechanism to anticipate actions from egocentric video. Experiments on two datasets show the superiority of the approach with respect to prior art and highlight generalization over datasets and tasks: anticipation, early recognition, and recognition.

Acknowledgment

This research is supported by Piano della Ricerca 2016-2018, linea di Intervento 2 of DMI, University of Catania.

Appendix A Implementation Details and Training Procedure of the Proposed Method

This section reports the implementation and training details of both the proposed and compared methods. A diagram of our architecture is reported in Figure 8 for the convenience of the reader. The reader is referred to the paper for a description of the architecture.

A.1 Architectural Details of RU-LSTM and MATT

We use a Batch Normalized Inception CNN [26] (BNInception) in the spatial and flow branches and consider the $1024$-dimensional vectors produced by the last global average pooling layer of the network as output representations. Optical flows are extracted using the TVL1 algorithm [63]. Specifically, we use the pre-computed optical flows provided by the authors in the case of EPIC-Kitchens (see http://EPIC-Kitchens.github.io/) and compute optical flows on EGTEA Gaze+ using the code provided in https://github.com/feichtenhofer/gpu_flow with default parameters. At test time, the CNNs are fed with input images and optical flows resized to $456\times 256$ pixels. Note that, due to global average pooling, the output of the BNInception CNN will be a $1024$ feature vector regardless size of the input image. We found this setting leading to better performance as compared to extracting a $224\times 224$ crop from the center of the image. For the object branch, we use a Faster R-CNN object detector [20] with a ResNet-101 backbone [22], as implemented in [21]. Both the Rolling LSTM (R-LSTM) and the Unrolling LSTM (U-LSTM) contain a single layer with $1024$ hidden units. Dropout with $p=0.8$ is applied to the input of each LSTM and to the input of the final fully connected layer used to obtain class scores. The Modality ATTention network (MATT) is a feed-forward network with three fully connected layers containing respectively $h/4$, $h/8$ and $3$ hidden units, where $h=6144$ is the dimension of the input to the attention network (i.e., the concatenation of the hidden and cell states of $1024$ units each related to the three R-LSTMs). Dropout with $p=0.8$ is applied to the input of the second and third layers of the attention network to avoid over-fitting. ReLU activation function are used within the attention network.

A.2 Training Procedure of RU-LSTM and MATT

While the proposed architecture could be in principle trained in an end-to-end fashion, we found it extremely challenging to avoid over-fitting during end-to-end training. This is mainly due to the indirect relationship between input video and future actions. Indeed, differently from action recognition, where the objects and actions to be recognized are present or take place in the input video, in the case of action anticipation, the system should be able to anticipate objects and actions which do not always appear in the input video, which makes it hard to learn good representations end-to-end. To avoid over-fitting, the proposed architecture is trained as follows. First, we independently train the spatial and motion CNNs for the task of egocentric action recognition within the framework of TSN [61]. Specifically, we set the number of segments to $3$ and train the TSN models with Stochastic Gradient Descent (SGD) using standard cross entropy for $160$ epochs with an initial learning rate equal to $0.001$, which is decreased by a factor of $10$ after $80$ epochs. We use a mini-batch size of $64$ samples and train the models on a single Titan X. For all other parameters, we use the values recommended in [61]. We train the object detector to recognize the $352$ object classes of the EPIC-Kitchens dataset. We use the same object detector trained on EPIC-Kitchens when performing experiments on EGTEA Gaze+, as the latter dataset does not contain object bounding box annotations. This training procedure allows to learn the parameters $\theta^{1}$, $\theta^{2}$ and $\theta^{3}$ of the representation functions related to the three modalities (i.e., RGB, Flow, OBJ). After this procedure, these parameters are fixed and they are no more optimized. For efficiency, we pre-compute representations over the whole dataset.

Each branch of the RU-LSTM is training with SGD using the cross entropy loss with a fixed learning rate equal to $0.01$ and momentum equal to $0.9$. Each branch is first pre-trained with Sequence Completion Pre-training (SCP). Specifically, appearance and motion branches are trained for $100$ epochs, whereas the object branch is trained for $200$ epochs. Branches are then fine-tuned for the action anticipation task. Once each branch has been trained, the complete architecture with three branches is assembled to form a three-branch network and the model is further fine-tuned for $100$ epochs using cross entropy and the same learning parameters. In Figure 8 an example of a two-branches architecture is shown.

In the case of early action recognition, each branch is trained for $200$ epochs (both SCP and main task) with a fixed learning rate equal to $0.01$ and momentum equal to $0.9$.

Note that, in order to improve performances, we apply early stopping at each training stage. This is done by choosing the iterations of the intermediate and final models which obtain the best Top-5 action anticipation accuracy for the anticipation time $\tau_{a}=1s$ on the validation set. In the case of early action recognition, we choose the epoch obtaining the best average Top-1 action accuracy across observation rates. The same early stopping strategy is applied to all the methods for fair comparison. The proposed RU-LSTM architecture has been implemented using the PyTorch library [43]. The code will be provided upon publication, together with all details and data useful to replicate the results.

A.2.1 Note on End-To-End Training

We chose to fix the feature extractors in our work as we experienced over-fitting when training the model end-to-end. Specifically, we tried the following: (1) Training the RGB branch end-to-end from scratch (except the CNN, which is pre-trained on Imagenet), (2) Pre-training the CNN on the action recognition task with TSN, then training the RGB branch end-to-end, (3) Training the RGB branch using fixed representations as described in the paper, then fine-tuning the CNN + RU-LSTM model end-to-end. In our experiments, (1) and (2) led to poor performance already at the sequence-completion stage, while (3) did not improve performance. Our insight is that the indirect relationship between the observed scene and the action yet to take place can make learning representations end-to-end much more difficult than in the case of recognition. For instance, when anticipating the action “take cup”, the object “cup” may or may not be present in the observed video segment, which makes unclear what visual features the CNN should extract.

A.3 Inference At a Fixed Anticipation Time

Our model makes multiple anticipations at time-steps $7-14$. Also, since the predictions are updated as the video is processed and more evidence is acquired, such predictions may indeed be inconsistent (with anticipations performed closer to the beginning of the action being more likely to be correct). However, it should be noted that each prediction is deemed to be specific to a given anticipation time. For instance, at time-step $11$, the model tries to anticipate actions happening in $1s$. Therefore, the proposed approach can be used to anticipate actions at a fixed anticipation time by processing the buffered video up to the related time-step, discarding all other predictions. E.g., if the anticipation time is set to $\tau_{a}=1s$, the model should process the last $11$ time-steps.

A.4 Choice of Parameters $\alpha$, $S_{enc}$ and $S_{ant}$.

In this work, we set $\alpha=0.25s$ and $S_{ant}=8$ to generalize the settings of the EPIC-Kitchens anticipation challenge. Indeed, in these settings, we can anticipate actions up to $2s$ in advance ($8\times 0.25s$), while still being able to produce anticipations at anticipation time $\tau_{a}=1s$ ($4\times 0.25s$) as required for the challenge. We investigated the effect of $S_{enc}$ when we fix $\alpha=0.25s$ and $S_{dec}=8$. We noted that the choice of $S_{enc}$ affects performance lightly and hence chose $S_{enc}=6$ to maximize action anticipation performance for anticipation time $\tau_{a}=1s$.

Appendix B Implementation Details of the Compared Methods

Since no official public implementation is available for the compared methods, we performed experiments using our own implementations. In this section, we report the implementation details of each of the compared method.

B.1 Deep Multimodal Regressor (DMR)

We implement the Deep Multimodal Regressor proposed in [58] setting the number of multi-modal branches with interleaved units to $k=3$. For fair comparisons, we substituted the AlexNet backbone originally considered in [58] with a BNInception CNN pre-trained on ImageNet. The CNN is trained to anticipate future representations extracted using BNInception pre-trained on ImageNet using the procedure proposed by the authors. Specifically, we perform mode update every epoch. Since training an SVM with large number of classes is challenging (in our settings, we have $2,513$ different action classes), we substituted the SVM with a Multi Layer Perceptron (MLP) with $1024$ hidden units and dropout with $p=0.8$ applied to the input of the first and second layer. To comply with the pipeline proposed in [58], we pre-train the model on our training split of EPIC-Kitchens in an unsupervised fashion and train the MLP separately on representations pre-extracted from the training set using the optimal modes found at training time. As a result, during the training of the MLP, the weights of the CNN are not optimized. The DMR architecture is trained with Stochastic Gradient Descent using a fixed learning rate equal to $0.1$ and a momentum equal to $0.9$. The network is trained for several epochs until the validation loss saturates. Note that training the CNN on the EPIC-Kitchens dataset takes several days on a single Titan X GPU using our implementation. After training, we apply early stopping by selecting the iteration with the lowest validation loss. The MLP is then trained with Stochastic Gradient Descent with fixed learning rate equal to $0.01$ and momentum equal to $0.9$. Early stopping is applied also in this case considering the iteration of the MLP achieving the highest Top-5 action accuracy on the validation set.

B.2 Anticipation TSN (ATSN)

We implement this model considering the TSN architecture used to pre-train the CNNs employed in the RGB and Flow branches of our architecture. We modify the network to output verb and noun scores and train it summing the cross entropy losses applied independently to verbs and nouns as specified in [8]. At test time, we obtain action probabilities by assuming independence of verbs and nouns as follows: $p(a=(v,n)|x)=p(v|x)\cdot p(n|x)$, where $a=(v,n)$ is an action involving verb $v$ and noun $n$, $x$ is the input sample, whereas $p(v|x)$ and $p(n|x)$ are the probabilities computed directly by the network.

B.3 ATSN + VNMCE Loss (MCE)

This method is implemented training the TSN architecture used for ATSN with the Verb-Noun Marginal Cross Entropy Loss proposed in [16]. We used the official code provided by the authors (https://github.com/fpv-iplab/action-anticipation-losses/).

B.4 Encoder-Decoder LSTM (ED)

We implement this model following the details specified in [18]. For fair comparison with respect to the proposed method, the model takes RGB and Flow features obtained using the representation functions as input for the RGB and Flow modalities used in our RU architecture. Differently from [18], we do not include a reinforcement learning term in the loss as our aim is not to distinguish the action from the background as early as possible as proposed in [18]. The hidden state of the LSTMs is set to $2048$ units. The model encodes representations for $20$ steps, while decoding is carried out for $10$ steps at a step-size of $0.25s$. The architecture is trained on top of pre-extracted representations for $100$ epochs with the Adam optimizer and a learning rate of $0.001$.

B.5 Feedback-Network LSTM (FN)

The method proposed in [19] has been implemented considering the most performing architecture investigated by the authors, which comprises the “optional” LSTM layer and performs fusion by concatenation. The network uses our proposed video processing strategy. For fair comparison, we implement the network as a two-stream architecture with two branches processing independently RGB and Flow features. Final predictions are obtained with late fusion (equal weights for the two modalities). We use the representation functions of our architecture to obtain RGB and Flow features. The model has hidden layers of $1024$ units, which in our experiments leaded to improved results with respect to the $128$ features proposed by the authors [19]. The model is trained using the same parameters used in the proposed architecture.

B.6 RL & EL

These two methods are implemented considering a single LSTM with the same parameters of our Rolling LSTM. Similarly to FN, the models are trained as two-stream models with late fusion used to obtain final predictions. The input RGB and Flow features are computed using the representation functions considered in our architecture. The models are trained with the same parameters used in the proposed architecture. RL is trained using the ranking loss on the detection score proposed in [39], whereas EL is trained using the exponential anticipation loss proposed in [28].

Appendix C Additional Results

This section reports the full set of anticipation and recognition results on EPIC-Kitchens, including precision and recall, as well as the full table of comparisons of the proposed method on EGTEA Gaze+ for action recognition.

Table 8 compares the proposed method with respect to the competitors according to the full set of measures proposed along with the egocentric action anticipation challenge [8], including precision and recall (which could not be included in the paper due to space limits). The proposed approach outperforms all competitors also according to precision and recall on S1 and S2, except for average verb precision, where it is outperform by the two-stream CNN. Note that, coherently with Top-1 and Top-5 accuracy, the proposed method achieves large gains for noun precision and recall. Also note the small drop in performance between Top-1 noun accuracy and average noun recall on S1 (from $22.78\%$ to $19.81\%$), which highlights balanced noun predictions.

Table 9 compares the proposed method with respect to the competitors according to the full set of measures proposed with the egocentric action recognition challenge [8]. Similarly to Table 8, this includes precision and recall, which could not be included in the paper due to space limits. Similarly to what observed in the case of top-1 and top-5 accuracy, the proposed method outperforms the competitors according to most of the considered measures, despite not being explicitly designed to tackle the recognition task (i.e., our architecture was designed for the egocentric action anticipation task.)

Table 10 compares the proposed RU method against the state-of-the-art when tackling the task of egocentric action recognition on EGTEA Gaze+. It is worth noting that the proposed method outperforms many recent approaches by significant margins. It is also comparable with other state-of-the-art approaches such as the ones proposed in [55, 56]. Again, note that our architecture generalizes despite not being explicitly designed for the recognition task.

Appendix D EPIC-Kitchens Egocentric Action Anticipation Challenge Leaderboards

The proposed RULSTM approach has been used to participate in the EPIC-Kitchens egocentric action anticipation competition. Specifically, we considered an ensemble model including features extracted using a BNInception and a ResNet-50 CNN trained for action recognition. Figure 9 reports a screenshot of the EPIC-Kitchens egocentric action anticipation challenge at the end of the competition. The screenshot has been acquired from https://epic-kitchens.github.io/ on the $1^{st}$ of August, 2019. Note that our submission (team name “DMI-UNICT”) is ranked first on both S1 and S2.

Appendix E Additional Qualitative Examples

Figure 10 reports qualitative results of three additional success action anticipation examples. For improved clarity, we report frames with and without optical flows for each example. In the top example, MATT assigns a small weight to the object branch as the contextual appearance features (i.e., RGB) are already enough to reliably anticipate the next actions. In the middle example object detection is fundamental to correctly anticipate “put down spoon”, as soon as the object is detected. The bottom example shows a complex scene with many objects. The ability to correctly recognize objects is fundamental to anticipate certain actions (i.e., “wash spoon”). The algorithm can anticipate “wash” well in advance. As soon as the spoon is detected ($\tau_{a}=2s$), “wash spoon” is correctly anticipated. Note that, even if the spoon is not correctly detected at time $\tau_{a}=0.5s$, “wash spoon” is still correctly anticipated.

Figure 11 reports three failure examples. In the top example, the model fails to predict “adjust chair”, probably due to the inability of the object detector to identify the chair. Note that, when the object “pan” on the table is detected, “take curry” is wrongly anticipated. In the middle example, the algorithm successfully detects the fridge and tries to anticipate “close fridge” and some actions involving the “take” action, with wrong objects. This is probably due to the inability of the detector to detect “mozzarella”, which is not yet appearing in the scene. In the bottom example, the method tries to anticipate actions involving “jar”, as soon as “jar” is detected. This misleads the algorithm as the correct action is “pour coffee”.

The reader is referred to the videos in the supplementary material for additional success and failure qualitative examples. The supplementary material also reports qualitative examples of the proposed method when applied to the problem of early action recognition.