Recurrent Models for Situation Recognition

TL;DR

This paper introduces RNN-based models for predicting structured image situations, achieving state-of-the-art accuracy and demonstrating transferability to image captioning tasks involving human-object interactions.

Contribution

It presents a novel RNN approach for situation recognition that outperforms previous CRF-based models and shows effective transfer to captioning tasks.

Findings

Achieved state-of-the-art accuracy on imSitu dataset.

RNN models outperform CRF-based approaches.

Transferred features improve image captioning of human-object interactions.

Abstract

This work proposes Recurrent Neural Network (RNN) models to predict structured 'image situations' -- actions and noun entities fulfilling semantic roles related to the action. In contrast to prior work relying on Conditional Random Fields (CRFs), we use a specialized action prediction network followed by an RNN for noun prediction. Our system obtains state-of-the-art accuracy on the challenging recent imSitu dataset, beating CRF-based models, including ones trained with additional data. Further, we show that specialized features learned from situation prediction can be transferred to the task of image captioning to more accurately describe human-object interactions.

Tables

Table 1. Table 1: Situation prediction results on the full imSitu dev set (see text for detail).

		top-1 predicted verb			top-5 predicted verbs			ground truth verbs		mean
		verb	value	value-all	verb	value	value-all	value	value-all
I)	Discrete Classifier [33]	26.4	4.0	0.4	51.1	7.8	0.6	14.4	0.9	13.2
	Image Regression CRF [33]	32.25	24.56	14.28	58.64	42.68	22.75	65.90	29.50	36.32
	Tensor Composition CRF [32]	31.73	24.04	13.73	58.06	42.64	22.70	68.73	32.14	36.72
	Tensor Comp. + Image Reg. CRF [32]	32.91	25.39	14.87	59.92	44.50	24.04	69.39	33.17	38.02
	Above + Extra 5M Images [32]	34.20	26.56	15.61	62.21	46.72	25.66	70.80	34.82	39.57
II)	Baseline RNN Method
Fig. 2a	No Vision, RNN for Nouns	-	-	-	-	-	-	52.12	17.62	-
III)	Joint Prediction       –       VGG jointly fine-tuned for Action and Noun Prediction
Fig. 2b	VGG, RNN for Actions & Nouns	26.52	20.08	11.80	52.37	38.32	20.90	68.27	32.67	33.87
\Centerstack[l] IV) Fig. 2c	VGG, Actions class., RNN for Nouns	23.04	17.65	10.70	44.63	33.18	18.83	68.98	33.73	31.34
	Joint Prediction       –       VGG fine-tuned for Action Prediction Only
	VGG, Actions class., RNN for Nouns	35.35	26.80	15.77	61.42	44.84	24.31	68.44	32.98	38.74
	VGG, Actions class., RNN for Nouns (reversed)	35.35	26.82	15.60	61.42	44.92	24.25	68.56	32.84	38.72
	Joint Prediction       –       VGG fine-tuned for Action Prediction first, then jointly with Noun Prediction
	VGG, Actions class., RNN for Nouns	34.76	26.29	15.46	60.31	44.31	24.30	68.82	33.42	38.46
V)	Action Prediction Only
	VGG, Actions class. (no weighted loss)	34.43	-	-	61.06	-	-	-	-	-
	VGG, Actions class.	35.35	-	-	61.42	-	-	-	-	-
	Fusion (no weighted loss)	35.53	-	-	63.04	-	-	-	-	-
	Fusion	36.11	-	-	63.11	-	-	-	-	-
	Noun Prediction Only
	VGG+RNN for Nouns	-	-	-	-	-	-	68.57	33.12	-
	VGG+RNN for Nouns, VGG fine-tuned (ft)	-	-	-	-	-	-	70.48	35.56	-
	VGG+RNN with Attention for Nouns	-	-	-	-	-	-	69.31	33.67	-
	VGG+RNN with Attention for Nouns (ft)	-	-	-	-	-	-	69.87	34.69	-
\Centerstack[l] VI) Fig. 2d	Separate Action and Noun Prediction
	Fusion for Actions, VGG+RNN for Nouns (ft)	36.11	27.74	16.60	63.11	47.09	26.48	70.48	35.56	40.40
	henceforth ref. to as Fusion, VGG+RNN

Table 2. Table 2: Situation prediction results on the full imSitu test set.

	top-1 predicted verb			top-5 predicted verbs			ground truth verbs		mean
	verb	value	value-all	verb	value	value-all	value	value-all
Image Regression CRF [33]	32.34	24.64	14.19	58.88	42.76	22.55	65.66	28.96	36.25
Tensor Comp. + Image Reg. CRF [32]	32.96	25.32	14.57	60.12	44.64	24.00	69.20	32.97	37.97
Above + Extra 5M Images [32]	34.12	26.45	15.51	62.59	46.88	25.46	70.44	34.38	39.48
Fusion, VGG+RNN	35.90	27.45	16.36	63.08	46.88	26.06	70.27	35.25	40.16

Table 3. Table 3: Situation prediction results on the rare portion of the imSitu test set. Along with better verb prediction accuracy, our method also produces more accurate role values given GT verbs, indicating better generalization probably due to the use of shared parameters and word embeddings.

	top-1 predicted verb			top-5 predicted verbs			ground truth verbs		mean
	verb	value	value-all	verb	value	value-all	value	value-all
Image Regression CRF [33]	20.61	11.79	3.07	44.75	24.85	5.98	50.37	9.31	21.34
Tensor Comp. + Image Reg. CRF [32]	19.96	11.57	2.30	44.89	25.26	4.87	53.39	10.15	21.55
Above + Extra 5M Images [32]	20.32	11.87	2.52	47.07	27.50	6.35	55.72	12.28	22.95
Fusion, VGG+RNN	22.07	12.96	3.37	47.83	27.89	6.85	56.38	13.79	23.89

Table 4. Table 4: Caption generation model performance on the COCO test set (5000 images) of Karpathy et al .   [ 11 ] . B@N , M , C , and S indicate BLEU@N  [ 18 ] , METEOR  [ 14 ] , CIDEr  [ 25 ] , and SPICE  [ 2 ] respectively. † indicates a different split of 4000 images and Σ Σ \Sigma indicates an ensemble of models. Bold values indicate the highest value for metrics obtained using a single model.

Methods	B@1	B@2	B@3	B@4	M	C	S
LRCN [6]	62.8	44.2	30.4	21.0	-	-	-
img-gLSTM [10]	64.7	45.9	31.1	21.4	20.4	67.7	-
NIC [26]${}^{\text{†},\Sigma }$	66.6	46.1	32.9	24.6	-	-	-
img-gLSTM [10]	67.0	49.1	35.8	26.4	22.7	81.3	-
Hard-Attention [29]	71.8	50.4	35.7	25.0	23.0	-	-
Soft-Attention [29]	70.7	49.2	34.4	24.3	23.9	-	-
ATT-FCN [34]Σ	70.9	53.7	40.2	30.4	24.3	-	-
NeuralTalk2 [1] (Ours)	70.8	53.7	40.1	30.1	24.5	93.0	17.3
Image + Actions (Ours)	71.5	54.6	40.9	30.9	24.7	94.5	17.6
Image + Nouns (Ours)	71.5	54.6	41.1	31.1	24.8	95.2	17.7

Table 5. Table 5: Caption generation model performance on the COCO test2014 online leaderboard. We list results that have been published and highlight our implemented baseline and methods. Note that the top methods use ensembles, better model architectures, and other engineering tricks such as scheduled sampling, beyond the scope of this work. The c5 test setting uses 5 reference captions and c40 uses 40 reference captions. Σ Σ \Sigma indicates an ensemble of models, ? ? ? indicates unspecified if ensemble.

Methods	BLEU-1		BLEU-2		BLEU-3		BLEU-4		METEOR		ROUGE		CIDEr
	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40	c5	c40
ATT-FCN [34]Σ	73.1	90.0	56.5	81.5	42.4	70.9	31.6	59.9	25.0	33.5	53.5	68.2	94.3	95.8
OriolVinyals [27]Σ	71.3	89.5	54.2	80.2	40.7	69.4	30.9	58.7	25.4	34.6	53.0	68.2	94.3	94.6
MSR_Captivator [5]?	71.5	97.0	54.3	81.9	47.0	71.0	38.0	61.0	24.8	33.9	52.6	68.0	93.1	93.7
Q.Wu [28]?	72.5	89.2	55.6	80.3	41.4	69.4	30.6	58.2	24.6	32.9	52.8	67.2	91.1	92.4
NeuralTalk2 [1] (Ours)	70.6	87.9	53.2	77.8	39.2	66.1	29.0	54.7	24.2	32.4	51.9	66.0	88.1	89.1
Image + Actions (Ours)	71.1	88.6	53.9	79.0	40.1	67.7	30.1	56.7	24.4	33.0	52.3	66.8	90.1	90.7
Image + Roles (Ours)	71.2	88.7	54.0	79.4	40.3	68.2	30.2	57.2	24.6	33.2	52.4	67.0	90.7	91.8

Equations

$$p(S|I;\theta )=\frac{1}{Z}\cdot {\psi }_v(v|I;\theta )\cdot \prod _{\begin{array}{c}(r_i,n_i)\\ r_i\in R_v,n_i\in N\cup \{\varnothing \}\end{array}}{\psi }_r(v,r_i,n_i|I;\theta ).$$

$$p(S|I;\theta )$$

$$=p\left(v,n_1,\cdots ,n_{|R_v|}|I;\theta \right)$$

$$=p(v|I;\theta )\prod _{t=1}^{|R_v|}p\left(n_t|v,n_1,\cdots ,n_{t-1},I;\theta \right).$$

Full text

1 Introduction

Recognition of actions and human-object interactions in still images has been widely studied in computer vision. Early datasets and approaches focused on identifying a relatively small number of actions, such as 10 in PASCAL VOC [7] and 40 in the Stanford Dataset [30]. Newer and larger datasets such as MPII Human Pose [19] have enlarged the number of action classes to around 400. The COCO-A [21] and HICO [4] datasets aim to recognize interactions between multiple humans, and humans and objects, expanding the scope of recognition to outputs such as human-riding-bicycle, human-repairing-bicycle, human-riding-horse, etc.

Of late, the focus has shifted to predicting even more structured outputs, tackling higher-level questions such as who is doing what and with which object. The recently introduced imSitu Dataset [33] generalizes the task of action recognition to ‘situation recognition’ — the recognition of all entities fulfilling semantic roles in an instance of an action performed by a human or non-human actor. Given a particular action, situations are represented by a set of relevant (semantic role: noun entity) pairs. An example image and associated situation from imSitu are shown in Fig. 1, where “a woman arranging flowers in a vase on the countertop” is represented by Action: arranging, {(Agent: woman), (Item: flowers), (Tool: vase), (Place: countertop)}. As another example, “A horse rearing outside” can be mapped to Action: rearing, {(Agent: horse), (Place: outside)}. imSitu consists of 504 actions, 1,700 semantic roles, and 11,000 noun entities resulting in around 200,000 unique situations. Along with the dataset, Yatskar et al. [33, 32] also introduced Conditional Random Field (CRF) models to predict situations given an image. In our work, we propose and train Recurrent Neural Networks (RNNs) to predict such situations and outperform the previously state of the art CRFs.

Our use of RNNs for situation prediction is motivated by their popularity for tasks like image caption generation, where they have proven to be successful at capturing grammar and forming coherent sentences linking multiple concepts. The standard framework for caption generation involves feeding high-level features from a CNN, often trained for image classification on ImageNet [22], into an RNN that proceeds to generate one word of the caption at a time [11, 26, 27, 6, 34]. Situation recognition involves the prediction of a sequence of noun entities for a particular action, so it can be viewed as a more structured version of the captioning task with a grammar that is fixed given an action.

Figure 1 gives an overview of our best proposed system. First, we predict the action verb using the specialized action recognition architecture of [16], which fuses features from a detected person box with a global representation of the image. Conditioned on the action, we treat the prediction of noun entities as a sequence generation problem and use an RNN. Details of our model, along with several baselines, will be given in Section 2. Through extensive experiments (Section 3) we found that using separate networks for predicting the action verb and the noun entities produces higher accuracy than jointly training a visual representation for the two tasks. Finally, in Section 4 we explore how knowledge gained from situation prediction can obtain meaningful improvements for image captioning on the MSCOCO dataset [15] through feature transfer.

2 The Situation Prediction Task and Methods

Situations are based on a discrete set of action verbs $V$, noun entities $N$, and semantic roles $R$. Each verb $v\in V$ is paired with a unique frame $f\in F$ derived from FrameNet [8], a lexicon for semantic role labeling. A frame is a collection of semantic roles $R_{v}\subset R$ which are associated with the verb $v$. For example, the semantic roles {Agent, Item, Tool, Place$\}\subset R$ are associated with the verb arranging. In an instantiation of an action in an image, each semantic role is fulfilled by some noun $n\in N\cup\{\varnothing\}$, where $\varnothing$ indicates that the value is either not known or does not apply. The set of nouns $N$ is derived from WordNet [17]. An instance of an action $v$ in an image $I$ forms a realized frame $F_{(I,v)}$ in which each semantic role is associated with some noun $n$, i.e. $F_{(I,v)}=\{(r_{i},n_{i}):r_{i}\in R_{v},n_{i}\in N\cup\{\varnothing\},i=1,\cdots,|R_{v}|\}$. Finally, a situation $S$ is the pair of action and realized frame for that action, $S=\{v,F_{(I,v)}\}$. The task of situation prediction is to predict an action verb and its associated realized frame given an image. Though each image is annotated with a single verb, multiple situations might be applicable for an image due to the choice of nouns used to form a realized frame. For example, one might use the term countertop instead of kitchen as the noun associated with the semantic role of Place in Fig. 1. To account for this multiplicity, the imSitu dataset provides three independently labeled situations per image.

The authors who introduced situtation prediction also proposed a CRF-based approach for the task [33]. They decompose the structured prediction of a situation, $S=\{v,F_{(I,v)}\}$, over the verb $v$ and semantic role value pairs $(r,n)$ in the realized frame $F_{(I,v)}$. They learn a potential function $\psi_{v}(v;\theta)$ for every verb, and a potential function for every verb, semantic role, noun entity tuple $\psi_{r}(v,r,n;\theta)$ ($v\in V$, $r\in R_{v}$, $n\in N\cup\{\varnothing\}$), where $\theta$ denotes the parameters of the deep neural network used to predict these potentials. The probability of a particular situation $S$ given input image $I$ can thus be represented by:

[TABLE]

The CRF normalization constant $Z$ required for computing the loss during training is obtained by predicting the potentials for all valid tuples found in the training set and then summing them. The potentials are predicted using a fully connected layer on top of the fc7 layer of the VGG-16 network [23]. During inference time, all valid tuples are scored and ranked. A difficulty with this approach is the large number of potentials that need to be predicted: 504 for all possible verbs and 121,381 for all valid verb, semantic role, noun entity tuples. Further, this model does not explicitly account for the fact that nouns are shared across semantic roles, though it is possible that the deep neural network implicitly learns such representations. In order to explicitly enforce the sharing of information and reduce the number of parameters, the follow-up work by Yatskar et al. [32] further decomposes the potentials as a tensor product over verbs, semantic roles, and noun entities. This makes for a complex model, details of which can be found in [32].

We take an alternate view of situation prediction by observing that given a verb $v$, the set of semantic roles $R_{v}$ associated with it is fixed. For example, given the verb arranging, we know that we have to predict relevant noun entities for the semantic roles of $R_{arranging}$={Agent, Item, Tool, Place} (see Fig. 1). Conditioned on a given verb, if we assume some arbitrary but fixed ordering over these semantic roles, we can reduce the problem to that of sequential prediction of noun entities corresponding to the semantic roles. We decompose $p(S|I;\theta)$ as:

[TABLE]

Note that if an arbitrary but fixed ordering is chosen for semantic roles belonging to every verb, then Eq. (3) follows from Eq. (2) as the correspondence of nouns to roles is implicit. In our implementation, we use the semantic role ordering provided in the dataset, which was derived from FrameNet [8]. We explore the sensitivity of methods to the specific ordering in the experiments of Section 3, and find that the accuracy is affected only to a very small degree.

We represent each $p\left(n_{t}|v,n_{1},\cdots,n_{t-1},I;\theta\right)$ in Eq. (4) with a softmax over all the noun entities in the training dataset, referred to as the noun vocabulary. This is a standard formulation first introduced for natural language translation [24] and widely adopted for image captioning [15, 26, 27, 29]. Similar to these works, we use a softmax classification loss with the corresponding ground truth noun entity as the target at every prediction step.

It is worth pointing out that both formulations, those of CRF-based structured prediction (Eq. (1)) and sequential prediction (Eq. (4)), are equally powerful in their representational abilities as both model the joint probability of the verb and noun entities in a proposed situation. At inference time, in the CRF approach of [32, 33], all valid tuples of verb and noun entities are evaluated and the most likely one is reported, while in our sequential approach, we perform approximate inference by selecting the most likely noun entity at each step. Despite this limitation, we obtain satisfactory empirical results (we also experimented with beam search but did not see an improvement).

Next, we present the progression of models we developed, starting with a language-only baseline and ending in our highest-performing method illustrated in Figure 1.

A) No vision, RNN for Nouns. In order to verify that sequential situation prediction can actually work and that an RNN can memorize the specific ordering of semantic roles for each verb, we propose a basic language-only model that only tries to predict noun entities given the ground truth verb. This model also acts as a strong baseline by exploiting bias in the dataset labeling as it does not use any visual feature input. This model is depicted in Fig. 2a. The ground truth verb is fed in at the first time step. Note that it is essential to feed in the verb at the first time step as the ordering and number of semantic roles for which noun entities are produced is decided by the choice of verb. At the following time step, the RNN tries to predict the noun entity associated with the first semantic role in the arbitrarily selected but fixed ordering, and so on, until a noun entity is predicted for each semantic role for that verb. In line with prior work [24, 26], we feed in the initial verb and the output of the previous time step as a one-hot vector through a word embedding layer. As will be discussed in the next section, this RNN can indeed memorize the arbitrary semantic role ordering to make noun entity predictions in the appropriate order.

B) Shared network, RNN for Actions & Nouns. The next natural step is to extend the above no-vision model to use image features and predict the action as well. This model is shown in Fig. 2b. After consuming the fc7 image features from a VGG-16 network at the first time step, the model predicts the action at the second time step and then continues on to predict noun entities. The noun vocabulary (space of all noun entities) is extended with that of possible actions to allow the prediction of both. Note that we use the ground truth action as input during training and the predicted action during testing. At inference time, we enforce that only an action can be predicted at the second time step, followed by noun entities only thereafter.

C) Shared network, Actions classifier, RNN for Nouns. Since situation recognition has such a strong up-front dependence on the action verb, the next question we want to explore is whether we can improve performance by breaking off the action prediction into a specialized task, instead of treating it the same as the other roles. It also helps that imSitu has many fewer verbs (504) than noun entities (11K), giving us enough data to train a dedicated action classifier. Accordingly, our second model predicts actions using a separate fully-connected classification layer on top of the fc7 layer of the VGG-16 network as shown in Fig. 2c. At the first step of the RNN, we feed in the one-hot representation of the action (at training time, we use the ground truth action and at test time, the predicted action). At the second time step, we feed in the fc7 image features to the RNN to predict noun entities. Our experiments will investigate how to train the VGG network to get the highest accuracy for the overall task. One option is to train it solely for action prediction and another is to jointly train it for both action and noun prediction. Interestingly, our results in Section 3 will show that the former strategy works better.

D) Separate networks, Actions classifier, RNN for Nouns. The lack of success of joint training leads to the question of whether we can do even better by not sharing parameters between action and noun entity prediction. Accordingly, our final model decouples the two tasks and uses two separate networks that are independently fine-tuned, as depicted in Fig. 2d. For predicting actions, we use the feature fusion network of [16] which obtained state-of-the-art performance on the HICO dataset [4]. This network (called Fusion in the following) combines local features from detected human boxes and global features from the whole image to make predictions that are then pooled. It defaults to the full image in case no human is detected in the image. As a large number of images in the imSitu dataset feature humans, this is a reasonable choice of architecture. Along with a vanilla RNN for predicting noun entities, we will also report experiments with an attention model based on [29] which consumes image features through a soft attention module at each time step. Note that instead of the fc7 features, the attention-based RNN uses the conv5 feature map.

3 Situation Prediction Experiments

Implementation Details. We use the simplified Long-Short Term Memory (LSTM) cell [9, 35] as our RNN model. We use a single-layer LSTM and with input and hidden layer sizes of 512. We did not observe any significant improvement by using larger layer sizes or more layers. The imSitu dataset has a total of 504 actions and 11,790 noun entities, leading to an LSTM output layer size of 11,790 in the case of models A, C, and D and 11,790+504 in the case of model B. We train all our RNNs with Adam [13] using an initial learning rate of 4e-4, decayed by a factor of 10 every 28,800 iterations using a batch size of 64. For noun entity prediction, we first train the RNN for 60k iterations. We then turn on fine-tuning for the CNN with an initial learning rate of 1e-5 and use Adam with the same learning rate decay scheme for an additional 100k iterations. The Fusion network [16] is trained using stochastic gradient descent with momentum using a learning rate of 5e-5 for 70k iterations. Person boxes are detected using the Faster-RCNN [20] with a confidence threshold of 0.8. Similar to [16], we use a weighted loss during action prediction, unless otherwise specified. The weight for a class is inversely proportional to its frequency in the training set. Using weighted loss or beam search for noun entity prediction did not help. We only train on the imSitu train set of 75k images. During training, we evaluate the model on the dev set of 25k images and retain the best-performing model. Finally, we evaluate the best model on the imSitu test set of 25k images. All hyperparameters are tuned on the dev set.

Metrics. We evaluate performance on action verb predictions (verb), and (semantic role: noun entity) pair predictions (value, value-all) as well as the average across all measures (mean), as proposed in [32]. Value-all measures the percentage of predictions for which all of the (semantic role: noun entity) pairs of an action verb matched with at least 1 of the 3 ground truth (GT) annotations, while Value measures the percentage of pairs which matched at least one of the three GT annotations. We report accuracy at top-1, top-5 action verb predictions and given the GT verb. Similar to [32], we also report performance on examples with ten or fewer samples in the imSitu training set (rare setting).