Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph

TL;DR

This paper introduces a gated spatio-temporal energy graph model for reasoning about complex visual relationships in videos, leveraging temporal information and adaptive relation learning to improve performance.

Contribution

It proposes a novel gated energy function within a CRF framework for video relationship reasoning, enabling efficient and adaptive modeling of spatio-temporal dependencies.

Findings

Achieves state-of-the-art results on ImageNet Video and Charades datasets.

Effectively models complex relationships requiring multiple observations.

Demonstrates improved accuracy in detection, tagging, and recognition tasks.

Abstract

Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts. For example, a relationship 'man, open, door' involves a complex relation 'open' between concrete entities 'man, door'. While much of the existing work has studied this problem in the context of still images, understanding visual relationships in videos has received limited attention. Due to their temporal nature, videos enable us to model and reason about a more comprehensive set of visual relationships, such as those requiring multiple (temporal) observations (e.g., 'man, lift up, box' vs. 'man, put down, box'), as well as relationships that are often correlated through time (e.g., 'woman, pay, money' followed by 'woman, buy, coffee'). In this paper, we construct a Conditional Random Field on a fully-connected spatio-temporal graph that exploits the statistical dependency between relational entities spatially and temporally. We introduce a novel gated energy function parametrization that learns adaptive relations conditioned on visual observations. Our model optimization is computationally efficient, and its space computation complexity is significantly amortized through our proposed parameterization. Experimental results on benchmark video datasets (ImageNet Video and Charades) demonstrate state-of-the-art performance across three standard relationship reasoning tasks: Detection, Tagging, and Recognition.

Figures

Equations

$$\begin{array}{rl}& f_{\theta }^{\phi }(S,X_t^k,t,t^{\prime },k,k^{\prime },y_t^k,y_{t^{\prime }}^{k^{\prime }})\\ =& f_{\theta }^{\phi }(X_t^k,t,t^{\prime },k,k^{\prime },y_t^k,y_{t^{\prime }}^{k^{\prime }})+u_{\theta }({⟨S^k⟩}_{y_t^k})\cdot v_{\theta }({⟨S^{k^{\prime }}⟩}_{y_{t^{\prime }}^{k^{\prime }}}),\end{array}$$

$$\footnotesize\begin{split}&-\mathrm{log}\,m_{t^{\prime},k^{\prime},t,k}(y_{t}^{k}|X)=\sum_{y_{t^{\prime}}^{k^{\prime}}}\varphi_{t,k,t^{\prime},k^{\prime}}(y_{t}^{k},y_{t^{\prime}}^{k^{\prime}}|X)Q(y_{t^{\prime}}^{k^{\prime}})\\ \approx&\left\{\begin{matrix}\sum_{y_{t^{\prime}}^{k^{\prime}}}\left\langle g^{kk^{\prime}}_{\theta}(X_{t}^{k})\otimes h^{kk^{\prime}}_{\theta}(X_{t}^{k})\right\rangle_{y_{t}^{k},y_{t^{\prime}}^{k^{\prime}}}Q(y_{t^{\prime}}^{k^{\prime}})&t=t^{\prime}\\ \sum_{y_{t^{\prime}}^{k^{\prime}}}K_{\sigma}\Big{(}t,t^{\prime}\Big{)}\left\langle r^{kk^{\prime}}_{\theta}(X_{t}^{k})\otimes s^{kk^{\prime}}_{\theta}(X_{t}^{k})\right\rangle_{y_{t}^{k},y_{t^{\prime}}^{k^{\prime}}}Q(y_{t^{\prime}}^{k^{\prime}})&t\neq t^{\prime}\end{matrix}\right..\end{split}$$

$$-\log {\mathbf{m}}_{t^{\prime },k^{\prime },t,k}\approx \mathrm{Query}\cdot {\mathrm{Key}}^{\top }\cdot \mathrm{Value},$$

$$\begin{split}&\left\langle{\bf m}_{t^{\prime},k^{\prime},t,k}\right\rangle_{y_{t}^{k}}=m_{t^{\prime},k^{\prime},t,k}(y_{t}^{k}|X)\\ &\left\langle\mathrm{Value}\right\rangle_{y_{t^{\prime}}^{k^{\prime}}}=\left\{\begin{matrix}Q(y_{t^{\prime}}^{k^{\prime}})&t=t^{\prime}\\ K_{\sigma}\Big{(}t,t^{\prime}\Big{)}\cdot Q(y_{t^{\prime}}^{k^{\prime}})&t\neq t^{\prime}\end{matrix}\right.\\ &\mathrm{Query}=\left\{\begin{matrix}g^{kk^{\prime}}_{\theta}(X_{t}^{k})&t=t^{\prime}\\ r^{kk^{\prime}}_{\theta}(X_{t}^{k})&t\neq t^{\prime}\end{matrix}\right.\\ &\mathrm{Key}=\left\{\begin{matrix}h^{kk^{\prime}}_{\theta}(X_{t}^{k})&t=t^{\prime}\\ s^{kk^{\prime}}_{\theta}(X_{t}^{k})&t\neq t^{\prime}\end{matrix}\right..\end{split}$$

$$\mathrm{softmax}\Big{(}\mathrm{Query}\cdot\mathrm{Key}^{\top}\Big{)}\cdot\mathrm{Value}$$

Full text

Supplementary for Video Relationship Reasoning using

Gated Spatio-Temporal Energy Graph

1 Towards Leveraging Language Priors

The work of [5] has emphasized the role of language priors in alleviating the challenge of learning relationship models from limited training data. Motivated by their work, we also study the role of incorporating language priors in our framework.

In Table. 1 in the main text, comparing UEG to UEG†, we have seen that language priors aid in improving the relationship reasoning performance. Considering our example in Sec. 3.1 in main text, when the training instance is {mother, pay, money}, we may also want to infer that {father, pay, money} is a more likely relationship as opposed to {cat, pay, money} (as mother and father are semantically similar compared to mother and cat). Likewise, we can also infer {mother, pay, check} from the semantic similarity between money and check.

[5] adopted a triplet loss for pairing word embeddings of object, predicate, and subject. However, their method required sampling of all the possible relationships and was also restricted to the number of entities spatially (e.g, $K=3$). Here, we present another way to make the parameterized pairwise energy also be gated by the prior knowledge in semantic space. We let the prior from semantic space be encoded as word embedding: $S=\{S^{k}\}_{k=1}^{K}$ in which $S^{k}\in\mathbb{R}^{|Y_{t}^{k}|\times d}$ denoting prior of labels with length $d$. We extend Eq. (3) in the main text as

[TABLE]

where $u_{\theta}(\cdot)\in\mathbb{R}$ and $v_{\theta}(\cdot)\in\mathbb{R}$ maps the label prior to a score. Eq. (1) suggests that the label transition from $Y_{t}^{k}$ to $Y_{t^{\prime}}^{k^{\prime}}$ can also attend to the affinity inferred from prior knowledge.

We performed a preliminary evaluation on the relation recognition task in the ImageNet Video dataset using $300$-dim Glove features [6] as word embeddings. For subject, predicate, object, and relation triplet, Acc@1 metric improves from $90.60$, $28.78$, $89.79$, and $25.01$ to $90.97$, $29.54$, $90.57$, and $26.48$.

2 Connection to Self Attention and Non-Local Means

In our main text, the message form (eq. (5)) with our observation-gated parametrization (eq. (3) with $t=t^{\prime}$) can be expressed as follows:

[TABLE]

The equation can be reformulated in matrix form:

[TABLE]

where

[TABLE]

We now link this message form with self attention in Machine Translation [10] and Non-Local Mean in Computer Vision [1]. Self-Attention is expressed as the form of

[TABLE]

with $\mathrm{Query}$, $\mathrm{Key}$, and $\mathrm{Value}$ depending on input (termed observation in our case).

In both Self Attention and our message form, the attended weights for $\mathrm{Value}$ is dependent on observation. The difference is that we do not have a row-wise softmax activation to make the attended weights sum to $1$. The derivation is also similar to Non-Local Means [1]. Note that Machine Translation [10] focuses on the updates for features across temporal regions, Non-Local Mean [1] focuses on the updates for the features across spatial regions, while ours focuses on the updates for the entities prediction (i.e., as a message passing).

3 Comparisons with SRNN [3] & GCN [11]

Here, we provide comparisons with Structural-RNN (SRNN) [3] and Graph Convolutional Network (GCN) [11] for comparisons. We note that these approaches are designed for video activity recognition, which cannot be directly applied in video visual relationship detection. In Fig. 1 (top-left), we show how we minimally modifying SRNN and GCN for evaluating them in video relationship detection. The main differences are: 1) our model constructs a fully-connected graph for entire video, while SRNN has a non-fully connected graph and GCN considers only building a graph on partial video ($\sim$32 frames), and 2) the message passing across node represents prediction’s dependency for our model, while it indicates temporally evolving edge features for SRNN and similarity-reweighted features for GCN.

4 Activity Recognition in Charades

Sigurdsson et al. [9] proposed Asynchronous Temporal Fields (AsyncTF) for recognizing $157$ video activities. As discussed in Related Work (see Sec. 2), video activity recognition is a downstream task of visual relationship learning: in Charades, each activity (in $157$ activities) is a combination of one category object and one category in verb. We now cast how our model be transformed into video activity recognition. First, we change the output sequence to be $Y=\{Y_{t}\}_{t=1}^{T}$, where $Y_{t}$ is the prediction of video activity. Then, we apply our Gated Spatio-Temporal Energy Graph on top of the sequence of activity predictions. In this design, we achieve the mAP of 33.3%. AsyncTF reported the mAP 18.3% for using only RGB values from a video.

5 Feature Representation in Pre-Reasoning Modules

ImageNet Video. We now provide the details for representing $X_{t}^{p}$, which is the predicate feature in $t_{th}$ chunk of the input instance. Note that we use the relation feature from prior work [8] (the feature can be downloaded form [7]) as our predicate feature. The feature comprises three components: (1) improved dense trajectory (iDT) features from subject trajectory, (2) improved dense trajectory (iDT) features from object trajectory, and (3) relative features describing the relative positions, size, and motion between subject and object trajectories. iDT features are able to capture the movement and also the low-level visual characteristics for an object moving in a short clip. The relative features are able to represent the relative spatio-temporal differences between subject trajectory and object trajectory. Next, the features are post-processed as bag-of-words features after applying a dictionary learning on the original features. Last, three sub-features are concatenated together for representing our predicate feature.

Charades. We use the output feature layer from I3D network [2] to represent our object ($X_{t}^{o}$), verb ($X_{t}^{v}$), and scene feature ($X_{t}^{s}$). The I3D network is pre-trained from Kinetics dataset [2] (the model can be downloaded from [12]) and the output feature layer is the layer before output logits.

6 Intractable Inference during Evaluation

In ImageNet Video dataset, during evaluation, for relation detection and tagging, we have to enumerate all the possible associations of subject or object tracklets. The number of possible associations grows exponentially by the factor of the number of chunks in a video, which will easily become computationally intractable. Note that the problem exists only during evaluation since the ground truth associations (for subject and object tracklets) are given during training. To overcome the issue, we apply the greedy association algorithm described in [8] for efficiently associating subject or object tracklets. The idea is as follows. First, we adopt the inference only in a chunk. Since the message does not pass across chunks, at this step, we don’t need to consider associations (for subject or object tracklets) across chunks. In a chunk, for a pair of subject and object tracklet, we have a predicted relation triplet. Then, from two overlapping chunks, we only associate the pair of the subject and object tracklets with the same predicted relation triplet and high tracklets vIoU (i.e., $>0.5$). Comparing to the original inference, this algorithm exponentially accelerates the time computational complexity. On the other hand, in Charades, we do not need associate object tracklets. Thus, the intractable computation complexity issue does not exist. The greedy associate algorithm is not required for Charades.

7 Training and Parametrization Details

We specify the training and parametrization details as follows.

ImageNet Video. Throughout all the experiments, we choose Adam [4] with learning rate $0.001$ as our optimizer, $32$ as our batch size, $30$ as the number of training epoch, and $3$ as the number of message passing. We initialize the marginals to be the marginals estimated from unary energy.

• •

Rank number $r$: 5

• •

$g_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times(|Y_{t}^{k}|\times r)$ fully-connected layer, resize to $|Y_{t}^{k}|\times r$

• •

$h_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times\times(|Y_{t^{\prime}}^{k^{\prime}}|\times r)$ fully-connected layer, resize to $|Y_{t^{\prime}}^{k^{\prime}}|\times r$

• •

$r_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times(|Y_{t}^{k}|\times r)$ fully-connected layer, resize to $|Y_{t}^{k}|\times r$

• •

$s_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times(|Y_{t^{\prime}}^{k^{\prime}}|\times r)$ fully-connected layer, resize to $|Y_{t^{\prime}}^{k^{\prime}}|\times r$

• •

$\sigma$: $10$

• •

$w_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times|Y_{t}^{k}|$ fully-connected layer

Charades: Throughout all the experiments, we choose SGD with learning rate $0.005$ as our optimizer, $40$ as our batch size, $5$ as the number of training epoch, and $5$ as the number of message passing. We initialize the marginals to be the marginals estimated from unary energy.

• •

Rank number $r$: 5

• •

$g_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times(|Y_{t}^{k}|\times r)$ fully-connected layer, resize to $|Y_{t}^{k}|\times r$

• •

$h_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times(|Y_{t^{\prime}}^{k^{\prime}}|\times r)$ fully-connected layer, resize to $|Y_{t^{\prime}}^{k^{\prime}}|\times r$

• •

$r_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times(|Y_{t}^{k}|\times r)$ fully-connected layer, resize to $|Y_{t}^{k}|\times r$

• •

$s_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times\times(|Y_{t^{\prime}}^{k^{\prime}}|\times r)$ fully-connected layer, resize to $|Y_{t^{\prime}}^{k^{\prime}}|\times r$

• •

$\sigma$: $300$

• •

$w_{\theta}^{kk^{\prime}}(X_{t}^{k})$: $|X_{t}^{k}|\times|Y_{t}^{k}|$ fully-connected layer

8 Parametrization in Leveraging Language Priors

Additional networks in the experiments towards leveraging language priors are parametrized as follows:

• •

$d$: 300 (because we use 300-dim. Glove [6] features)

• •

$u_{\theta}(\cdot)$: $d\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1$ fully-connected layer

• •

$v_{\theta}(\cdot)$: $d\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1024$ fully-connected layer, ReLU Activation, Dropout with rate $0.3$, $1024\times 1$ fully-connected layer

9 Category Set in Dataset

For clarity, we use bullet points for referring to the category choice in datasets for the different entity.

• •

subject / object in ImageNet Video (total $35$ categories)

• –

airplane, antelope, ball, bear, bicycle, bird, bus, car, cat, cattle, dog, elephant, fox, frisbee, giant panda, hamster, horse, lion, lizard, monkey, motorcycle, person, rabbit, red panda, sheep, skateboard, snake, sofa, squirrel, tiger, train, turtle, watercraft, whale, zebra

• •

predicate in ImageNet Video (total $132$ categories)

• –

taller, swim behind, walk away, fly behind, creep behind, lie with, move left, stand next to, touch, follow, move away, lie next to, walk with, move next to, creep above, stand above, fall off, run with, swim front, walk next to, kick, stand left, creep right, sit above, watch, swim with, fly away, creep beneath, front, run past, jump right, fly toward, stop beneath, stand inside, creep left, run next to, beneath, stop left, right, jump front, jump beneath, past, jump toward, sit front, sit inside, walk beneath, run away, stop right, run above, walk right, away, move right, fly right, behind, sit right, above, run front, run toward, jump past, stand with, sit left, jump above, move with, swim beneath, stand behind, larger, walk past, stop front, run right, creep away, move toward, feed, run left, lie beneath, fly front, walk behind, stand beneath, fly above, bite, fly next to, stop next to, fight, walk above, jump behind, fly with, sit beneath, sit next to, jump next to, run behind, move behind, swim right, swim next to, hold, move past, pull, stand front, walk left, lie above, ride, next to, move beneath, lie behind, toward, jump left, stop above, creep toward, lie left, fly left, stop with, walk toward, stand right, chase, creep next to, fly past, move front, run beneath, creep front, creep past, play, lie inside, stop behind, move above, sit behind, faster, lie right, walk front, drive, swim left, jump away, jump with, lie front, left

• •

verb in Charades (total $33$ categories)

• –

awaken, close, cook, dress, drink, eat, fix, grasp, hold, laugh, lie, make, open, photograph, play, pour, put, run, sit, smile, sneeze, snuggle, stand, take, talk, throw, tidy, turn, undress, walk, wash, watch, work

• •

object in Charades (total $38$ categories)

• –

None, bag, bed, blanket, book, box, broom, chair, closet/cabinet, clothes, cup/glass/bottle, dish, door, doorknob, doorway, floor, food, groceries, hair, hands, laptop, light, medicine, mirror, paper/notebook, phone/camera, picture, pillow, refrigerator, sandwich, shelf, shoe, sofa/couch, table, television, towel, vacuum, window

• •

scene in Charades (total $16$ categories)

• –

Basement, Bathroom, Bedroom, Closet / Walk-in closet / Spear closet, Dining room, Entryway, Garage, Hallway, Home Office / Study, Kitchen, Laundry room, Living room, Other, Pantry, Recreation room / Man cave, Stairs