# Semantics to Space(S2S): Embedding semantics into spatial space for zero-shot verb-object query inferencing

## TL;DR

This paper introduces S2S, a novel deep zero-shot learning model that embeds semantic information into the visual spatial space for improved human-object interaction inference with VO queries, outperforming existing methods.

## Contribution

The paper proposes the Semantics-to-Space (S2S) architecture that embeds semantics into visual space, enabling better zero-shot human-object interaction inference.

## Key findings

- Outperforms state-of-the-art ZSL methods on VT60 dataset
- Embedding semantics into spatial space improves inference accuracy
- Model consistently enhances performance across different ZSL architectures

## Abstract

We present a novel deep zero-shot learning (ZSL) model for inferencing human-object-interaction with verb-object (VO) query. While the previous two-stream ZSL approaches only use the semantic/textual information to be fed into the query stream, we seek to incorporate and embed the semantics into the visual representation stream as well. Our approach is powered by Semantics-to-Space (S2S) architecture where semantics derived from the residing objects are embedded into a spatial space of the visual stream. This architecture allows the co-capturing of the semantic attributes of the human and the objects along with their location/size/silhouette information. To validate, we have constructed a new dataset, Verb-Transferability 60 (VT60). VT60 provides 60 different VO pairs with overlapping verbs tailored for testing two-stream ZSL approaches with VO query. Experimental evaluations show that our approach not only outperforms the state-of-the-art, but also shows the capability of consistently improving performance regardless of which ZSL baseline architecture is used.