# Actions Generation from Captions

## TL;DR

This paper introduces a new dataset and a novel model combining attention and GANs for generating continuous action sequences from caption descriptions, advancing sequence transduction in NLP.

## Contribution

The work presents a new caption-action dataset and a combined attention-GAN model for continuous sequence generation, addressing a less-explored area in sequence transduction.

## Key findings

- The model achieves plausible action generation from captions.
- The dataset enables studying continuous sequence transduction.
- The combined model outperforms baseline approaches.

## Abstract

Sequence transduction models have been widely explored in many natural language processing tasks. However, the target sequence usually consists of discrete tokens which represent word indices in a given vocabulary. We barely see the case where target sequence is composed of continuous vectors, where each vector is an element of a time series taken successively in a temporal domain. In this work, we introduce a new data set, named Action Generation Data Set (AGDS) which is specifically designed to carry out the task of caption-to-action generation. This data set contains caption-action pairs. The caption is comprised of a sequence of words describing the interactive movement between two people, and the action is a captured sequence of poses representing the movement. This data set is introduced to study the ability of generating continuous sequences through sequence transduction models. We also propose a model to innovatively combine Multi-Head Attention (MHA) and Generative Adversarial Network (GAN) together. In our model, we have one generator to generate actions from captions and three discriminators where each of them is designed to carry out a unique functionality: caption-action consistency discriminator, pose discriminator and pose transition discriminator. This novel design allowed us to achieve plausible generation performance which is demonstrated in the experiments.