# Attending to Future Tokens For Bidirectional Sequence Generation

## TL;DR

This paper introduces a novel bidirectional sequence generation method using placeholder tokens, enabling models to consider both past and future tokens during generation, which improves performance on conversational tasks.

## Contribution

It proposes a new approach to bidirectional sequence generation with placeholder tokens, enhancing context utilization beyond traditional unidirectional models.

## Key findings

- Bidirectional model outperforms baselines on conversational tasks.
- Placeholder tokens effectively incorporate future context.
- Significant performance improvements demonstrated.

## Abstract

Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversational tasks where the proposed bidirectional model outperforms competitive baselines by a large margin.