An FNet based Auto Encoder for Long Sequence News Story Generation

TL;DR

This paper introduces an FNet-based autoencoder for generating long news stories, demonstrating efficiency advantages over BERT-based models and analyzing output variations with different seed texts.

Contribution

It presents a novel autoencoder architecture based on FNet for long sequence news generation, highlighting efficiency benefits and output analysis.

Findings

FNet autoencoder trains 80% faster on GPUs and 70% faster on TPUs than BERT.

The model effectively generates news stories from seed texts.

Output quality varies with different seed inputs.

Abstract

In this paper, we design an auto encoder based off of Google's FNet Architecture in order to generate text from a subset of news stories contained in Google's C4 dataset. We discuss previous attempts and methods to generate text from autoencoders and non LLM Models. FNET poses multiple advantages to BERT based encoders in the realm of efficiency which train 80% faster on GPUs and 70% faster on TPUs. We then compare outputs of how this autencoder perfroms on different epochs. Finally, we analyze what outputs the encoder produces with different seed text.