End-to-end Document Recognition and Understanding with Dessurt

TL;DR

Dessurt is a versatile end-to-end transformer model for document understanding that integrates text recognition and understanding, enabling fine-tuning across diverse document tasks without external recognition models.

Contribution

It introduces Dessurt, a simple, flexible transformer architecture capable of handling multiple document understanding tasks in an end-to-end manner.

Findings

Effective on 9 dataset-task combinations

Does not require external recognition models

Handles diverse document domains and tasks

Abstract

We introduce Dessurt, a relatively simple document understanding transformer capable of being fine-tuned on a greater variety of document tasks than prior methods. It receives a document image and task string as input and generates arbitrary text autoregressively as output. Because Dessurt is an end-to-end architecture that performs text recognition in addition to the document understanding, it does not require an external recognition model as prior methods do. Dessurt is a more flexible model than prior methods and is able to handle a variety of document domains and tasks. We show that this model is effective at 9 different dataset-task combinations.