The Credibility Transformer

TL;DR

The paper introduces the Credibility Transformer, a novel architecture that enhances Transformer models for tabular data by incorporating a credibility mechanism, resulting in more stable training and superior predictive performance.

Contribution

It proposes a new credibility mechanism for Transformers applied to tabular data, improving training stability and predictive accuracy over existing models.

Findings

Enhanced training stability with the credibility mechanism

Superior predictive performance compared to state-of-the-art models

Effective application of Transformers to tabular data

Abstract

Inspired by the large success of Transformers in Large Language Models, these architectures are increasingly applied to tabular data. This is achieved by embedding tabular data into low-dimensional Euclidean spaces resulting in similar structures as time-series data. We introduce a novel credibility mechanism to this Transformer architecture. This credibility mechanism is based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information. We demonstrate that this novel credibility mechanism is very beneficial to stabilize training, and our Credibility Transformer leads to predictive models that are superior to state-of-the-art deep learning models.