Advancing Sequential Numerical Prediction in Autoregressive Models

TL;DR

This paper introduces Numerical Token Integrity Loss (NTIL), a novel method that enhances autoregressive models' ability to generate coherent numerical sequences by preserving ordinal relationships and sequence integrity.

Contribution

The paper proposes NTIL, a dual-level loss function that improves numerical sequence prediction in autoregressive models, addressing limitations of standard token-based approaches.

Findings

NTIL significantly improves numerical prediction accuracy.

NTIL effectively preserves ordinal relationships in sequences.

NTIL integrates well with large language models.

Abstract

Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.