NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder

TL;DR

NIERT introduces a transformer-based encoder-only model for scattered data interpolation, unifying observed and target points in a shared representation space and leveraging pre-training on synthetic functions for improved accuracy across domains.

Contribution

The paper proposes NIERT, a novel encoder-only transformer model that unifies data representations and uses pre-training to enhance interpolation accuracy and generalization.

Findings

NIERT achieves 4.3 to 14.3 times lower MAE on synthetic datasets.

NIERT outperforms existing methods with 1.7 to 8.7 times lower MSE on real-world datasets.

Pre-training on synthetic functions improves performance across multiple interpolation domains.

Abstract

Interpolation for scattered data is a classical problem in numerical analysis, with a long history of theoretical and practical contributions. Recent advances have utilized deep neural networks to construct interpolators, exhibiting excellent and generalizable performance. However, they still fall short in two aspects: \textbf{1) inadequate representation learning}, resulting from separate embeddings of observed and target points in popular encoder-decoder frameworks and \textbf{2) limited generalization power}, caused by overlooking prior interpolation knowledge shared across different domains. To overcome these limitations, we present a \textbf{N}umerical \textbf{I}nterpolation approach using \textbf{E}ncoder \textbf{R}epresentation of \textbf{T}ransformers (called \textbf{NIERT}). On one hand, NIERT utilizes an encoder-only framework rather than the encoder-decoder structure. This way, NIERT can embed observed and target points into a unified encoder representation space, thus effectively exploiting the correlations among them and obtaining more precise representations. On the other hand, we propose to pre-train NIERT on large-scale synthetic mathematical functions to acquire prior interpolation knowledge, and transfer it to multiple interpolation domains with consistent performance gain. On both synthetic and real-world datasets, NIERT outperforms the existing approaches by a large margin, i.e., 4.3$\sim$14.3$\times$ lower MAE on TFRD subsets, and 1.7/1.8/8.7$\times$ lower MSE on Mathit/PhysioNet/PTV datasets. The source code of NIERT is available at https://github.com/DingShizhe/NIERT.