Algebraformer: A Neural Approach to Linear Systems

TL;DR

Algebraformer is a Transformer-based neural model designed to efficiently solve ill-conditioned linear systems end-to-end, reducing computational complexity while maintaining accuracy in scientific computing tasks.

Contribution

It introduces a novel encoding scheme and a scalable Transformer architecture for solving linear systems, especially those that are ill-conditioned, with lower computational overhead.

Findings

Achieves competitive accuracy on linear problems.

Supports scalable inference with $O(n^2)$ memory complexity.

Reduces computational overhead compared to traditional methods.

Abstract

Recent work in deep learning has opened new possibilities for solving classical algorithmic tasks using end-to-end learned models. In this work, we investigate the fundamental task of solving linear systems, particularly those that are ill-conditioned. Existing numerical methods for ill-conditioned systems often require careful parameter tuning, preconditioning, or domain-specific expertise to ensure accuracy and stability. In this work, we propose Algebraformer, a Transformer-based architecture that learns to solve linear systems end-to-end, even in the presence of severe ill-conditioning. Our model leverages a novel encoding scheme that enables efficient representation of matrix and vector inputs, with a memory complexity of $O(n^2)$, supporting scalable inference. We demonstrate its effectiveness on application-driven linear problems, including interpolation tasks from spectral methods for boundary value problems and acceleration of the Newton method. Algebraformer achieves competitive accuracy with significantly lower computational overhead at test time, demonstrating that general-purpose neural architectures can effectively reduce complexity in traditional scientific computing pipelines.