Beyond Least Squares: Robust Regression Transformer (R2T)

TL;DR

This paper introduces R2T, a hybrid neural-symbolic model that enhances robust regression by combining transformer encoders, neural parameter prediction, and symbolic equations, significantly outperforming traditional methods under asymmetric structured noise.

Contribution

The paper presents a novel neural-symbolic architecture for robust regression that effectively handles asymmetric structured noise, surpassing existing least-squares and robust regression techniques.

Findings

Achieves median regression MSE of 6e-6 to 3.5e-5 on synthetic data.

Outperforms ordinary least squares by 10-300 times.

Significantly improves robustness against asymmetric structured noise.

Abstract

Robust regression techniques rely on least-squares optimization, which works well for Gaussian noise but fails in the presence of asymmetric structured noise. We propose a hybrid neural-symbolic architecture where a transformer encoder processes numerical sequences, a compression NN predicts symbolic parameters, and a fixed symbolic equation reconstructs the original sequence. Using synthetic data, the training objective is to recover the original sequence after adding asymmetric structured noise, effectively learning a symbolic fit guided by neural parameter estimation. Our model achieves a median regression MSE of 6e-6 to 3.5e-5 on synthetic wearable data, which is a 10-300 times improvement when compared with ordinary least squares fit and robust regression techniques such as Huber loss or SoftL1.