MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point

TL;DR

MCTensor is a PyTorch-based library that offers high-precision arithmetic for deep learning, enabling models to match or surpass the performance of standard precision models.

Contribution

It introduces a high-precision arithmetic library compatible with PyTorch, with optimized algorithms that improve accuracy without sacrificing performance.

Findings

Models with MCTensor in float16 match or outperform float32/float64 models.

MCTensor achieves high-precision computation with optimized PyTorch arithmetic.

Enhanced model accuracy demonstrated across various tasks.

Abstract

In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training. MCTensor is used in the same way as PyTorch Tensor: we implement multiple basic, matrix-level computation operators and NN modules for MCTensor with identical PyTorch interface. Our algorithms achieve high precision computation and also benefits from heavily-optimized PyTorch floating-point arithmetic. We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.