Eliminating Unstable Tests in Floating-Point Programs

TL;DR

This paper presents a formally verified program transformation that detects and corrects test instability caused by round-off errors in floating-point computations, ensuring reliable control flow in critical applications.

Contribution

A novel, formally proven transformation method that guarantees correct control flow in floating-point programs by addressing test instability issues.

Findings

Successfully applied to NASA's polygon containment algorithm

Guarantees correct control flow or issues warnings in unstable cases

Improves reliability of floating-point computations in safety-critical systems

Abstract

Round-off errors arising from the difference between real numbers and their floating-point representation cause the control flow of conditional floating-point statements to deviate from the ideal flow of the real-number computation. This problem, which is called test instability, may result in a significant difference between the computation of a floating-point program and the expected output in real arithmetic. In this paper, a formally proven program transformation is proposed to detect and correct the effects of unstable tests. The output of this transformation is a floating-point program that is guaranteed to return either the result of the original floating-point program when it can be assured that both its real and its floating-point flows agree or a warning when these flows may diverge. The proposed approach is illustrated with the transformation of the core computation of a polygon containment algorithm developed at NASA that is used in a geofencing system for unmanned aircraft systems.