Runtime Failure Hunting for Physics Engine Based Software Systems: How Far Can We Go?

TL;DR

This paper conducts the first large-scale empirical study on physics failures in physics engine software, analyzing their manifestations, detection methods, and developer perceptions to improve reliability and safety.

Contribution

It introduces a taxonomy of physics failures, evaluates various detection techniques, and provides insights from developers to enhance failure detection approaches.

Findings

Identified diverse physics failure manifestations

Evaluated deep learning and multimodal detection methods

Gathered developer insights for future improvements

Abstract

Physics Engines (PEs) are fundamental software frameworks that simulate physical interactions in applications ranging from entertainment to safety-critical systems. Despite their importance, PEs suffer from physics failures, deviations from expected physical behaviors that can compromise software reliability, degrade user experience, and potentially cause critical failures in autonomous vehicles or medical robotics. Current testing approaches for PE-based software are inadequate, typically requiring white-box access and focusing on crash detection rather than semantically complex physics failures. This paper presents the first large-scale empirical study characterizing physics failures in PE-based software. We investigate three research questions addressing the manifestations of physics failures, the effectiveness of detection techniques, and developer perceptions of current detection practices. Our contributions include: (1) a taxonomy of physics failure manifestations; (2) a comprehensive evaluation of detection methods including deep learning, prompt-based techniques, and large multimodal models; and (3) actionable insights from developer experiences for improving detection approaches. To support future research, we release PhysiXFails, code, and other materials at https://sites.google.com/view/physics-failure-detection.