When malloc() Never Returns NULL -- Reliability as an Illusion

TL;DR

This paper challenges the common belief that malloc() can reliably indicate memory failure, revealing that most modern OSs do not allow applications to handle such failures, which impacts software reliability and design.

Contribution

It uncovers the default behaviors of major OSs regarding memory allocation failure detection and discusses implications for software engineering practices.

Findings

Most OSs do not allow applications to detect malloc() failures.

Windows uniquely enables detection of memory allocation failures.

Relying on malloc() failure detection creates an illusion of reliability.

Abstract

For decades, the guidance given to software engineers has been to check the memory allocation results. This validation step is necessary to avoid crashes. However, in user mode, in modern operating systems (OS), such as Android, FreeBSD, iOS, and macOS, the caller does not have an opportunity to handle the memory allocation failures. This behavioral trait results from the actions of a system component called an out-of-memory (OOM) killer. We identify that the only mainstream OS that, by default, lets applications detect memory allocation failures is Microsoft Windows. The false expectation that an application can handle OOM errors can negatively impact its design. The presence of error-handling code creates an illusion of reliability and is wasteful in terms of lines of code and code size. We describe the current behavior of a sample of popular OSs during low-memory conditions and provide recommendations for engineering practices going forward.