# Differentially Testing Soundness and Precision of Program Analyzers

## TL;DR

This paper introduces an automated method for systematically comparing the soundness and precision of various program analyzers, revealing issues in many existing tools and aiding their improvement.

## Contribution

It presents the first automated differential testing technique for program analyzers, enabling systematic evaluation of their soundness and precision.

## Key findings

- Most analyzers had soundness or precision issues
- The technique detected issues across six state-of-the-art analyzers
- Implications for analyzer design and usage were analyzed

## Abstract

In the last decades, numerous program analyzers have been developed both by academia and industry. Despite their abundance however, there is currently no systematic way of comparing the effectiveness of different analyzers on arbitrary code. In this paper, we present the first automated technique for differentially testing soundness and precision of program analyzers. We used our technique to compare six mature, state-of-the art analyzers on tens of thousands of automatically generated benchmarks. Our technique detected soundness and precision issues in most analyzers, and we evaluated the implications of these issues to both designers and users of program analyzers.