Learning Heuristics for Template-based CEGIS of Loop Invariants with Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning approach to automatically learn heuristics for template-based CEGIS in loop invariant synthesis, improving performance over existing solvers.

Contribution

It presents a systematic method to learn heuristics for CEGIS using reinforcement learning, replacing ad-hoc heuristic design.

Findings

Learned heuristics improve CEGIS performance

Outperforms state-of-the-art CEGIS solvers

Has advantages over non-CEGIS solvers in CHC solving

Abstract

Loop-invariant synthesis is the basis of program verification. Due to the undecidability of the problem in general, a tool for invariant synthesis necessarily uses heuristics. Despite the common belief that the design of heuristics is vital for the performance of a synthesizer, heuristics are often engineered by their developers based on experience and intuition, sometimes in an \emph{ad-hoc} manner. In this work, we propose an approach to systematically learning heuristics for template-based CounterExample-Guided Inductive Synthesis (CEGIS) with reinforcement learning. As a concrete example, we implement the approach on top of PCSat, which is an invariant synthesizer based on template-based CEGIS. Experiments show that PCSat guided by the heuristics learned by our framework not only outperforms existing state-of-the-art CEGIS-based solvers such as HoICE and the neural solver Code2Inv, but also has slight advantages over non-CEGIS-based solvers such as Eldarica and Spacer in linear Constrained Horn Clause (CHC) solving.