Experimental Aspects of Synthesis

TL;DR

This paper reviews and analyzes experimental evaluation methods for LTL synthesis tools, proposing a framework to improve comparability and address complexities in modern synthesis workflows.

Contribution

It provides a comprehensive survey of existing evaluation practices and introduces a framework to standardize and improve future experimental comparisons of synthesis tools.

Findings

Previous tools have incompatible scopes and semantics

A proposed framework reduces evaluation inconsistencies

Guidelines for conducting convincing experimental evaluations

Abstract

We discuss the problem of experimentally evaluating linear-time temporal logic (LTL) synthesis tools for reactive systems. We first survey previous such work for the currently publicly available synthesis tools, and then draw conclusions by deriving useful schemes for future such evaluations. In particular, we explain why previous tools have incompatible scopes and semantics and provide a framework that reduces the impact of this problem for future experimental comparisons of such tools. Furthermore, we discuss which difficulties the complex workflows that begin to appear in modern synthesis tools induce on experimental evaluations and give answers to the question how convincing such evaluations can still be performed in such a setting.