Should We Learn Probabilistic Models for Model Checking? A New Approach and An Empirical Study

TL;DR

This paper investigates the effectiveness of learning probabilistic models for system analysis, proposing an evolutionary approach to improve generalization, and empirically evaluating whether learned models outperform sampling-based estimates.

Contribution

It introduces an evolution-based method for learning probabilistic models and provides an empirical study comparing learned models to sampling methods.

Findings

Learning effectiveness can be limited in some cases.

The proposed evolutionary approach offers better control over model generalization.

Empirical results compare the accuracy of learned models versus sampling estimates.

Abstract

Many automated system analysis techniques (e.g., model checking, model-based testing) rely on first obtaining a model of the system under analysis. System modeling is often done manually, which is often considered as a hindrance to adopt model-based system analysis and development techniques. To overcome this problem, researchers have proposed to automatically "learn" models based on sample system executions and shown that the learned models can be useful sometimes. There are however many questions to be answered. For instance, how much shall we generalize from the observed samples and how fast would learning converge? Or, would the analysis result based on the learned model be more accurate than the estimation we could have obtained by sampling many system executions within the same amount of time? In this work, we investigate existing algorithms for learning probabilistic models for model checking, propose an evolution-based approach for better controlling the degree of generalization and conduct an empirical study in order to answer the questions. One of our findings is that the effectiveness of learning may sometimes be limited.