A Planning Approach to Monitoring Behavior of Computer Programs

TL;DR

This paper introduces a novel AI planning-based method to monitor and understand program behavior from system call traces, enhancing malware detection robustness through semantic analysis.

Contribution

It presents a new approach that models operating system behavior with AI planning, enabling semantic analysis of system calls for improved malware detection.

Findings

Effective in distinguishing malicious from benign behavior

Robust against obfuscation techniques

Validated on real system call traces

Abstract

We describe a novel approach to monitoring high level behaviors using concepts from AI planning. Our goal is to understand what a program is doing based on its system call trace. This ability is particularly important for detecting malware. We approach this problem by building an abstract model of the operating system using the STRIPS planning language, casting system calls as planning operators. Given a system call trace, we simulate the corresponding operators on our model and by observing the properties of the state reached, we learn about the nature of the original program and its behavior. Thus, unlike most statistical detection methods that focus on syntactic features, our approach is semantic in nature. Therefore, it is more robust against obfuscation techniques used by malware that change the outward appearance of the trace but not its effect. We demonstrate the efficacy of our approach by evaluating it on actual system call traces.