Specification-Based Protocol Obfuscation

TL;DR

This paper introduces a protocol obfuscation method based on transforming message format specifications to hinder reverse engineering, demonstrated on Modbus and HTTP with acceptable performance overheads.

Contribution

It presents a novel obfuscation technique using AST transformations of protocol messages and a framework for automatic implementation, applied to real protocols.

Findings

Increased complexity of obfuscated protocol binaries

Acceptable execution time and memory overheads

Effective obfuscation demonstrated on Modbus and HTTP

Abstract

This paper proposes a new obfuscation technique of a communication protocol that is aimed at making the reverse engineering of the protocol more complex. The obfuscation is based on the transformation of protocol message format specification. The obfuscating transformations are applied to the Abstract Syntax Tree (AST) representation of the messages and mainly concern the ordering or aggregation of the AST nodes. The paper also presents the design of a framework that implements the proposed obfuscation technique by automatically generating, from the specification of the message format, a library performing the corresponding transformations. Finally, our framework is applied to two real application protocols (Modbus and HTTP) to illustrate the relevance and efficiency of the proposed approach. Various metrics recorded from the experiments show the significant increase of the complexity of the obfuscated protocol binary compared to the non-obfuscated code. It is also shown that the execution time and memory overheads remain acceptable for a practical deployment of the approach in operation.