Refining Network Message Segmentation with Principal Component Analysis

TL;DR

This paper introduces a PCA-based method to improve message segmentation accuracy in protocol reverse engineering, significantly enhancing field inference in network traffic analysis.

Contribution

It presents a novel application of PCA to refine message segmentation boundaries, improving the accuracy of protocol message format inference.

Findings

Median improvement of message format accuracy up to 100%

Effective in real-world protocol analysis

Enhances subsequent message analysis tasks

Abstract

Reverse engineering of undocumented protocols is a common task in security analyses of networked services. The communication itself, captured in traffic traces, contains much of the necessary information to perform such a protocol reverse engineering. The comprehension of the format of unknown messages is of particular interest for binary protocols that are not human-readable. One major challenge is to discover probable fields in a message as the basis for further analyses. Given a set of messages, split into segments of bytes by an existing segmenter, we propose a method to refine the approximation of the field inference. We use principle component analysis (PCA) to discover linearly correlated variance between sets of message segments. We relocate the boundaries of the initial coarse segmentation to more accurately match with the true fields. We perform different evaluations of our method to show its benefit for the message format inference and subsequent analysis tasks from literature that depend on the message format. We can achieve a median improvement of the message format accuracy across different real-world protocols by up to 100 %.