Toward the Detection of Polyglot Files

TL;DR

This paper investigates the challenge of detecting polyglot files that can evade file format identification, proposing machine learning models like Malconv2 and Catboost to improve detection accuracy for malware filtering.

Contribution

The study evaluates existing file identification tools and introduces machine learning models that effectively detect polyglot files, enhancing malware detection pipelines.

Findings

Malconv2 achieved 95.16% recall in polyglot detection.

Catboost achieved 95.45% recall, outperforming other models.

Machine learning models can be integrated into malware detection systems for better filtering.

Abstract

Standardized file formats play a key role in the development and use of computer software. However, it is possible to abuse standardized file formats by creating a file that is valid in multiple file formats. The resulting polyglot (many languages) file can confound file format identification, allowing elements of the file to evade analysis.This is especially problematic for malware detection systems that rely on file format identification for feature extraction. File format identification processes that depend on file signatures can be easily evaded thanks to flexibility in the format specifications of certain file formats. Although work has been done to identify file formats using more comprehensive methods than file signatures, accurate identification of polyglot files remains an open problem. Since malware detection systems routinely perform file format-specific feature extraction, polyglot files need to be filtered out prior to ingestion by these systems. Otherwise, malicious content could pass through undetected. To address the problem of polyglot detection we assembled a data set using the mitra tool. We then evaluated the performance of the most commonly used file identification tool, file. Finally, we demonstrated the accuracy, precision, recall and F1 score of a range of machine and deep learning models. Malconv2 and Catboost demonstrated the highest recall on our data set with 95.16% and 95.45%, respectively. These models can be incorporated into a malware detector's file processing pipeline to filter out potentially malicious polyglots before file format-dependent feature extraction takes place.