MalwarePT: A Binary-Level Foundation Model for Malware Analysis

TL;DR

MalwarePT is a binary-level foundation model pretrained on Windows PE code-section bytes, demonstrating improved performance across malware analysis tasks at different granularities.

Contribution

Introduces MalwarePT, a binary-level foundation model with byte-pair encoding tokenization, capable of transfer learning across multiple malware analysis tasks.

Findings

Pretraining significantly improves API call prediction and functionality classification.

Increasing BPE vocabulary size enhances performance, with optimal at 1,024 tokens.

MalwarePT outperforms neural network baselines in malware detection at low FPR.

Abstract

Automated malware analysis increasingly relies on machine learning, yet most existing methods remain task-specific and depend on handcrafted features or narrowly scoped models. Recent developments in binary-level foundation models suggest a path toward reusable program representations, but their application to malware analysis remains underexplored, and most still operate at byte-level tokenization, limiting their ability to capture multi-byte code patterns. In this work, we introduce MalwarePT, a binary-level foundation model for malware analysis built on a ModernBERT-style encoder and pretrained with masked language modeling on Windows PE code-section bytes. We study whether a single pretrained encoder can transfer across malware-analysis tasks at different granularities, and how tokenization design affects that transfer. We train a byte-pair encoding (BPE) tokenizer on code-section bytes to compress frequent multi-byte patterns within a fixed context budget. We evaluate MalwarePT on three downstream tasks spanning token-, function-, and document-level prediction: API call prediction, functionality classification, and malware (program) detection under temporal drift. Our evaluation demonstrates that pretraining yields substantial gains for API call prediction and functionality classification, and that increasing the BPE vocabulary beyond the byte-level baseline improves performance, with the strongest overall tradeoff at a vocabulary size of 1,024 tokens. In malware detection at FPR ~ 0.001, MalwarePT outperforms the neural network baselines, and is complementary to feature-engineering models that rely on PE structure. We also compare against existing binary foundation models and show that MalwarePT's design choices yield gains across all downstream tasks.