Magika: AI-Powered Content-Type Detection

TL;DR

Magika is an AI-powered content-type detection tool that uses a lightweight deep learning model to accurately identify over 100 data formats, outperforming existing tools and supporting critical security and operating system functions.

Contribution

We introduce Magika, a novel deep learning-based content-type detection system that is highly accurate, resource-efficient, and open source, with real-world adoption and ongoing development.

Findings

Achieves 99% F1 score across 100+ content types

Outperforms existing content-type detection tools

Supports integration with security and email systems

Abstract

The task of content-type detection -- which entails identifying the data encoded in an arbitrary byte sequence -- is critical for operating systems, development, reverse engineering environments, and a variety of security applications. In this paper, we introduce Magika, a novel AI-powered content-type detection tool. Under the hood, Magika employs a deep learning model that can execute on a single CPU with just 1MB of memory to store the model's weights. We show that Magika achieves an average F1 score of 99% across over a hundred content types and a test set of more than 1M files, outperforming all existing content-type detection tools today. In order to foster adoption and improvements, we open source Magika under an Apache 2 license on GitHub and make our model and training pipeline publicly available. Our tool has already seen adoption by the Gmail email provider for attachment scanning, and it has been integrated with VirusTotal to aid with malware analysis. We note that this paper discusses the first iteration of Magika, and a more recent version already supports more than 200 content types. The interested reader can see the latest development on the Magika GitHub repository, available at https://github.com/google/magika.