Mill.jl and JsonGrinder.jl: automated differentiable feature extraction for learning from raw JSON data

TL;DR

This paper introduces Mill.jl and JsonGrinder.jl, two libraries that automate the conversion of raw JSON data into differentiable models, reducing manual feature engineering and enabling learning directly from hierarchical JSON structures.

Contribution

The authors present a novel automated pipeline for transforming raw JSON data into differentiable models, streamlining feature extraction for machine learning tasks.

Findings

Automates feature extraction from raw JSON data

Enables learning directly from hierarchical data structures

Reduces manual preprocessing effort

Abstract

Learning from raw data input, thus limiting the need for manual feature engineering, is one of the key components of many successful applications of machine learning methods. While machine learning problems are often formulated on data that naturally translate into a vector representation suitable for classifiers, there are data sources, for example in cybersecurity, that are naturally represented in diverse files with a unifying hierarchical structure, such as XML, JSON, and Protocol Buffers. Converting this data to vector (tensor) representation is generally done by manual feature engineering, which is laborious, lossy, and prone to human bias about the importance of particular features. Mill and JsonGrinder is a tandem of libraries, which fully automates the conversion. Starting with an arbitrary set of JSON samples, they create a differentiable machine learning model capable of infer from further JSON samples in their raw form.