Eywa: Automating Model Based Testing using LLMs

TL;DR

Eywa leverages large language models to automatically generate detailed protocol models from natural language sources, enabling effective model-based testing of network protocols with minimal user effort and discovering numerous previously unknown bugs.

Contribution

This work introduces Eywa, a novel framework that automates protocol model creation using LLMs, overcoming previous barriers in model-based testing of network protocols.

Findings

Discovered 32 unique bugs across DNS, BGP, and SMTP implementations.

Successfully generated rich protocol models from natural language sources.

Reduced user effort in protocol modeling to minimal input.

Abstract

Model-based testing (MBT), whereby a model of the system under test is analyzed to generate high-coverage test cases, has been used to test protocol implementations. A key barrier to the use of MBT is the need for users to understand protocol RFCs in detail to create a compliant model. Our new approach to MBT uses LLMs to automatically build rich models of intended protocol behavior from knowledge embedded in RFCs, blogs, and other natural language sources. Our approach addresses key challenges with using LLMs, including hallucinations and their inability to monolithically generate complex protocol models. We realize our approach through a novel protocol testing framework Eywa,and demonstrate its effectiveness through extensive case studies of DNS and BGP and a smaller study of SMTP. Despite minimal user effort, applying Eywa enabled the discovery of 32 unique bugs across widely used DNS, BGP, and SMTP implementations, 15 of which were previously undiscovered despite extensive prior testing with manually crafted models.