VetIoT: On Vetting IoT Defenses Enforcing Policies at Runtime

TL;DR

VetIoT is an automated, standardized platform for evaluating IoT security defenses that enforce policies at runtime, improving reproducibility and enabling large-scale comparative analysis.

Contribution

We introduce VetIoT, a fully automated evaluation platform that streamlines testing and comparison of IoT defense solutions enforcing runtime policies.

Findings

Successfully reproduced and assessed four runtime policy enforcement solutions.

Enhanced evaluation reproducibility and comparability for IoT defenses.

Enabled stress and differential testing of IoT security solutions.

Abstract

Smart homes, powered by programmable IoT platforms, often face safety and security issues. A class of defense solutions dynamically enforces policies that capture the expected behavior of the IoT system. Despite numerous innovations, these solutions are under-vetted. The primary reason lies in their evaluation approach -- they are self-assessed in isolated virtual testbeds with hand-crafted orchestrated scenarios that require manual interactions using the platform's user-interface (UI). Such non-uniform evaluation setups limit reproducibility and comparative analysis. Closing this gap in the traditional way requires a significant upfront manual effort, causing researchers to turn away from large-scale comparative empirical evaluation. To address this, we propose VetIoT -- a highly automated, uniform evaluation platform -- to vet the defense solutions that hinge on runtime policy enforcement. Given a defense solution, VetIoT readily instantiates a virtual testbed to deploy and evaluate the solution. VetIoT replaces manual UI-based interactions with an automated event simulator and manual inspection of test outcomes with an automated comparator. VetIoT incorporates automated event generators to feed events to the event simulator. We developed a prototype of VetIoT, which successfully reproduced and comparatively assessed four runtime policy enforcement solutions. VetIoT's stress testing and differential testing capabilities make it a promising tool for future research and evaluation.