Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices

TL;DR

Polymorph is a novel framework that enables energy-efficient, real-time multi-label video classification on embedded devices by dynamically activating specialized lightweight adapters based on video content.

Contribution

It introduces a context-aware, modular approach using lightweight adapters that adapt to label co-occurrence, reducing energy and latency while maintaining high accuracy.

Findings

Achieves 40% lower energy consumption compared to baselines.

Improves mean Average Precision (mAP) by 9 points on TAO dataset.

Enables scalable, real-time multi-label classification on embedded hardware.

Abstract

Real-time multi-label video classification on embedded devices is constrained by limited compute and energy budgets. Yet, video streams exhibit structural properties such as label sparsity, temporal continuity, and label co-occurrence that can be leveraged for more efficient inference. We introduce Polymorph, a context-aware framework that activates a minimal set of lightweight Low Rank Adapters (LoRA) per frame. Each adapter specializes in a subset of classes derived from co-occurrence patterns and is implemented as a LoRA weight over a shared backbone. At runtime, Polymorph dynamically selects and composes only the adapters needed to cover the active labels, avoiding full-model switching and weight merging. This modular strategy improves scalability while reducing latency and energy overhead. Polymorph achieves 40% lower energy consumption and improves mAP by 9 points over strong baselines on the TAO dataset. Polymorph is open source at https://github.com/inference-serving/polymorph/.