EdgeFM: Efficient Edge Inference for Vision-Language Models

TL;DR

EdgeFM is a lightweight, cross-platform inference framework for vision-language models that optimizes performance and reduces latency on edge devices, surpassing traditional vendor-specific solutions.

Contribution

It introduces a modular, agent-driven approach to generate optimized kernels, enabling better cross-platform deployment and performance for industrial edge applications.

Findings

Achieves up to 1.49x speedup over TensorRT-Edge-LLM on NVIDIA Orin.

Supports mainstream platforms including x86 and NVIDIA Orin SoCs.

Provides an open-source, production-grade solution for edge industrial scenarios.

Abstract

Vision-language models (VLMs) have demonstrated strong applicability in edge industrial applications, yet their deployment remains severely constrained by requirements for deterministic low latency and stable execution under resource limitations. Existing frameworks either rely on bloated general-purpose designs or force developers into opaque, hardware-specific closed-source ecosystems, leading to hardware lock-in limitation and poor cross-platform adaptability. Observing that modern AI agents can efficiently search and tune configurations to generate highly optimized low-level kernels for standard LLM operators, we propose EdgeFM, a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment. EdgeFM removes non-essential features to reduce single-request latency, and encapsulates agent-tuned kernel optimizations as a modular library of reusable skills. By allowing direct invocation of these skills rather than waiting for closed-source implementations, it effectively closes the performance gap long dominated by proprietary toolchains. The framework natively supports mainstream platforms including x86 and NVIDIA Orin SoCs, and represents the first end-to-end VLA deployment on the domestic Horizon Journey platform, enhancing cross-platform portability. In most cases, it yields clearly better inference performance than conventional vendor-specific toolchains, achieving up to 1.49 times speedup over TensorRT-Edge-LLM on the NVIDIA Orin platform. Experimental results show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios.