TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices

TL;DR

This paper evaluates and optimizes small language models for agentic tasks on edge devices, demonstrating that hybrid optimization strategies significantly improve their accuracy and stability for autonomous, privacy-preserving AI applications.

Contribution

It introduces a comprehensive evaluation framework and hybrid optimization methods for small language models, enabling effective agentic tasks on edge devices.

Findings

Medium-sized models outperform ultra-compact models in accuracy.

Hybrid optimization achieves up to 65.74% overall accuracy.

Small models can be effective for autonomous AI on edge devices.

Abstract

This paper investigates the effectiveness of small language models (SLMs) for agentic tasks (function/tool/API calling) with a focus on running agents on edge devices without reliance on cloud infrastructure. We evaluate SLMs using the Berkeley Function Calling Leaderboard (BFCL) framework and describe parameter-driven optimization strategies that include supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), reinforcement learning (RL)-based optimization, preference alignment via Direct Preference Optimization (DPO), and hybrid methods. We report results for models including TinyAgent, TinyLlama, Qwen, and xLAM across BFCL categories (simple, multiple, parallel, parallel-multiple, and relevance detection), both in live and non-live settings, and in multi-turn evaluations. We additionally detail a DPO training pipeline constructed from AgentBank data (e.g., ALFRED), including our conversion of SFT data to chosen-rejected pairs using TinyLlama responses as rejected outputs and manual validation. Our results demonstrate clear accuracy differences across model scales where medium-sized models (1-3B parameters) significantly outperform ultra-compact models (<1B parameters), achieving up to 65.74% overall accuracy, and 55.62% multi-turn accuracy with hybrid optimization. This study highlights the importance of hybrid optimization strategies that enable small language models to deliver accurate, efficient, and stable agentic AI on edge devices, making privacy-preserving, low-latency autonomous agents practical beyond the cloud.