Camera Control at the Edge with Language Models for Scene Understanding

TL;DR

The paper introduces OPUS, a cost-effective LLM-based system for controlling PTZ cameras using natural language, improving environmental understanding and outperforming traditional methods in benchmark tests.

Contribution

It presents a novel framework that transfers knowledge from large language models to smaller ones for edge deployment in camera control, enhancing environmental awareness without specialized sensors.

Findings

Achieved 35% improvement over advanced prompting techniques.

20% higher task accuracy than closed-source models like Gemini Pro.

Significantly outperformed traditional language model approaches.

Abstract

In this paper, we present Optimized Prompt-based Unified System (OPUS), a framework that utilizes a Large Language Model (LLM) to control Pan-Tilt-Zoom (PTZ) cameras, providing contextual understanding of natural environments. To achieve this goal, the OPUS system improves cost-effectiveness by generating keywords from a high-level camera control API and transferring knowledge from larger closed-source language models to smaller ones through Supervised Fine-Tuning (SFT) on synthetic data. This enables efficient edge deployment while maintaining performance comparable to larger models like GPT-4. OPUS enhances environmental awareness by converting data from multiple cameras into textual descriptions for language models, eliminating the need for specialized sensory tokens. In benchmark testing, our approach significantly outperformed both traditional language model techniques and more complex prompting methods, achieving a 35% improvement over advanced techniques and a 20% higher task accuracy compared to closed-source models like Gemini Pro. The system demonstrates OPUS's capability to simplify PTZ camera operations through an intuitive natural language interface. This approach eliminates the need for explicit programming and provides a conversational method for interacting with camera systems, representing a significant advancement in how users can control and utilize PTZ camera technology.