Embedded AI Companion System on Edge Devices

TL;DR

This paper presents a memory-efficient AI companion system for edge devices that balances real-time interaction with long-term memory management, demonstrating improved performance over baseline models within resource constraints.

Contribution

It introduces a novel alternating memory paradigm and a comprehensive benchmark for evaluating AI companions on embedded hardware.

Findings

Outperforms raw LLM without memory on most metrics

Performs comparably to GPT-3.5 with 16k context window

Efficiently balances latency and personalization on edge devices

Abstract

Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.