Efficiently Enhancing General Agents With Hierarchical-categorical Memory

TL;DR

This paper introduces EHC, a novel general agent that leverages hierarchical memory retrieval and experience learning to adapt and perform well on complex multi-modal tasks without requiring parameter updates.

Contribution

EHC combines hierarchical memory retrieval with task-category experience learning, enabling continuous adaptation and high performance without parameter updates.

Findings

EHC outperforms existing methods on multiple datasets.

EHC achieves state-of-the-art results in multi-modal tasks.

EHC demonstrates effective continuous learning capabilities.

Abstract

With large language models (LLMs) demonstrating remarkable capabilities, there has been a surge in research on leveraging LLMs to build general-purpose multi-modal agents. However, existing approaches either rely on computationally expensive end-to-end training using large-scale multi-modal data or adopt tool-use methods that lack the ability to continuously learn and adapt to new environments. In this paper, we introduce EHC, a general agent capable of learning without parameter updates. EHC consists of a Hierarchical Memory Retrieval (HMR) module and a Task-Category Oriented Experience Learning (TOEL) module. The HMR module facilitates rapid retrieval of relevant memories and continuously stores new information without being constrained by memory capacity. The TOEL module enhances the agent's comprehension of various task characteristics by classifying experiences and extracting patterns across different categories. Extensive experiments conducted on multiple standard datasets demonstrate that EHC outperforms existing methods, achieving state-of-the-art performance and underscoring its effectiveness as a general agent for handling complex multi-modal tasks.