A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecasting

TL;DR

This paper presents LLM-AutoOpt, a hybrid framework that enhances hyperparameter optimization for time-series forecasting by integrating Bayesian Optimization with large language models to incorporate structured prior knowledge and reasoning.

Contribution

It introduces a novel LLM-augmented HPO framework that encodes meta-knowledge into prompts, improving efficiency, interpretability, and performance over traditional methods.

Findings

Improved forecasting accuracy with LLM-AutoOpt.

Enhanced interpretability of hyperparameter decisions.

Better performance than baseline BO and LLM methods.

Abstract

Hyperparameter optimization (HPO) plays a central role in the performance of deep learning models, yet remains computationally expensive and difficult to interpret, particularly for time-series forecasting. While Bayesian Optimization (BO) is a standard approach, it typically treats tuning tasks independently and provides limited insight into its decisions. Recent advances in large language models (LLMs) offer new opportunities to incorporate structured prior knowledge and reasoning into optimization pipelines. We introduce LLM-AutoOpt, a hybrid HPO framework that combines BO with LLM-based contextual reasoning. The framework encodes dataset meta-features, model descriptions, historical optimization outcomes, and target objectives as structured meta-knowledge within LLM prompts, using BO to initialize the search and mitigate cold-start effects. This design enables context-aware and stable hyperparameter refinement while exposing the reasoning behind optimization decisions. Experiments on a multivariate time series forecasting benchmark demonstrate that LLM-AutoOpt achieves improved predictive performance and more interpretable optimization behavior compared to BO and LLM baselines without meta-knowledge.