Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs

TL;DR

This paper proposes a formal, graph-based multi-objective hyperparameter selection method for large language models that guarantees reliability and efficiently incorporates prior knowledge about hyperparameter relationships.

Contribution

It introduces reliability graph-based Pareto testing (RG-PT), a novel framework that maintains formal reliability guarantees while leveraging structured hyperparameter relationships.

Findings

RG-PT outperforms existing methods in hyperparameter exploration efficiency.

It guarantees false discovery rate control in hyperparameter selection.

Experimental results validate the effectiveness of RG-PT in LLM settings.

Abstract

The selection of hyperparameters, such as prompt templates in large language models (LLMs), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can be inferred from prior information and held-out data -- e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees or are unable to incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objective hyperparameter selection framework that maintains formal reliability guarantees in terms of false discovery rate (FDR), while accounting for known relationships among hyperparameters via a directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such as learn-then-test (LTT) and Pareto testing (PT) through a more efficient exploration of the hyperparameter space.