To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO

TL;DR

This paper introduces TempNet, a novel framework using distributionally robust optimization to predict personalized temperatures for large foundation models, enhancing their performance during training and inference.

Contribution

The paper proposes a new TempNet framework with a robust loss and theoretical basis, capable of predicting personalized temperatures for LFMs, improving their adaptability and performance.

Findings

TempNet significantly improves LLM and CLIP model performance.

The framework is transferable to new tasks and models.

Experimental results validate the effectiveness of TempNet.

Abstract

The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax function in LLMs, which is crucial for next token generation, and it scales the similarities in the contrastive loss for training CLIP models. A significant question remains: Is it viable to learn a neural network to predict a personalized temperature of any input data for enhancing LFMs"? In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs. Our solution is composed of a novel learning framework with a robust loss underpinned by constrained distributionally robust optimization (DRO), and a properly designed TempNet with theoretical inspiration. TempNet can be trained together with a large foundation model from scratch or learned separately given a pretrained foundation model. It is not only useful for predicting personalized temperature to promote the training of LFMs but also generalizable and transferable to new tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly improves the performance of existing solutions or models, e.g. Table 1. The code to reproduce the experimental results in this paper can be found at https://github.com/zhqiu/TempNet.