Leveraging semantic similarity for experimentation with AI-generated treatments

TL;DR

This paper introduces a novel kernel-based representation learning method for high-dimensional treatments generated by LLMs, enabling more effective online experimentation and causal effect estimation.

Contribution

It proposes double kernel representation learning with an efficient algorithm and convergence guarantees, advancing the analysis of complex AI-generated treatments.

Findings

Effective low-dimensional representations of treatments

Improved adaptive online experimentation strategies

Numerical experiments demonstrating method efficacy

Abstract

Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main methodological challenge in this setting is representing these high-dimensional treatments without losing their semantic meaning or rendering analysis intractable. Here, we address this problem by focusing on learning low-dimensional representations that capture the underlying structure of such treatments. These representations enable downstream applications such as guiding generative models to produce meaningful treatment variants and facilitating adaptive assignment in online experiments. We propose double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an alternating-minimization algorithm that learns these representations efficiently from data and provides convergence guarantees under a low-rank factor model. As an application of this framework, we introduce an adaptive design strategy for online experimentation and demonstrate the method's effectiveness through numerical experiments.