Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning

TL;DR

This paper demonstrates that small language models, trained with upside-down reinforcement learning and synthetic data from large models, can effectively perform multimodal search and recommendation tasks with near state-of-the-art relevance and diversity.

Contribution

The authors introduce a framework combining upside-down reinforcement learning and synthetic data distillation to train small language models for multimodal tasks, achieving competitive performance.

Findings

SLMs achieve within 6% of large LLMs in relevance and diversity.

The approach reduces inference latency and memory use significantly.

Small models can handle complex multimodal tasks effectively.

Abstract

In this work, we investigate how small language models (SLMs) can be scaled to support multimodal search and recommendation use cases while remaining efficient enough for real-time, resource-constrained deployments. We present a framework that combines upside-down reinforcement learning with synthetic data distillation from a large language model (Llama-3) to train a 100M-parameter GPT-2 model for multitask prompt generation. Despite being up to 80 times smaller than state-of-the-art large language models (LLMs), our SLM achieves relevance and diversity scores within 6% of competitive baselines such as Llama-3 8B, Qwen3 8B, and Ministral 8B. These results demonstrate that SLMs can effectively handle multimodal search and recommendation tasks, while dramatically reducing inference latency and memory overhead. Our study highlights the potential of lightweight models as practical engines for scalable multimodal discovery, bridging the gap between cutting-edge research and real-world multimodal applications such as media recommendations and creative content generation.