DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications

TL;DR

This paper introduces DynamicEmbedding, a system extension for TensorFlow that enables models to handle unlimited sparse features and evolve dynamically, improving scalability and performance in large-scale applications.

Contribution

It proposes a novel neuron model, DynamicCell, and a system design that decouples model content from form, allowing continuous growth without redefining the model.

Findings

Successfully deployed in production for over a year

Achieved significant accuracy improvements in Google Smart Campaigns

Enables models to handle arbitrary sparse features and evolve seamlessly

Abstract

One of the limitations of deep learning models with sparse features today stems from the predefined nature of their input, which requires a dictionary be defined prior to the training. With this paper we propose both a theory and a working system design which remove this limitation, and show that the resulting models are able to perform better and efficiently run at a much larger scale. Specifically, we achieve this by decoupling a model's content from its form to tackle architecture evolution and memory growth separately. To efficiently handle model growth, we propose a new neuron model, called DynamicCell, drawing inspiration from from the free energy principle [15] to introduce the concept of reaction to discharge non-digestive energy, which also subsumes gradient descent based approaches as its special cases. We implement DynamicCell by introducing a new server into TensorFlow to take over most of the work involving model growth. Consequently, it enables any existing deep learning models to efficiently handle arbitrary number of distinct sparse features (e.g., search queries), and grow incessantly without redefining the model. Most notably, one of our models, which has been reliably running in production for over a year, is capable of suggesting high quality keywords for advertisers of Google Smart Campaigns and achieved significant accuracy gains based on a challenging metric -- evidence that data-driven, self-evolving systems can potentially exceed the performance of traditional rule-based approaches.