# Just-in-Time Dynamic-Batching

## TL;DR

This paper introduces a dynamic batching method for deep learning models with variable computation graphs, significantly improving efficiency by leveraging just-in-time compilation techniques.

## Contribution

It presents a novel dynamic batching approach integrated with MXNet Gluon's JIT, optimizing computation for models with dynamic graph structures.

## Key findings

- Up to 6.25x speed-up on a tree-LSTM semantic relatedness task
- Analysis of the trade-off between graph analysis time and batching effectiveness
- Extension of JIT framework for dynamic graph batching

## Abstract

Batching is an essential technique to improve computation efficiency in deep learning frameworks. While batch processing for models with static feed-forward computation graphs is straightforward to implement, batching for dynamic computation graphs such as syntax trees or social network graphs is challenging due to variable computation graph structure across samples. Through simulation and analysis of a Tree-LSTM model, we show the key trade-off between graph analysis time and batching effectiveness in dynamic batching. Based on this finding, we propose a dynamic batching method as an extension to MXNet Gluon's just-in-time compilation (JIT) framework. We show empirically that our method yields up to 6.25 times speed-up on a common dynamic workload, a tree-LSTM model for the semantic relatedness task.