Automatic Synthesis of Parallel Unix Commands and Pipelines with KumQuat
Jiasi Shen, Martin Rinard, Nikos Vasilakis

TL;DR
KumQuat automatically generates data parallel implementations of Unix commands and pipelines, enabling efficient parallel processing by synthesizing correct combine operators through a domain-specific language.
Contribution
The paper introduces KumQuat, a system that automatically synthesizes combiners for parallelizing Unix pipelines, improving automation and efficiency in data stream processing.
Findings
Successfully synthesizes combiners for 113 out of 121 commands.
Median synthesis time is 60 seconds.
Parallelized scripts show effective performance improvements.
Abstract
We present KumQuat, a system for automatically generating data parallel implementations of Unix shell commands and pipelines. The generated parallel versions split input streams, execute multiple instantiations of the original pipeline commands to process the splits in parallel, then combine the resulting parallel outputs to produce the final output stream. KumQuat automatically synthesizes the combine operators, with a domain-specific combiner language acting as a strong regularizer that promotes efficient inference of correct combiners. We evaluate KumQuat on 70 benchmark scripts that together have a total of 427 stages. KumQuat synthesizes a correct combiner for 113 of the 121 unique commands that appear in these benchmark scripts. The synthesis times vary between 39 seconds and 331 seconds with a median of 60 seconds. We present experimental results that show that these combiners…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsParallel Computing and Optimization Techniques · Software Testing and Debugging Techniques · Software System Performance and Reliability
