Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture
Daria Bakshandaeva, Denis Dimitrov, Vladimir Arkhipkin, Alex, Shonenkov, Mark Potanin, Denis Karachev, Andrey Kuznetsov, Anton Voronov,, Vera Davydova, Elena Tutubalina, Aleksandr Petiushko

TL;DR
This paper introduces the Fusion Brain challenge, a competition aimed at developing a universal multimodal, multitask AI architecture capable of handling diverse vision and language tasks efficiently.
Contribution
It presents the first multimodal multitask architecture baseline using a frozen foundation model trained in fusion mode, demonstrating competitive performance and energy efficiency.
Findings
Fusion approach is competitive with task-specific models.
Proposed architecture is more energy-efficient.
New multilingual handwritten dataset with 94,128 pairs.
Abstract
Supporting the current trend in the AI community, we present the AI Journey 2021 Challenge called Fusion Brain, the first competition which is targeted to make the universal architecture which could process different modalities (in this case, images, texts, and code) and solve multiple tasks for vision and language. The Fusion Brain Challenge combines the following specific tasks: Code2code Translation, Handwritten Text recognition, Zero-shot Object Detection, and Visual Question Answering. We have created datasets for each task to test the participants' submissions on it. Moreover, we have collected and made publicly available a new handwritten dataset in both English and Russian, which consists of 94,128 pairs of images and texts. We also propose a multimodal and multitask architecture - a baseline solution, in the center of which is a frozen foundation model and which has been…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Natural Language Processing Techniques
