TD-Interpreter: Enhancing the Understanding of Timing Diagrams with Visual-Language Learning

TL;DR

TD-Interpreter is a novel ML tool that uses multimodal learning to help engineers understand complex timing diagrams through an interactive visual question-answer environment, improving upon existing models.

Contribution

It introduces a specialized multimodal learning approach with synthetic data generation for understanding timing diagrams, enhancing design and verification workflows.

Findings

Outperformed untuned GPT-4o on benchmarks

Demonstrated effectiveness in assisting engineers with TDs

Developed a synthetic data workflow for training

Abstract

We introduce TD-Interpreter, a specialized ML tool that assists engineers in understanding complex timing diagrams (TDs), originating from a third party, during their design and verification process. TD-Interpreter is a visual question-answer environment which allows engineers to input a set of TDs and ask design and verification queries regarding these TDs. We implemented TD-Interpreter with multimodal learning by fine-tuning LLaVA, a lightweight 7B Multimodal Large Language Model (MLLM). To address limited training data availability, we developed a synthetic data generation workflow that aligns visual information with its textual interpretation. Our experimental evaluation demonstrates the usefulness of TD-Interpreter which outperformed untuned GPT-4o by a large margin on the evaluated benchmarks.