VisTR: Visualizations as Representations for Time-series Table Reasoning

TL;DR

VisTR introduces a novel framework that uses visualizations as core representations to improve time-series table reasoning, enabling better pattern recognition, interpretability, and multimodal interaction.

Contribution

It presents a new approach that integrates visualizations with multimodal LLMs for scalable, interpretable, and accurate time-series reasoning.

Findings

Enhanced reasoning accuracy with visualization-based representations

Effective multimodal alignment between visualizations and user inputs

Scalable retrieval mechanisms for large-scale time-series data

Abstract

Time-series table reasoning interprets temporal patterns and relationships in data to answer user queries. Despite recent advancements leveraging large language models (LLMs), existing methods often struggle with pattern recognition, context drift in long time-series data, and the lack of visual-based reasoning capabilities. To address these challenges, we propose VisTR, a framework that places visualizations at the core of the reasoning process. Specifically, VisTR leverages visualizations as representations to bridge raw time-series data and human cognitive processes. By transforming tables into fixed-size visualization references, it captures key trends, anomalies, and temporal relationships, facilitating intuitive and interpretable reasoning. These visualizations are aligned with user input, i.e., charts, text, and sketches, through a fine-tuned multimodal LLM, ensuring robust cross-modal alignment. To handle large-scale data, VisTR integrates pruning and indexing mechanisms for scalable and efficient retrieval. Finally, an interactive visualization interface supports seamless multimodal exploration, enabling users to interact with data through both textual and visual modalities. Quantitative evaluations demonstrate the effectiveness of VisTR in aligning multimodal inputs and improving reasoning accuracy. Case studies further illustrate its applicability to various time-series reasoning and exploration tasks.