Learning Physical Simulation with Message Passing Transformer

TL;DR

This paper introduces a novel Message Passing Transformer architecture for physical simulation, combining graph neural networks, a specialized attention mechanism, and a Fourier-based loss to improve long-term accuracy in dynamical systems.

Contribution

The paper presents a universal GNN-based architecture with a new attention mechanism and Fourier loss, enhancing simulation accuracy and efficiency over existing methods.

Findings

Achieves significant accuracy improvements in long-term dynamical system simulations.

Introduces Hadamard-Product Attention for fine-grained feature focus.

Employs Graph Fourier Loss for balanced energy component optimization.

Abstract

Machine learning methods for physical simulation have achieved significant success in recent years. We propose a new universal architecture based on Graph Neural Network, the Message Passing Transformer, which incorporates a Message Passing framework, employs an Encoder-Processor-Decoder structure, and applies Graph Fourier Loss as loss function for model optimization. To take advantage of the past message passing state information, we propose Hadamard-Product Attention to update the node attribute in the Processor, Hadamard-Product Attention is a variant of Dot-Product Attention that focuses on more fine-grained semantics and emphasizes on assigning attention weights over each feature dimension rather than each position in the sequence relative to others. We further introduce Graph Fourier Loss (GFL) to balance high-energy and low-energy components. To improve time performance, we precompute the graph's Laplacian eigenvectors before the training process. Our architecture achieves significant accuracy improvements in long-term rollouts for both Lagrangian and Eulerian dynamical systems over current methods.