Diffusing Graph Attention
Daniel Glickman, Eran Yahav

TL;DR
Graph Diffuser introduces a novel Transformer-based model for graphs that effectively captures long-range interactions by learning structural and positional relationships, outperforming existing methods across multiple benchmarks.
Contribution
The paper proposes Graph Diffuser, a new architecture that integrates learned structural relationships into Graph Transformers to better model long-range dependencies.
Findings
Outperforms state-of-the-art on eight benchmarks
Effectively captures long-range interactions in graphs
Provides intuitive visualizations of learned relationships
Abstract
The dominant paradigm for machine learning on graphs uses Message Passing Graph Neural Networks (MP-GNNs), in which node representations are updated by aggregating information in their local neighborhood. Recently, there have been increasingly more attempts to adapt the Transformer architecture to graphs in an effort to solve some known limitations of MP-GNN. A challenging aspect of designing Graph Transformers is integrating the arbitrary graph structure into the architecture. We propose Graph Diffuser (GD) to address this challenge. GD learns to extract structural and positional relationships between distant nodes in the graph, which it then uses to direct the Transformer's attention and node representation. We demonstrate that existing GNNs and Graph Transformers struggle to capture long-range interactions and how Graph Diffuser does so while admitting intuitive visualizations.…
| PART | DESCRIPTION |
|---|---|
| Dendrite | Input terminal |
| Axon | Output terminal |
| Soma | Cell body (contains cell nucleus) |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Graph Neural Networks · Complex Network Analysis Techniques · Functional Brain Connectivity Studies
MethodsAttention Is All You Need · Linear Layer · Absolute Position Encodings · Label Smoothing · Softmax · Adam · Layer Normalization · Residual Connection · Dense Connections · Multi-Head Attention
Formatting Instructions for ICLR 2023
Conference Submissions
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213, USA
{hippo,brain,jen}@cs.cranberry-lemon.edu
&Ji Q. Ren & Yevgeny LeNet
Department of Computational Neuroscience
University of the Witwatersrand
Joburg, South Africa
{robot,net}@wits.ac.za
\ANDCoauthor
Affiliation
Address
email Use footnote for providing further information about author (webpage, alternative address)—not for acknowledging funding agencies. Funding acknowledgements go at the end of the paper.
Abstract
The abstract paragraph should be indented 1/2 inch (3 picas) on both left and right-hand margins. Use 10 point type, with a vertical spacing of 11 points. The word Abstract must be centered, in small caps, and in point size 12. Two line spaces precede the abstract. The abstract must be limited to one paragraph.
1 Submission of conference papers to ICLR 2023
ICLR requires electronic submissions, processed by https://openreview.net/. See ICLR’s website for more instructions.
If your paper is ultimately accepted, the statement \iclrfinalcopy should be inserted to adjust the format to the camera ready requirements.
The format for the submissions is a variant of the NeurIPS format. Please read carefully the instructions below, and follow them faithfully.
1.1 Style
Papers to be submitted to ICLR 2023 must be prepared according to the instructions presented here.
Authors are required to use the ICLR LaTeX style files obtainable at the ICLR website. Please make sure you use the current files and not previous versions. Tweaking the style files may be grounds for rejection.
1.2 Retrieval of style files
The style files for ICLR and other conference information are available online at:
The file iclr2023_conference.pdf contains these instructions and illustrates the various formatting requirements your ICLR paper must satisfy. Submissions must be made using LaTeX and the style files iclr2023_conference.sty and iclr2023_conference.bst (to be used with LaTeX2e). The file iclr2023_conference.tex may be used as a “shell” for writing your paper. All you have to do is replace the author, title, abstract, and text of the paper with your own.
The formatting instructions contained in these style files are summarized in sections 2, 3, and 4 below.
2 General formatting instructions
The text must be confined within a rectangle 5.5 inches (33 picas) wide and 9 inches (54 picas) long. The left margin is 1.5 inch (9 picas). Use 10 point type with a vertical spacing of 11 points. Times New Roman is the preferred typeface throughout. Paragraphs are separated by 1/2 line space, with no indentation.
Paper title is 17 point, in small caps and left-aligned. All pages should start at 1 inch (6 picas) from the top of the page.
Authors’ names are set in boldface, and each name is placed above its corresponding address. The lead author’s name is to be listed first, and the co-authors’ names are set to follow. Authors sharing the same address can be on the same line.
Please pay special attention to the instructions in section 4 regarding figures, tables, acknowledgments, and references.
There will be a strict upper limit of 9 pages for the main text of the initial submission, with unlimited additional pages for citations.
3 Headings: first level
First level headings are in small caps, flush left and in point size 12. One line space before the first level heading and 1/2 line space after the first level heading.
3.1 Headings: second level
Second level headings are in small caps, flush left and in point size 10. One line space before the second level heading and 1/2 line space after the second level heading.
3.1.1 Headings: third level
Third level headings are in small caps, flush left and in point size 10. One line space before the third level heading and 1/2 line space after the third level heading.
4 Citations, figures, tables, references
These instructions apply to everyone, regardless of the formatter being used.
4.1 Citations within the text
Citations within the text should be based on the natbib package and include the authors’ last names and year (with the “et al.” construct for more than two authors). When the authors or the publication are included in the sentence, the citation should not be in parenthesis using \citet{} (as in “See Hinton06 for more information.”). Otherwise, the citation should be in parenthesis using \citep{} (as in “Deep learning shows promise to make progress towards AI (Bengio+chapter2007).”).
The corresponding references are to be listed in alphabetical order of authors, in the References section. As to the format of the references themselves, any style is acceptable as long as it is used consistently.
4.2 Footnotes
Indicate footnotes with a number111Sample of the first footnote in the text. Place the footnotes at the bottom of the page on which they appear. Precede the footnote with a horizontal rule of 2 inches (12 picas).222Sample of the second footnote
4.3 Figures
All artwork must be neat, clean, and legible. Lines should be dark enough for purposes of reproduction; art work should not be hand-drawn. The figure number and caption always appear after the figure. Place one line space before the figure caption, and one line space after the figure. The figure caption is lower case (except for first word and proper nouns); figures are numbered consecutively.
Make sure the figure caption does not get separated from the figure. Leave sufficient space to avoid splitting the figure and figure caption.
You may use color figures. However, it is best for the figure captions and the paper body to make sense if the paper is printed either in black/white or in color.
4.4 Tables
All tables must be centered, neat, clean and legible. Do not use hand-drawn tables. The table number and title always appear before the table. See Table 1.
Place one line space before the table title, one line space after the table title, and one line space after the table. The table title must be lower case (except for first word and proper nouns); tables are numbered consecutively.
5 Default Notation
In an attempt to encourage standardized notation, we have included the notation file from the textbook, Deep Learning goodfellow2016deep available at https://github.com/goodfeli/dlbook_notation/. Use of this style is not required and can be disabled by commenting out math_commands.tex.
Numbers and Arrays
[TABLE]
Sets and Graphs
[TABLE]
Indexing
[TABLE]
Calculus
[TABLE]
Probability and Information Theory
[TABLE]
Functions
[TABLE]
6 Final instructions
Do not change any aspects of the formatting parameters in the style files. In particular, do not modify the width or length of the rectangle the text should fit into, and do not change font sizes (except perhaps in the References section; see below). Please note that pages should be numbered.
7 Preparing PostScript or PDF files
Please prepare PostScript or PDF files with paper size “US Letter”, and not, for example, “A4”. The -t letter option on dvips will produce US Letter files.
Consider directly generating PDF files using pdflatex (especially if you are a MiKTeX user). PDF figures must be substituted for EPS figures, however.
Otherwise, please generate your PostScript and PDF files with the following commands:
dvips mypaper.dvi -t letter -Ppdf -G0 -o mypaper.ps ps2pdf mypaper.ps mypaper.pdf
7.1 Margins in LaTeX
Most of the margin problems come from figures positioned by hand using \special or other commands. We suggest using the command \includegraphics from the graphicx package. Always specify the figure width as a multiple of the line width as in the example below using .eps graphics
\usepackage[dvips]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.eps}
or
\usepackage[pdftex]{graphicx} ... \includegraphics[width=0.8\linewidth]{myfile.pdf}
for .pdf graphics. See section 4.4 in the graphics bundle documentation (http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps)
A number of width problems arise when LaTeX cannot properly hyphenate a line. Please give LaTeX hyphenation hints using the - command.
Author Contributions
If you’d like to, you may include a section for author contributions as is done in many journals. This is optional and at the discretion of the authors.
Acknowledgments
Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those to funding agencies, go at the end of the paper.
Appendix A Appendix
You may include other additional sections here.
