JMLR 2023

GNNBenchmark

Vijay Prakash Dwivedi, Chaitanya K. Joshi, T. Laurent, Yoshua Bengio, X. Bresson

Main figure for GNNBenchmark

Abstract

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository. has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

Benchmark Results

Tested on 11 datasets 18 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Classic

#ModelAccuracy ↑Paper
1
GCN
actual #1 vs paper #193 192 places worse
0.9302 ±0.0011
best reported
actual #1 vs paper #193 192 places worse
paper 0.7747 ±0.0085
paper 0.7747 ±0.0085 Δ at pub 0.1555
2
RSE + RGCN (70b)
0.8852 ±0.0019
3
LEMP+T
0.8825
9
SAGE
0.8771
31
GAT
actual #31 vs paper #209 178 places worse
0.8533
best reported
actual #31 vs paper #209 178 places worse
paper 0.7691 ±0.0082
54
GIN
actual #54 vs paper #218 164 places worse
0.8303 ±0.0021
best reported
actual #54 vs paper #218 164 places worse
paper 0.7586 ±0.0058
69
GraphSAGE
actual #69 vs paper #236 167 places worse
0.8146 ±0.0010
best reported
actual #69 vs paper #236 167 places worse
paper 0.7477 ±0.0095
193
MoNet-PE
0.7748 ±0.0071
This paper
194
MoNet
0.7743 ±0.0067
This paper
206
vanilla GCN
0.7710 ±0.0083
This paper
237
MLP
actual #237 vs paper #306 69 places worse
0.7469 ±0.0085
best reported
actual #237 vs paper #306 69 places worse
paper 0.5945 ±0.0233
paper 0.5945 ±0.0233 Δ at pub 0.1253 Δ today 0.1524
#ModelMAE ↓Paper
1
Ours(100K)
0.9200 ±0.0090
3
GatedGCN-E-PE
0.9960 ±0.0080
This paper
2
GatedGCN-PE/E(500K)
0.9960 ±0.0080
4
3WLGNN-E
1.0420 ±0.0640
This paper
8
3WLGNN
1.1540 ±0.0500
This paper
10
GAT
actual #10 vs paper #19 9 places worse
1.2400 ±0.0110
best reported
actual #10 vs paper #19 9 places worse
paper 1.4410 ±0.0230
paper 1.4410 ±0.0230 Δ today 0.2010
13
GCN
actual #13 vs paper #17 4 places worse
1.2800 ±0.0110
best reported
actual #13 vs paper #17 4 places worse
paper 1.3720 ±0.0200
paper 1.3720 ±0.0200 Δ today 0.0920
14
GatedGCN-E
1.2950 ±0.0160
This paper
17
GatedGCN
1.3520 ±0.0340
This paper
18
MoNet
1.3950 ±0.0270
This paper
19
GraphSAGE
1.4310 ±0.0100
This paper
20
vanilla GCN
1.4830 ±0.0140
This paper
22
MLP
1.7440 ±0.0160
This paper
23
GIN
1.8940 ±0.0240
This paper
24
RingGNN-E
3.7690 ±1.0120
This paper
25
RingGNN
20.2640 ±7.5490
This paper

TU Dortmund

#ModelAccuracy ↑Paper
1
MSH-GNN
0.9410 ±0.0330
2
GRDL
0.8260 ±0.0120
3
GMN
0.8225
32
GIN
actual #32 vs paper #308 276 places worse
0.7880 ±0.0410
best reported
actual #32 vs paper #308 276 places worse
paper 0.7412 ±0.0336

Difference within reported uncertainty; not counted as underperformance.

paper 0.7412 ±0.0336 Δ at pub 0.0198 Δ today 0.0468
49
GAT
actual #49 vs paper #131 82 places worse
0.7628 ±0.0241
actual #49 vs paper #131 82 places worse
paper 0.7628 ±0.0241

Improvement within reported uncertainty; not highlighted.

paper 0.7628 ±0.0241 Δ at pub 0.0038
67
SAGE
0.7726 ±0.0228
95
GatedGCN
0.7672 ±0.0311
This paper
111
MoNet
0.7645 ±0.0289
This paper
144
vanilla GCN
0.7610 ±0.0241
This paper
149
MLP
actual #149 vs paper #163 14 places worse
0.7582 ±0.0291
actual #149 vs paper #163 14 places worse
paper 0.7582 ±0.0291

Improvement within reported uncertainty; not highlighted.

paper 0.7582 ±0.0291 Δ at pub 0.0152
155
GraphSAGE
actual #155 vs paper #190 35 places worse
0.7556 ±0.0191
actual #155 vs paper #190 35 places worse
paper 0.7556 ±0.0191
paper 0.7556 ±0.0191 Δ at pub 0.0096
472
RingGNN
0.6756 ±0.0755
This paper
485
3WLGNN
0.6468 ±0.0588
This paper
#ModelAccuracy ↑Paper
1
GMN
0.7866
2
MemGNN
0.7550
3
TFGW SP
0.7510 ±0.0500
17
GatedGCN
0.7000 ±0.0494
This paper
20
GAT
actual #20 vs paper #20
0.6850 ±0.0524
actual #20 vs paper #20
paper 0.6850 ±0.0524
paper 0.6850 ±0.0524 Δ at pub 0.1440
26
GraphSAGE
actual #26 vs paper #26
0.6817 ±0.0545
actual #26 vs paper #26
paper 0.6817 ±0.0545

Improvement within reported uncertainty; not highlighted.

paper 0.6817 ±0.0545 Δ at pub 0.0787
32
GIN
actual #32 vs paper #32
0.6767 ±0.0583
actual #32 vs paper #32
paper 0.6767 ±0.0583
paper 0.6767 ±0.0583 Δ at pub 0.0547
36
GCN
0.6650 ±0.0691
42
vanilla GCN
0.6583 ±0.0461
This paper
67
MoNet
0.6300 ±0.0809
This paper
85
3WLGNN
0.6100 ±0.0680
This paper
124
MLP
actual #124 vs paper #124
0.5583 ±0.0352
actual #124 vs paper #124
paper 0.5583 ±0.0352
paper 0.5583 ±0.0352 Δ at pub 0.2503
172
RingGNN
0.4533 ±0.0452
This paper

OGB

#ModelHits@50 ↑Paper
1
PROXI
0.7650 ±0.0027
2
OGB Leader
0.7129 ±0.0018
3
GIDN
0.7096 ±0.0055
9
GCN
actual #9 vs paper #75 66 places worse
0.4896 ±0.0114
actual #9 vs paper #75 66 places worse
paper 0.4896 ±0.0114
prev best: OGB (NeurIPS 2020)

Improvement within reported uncertainty; not highlighted.

paper 0.4896 ±0.0114 Δ at pub 0.0182 Δ today 0.1725
28
SAGE
0.5944 ±0.0137
34
GAT
actual #34 vs paper #56 22 places worse
0.5150 ±0.0096
actual #34 vs paper #56 22 places worse
paper 0.5150 ±0.0096
paper 0.5150 ±0.0096 Δ at pub 0.0661 Δ today 0.0493
41
GraphSAGE
actual #41 vs paper #56 15 places worse
0.5463 ±0.0112
best reported
actual #41 vs paper #56 15 places worse
paper 0.5162 ±0.0069
best at pub: OGB (NeurIPS 2020)
paper 0.5162 ±0.0069 Δ at pub 0.0301 Δ today 0.0301
51
GatedGCN-PE
0.5285 ±0.0135
This paper
53
GatedGCN
0.5263 ±0.0117
This paper
59
vanilla GCN
0.5042 ±0.0113
This paper
71
GatedGCN-E
0.4921 ±0.0156
This paper
94
MLP
actual #94 vs paper #110 16 places worse
0.2035 ±0.0217
actual #94 vs paper #110 16 places worse
paper 0.2035 ±0.0217

Improvement within reported uncertainty; not highlighted.

paper 0.2035 ±0.0217 Δ at pub 0.0037 Δ today 0.2403
95
Matrix Fact.
0.4421 ±0.0045
This paper
99
GIN
0.4173 ±0.0228
This paper
105
MoNet
0.3614 ±0.0219
This paper

GNNBenchmark

#ModelAccuracy ↑Paper
1
Schrödinger
0.9913 ±0.0004
2
CNN
0.9907 ±0.0007
3
MPNN
0.9895 ±0.0006
11
GatedGCN
actual #11 vs paper #90 79 places worse
0.9855 ±0.0002
best reported
actual #11 vs paper #90 79 places worse
paper 0.9734 ±0.0014
32
GIN
actual #32 vs paper #110 78 places worse
0.9833 ±0.0011
best reported
actual #32 vs paper #110 78 places worse
paper 0.9648 ±0.0025
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)
paper 0.9648 ±0.0025 Δ at pub 0.0075 Δ today 0.0184
86
GAT
actual #86 vs paper #121 35 places worse
0.9755 ±0.0003
best reported
actual #86 vs paper #121 35 places worse
paper 0.9554 ±0.0021
95
GraphSAGE
0.9731 ±0.0010
This paper
97
SAGE
0.9731 ±0.0010
126
MLP
0.9534 ±0.0014
This paper
129
3WLGNN
0.9507 ±0.0096
This paper
135
GCN
actual #135 vs paper #141 6 places worse
0.9405 ±0.0005
best reported
actual #135 vs paper #141 6 places worse
paper 0.9012 ±0.0014
paper 0.9012 ±0.0014 Δ at pub 0.0257 Δ today 0.0393
137
MoNet
actual #137 vs paper #137
0.9111
best reported
actual #137 vs paper #137
paper 0.9081 ±0.0003
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)
paper 0.9081 ±0.0003 Δ at pub 0.0030
139
vanilla GCN
0.9071 ±0.0022
This paper
153
RingGNN
0.1135
This paper
#ModelAccuracy ↑Paper
1
RandAlign
0.7942 ±0.0007
2
differential encoding
0.7907 ±0.0027
3
PPGT
0.7856 ±0.0070
62
GatedGCN
actual #62 vs paper #90 28 places worse
0.7133 ±0.0039
best reported
actual #62 vs paper #90 28 places worse
paper 0.6731 ±0.0031
paper 0.6731 ±0.0031 Δ at pub 0.0402
92
GraphSAGE
actual #92 vs paper #104 12 places worse
0.6690 ±0.0020
best reported
actual #92 vs paper #104 12 places worse
paper 0.6577 ±0.0031
paper 0.6577 ±0.0031 Δ at pub 0.0113
101
GAT
actual #101 vs paper #112 11 places worse
0.6611 ±0.0098
best reported
actual #101 vs paper #112 11 places worse
paper 0.6422 ±0.0046
paper 0.6422 ±0.0046 Δ at pub 0.0189
103
MoNet
actual #103 vs paper #129 26 places worse
0.6591 ±0.0251
best reported
actual #103 vs paper #129 26 places worse
paper 0.5465 ±0.0052
paper 0.5465 ±0.0052 Δ at pub 0.1126
104
SAGE
0.6577 ±0.0031
109
GIN
actual #109 vs paper #129 20 places worse
0.6510 ±0.0050
best reported
actual #109 vs paper #129 20 places worse
paper 0.5525 ±0.0153
paper 0.5525 ±0.0153 Δ at pub 0.0985
123
3WLGNN
0.5917 ±0.0159
This paper
124
GCN
actual #124 vs paper #131 7 places worse
0.5839 ±0.0073
best reported
actual #124 vs paper #131 7 places worse
paper 0.5414 ±0.0039
paper 0.5414 ±0.0039 Δ at pub 0.0425
126
MLP
actual #126 vs paper #126
0.5634 ±0.0018
actual #126 vs paper #126
paper 0.5634 ±0.0018
paper 0.5634 ±0.0018 Δ at pub 0.0274
129
vanilla GCN
0.5571 ±0.0038
This paper
140
RingGNN
0.1930 ±0.1611
This paper
#ModelAccuracy ↑Paper
1
GRPE-Small
0.8159 ±0.0019
2
TANGO-GPS
0.8011 ±0.0014
3
GRIT
0.8003 ±0.0028
55
GatedGCN-PE
0.7608 ±0.0020
This paper
54
GatedGCN
0.7608 ±0.0020
67
GAT
actual #67 vs paper #90 23 places worse
0.7059 ±0.0045
best reported
actual #67 vs paper #90 23 places worse
paper 0.5773 ±0.0032
paper 0.5773 ±0.0032 Δ at pub 0.1286 Δ today 0.1286
70
GCN
actual #70 vs paper #102 32 places worse
0.6903 ±0.0137
best reported
actual #70 vs paper #102 32 places worse
paper 0.4783 ±0.0151
paper 0.4783 ±0.0151 Δ at pub 0.2067 Δ today 0.2120
76
MoNet
actual #76 vs paper #88 12 places worse
0.6641 ±0.0054
best reported
actual #76 vs paper #88 12 places worse
paper 0.5806 ±0.0013
79
GIN
actual #79 vs paper #87 8 places worse
0.6472 ±0.0155
best reported
actual #79 vs paper #87 8 places worse
paper 0.5838 ±0.0024
paper 0.5838 ±0.0024 Δ at pub 0.0633 Δ today 0.0633
80
GraphSAGE
actual #80 vs paper #94 14 places worse
0.6388 ±0.0011
best reported
actual #80 vs paper #94 14 places worse
paper 0.5045 ±0.0014
best today: IJCAI--23 Formatting Instructions (IJCAI 2023)
paper 0.5045 ±0.0014 Δ today 0.1343
85
3WLGNN
actual #85 vs paper #90 5 places worse
0.6040 ±0.0042
best reported
actual #85 vs paper #90 5 places worse
paper 0.5713 ±0.0654

Difference within reported uncertainty; not counted as underperformance.

paper 0.5713 ±0.0654 Δ at pub 0.0327
93
vanilla GCN
0.5344 ±0.0203
This paper
104
RingGNN
0.4242 ±0.2006
This paper
114
MLP
0.2097 ±0.0000
This paper
#ModelAccuracy ↑Paper
1
DeepGraph (48)
0.9066 ±0.0006
2
GRIT (our run)
0.9041 ±0.0023
3
GRIT 2-QiQRW
0.9016 ±0.0045
67
GatedGCN-PE
0.8651 ±0.0009
This paper
68
GatedGCN (Rand sign(EigVecs))
0.8651 ±0.0009
This paper
66
GatedGCN
0.8651 ±0.0008
69
GatedGCN (Abs(EigVecs))
0.8639 ±0.0004
This paper
72
RingGNN
0.8625 ±0.0001
This paper
74
GatedGCN (EigVecs-2)
0.8603 ±0.0009
This paper
78
GatedGCN (Rand node ordering)
0.8577 ±0.0004
This paper
85
3WLGNN
0.8566 ±0.0035
This paper
88
GCN
actual #88 vs paper #92 4 places worse
0.8550 ±0.0004
actual #88 vs paper #92 4 places worse
paper 0.8550 ±0.0004
paper 0.8550 ±0.0004 Δ at pub 0.1361 Δ today 0.0012
89
GIN
0.8559 ±0.0001
This paper
91
MoNet
actual #91 vs paper #92 1 place worse
0.8558 ±0.0004
best reported
actual #91 vs paper #92 1 place worse
paper 0.8548 ±0.0004
107
GAT
actual #107 vs paper #112 5 places worse
0.7827 ±0.0019
best reported
actual #107 vs paper #112 5 places worse
paper 0.7582 ±0.0182

Difference within reported uncertainty; not counted as underperformance.

paper 0.7582 ±0.0182 Δ at pub 0.0245 Δ today 0.0245
119
vanilla GCN
0.6388 ±0.0007
This paper
123
MLP
0.5052
This paper
124
GraphSAGE
0.5052 ±0.0000
This paper
#ModelAccuracy ↑Paper
8
GCN
actual #8 vs paper #8
1.0000
actual #8 vs paper #8
paper 1.0000
paper 1.0000 Δ at pub 1.0000
1
PPGN
1.0000
2
RW-AgentNet
1.0000
3
AgentNet
1.0000
33
MoNet
0.9997 ±0.0033
This paper
35
GraphSAGE
0.9993 ±0.0047
This paper
34
GAT
0.9993 ±0.0047
This paper
36
GatedGCN
actual #36 vs paper #36
0.9960 ±0.0108
actual #36 vs paper #36
paper 0.9960 ±0.0108
paper 0.9960 ±0.0108 Δ at pub 0.9960
37
GatedGCN (Rand sign(EigVecs))
0.9960 ±0.0108
This paper
38
GatedGCN (Abs(EigVecs))
0.9943 ±0.0113
This paper
40
GIN
actual #40 vs paper #40
0.9933 ±0.0133
actual #40 vs paper #40
paper 0.9933 ±0.0133
paper 0.9933 ±0.0133 Δ at pub 0.8933
46
3WLGNN
actual #46 vs paper #47 1 place worse
0.9780 ±0.1090
best reported
actual #46 vs paper #47 1 place worse
paper 0.9570 ±0.1485

Difference within reported uncertainty; not counted as underperformance.

paper 0.9570 ±0.1485 Δ at pub 0.0210 Δ today 0.0210
69
RingGNN
actual #69 vs paper #69
0.1723 ±0.0633
actual #69 vs paper #69
paper 0.1723 ±0.0633
paper 0.1723 ±0.0633 Δ at pub 0.0723
79
All MP-GCNs
0.1000
This paper
#ModelF1 ↑Paper
1
TGT-Agx4
0.8710 ±0.0010
2
TGT-Agx3
0.8660 ±0.0010
3
GIG-GatedGCN
0.8630 ±0.0010
8
GatedGCN
actual #8 vs paper #11 3 places worse
0.8380 ±0.0020
best reported
actual #8 vs paper #11 3 places worse
paper 0.7910 ±0.0030
paper 0.7910 ±0.0030 Δ at pub 0.0170 Δ today 0.0470
9
GatedGCN-E
0.8380 ±0.0020
This paper
12
RingGNN
0.7040 ±0.0030
This paper
15
3WLGNN
0.6940 ±0.0730
This paper
16
k-NN Heuristic
0.6930
This paper
18
GAT
0.6710 ±0.0020
This paper
19
GraphSAGE
0.6650 ±0.0030
This paper
20
GIN
0.6560 ±0.0030
This paper
21
GCN
actual #21 vs paper #21
0.6430 ±0.0010
actual #21 vs paper #21
paper 0.6430 ±0.0010
paper 0.6430 ±0.0010 Δ at pub 0.0130
22
MoNet
0.6410 ±0.0020
This paper
23
vanilla GCN
0.6300 ±0.0010
This paper
24
MLP
0.5440 ±0.0010
This paper
SOTA performance (not significantly different from the best) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Improved at pub — later paper surpassed this result Best today — OK at publication; today's leaderboard best is elsewhere