ICLR 2022

Gradient Gating for Deep Multi-Rate Learning on Graphs

T. Konstantin Rusch, B. Chamberlain, Michael W. Mahoney, Michael M. Bronstein, Siddhartha Mishra

Main figure for Gradient Gating for Deep Multi-Rate Learning on Graphs

Abstract

We present Gradient Gating (G$^2$), a novel framework for improving the performance of Graph Neural Networks (GNNs). Our framework is based on gating the output of GNN layers with a mechanism for multi-rate flow of message passing information across nodes of the underlying graph. Local gradients are harnessed to further modulate message passing updates. Our framework flexibly allows one to use any basic GNN layer as a wrapper around which the multi-rate gradient gating mechanism is built. We rigorously prove that G$^2$ alleviates the oversmoothing problem and allows the design of deep GNNs. Empirical results are presented to demonstrate that the proposed framework achieves state-of-the-art performance on a variety of graph learning tasks, including on large-scale heterophilic graphs.

Benchmark Results

Tested on 7 datasets 21 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Heterophilic Graphs

#ModelAccuracy ↑Paper
1
P^2GNN
0.9672 ±0.0147
2
DoG
0.9660 ±0.0210
3
GPF-plus
0.9583 ±0.0419
28
GPRGNN
actual #28 vs paper #269 241 places worse
0.9295 ±0.0131
best reported
actual #28 vs paper #269 241 places worse
paper 0.7838 ±0.0436
paper 0.7838 ±0.0436 Δ at pub 0.0622 Δ today 0.1457
34
MLP
actual #34 vs paper #230 196 places worse
0.9230 ±0.0070
best reported
actual #34 vs paper #230 196 places worse
paper 0.8081 ±0.0475
39
GGCN
actual #39 vs paper #148 109 places worse
0.9213 ±0.0305
best reported
actual #39 vs paper #148 109 places worse
paper 0.8486 ±0.0455
best today: Graph Wave Networks (WWW 2025)
paper 0.8486 ±0.0455 Δ today 0.0727
40
H2GCN
actual #40 vs paper #148 108 places worse
0.9189 ±0.0393
best reported
actual #40 vs paper #148 108 places worse
paper 0.8486 ±0.0723
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)

Difference within reported uncertainty; not counted as underperformance.

paper 0.8486 ±0.0723 Δ at pub 0.0004 Δ today 0.0703
53
GraphSAGE
actual #53 vs paper #193 140 places worse
0.9000 ±0.0365
best reported
actual #53 vs paper #193 140 places worse
paper 0.8243 ±0.0614

Difference within reported uncertainty; not counted as underperformance.

paper 0.8243 ±0.0614 Δ today 0.0757
61
FAGCN
actual #61 vs paper #193 132 places worse
0.8961 ±0.0152
best reported
actual #61 vs paper #193 132 places worse
paper 0.8243 ±0.0689

Difference within reported uncertainty; not counted as underperformance.

paper 0.8243 ±0.0689 Δ today 0.0718
70
GCNII
actual #70 vs paper #280 210 places worse
0.8918 ±0.0443
best reported
actual #70 vs paper #280 210 places worse
paper 0.7757 ±0.0383
paper 0.7757 ±0.0383 Δ at pub 0.0289 Δ today 0.1161
99
G^2-GraphSAGE
0.8757 ±0.0386
This paper
141
GloGNN
actual #141 vs paper #164 23 places worse
0.8524 ±0.0490
best reported
actual #141 vs paper #164 23 places worse
paper 0.8432 ±0.0415

Difference within reported uncertainty; not counted as underperformance.

paper 0.8432 ±0.0415 Δ today 0.0092
151
G^2-GCN
0.8486 ±0.0324
This paper
161
G^2-GAT
0.8459 ±0.0555
This paper
187
F^2GAT
0.8270 ±0.0595
This paper
189
SAGE
0.8269
211
GAT
actual #211 vs paper #576 365 places worse
0.8162 ±0.0645
best reported
actual #211 vs paper #576 365 places worse
paper 0.5216 ±0.0663
paper 0.5216 ±0.0663 Δ at pub 0.2764 Δ today 0.2946
242
ResGatedGCN
0.8000 ±0.0557
This paper
257
GCN
actual #257 vs paper #568 311 places worse
0.7933 ±0.0447
best reported
actual #257 vs paper #568 311 places worse
paper 0.5514 ±0.0516
best today: Graph Wave Networks (WWW 2025)
paper 0.5514 ±0.0516 Δ at pub 0.2326 Δ today 0.2419
276
MixHop
0.7784 ±0.0773
This paper
325
LINKX
0.7460 ±0.0837
This paper
415
Geom-GCN
actual #415 vs paper #425 10 places worse
0.6760
best reported
actual #415 vs paper #425 10 places worse
paper 0.6676 ±0.0272

Difference within reported uncertainty; not counted as underperformance.

paper 0.6676 ±0.0272 Δ at pub 0.0084
499
PairNorm
actual #499 vs paper #499
0.6030 ±0.0430
best reported
actual #499 vs paper #499
paper 0.6027 ±0.0434
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)

Difference within reported uncertainty; not counted as underperformance.

paper 0.6027 ±0.0434 Δ at pub 0.0003 Δ today 0.0003
#ModelAccuracy ↑Paper
1
P^2GNN
0.9541 ±0.0272
2
ACM-GCN
0.9480 ±0.0380
3
RF-GCN
0.9429
15
GGCN
actual #15 vs paper #79 64 places worse
0.9230 ±0.0367
best reported
actual #15 vs paper #79 64 places worse
paper 0.8568 ±0.0663

Difference within reported uncertainty; not counted as underperformance.

paper 0.8568 ±0.0663 Δ today 0.0662
22
GPRGNN
actual #22 vs paper #177 155 places worse
0.9157 ±0.0196
best reported
actual #22 vs paper #177 155 places worse
paper 0.8027 ±0.0811

Difference within reported uncertainty; not counted as underperformance.

paper 0.8027 ±0.0811 Δ at pub 0.0351 Δ today 0.1130
24
MLP
actual #24 vs paper #158 134 places worse
0.9136 ±0.0070
best reported
actual #24 vs paper #158 134 places worse
paper 0.8189 ±0.0640

Difference within reported uncertainty; not counted as underperformance.

paper 0.8189 ±0.0640 Δ at pub 0.0297 Δ today 0.0947
35
GCNII
actual #35 vs paper #203 168 places worse
0.8918 ±0.0396
best reported
actual #35 vs paper #203 168 places worse
paper 0.7786 ±0.0379
paper 0.7786 ±0.0379 Δ at pub 0.0640 Δ today 0.1132
38
FAGCN
actual #38 vs paper #185 147 places worse
0.8852 ±0.0133
best reported
actual #38 vs paper #185 147 places worse
paper 0.7919 ±0.0979

Difference within reported uncertainty; not counted as underperformance.

paper 0.7919 ±0.0979 Δ at pub 0.0189 Δ today 0.0933
46
G^2-GAT
0.8730 ±0.0484
This paper
61
G^2-GCN
0.8649 ±0.0527
This paper
68
H2GCN
actual #68 vs paper #136 68 places worse
0.8623 ±0.0471
best reported
actual #68 vs paper #136 68 places worse
paper 0.8270 ±0.0528
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)

Difference within reported uncertainty; not counted as underperformance.

paper 0.8270 ±0.0528 Δ today 0.0353
70
G^2-GraphSAGE
0.8622 ±0.0490
This paper
74
GloGNN
actual #74 vs paper #119 45 places worse
0.8595 ±0.0510
best reported
actual #74 vs paper #119 45 places worse
paper 0.8351 ±0.0426

Difference within reported uncertainty; not counted as underperformance.

paper 0.8351 ±0.0426 Δ at pub 0.0244 Δ today 0.0244
120
F^2GAT
0.8351 ±0.0670
This paper
123
LINKX
actual #123 vs paper #203 80 places worse
0.8346 ±0.0061
best reported
actual #123 vs paper #203 80 places worse
paper 0.7784 ±0.0581

Difference within reported uncertainty; not counted as underperformance.

paper 0.7784 ±0.0581 Δ today 0.0562
138
SAGE
0.8269
157
GraphSAGE
actual #157 vs paper #222 65 places worse
0.8196 ±0.0739
best reported
actual #157 vs paper #222 65 places worse
paper 0.7595 ±0.0501
best at pub: Graph-Coupled Oscillator Networks (ICML 2022)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7595 ±0.0501 Δ at pub 0.0005 Δ today 0.0601
158
GAT
actual #158 vs paper #358 200 places worse
0.8190 ±0.0540
best reported
actual #158 vs paper #358 200 places worse
paper 0.6189 ±0.0505
paper 0.6189 ±0.0505 Δ at pub 0.2001
170
GCN
actual #170 vs paper #372 202 places worse
0.8082 ±0.0360
best reported
actual #170 vs paper #372 202 places worse
paper 0.6054 ±0.0530
paper 0.6054 ±0.0530 Δ at pub 0.1866 Δ today 0.2028
183
MixHop
actual #183 vs paper #249 66 places worse
0.7950 ±0.0630
best reported
actual #183 vs paper #249 66 places worse
paper 0.7351 ±0.0634

Difference within reported uncertainty; not counted as underperformance.

paper 0.7351 ±0.0634 Δ at pub 0.0599
251
ResGatedGCN
0.7351 ±0.0495
This paper
342
Geom-GCN
actual #342 vs paper #372 30 places worse
0.6412
best reported
actual #342 vs paper #372 30 places worse
paper 0.6054 ±0.0367

Difference within reported uncertainty; not counted as underperformance.

paper 0.6054 ±0.0367 Δ at pub 0.0027 Δ today 0.0358
389
PairNorm
0.5892 ±0.0315
This paper
#ModelAccuracy ↑Paper
1
DiGGR
0.4535 ±0.0353
2
TRIGON
0.4402 ±0.0028
3
DoG
0.4320 ±0.0120
38
GCNII
actual #38 vs paper #114 76 places worse
0.4082 ±0.0179
best reported
actual #38 vs paper #114 76 places worse
paper 0.3744 ±0.0130
paper 0.3744 ±0.0130 Δ today 0.0338
42
MLP
actual #42 vs paper #158 116 places worse
0.4026 ±0.0108
best reported
actual #42 vs paper #158 116 places worse
paper 0.3653 ±0.0070
paper 0.3653 ±0.0070 Δ at pub 0.0365 Δ today 0.0373
52
GPRGNN
actual #52 vs paper #247 195 places worse
0.3992 ±0.0067
best reported
actual #52 vs paper #247 195 places worse
paper 0.3463 ±0.0122
paper 0.3463 ±0.0122 Δ at pub 0.0175 Δ today 0.0529
56
FAGCN
actual #56 vs paper #240 184 places worse
0.3970
best reported
actual #56 vs paper #240 184 places worse
paper 0.3487 ±0.0125
paper 0.3487 ±0.0125 Δ at pub 0.0483
58
GloGNN
actual #58 vs paper #116 58 places worse
0.3959 ±0.0029
best reported
actual #58 vs paper #116 58 places worse
paper 0.3735 ±0.0130
62
GGCN
actual #62 vs paper #113 51 places worse
0.3929 ±0.0048
best reported
actual #62 vs paper #113 51 places worse
paper 0.3754 ±0.0156

Difference within reported uncertainty; not counted as underperformance.

paper 0.3754 ±0.0156 Δ today 0.0175
68
GraphSAGE
actual #68 vs paper #258 190 places worse
0.3899 ±0.0085
best reported
actual #68 vs paper #258 190 places worse
paper 0.3423 ±0.0099
best at pub: Deformable Graph Transformer (2022)
paper 0.3423 ±0.0099 Δ at pub 0.0101 Δ today 0.0476
71
H2GCN
actual #71 vs paper #198 127 places worse
0.3885 ±0.0117
best reported
actual #71 vs paper #198 127 places worse
paper 0.3570 ±0.0100
paper 0.3570 ±0.0100 Δ at pub 0.0270 Δ today 0.0315
87
Geom-GCN
actual #87 vs paper #308 221 places worse
0.3810
best reported
actual #87 vs paper #308 221 places worse
paper 0.3159 ±0.0115
paper 0.3159 ±0.0115 Δ at pub 0.0651
121
G^2-GAT
0.3730 ±0.0087
This paper
127
G^2-GraphSAGE
0.3714 ±0.0101
This paper
132
G^2-GCN
0.3709 ±0.0116
This paper
135
GCN
actual #135 vs paper #408 273 places worse
0.3690
best reported
actual #135 vs paper #408 273 places worse
paper 0.2732 ±0.0110
paper 0.2732 ±0.0110 Δ at pub 0.0958
153
F^2GAT
0.3665 ±0.0113
This paper
162
MixHop
actual #162 vs paper #297 135 places worse
0.3648 ±0.0120
best reported
actual #162 vs paper #297 135 places worse
paper 0.3222 ±0.0234
best at pub: Deformable Graph Transformer (2022)
paper 0.3222 ±0.0234 Δ at pub 0.0357 Δ today 0.0426
180
LINKX
0.3610 ±0.0155
This paper
182
GAT
actual #182 vs paper #406 224 places worse
0.3605 ±0.0035
best reported
actual #182 vs paper #406 224 places worse
paper 0.2744 ±0.0089
paper 0.2744 ±0.0089 Δ at pub 0.0816 Δ today 0.0861
183
ResGatedGCN
0.3602 ±0.0119
This paper
266
GIN
0.3410 ±0.0030
408
PairNorm
0.2740 ±0.0124
This paper
#ModelAccuracy ↑Paper
1
Hetero-S (GAT 8-layer)
0.8693
2
Dir-GNN+CNA
0.8586 ±0.0180
3
GAT
actual #3 vs paper #206 203 places worse
0.8560
best reported
actual #3 vs paper #206 203 places worse
paper 0.4836 ±0.0158
43
G2-GraphSAGE
0.7140 ±0.0238
This paper
45
GloGNN
actual #45 vs paper #56 11 places worse
0.7133 ±0.0242
best reported
actual #45 vs paper #56 11 places worse
paper 0.6978 ±0.0242

Difference within reported uncertainty; not counted as underperformance.

paper 0.6978 ±0.0242 Δ at pub 0.0143 Δ today 0.0155
48
GGCN
0.7114 ±0.0184
This paper
52
GCN
actual #52 vs paper #262 210 places worse
0.7033
best reported
actual #52 vs paper #262 210 places worse
paper 0.3844 ±0.0192
paper 0.3844 ±0.0192 Δ at pub 0.2916 Δ today 0.3189
64
LINKX
0.6842 ±0.0138
This paper
71
F2GAT
0.6781 ±0.0205
This paper
72
GPRGNN
actual #72 vs paper #215 143 places worse
0.6748 ±0.0040
best reported
actual #72 vs paper #215 143 places worse
paper 0.4658 ±0.0171
paper 0.4658 ±0.0171 Δ at pub 0.1989 Δ today 0.2090
99
G2-GAT
0.6412 ±0.0196
This paper
103
GCNII
0.6386 ±0.0304
This paper
107
FAGCN
actual #107 vs paper #164 57 places worse
0.6344 ±0.0205
best reported
actual #107 vs paper #164 57 places worse
paper 0.5522 ±0.0319
paper 0.5522 ±0.0319 Δ at pub 0.0648 Δ today 0.0822
108
SAGE
0.6340 ±0.0300
111
PairNorm
0.6274 ±0.0282
This paper
115
GraphSAGE
actual #115 vs paper #143 28 places worse
0.6215 ±0.0042
best reported
actual #115 vs paper #143 28 places worse
paper 0.5873 ±0.0168
paper 0.5873 ±0.0168 Δ today 0.0342
124
Geom-GCN
actual #124 vs paper #137 13 places worse
0.6090
best reported
actual #124 vs paper #137 13 places worse
paper 0.6000 ±0.0281

Difference within reported uncertainty; not counted as underperformance.

paper 0.6000 ±0.0281 Δ at pub 0.0090 Δ today 0.0090
129
MixHop
0.6050 ±0.0253
This paper
133
H2GCN
actual #133 vs paper #136 3 places worse
0.6030 ±0.0231
best reported
actual #133 vs paper #136 3 places worse
paper 0.6011 ±0.0215

Difference within reported uncertainty; not counted as underperformance.

paper 0.6011 ±0.0215 Δ at pub 0.0019
161
G2-GCN
0.5583 ±0.0288
This paper
169
MLP
actual #169 vs paper #217 48 places worse
0.5429
best reported
actual #169 vs paper #217 48 places worse
paper 0.4621 ±0.0299
paper 0.4621 ±0.0299 Δ at pub 0.0199 Δ today 0.0808
199
ResGatedGCN
0.4982 ±0.0271
This paper
#ModelAccuracy ↑Paper
1
IBG-NN
0.7763 ±0.0179
2
Dir-GNN+CNA
0.7747 ±0.0128
3
DHGNN
0.7684 ±0.0161
19
GloGNN
actual #19 vs paper #57 38 places worse
0.7121 ±0.0184
best reported
actual #19 vs paper #57 38 places worse
paper 0.5754 ±0.0139
paper 0.5754 ±0.0139 Δ at pub 0.0034 Δ today 0.1367
30
G2-GraphSAGE
0.6426 ±0.0238
This paper
39
H2GCN
actual #39 vs paper #202 163 places worse
0.6190 ±0.0140
best reported
actual #39 vs paper #202 163 places worse
paper 0.3648 ±0.0186
paper 0.3648 ±0.0186 Δ at pub 0.0427 Δ today 0.2542
40
LINKX
0.6181 ±0.0180
This paper
46
GCN
actual #46 vs paper #235 189 places worse
0.6067
best reported
actual #46 vs paper #235 189 places worse
paper 0.3152 ±0.0071
paper 0.3152 ±0.0071 Δ at pub 0.2418 Δ today 0.2915
67
GAT
actual #67 vs paper #199 132 places worse
0.5630 ±0.0220
best reported
actual #67 vs paper #199 132 places worse
paper 0.3677 ±0.0168
paper 0.3677 ±0.0168 Δ at pub 0.1953
69
GGCN
actual #69 vs paper #71 2 places worse
0.5567 ±0.0187
best reported
actual #69 vs paper #71 2 places worse
paper 0.5517 ±0.0158

Difference within reported uncertainty; not counted as underperformance.

paper 0.5517 ±0.0158 Δ today 0.0050
76
GPRGNN
actual #76 vs paper #235 159 places worse
0.5435 ±0.0087
best reported
actual #76 vs paper #235 159 places worse
paper 0.3161 ±0.0124
paper 0.3161 ±0.0124 Δ at pub 0.1742 Δ today 0.2274
88
PairNorm
0.5044 ±0.0204
This paper
100
F2GAT
0.4732 ±0.0243
This paper
105
G2-GAT
0.4648 ±0.0141
This paper
120
SAGE
0.4460 ±0.0130
127
MixHop
0.4380 ±0.0148
This paper
128
GraphSAGE
actual #128 vs paper #150 22 places worse
0.4378 ±0.0190
best reported
actual #128 vs paper #150 22 places worse
paper 0.4161 ±0.0074

Difference within reported uncertainty; not counted as underperformance.

paper 0.4161 ±0.0074 Δ today 0.0217
139
FAGCN
actual #139 vs paper #139
0.4259 ±0.0079
actual #139 vs paper #139
paper 0.4259 ±0.0079

Improvement within reported uncertainty; not highlighted.

paper 0.4259 ±0.0079 Δ at pub 0.0016
167
G2-GCN
0.3962 ±0.0291
This paper
172
MLP
actual #172 vs paper #243 71 places worse
0.3930 ±0.0079
best reported
actual #172 vs paper #243 71 places worse
paper 0.2877 ±0.0156
paper 0.2877 ±0.0156 Δ at pub 0.0458 Δ today 0.1053
176
GCNII
actual #176 vs paper #178 2 places worse
0.3881 ±0.0197
best reported
actual #176 vs paper #178 2 places worse
paper 0.3847 ±0.0158

Difference within reported uncertainty; not counted as underperformance.

paper 0.3847 ±0.0158 Δ today 0.0034
179
Geom-GCN
actual #179 vs paper #181 2 places worse
0.3815 ±0.0092
actual #179 vs paper #181 2 places worse
paper 0.3815 ±0.0092

Improvement within reported uncertainty; not highlighted.

paper 0.3815 ±0.0092 Δ at pub 0.0001
192
ResGatedGCN
0.3760 ±0.0180
This paper

LINKX Benchmarks

#ModelAccuracy ↑Paper
1
LargeScaleNet
0.6582 ±0.0036
2
S² DirGCN
0.6495 ±0.0033
3
FaberNet
0.6462 ±0.0101
7
G^2-GraphSAGE
0.6330 ±0.0184
This paper
13
LINKX
0.5600 ±0.0134
This paper
16
GloGNN
actual #16 vs paper #16
0.5480 ±0.0030
best reported
actual #16 vs paper #16
paper 0.5468 ±0.0034

Difference within reported uncertainty; not counted as underperformance.

paper 0.5468 ±0.0034 Δ at pub 0.0012
17
LINK
0.5397 ±0.0018
This paper
19
MixHop
actual #19 vs paper #19
0.5181 ±0.0017
actual #19 vs paper #19
paper 0.5181 ±0.0017
paper 0.5181 ±0.0017 Δ at pub 0.0571
22
GAT
actual #22 vs paper #30 8 places worse
0.4937 ±0.0020
best reported
actual #22 vs paper #30 8 places worse
paper 0.4605 ±0.0051
paper 0.4605 ±0.0051 Δ today 0.0332
25
ACM-GCN
actual #25 vs paper #25
0.4841 ±0.0030
best reported
actual #25 vs paper #25
paper 0.4737 ±0.0059
paper 0.4737 ±0.0059 Δ today 0.0104
26
GCNII
0.4721 ±0.0028
This paper
31
GCN
0.4602 ±0.0026
This paper
32
GPR-GNN
actual #32 vs paper #34 2 places worse
0.4597 ±0.0026
best reported
actual #32 vs paper #34 2 places worse
paper 0.4507 ±0.0021
paper 0.4507 ±0.0021 Δ at pub 0.0090
53
APPNP
0.3815 ±0.0026
This paper
56
MLP
0.3670 ±0.0021
This paper
#ModelAccuracy ↑Paper
1
FaberNet
0.7510 ±0.0003
2
LargeScaleNet
0.7505 ±0.0005
3
DirGNN
0.7395 ±0.0005
7
G^2-GraphSAGE
0.6950 ±0.0039
This paper
10
GloGNN
0.6209 ±0.0027
This paper
11
LINKX
0.6195 ±0.0012
This paper
12
LINK
0.6039 ±0.0007
This paper
14
ACM-GCN
0.5514 ±0.0016
This paper
15
MixHop
0.5216 ±0.0009
This paper
16
GCN
actual #16 vs paper #22 6 places worse
0.5102 ±0.0006
best reported
actual #16 vs paper #22 6 places worse
paper 0.4565 ±0.0004
paper 0.4565 ±0.0004 Δ at pub 0.0005 Δ today 0.0537
23
GAT
actual #23 vs paper #23
0.4540 ±0.0040
best reported
actual #23 vs paper #23
paper 0.4537 ±0.0044

Difference within reported uncertainty; not counted as underperformance.

paper 0.4537 ±0.0044 Δ at pub 0.0003
28
GPR-GNN
0.4019 ±0.0003
This paper
30
GCNII
0.3788 ±0.0069
This paper
31
APPNP
0.3219 ±0.0007
This paper
32
MLP
actual #32 vs paper #32
0.3134 ±0.0005
actual #32 vs paper #32
paper 0.3134 ±0.0005

Improvement within reported uncertainty; not highlighted.

paper 0.3134 ±0.0005 Δ at pub 0.0004 Δ today 0.0016
SOTA performance (not significantly different from the best) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Best today — OK at publication; today's leaderboard best is elsewhere