ICML 2024

Sign is Not a Remedy: Multiset-to-Multiset Message Passing for Learning on Heterophilic Graphs

Langzhang Liang, Sunwoo Kim, Kijung Shin, Zenglin Xu, Shirui Pan, Yuan Qi

Main figure for Sign is Not a Remedy: Multiset-to-Multiset Message Passing for Learning on Heterophilic Graphs

Abstract

Graph Neural Networks (GNNs) have gained significant attention as a powerful modeling and inference method, especially for homophilic graph-structured data. To empower GNNs in heterophilic graphs, where adjacent nodes exhibit dissimilar labels or features, Signed Message Passing (SMP) has been widely adopted. However, there is a lack of theoretical and empirical analysis regarding the limitations of SMP. In this work, we unveil some potential pitfalls of SMP and their remedies. We first identify two limitations of SMP: undesirable representation update for multi-hop neighbors and vulnerability against oversmoothing issues. To overcome these challenges, we propose a novel message passing function called Multiset to Multiset GNN(M2M-GNN). Our theoretical analyses and extensive experiments demonstrate that M2M-GNN effectively alleviates the aforementioned limitations of SMP, yielding superior performance in comparison

Proposed method: M2M-GNN

Benchmark Results

Tested on 6 datasets 28 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

Heterophilic Graphs

#ModelAccuracy ↑Paper
1
P^2GNN
0.9672 ±0.0147
2
DoG
0.9660 ±0.0210
3
GPF-plus
0.9583 ±0.0419
5
ACM-GCN
actual #5 vs paper #91 86 places worse
0.9557 ±0.0360
best reported
actual #5 vs paper #91 86 places worse
paper 0.8784 ±0.0440
30
GPR-GNN
actual #30 vs paper #269 239 places worse
0.9291 ±0.0132
best reported
actual #30 vs paper #269 239 places worse
paper 0.7838 ±0.0430
paper 0.7838 ±0.0430 Δ at pub 0.1453
34
MLP
actual #34 vs paper #230 196 places worse
0.9230 ±0.0070
best reported
actual #34 vs paper #230 196 places worse
paper 0.8081 ±0.0470
39
GGCN
actual #39 vs paper #148 109 places worse
0.9213 ±0.0305
best reported
actual #39 vs paper #148 109 places worse
paper 0.8486 ±0.0450
best today: Graph Wave Networks (WWW 2025)
paper 0.8486 ±0.0450 Δ today 0.0727
40
H2GCN
actual #40 vs paper #148 108 places worse
0.9189 ±0.0393
best reported
actual #40 vs paper #148 108 places worse
paper 0.8486 ±0.0720

Difference within reported uncertainty; not counted as underperformance.

paper 0.8486 ±0.0720 Δ at pub 0.0181 Δ today 0.0703
53
GraphSAGE
0.9000 ±0.0365
61
FAGCN
actual #61 vs paper #289 228 places worse
0.8961 ±0.0152
best reported
actual #61 vs paper #289 228 places worse
paper 0.7700 ±0.0770
paper 0.7700 ±0.0770 Δ at pub 0.1261
69
M2M-GNN
0.8919 ±0.0450
This paper
68
DMP
actual #68 vs paper #431 363 places worse
0.8919
best reported
actual #68 vs paper #431 363 places worse
paper 0.6608 ±0.0700
paper 0.6608 ±0.0700 Δ at pub 0.1556 Δ today 0.2311
114
Ord. GNN
0.8622 ±0.0410
This paper
164
AERO-GNN
0.8435 ±0.0520
This paper
175
GOAL
actual #175 vs paper #175
0.8362 ±0.0670
actual #175 vs paper #175
paper 0.8362 ±0.0670

Improvement within reported uncertainty; not highlighted.

paper 0.8362 ±0.0670 Δ at pub 0.1162
211
GAT
actual #211 vs paper #576 365 places worse
0.8162 ±0.0645
best reported
actual #211 vs paper #576 365 places worse
paper 0.5216 ±0.0660
paper 0.5216 ±0.0660 Δ at pub 0.2784 Δ today 0.2946
257
GCN
actual #257 vs paper #568 311 places worse
0.7933 ±0.0447
best reported
actual #257 vs paper #568 311 places worse
paper 0.5514 ±0.0510
best today: Graph Wave Networks (WWW 2025)
paper 0.5514 ±0.0510 Δ at pub 0.2326 Δ today 0.2419
#ModelAccuracy ↑Paper
1
P^2GNN
0.9541 ±0.0272
2
ACM-GCN
actual #2 vs paper #87 85 places worse
0.9480 ±0.0380
best reported
actual #2 vs paper #87 85 places worse
paper 0.8514 ±0.0600
3
RF-GCN
0.9429
11
GPR-GNN
actual #11 vs paper #177 166 places worse
0.9279 ±0.0312
best reported
actual #11 vs paper #177 166 places worse
paper 0.8027 ±0.0810
paper 0.8027 ±0.0810 Δ at pub 0.1252
15
GGCN
actual #15 vs paper #79 64 places worse
0.9230 ±0.0367
best reported
actual #15 vs paper #79 64 places worse
paper 0.8568 ±0.0660

Difference within reported uncertainty; not counted as underperformance.

paper 0.8568 ±0.0660 Δ today 0.0662
24
MLP
actual #24 vs paper #158 134 places worse
0.9136 ±0.0070
best reported
actual #24 vs paper #158 134 places worse
paper 0.8189 ±0.0640

Difference within reported uncertainty; not counted as underperformance.

paper 0.8189 ±0.0640 Δ at pub 0.0947
31
DMP
actual #31 vs paper #355 324 places worse
0.8919
best reported
actual #31 vs paper #355 324 places worse
paper 0.6273 ±0.0450
paper 0.6273 ±0.0450 Δ at pub 0.2427 Δ today 0.2646
38
FAGCN
actual #38 vs paper #145 107 places worse
0.8852 ±0.0133
best reported
actual #38 vs paper #145 107 places worse
paper 0.8241 ±0.0380
paper 0.8241 ±0.0380 Δ at pub 0.0611
53
Ord. GNN
0.8703 ±0.0470
This paper
64
M2M-GNN
0.8648 ±0.0610
This paper
68
H2GCN
actual #68 vs paper #136 68 places worse
0.8623 ±0.0471
best reported
actual #68 vs paper #136 68 places worse
paper 0.8270 ±0.0520

Difference within reported uncertainty; not counted as underperformance.

paper 0.8270 ±0.0520 Δ at pub 0.0240 Δ today 0.0353
81
GOAL
actual #81 vs paper #81
0.8568 ±0.0620
actual #81 vs paper #81
paper 0.8568 ±0.0620
paper 0.8568 ±0.0620 Δ at pub 0.3078
98
AERO-GNN
actual #98 vs paper #98
0.8480 ±0.0330
actual #98 vs paper #98
paper 0.8480 ±0.0330

Improvement within reported uncertainty; not highlighted.

paper 0.8480 ±0.0330 Δ at pub 0.0440
138
SAGE
0.8269
158
GAT
actual #158 vs paper #358 200 places worse
0.8190 ±0.0540
best reported
actual #158 vs paper #358 200 places worse
paper 0.6189 ±0.0500
paper 0.6189 ±0.0500 Δ at pub 0.2001
170
GCN
actual #170 vs paper #372 202 places worse
0.8082 ±0.0360
best reported
actual #170 vs paper #372 202 places worse
paper 0.6054 ±0.0530
paper 0.6054 ±0.0530 Δ at pub 0.1961 Δ today 0.2028
#ModelAccuracy ↑Paper
1
DiGGR
0.4535 ±0.0353
2
TRIGON
0.4402 ±0.0028
3
DoG
0.4320 ±0.0120
16
ACM-GCN
actual #16 vs paper #173 157 places worse
0.4198 ±0.0089
best reported
actual #16 vs paper #173 157 places worse
paper 0.3628 ±0.0100
paper 0.3628 ±0.0100 Δ at pub 0.0032 Δ today 0.0570
42
MLP
actual #42 vs paper #158 116 places worse
0.4026 ±0.0108
best reported
actual #42 vs paper #158 116 places worse
paper 0.3653 ±0.0070
paper 0.3653 ±0.0070 Δ at pub 0.0373
53
GPR-GNN
actual #53 vs paper #247 194 places worse
0.3992 ±0.0067
best reported
actual #53 vs paper #247 194 places worse
paper 0.3463 ±0.0120
paper 0.3463 ±0.0120 Δ at pub 0.0528 Δ today 0.0529
56
FAGCN
actual #56 vs paper #198 142 places worse
0.3970
best reported
actual #56 vs paper #198 142 places worse
paper 0.3567 ±0.0090
paper 0.3567 ±0.0090 Δ at pub 0.0403
62
GGCN
actual #62 vs paper #113 51 places worse
0.3929 ±0.0048
best reported
actual #62 vs paper #113 51 places worse
paper 0.3754 ±0.0150

Difference within reported uncertainty; not counted as underperformance.

paper 0.3754 ±0.0150 Δ at pub 0.0027 Δ today 0.0175
68
GraphSAGE
0.3899 ±0.0085
71
H2GCN
actual #71 vs paper #198 127 places worse
0.3885 ±0.0117
best reported
actual #71 vs paper #198 127 places worse
paper 0.3570 ±0.0100
paper 0.3570 ±0.0100 Δ at pub 0.0270 Δ today 0.0315
92
Ord. GNN
0.3799 ±0.0100
This paper
122
DMP
actual #122 vs paper #389 267 places worse
0.3726 ±0.0179
best reported
actual #122 vs paper #389 267 places worse
paper 0.2830 ±0.0270
paper 0.2830 ±0.0270 Δ at pub 0.0896
135
GCN
actual #135 vs paper #408 273 places worse
0.3690
best reported
actual #135 vs paper #408 273 places worse
paper 0.2732 ±0.0110
paper 0.2732 ±0.0110 Δ at pub 0.0958
147
M2M-GNN
0.3672 ±0.0160
This paper
157
AERO-GNN
0.3657 ±0.0110
This paper
163
GOAL
actual #163 vs paper #163
0.3646 ±0.0100
actual #163 vs paper #163
paper 0.3646 ±0.0100

Improvement within reported uncertainty; not highlighted.

paper 0.3646 ±0.0100 Δ at pub 0.0016
182
GAT
actual #182 vs paper #406 224 places worse
0.3605 ±0.0035
best reported
actual #182 vs paper #406 224 places worse
paper 0.2744 ±0.0080
paper 0.2744 ±0.0080 Δ at pub 0.0861
#ModelAccuracy ↑Paper
1
DMD++
0.9190 ±0.0260
2
F-GREAD-BS
0.9059 ±0.0380
3
APPNP
0.9010 ±0.0350
11
M2M-GNN
0.8901 ±0.0410
This paper
15
ACM-GCN
0.8843 ±0.0320
This paper
20
Ord. GNN
0.8804 ±0.0360
This paper
22
H2GCN
actual #22 vs paper #23 1 place worse
0.8790 ±0.0420
best reported
actual #22 vs paper #23 1 place worse
paper 0.8765 ±0.0490

Difference within reported uncertainty; not counted as underperformance.

paper 0.8765 ±0.0490 Δ today 0.0025
30
GOAL
0.8698 ±0.0440
This paper
32
GGCN
0.8686 ±0.0320
This paper
37
MLP
actual #37 vs paper #42 5 places worse
0.8629 ±0.0450
best reported
actual #37 vs paper #42 5 places worse
paper 0.8529 ±0.0330

Difference within reported uncertainty; not counted as underperformance.

paper 0.8529 ±0.0330 Δ at pub 0.0100
53
GAT
actual #53 vs paper #213 160 places worse
0.8430 ±0.0370
best reported
actual #53 vs paper #213 160 places worse
paper 0.4941 ±0.0400
paper 0.4941 ±0.0400 Δ at pub 0.3489
56
GPR-GNN
actual #56 vs paper #64 8 places worse
0.8412
best reported
actual #56 vs paper #64 8 places worse
paper 0.8294 ±0.0420

Difference within reported uncertainty; not counted as underperformance.

paper 0.8294 ±0.0420 Δ at pub 0.0118
60
FAGCN
actual #60 vs paper #89 29 places worse
0.8333
best reported
actual #60 vs paper #89 29 places worse
paper 0.7832 ±0.0630

Difference within reported uncertainty; not counted as underperformance.

paper 0.7832 ±0.0630 Δ at pub 0.0501
63
GCN
actual #63 vs paper #205 142 places worse
0.8310 ±0.0320
best reported
actual #63 vs paper #205 142 places worse
paper 0.5176 ±0.0300
paper 0.5176 ±0.0300 Δ at pub 0.3134
68
AERO-GNN
0.8124 ±0.0680
This paper
69
GraphSAGE
0.8118 ±0.0556
183
DMP
0.5641 ±0.0550
This paper

Heterophily Benchmark

#ModelAccuracy ↑Paper
1
UniGAP
0.6375 ±0.0110
2
H2GCN + HalfHop
0.5961 ±0.0100
3
GraphSAGE + HalfHop
0.5897 ±0.0150
7
H2GCN
actual #7 vs paper #196 189 places worse
0.5681 ±0.0090
best reported
actual #7 vs paper #196 189 places worse
paper 0.3647 ±0.0020
12
SAGE
0.5569 ±0.0023
14
GAT
actual #14 vs paper #105 91 places worse
0.5559 ±0.0027
best reported
actual #14 vs paper #105 91 places worse
paper 0.4909 ±0.0060
24
GCN
actual #24 vs paper #111 87 places worse
0.5411 ±0.0034
best reported
actual #24 vs paper #111 87 places worse
paper 0.4870 ±0.0060
47
ACM-GCN
actual #47 vs paper #189 142 places worse
0.5294 ±0.0023
best reported
actual #47 vs paper #189 142 places worse
paper 0.3862 ±0.0060
paper 0.3862 ±0.0060 Δ at pub 0.1432
84
MLP
actual #84 vs paper #169 85 places worse
0.5058
best reported
actual #84 vs paper #169 85 places worse
paper 0.4287 ±0.0040
paper 0.4287 ±0.0040 Δ at pub 0.0771
103
GPR-GNN
actual #103 vs paper #148 45 places worse
0.4937 ±0.0071
best reported
actual #103 vs paper #148 45 places worse
paper 0.4488 ±0.0030
best at pub: Evolving Computation Graphs (ICML 2023)
paper 0.4488 ±0.0030 Δ today 0.0449
105
M2M-GNN
0.4918 ±0.0060
This paper
137
FAGCN
actual #137 vs paper #156 19 places worse
0.4632 ±0.0250
best reported
actual #137 vs paper #156 19 places worse
paper 0.4412 ±0.0030

Difference within reported uncertainty; not counted as underperformance.

paper 0.4412 ±0.0030 Δ at pub 0.0220 Δ today 0.0220
142
AERO-GNN
0.4571 ±0.0050
This paper
165
GGCN
actual #165 vs paper #196 31 places worse
0.4300 ±0.0032
best reported
actual #165 vs paper #196 31 places worse
paper 0.3686 ±0.0040
paper 0.3686 ±0.0040 Δ at pub 0.0614
191
Ord. GNN
0.3852 ±0.0040
This paper
195
GOAL
0.3794 ±0.0030
This paper
198
DMP
0.3584 ±0.0040
This paper

LINKX Benchmarks

#ModelAccuracy ↑Paper
1
DC-GNN
0.8669 ±0.0022
2
M2M-GNN
0.8594 ±0.0040
This paper
3
GloGNN
0.8574 ±0.0042
15
GOAL
0.8418 ±0.0030
This paper
16
Ord. GNN
0.8365 ±0.0060
This paper
20
GPR-GNN
actual #20 vs paper #37 17 places worse
0.8354 ±0.0032
best reported
actual #20 vs paper #37 17 places worse
paper 0.8138 ±0.0010
paper 0.8138 ±0.0010 Δ at pub 0.0216 Δ today 0.0216
24
H2GCN
actual #24 vs paper #37 13 places worse
0.8320 ±0.0055
best reported
actual #24 vs paper #37 13 places worse
paper 0.8131 ±0.0060
paper 0.8131 ±0.0060 Δ at pub 0.0189
26
ACM-GCN
actual #26 vs paper #30 4 places worse
0.8298 ±0.0063
best reported
actual #26 vs paper #30 4 places worse
paper 0.8252 ±0.0090

Difference within reported uncertainty; not counted as underperformance.

paper 0.8252 ±0.0090 Δ at pub 0.0046
29
GIN
0.8268 ±0.0032
31
GCN
0.8247 ±0.0020
This paper
32
AERO-GNN
0.8247 ±0.0070
This paper
36
GAT
0.8153 ±0.0050
This paper
46
FAGCN
actual #46 vs paper #58 12 places worse
0.7697 ±0.0069
best reported
actual #46 vs paper #58 12 places worse
paper 0.7285 ±0.0050
paper 0.7285 ±0.0050 Δ at pub 0.0412
51
MLP
actual #51 vs paper #57 6 places worse
0.7534 ±0.0064
best reported
actual #51 vs paper #57 6 places worse
paper 0.7361 ±0.0040
paper 0.7361 ±0.0040 Δ at pub 0.0173
56
DMP
0.7385 ±0.0070
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Best today — OK at publication; today's leaderboard best is elsewhere