Pooling Architecture Search for Graph Classification

Lan Wei, Huan Zhao, Quanming Yao, Zhiqiang He

Main figure for Pooling Architecture Search for Graph Classification

Abstract

Graph classification is an important problem with applications across many domains, like chemistry and bioinformatics, for which graph neural networks (GNNs) have been state-of-the-art (SOTA) methods. GNNs are designed to learn node-level representation based on neighborhood aggregation schemes, and to obtain graph-level representation, pooling methods are applied after the aggregation operation in existing GNN models to generate coarse-grained graphs. However, due to highly diverse applications of graph classification, and the performance of existing pooling methods vary on different graphs. In other words, it is a challenging problem to design a universal pooling architecture to perform well in most cases, leading to a demand for data-specific pooling methods in real-world applications. To address this problem, we propose to use neural architecture search (NAS) to search for adaptive pooling architectures for graph classification. Firstly we designed a unified framework consisting of four modules: Aggregation, Pooling, Readout, and Merge, which can cover existing human-designed pooling methods for graph classification. Based on this framework, a novel search space is designed by incorporating popular operations in human-designed architectures. Then to enable efficient search, a coarsening strategy is proposed to continuously relax the search space, thus a differentiable search method can be adopted. Extensive experiments on six real-world datasets from three domains are conducted, and the results demonstrate the effectiveness and efficiency of the proposed framework.

Proposed method: PAS

Benchmark Results

Tested on 4 datasets

Tables show presented results in global context (best 3 papers + GCN + MLP).

TU Dortmund

#ModelAccuracy ↑Paper
1
MSH-GNN
0.9410 ±0.0330
2
GRDL
0.8260 ±0.0120
3
GMN
0.8225
29
ASAP
actual #29 vs paper #255 226 places worse
0.7892 ±0.0145
best reported
actual #29 vs paper #255 226 places worse
paper 0.7493 ±0.0357

Difference within reported uncertainty; not counted as underperformance.

paper 0.7493 ±0.0357 Δ at pub 0.0399
32
GIN
actual #32 vs paper #293 261 places worse
0.7880 ±0.0410
best reported
actual #32 vs paper #293 261 places worse
paper 0.7448 ±0.0278

Difference within reported uncertainty; not counted as underperformance.

paper 0.7448 ±0.0278 Δ at pub 0.0162 Δ today 0.0432
49
GAT
actual #49 vs paper #212 163 places worse
0.7800 ±0.0300
best reported
actual #49 vs paper #212 163 places worse
paper 0.7530 ±0.0372

Difference within reported uncertainty; not counted as underperformance.

paper 0.7530 ±0.0372 Δ at pub 0.0060 Δ today 0.0270
67
SAGE
actual #67 vs paper #325 258 places worse
0.7726 ±0.0228
best reported
actual #67 vs paper #325 258 places worse
paper 0.7375 ±0.0297

Difference within reported uncertainty; not counted as underperformance.

paper 0.7375 ±0.0297 Δ at pub 0.0351 Δ today 0.0351
98
PAS
0.7664 ±0.0329
This paper
132
DGCNN
actual #132 vs paper #339 207 places worse
0.7626
best reported
actual #132 vs paper #339 207 places worse
paper 0.7357 ±0.0469
best today: Orthogonal Graph Neural Networks (AAAI 2021)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7357 ±0.0469 Δ at pub 0.0197 Δ today 0.0269
148
GCN
actual #148 vs paper #271 123 places worse
0.7600 ±0.0320
best reported
actual #148 vs paper #271 123 places worse
paper 0.7484 ±0.0282

Difference within reported uncertainty; not counted as underperformance.

paper 0.7484 ±0.0282 Δ at pub 0.0096 Δ today 0.0116
149
MLP
0.7600 ±0.0320
154
SAGPool
actual #154 vs paper #359 205 places worse
0.7595 ±0.0452
best reported
actual #154 vs paper #359 205 places worse
paper 0.7312 ±0.0447

Difference within reported uncertainty; not counted as underperformance.

paper 0.7312 ±0.0447 Δ at pub 0.0277 Δ today 0.0283
155
GraphSAGE
0.7590 ±0.0320
172
MINCUTPOOL
0.7575 ±0.0380
This paper
206
GCN-JK
0.7539 ±0.0517
This paper
228
GraphNAS-WS
0.7520 ±0.0251
This paper
255
SAGE-JK
0.7494 ±0.0355
This paper
288
GAT-JK
0.7457 ±0.0405
This paper
320
Random
0.7394 ±0.0423
This paper
346
EA
0.7341 ±0.0298
This paper
358
Bayesian
0.7314 ±0.0239
This paper
360
GIN-JK
0.7312 ±0.0451
This paper
386
GraphNAS
0.7251 ±0.0336
This paper
395
MLP-Baseline
0.7239 ±0.0353
This paper
396
SNAG-WS
0.7233 ±0.0244
This paper
450
SNAG
0.7053 ±0.0311
This paper
#ModelAccuracy ↑Paper
1
GC-GNN (GCN)
0.9210 ±0.0810
2
SUGAR
0.8403 ±0.0133
3
Ortho-g-U-Nets
0.8387
15
SAGE
actual #15 vs paper #100 85 places worse
0.8088 ±0.0233
best reported
actual #15 vs paper #100 85 places worse
paper 0.7727 ±0.0406

Difference within reported uncertainty; not counted as underperformance.

paper 0.7727 ±0.0406 Δ at pub 0.0361 Δ today 0.0361
18
DiffPool
actual #18 vs paper #82 64 places worse
0.8064
best reported
actual #18 vs paper #82 64 places worse
paper 0.7775 ±0.0400
best at pub: Graph U-Nets (2019)
best today: Orthogonal Graph Neural Networks (AAAI 2021)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7775 ±0.0400 Δ at pub 0.0289 Δ today 0.0289
32
ASAP
actual #32 vs paper #95 63 places worse
0.7950 ±0.0180
best reported
actual #32 vs paper #95 63 places worse
paper 0.7735 ±0.0415

Difference within reported uncertainty; not counted as underperformance.

paper 0.7735 ±0.0415 Δ at pub 0.0215
37
DGCNN
actual #37 vs paper #108 71 places worse
0.7937
best reported
actual #37 vs paper #108 71 places worse
paper 0.7666 ±0.0403
best today: Orthogonal Graph Neural Networks (AAAI 2021)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7666 ±0.0403 Δ at pub 0.0271 Δ today 0.0271
43
GCN
actual #43 vs paper #72 29 places worse
0.7912 ±0.0307
best reported
actual #43 vs paper #72 29 places worse
paper 0.7812 ±0.0433
best at pub: Hierarchical Graph Capsule Network (AAAI 2020)

Difference within reported uncertainty; not counted as underperformance.

paper 0.7812 ±0.0433 Δ at pub 0.0100
46
PAS
0.7896 ±0.0368
This paper
56
SAGPool
actual #56 vs paper #137 81 places worse
0.7870 ±0.0229
best reported
actual #56 vs paper #137 81 places worse
paper 0.7506 ±0.0506

Difference within reported uncertainty; not counted as underperformance.

paper 0.7506 ±0.0506 Δ at pub 0.0364
69
MINCUTPOOL
actual #69 vs paper #74 5 places worse
0.7822 ±0.0054
best reported
actual #69 vs paper #74 5 places worse
paper 0.7803 ±0.0363

Difference within reported uncertainty; not counted as underperformance.

paper 0.7803 ±0.0363 Δ today 0.0019
73
GAT-JK
0.7809 ±0.0618
This paper
74
GIN
actual #74 vs paper #131 57 places worse
0.7805 ±0.0189
best reported
actual #74 vs paper #131 57 places worse
paper 0.7540 ±0.0368

Difference within reported uncertainty; not counted as underperformance.

paper 0.7540 ±0.0368 Δ at pub 0.0265
79
Random
0.7792 ±0.0482
This paper
84
GCN-JK
0.7769 ±0.0235
This paper
89
MLP-Baseline
0.7752 ±0.0390
This paper
96
SAGE-JK
0.7735 ±0.0420
This paper
99
GAT
actual #99 vs paper #127 28 places worse
0.7730 ±0.0368
best reported
actual #99 vs paper #127 28 places worse
paper 0.7556 ±0.0372

Difference within reported uncertainty; not counted as underperformance.

paper 0.7556 ±0.0372 Δ at pub 0.0174
103
Graph U-Net
0.7710 ±0.0517
This paper
107
GraphNAS-WS
0.7674 ±0.0455
This paper
109
GraphSAGE
0.7660
128
Bayesian
0.7555 ±0.0321
This paper
135
EA
0.7514 ±0.0301
This paper
136
GIN-JK
0.7513 ±0.0395
This paper
154
SNAG-WS
0.7351 ±0.0303
This paper
159
SNAG
0.7223 ±0.0386
This paper
161
GraphNAS
0.7198 ±0.0454
This paper
#ModelAccuracy ↑Paper
1
ID-GNN+BEC
0.8150 ±0.0240
2
GIN+BEC
0.8080 ±0.0330
3
GCN+BEC
0.7990 ±0.0340
4
GCN
actual #4 vs paper #93 89 places worse
0.7980 ±0.0230
best reported
actual #4 vs paper #93 89 places worse
paper 0.7267 ±0.0642

Difference within reported uncertainty; not counted as underperformance.

paper 0.7267 ±0.0642 Δ at pub 0.0133 Δ today 0.0713
14
GIN
actual #14 vs paper #114 100 places worse
0.7810 ±0.0350
best reported
actual #14 vs paper #114 100 places worse
paper 0.7167 ±0.0277
best at pub: Graph Homomorphism Convolution (ICML 2020)
paper 0.7167 ±0.0277 Δ at pub 0.0343 Δ today 0.0643
38
PAS
0.7510 ±0.0532
This paper
41
SAGPool
0.7487 ±0.0409
This paper
49
ASAP
0.7427 ±0.0397
This paper
56
GAT
actual #56 vs paper #56
0.7407 ±0.0453
actual #56 vs paper #56
paper 0.7407 ±0.0453

Improvement within reported uncertainty; not highlighted.

paper 0.7407 ±0.0453 Δ at pub 0.0277
57
EA
0.7400 ±0.0412
This paper
63
DGCNN
0.7367 ±0.0570
This paper
65
GraphNAS-WS
0.7360 ±0.0463
This paper
66
SNAG-WS
0.7360 ±0.0516
This paper
71
GCN-JK
0.7347 ±0.0445
This paper
75
GAT-JK
0.7327 ±0.0468
This paper
77
Graph U-Net
0.7317 ±0.0484
This paper
78
DiffPool
actual #78 vs paper #111 33 places worse
0.7186 ±0.0563
actual #78 vs paper #111 33 places worse
paper 0.7186 ±0.0563

Improvement within reported uncertainty; not highlighted.

paper 0.7186 ±0.0563 Δ at pub 0.0096
87
SAGE-JK
0.7287 ±0.0618
This paper
88
MLP-Baseline
0.7287 ±0.0520
This paper
93
Bayesian
0.7270 ±0.0335
This paper
94
MINCUTPOOL
actual #94 vs paper #132 38 places worse
0.7265 ±0.0075
best reported
actual #94 vs paper #132 38 places worse
paper 0.7077 ±0.0489

Difference within reported uncertainty; not counted as underperformance.

paper 0.7077 ±0.0489 Δ today 0.0188
96
SNAG
0.7250 ±0.0461
This paper
99
GraphSAGE
0.7230 ±0.0530
103
SAGE
0.7217 ±0.0529
This paper
106
Random
0.7210 ±0.0554
This paper
107
GIN-JK
0.7207 ±0.0486
This paper
126
GraphNAS
0.7110 ±0.0230
This paper
#ModelAccuracy ↑Paper
1
GraphSNN
0.7790 ±0.0360
2
SIN
0.7560 ±0.0320
3
CIN
0.7560 ±0.0370
5
GIN
actual #5 vs paper #59 54 places worse
0.7510 ±0.0510
best reported
actual #5 vs paper #59 54 places worse
paper 0.4980 ±0.0250
best at pub: Graph Homomorphism Convolution (ICML 2020)
paper 0.4980 ±0.0250 Δ at pub 0.0250 Δ today 0.2530
33
PAS
0.5220 ±0.0373
This paper
38
GCN
actual #38 vs paper #52 14 places worse
0.5190
best reported
actual #38 vs paper #52 14 places worse
paper 0.5040 ±0.0302

Difference within reported uncertainty; not counted as underperformance.

paper 0.5040 ±0.0302 Δ at pub 0.0150 Δ today 0.0150
43
DiffPool
actual #43 vs paper #69 26 places worse
0.5131 ±0.0072
best reported
actual #43 vs paper #69 26 places worse
paper 0.4953 ±0.0398

Difference within reported uncertainty; not counted as underperformance.

paper 0.4953 ±0.0398 Δ today 0.0178
45
MINCUTPOOL
actual #45 vs paper #77 32 places worse
0.5104 ±0.0070
best reported
actual #45 vs paper #77 32 places worse
paper 0.4900 ±0.0283

Difference within reported uncertainty; not counted as underperformance.

paper 0.4900 ±0.0283 Δ today 0.0204
46
GraphSAGE
0.5090
47
GIN-JK
0.5080 ±0.0302
This paper
48
ASAP
actual #48 vs paper #57 9 places worse
0.5078 ±0.0075
best reported
actual #48 vs paper #57 9 places worse
paper 0.5013 ±0.0344

Difference within reported uncertainty; not counted as underperformance.

paper 0.5013 ±0.0344 Δ today 0.0065
58
SNAG-WS
0.5000 ±0.0248
This paper
60
MLP-Baseline
0.4980 ±0.0428
This paper
61
Random
0.4980 ±0.0398
This paper
62
Bayesian
0.4980 ±0.0397
This paper
65
SAGE-JK
0.4973 ±0.0340
This paper
69
GAT
actual #69 vs paper #69
0.4967 ±0.0430
actual #69 vs paper #69
paper 0.4967 ±0.0430

Improvement within reported uncertainty; not highlighted.

paper 0.4967 ±0.0430 Δ at pub 0.0187
70
GAT-JK
0.4953 ±0.0384
This paper
73
SAGPool
0.4933 ±0.0490
This paper
74
SNAG
0.4933 ±0.0289
This paper
78
DGCNN
0.4900 ±0.0356
This paper
80
GCN-JK
0.4886 ±0.0496
This paper
82
Graph U-Net
0.4880 ±0.0319
This paper
83
EA
0.4860 ±0.0405
This paper
86
SAGE
0.4853 ±0.0543
This paper
89
GraphNAS-WS
0.4827 ±0.0350
This paper
95
GraphNAS
0.4693 ±0.0364
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best today — OK at publication; today's leaderboard best is elsewhere