Application of Graph Neural Networks and graph descriptors for graph classification

J. Adamczyk

Main figure for Application of Graph Neural Networks and graph descriptors for graph classification

Abstract

Graph classification is an important area in both modern research and industry. Multiple applications, especially in chemistry and novel drug discovery, encourage rapid development of machine learning models in this area. To keep up with the pace of new research, proper experimental design, fair evaluation, and independent benchmarks are essential. Design of strong baselines is an indispensable element of such works. In this thesis, we explore multiple approaches to graph classification. We focus on Graph Neural Networks (GNNs), which emerged as a de facto standard deep learning technique for graph representation learning. Classical approaches, such as graph descriptors and molecular fingerprints, are also addressed. We design fair evaluation experimental protocol and choose proper datasets collection. This allows us to perform numerous experiments and rigorously analyze modern approaches. We arrive to many conclusions, which shed new light on performance and quality of novel algorithms. We investigate application of Jumping Knowledge GNN architecture to graph classification, which proves to be an efficient tool for improving base graph neural network architectures. Multiple improvements to baseline models are also proposed and experimentally verified, which constitutes an important contribution to the field of fair model comparison.

Benchmark Results

Tested on 3 datasets SOTA at publication on 1 7 baseline values reported too low

Tables show presented results in global context (best 3 papers + GCN + MLP).

MoleculeNet

#ModelROC-AUC ↑Paper
1
ProtoMol
0.9140 ±0.0030
2
MMSG
0.9080 ±0.0050
3
MemGNN
0.9070
43
GIN
actual #43 vs paper #286 243 places worse
0.8646 ±0.0081
best reported
actual #43 vs paper #286 243 places worse
paper 0.6884
47
Morgan
0.8626
This paper
50
Fingerprints concatenation
0.8617
This paper
52
GAT
actual #52 vs paper #225 173 places worse
0.8600 ±0.0110
best reported
actual #52 vs paper #225 173 places worse
paper 0.7769
paper 0.7769 Δ at pub 0.0831
64
GCN
actual #64 vs paper #277 213 places worse
0.8540 ±0.0110
best reported
actual #64 vs paper #277 213 places worse
paper 0.7058
82
GraphSAGE
actual #82 vs paper #82
0.8451
actual #82 vs paper #82
paper 0.8451
paper 0.8451 Δ at pub 0.1991
92
RDKit
0.8386
This paper
113
MACCS
0.8278
This paper
141
LDP with additional features
0.8143
This paper
154
GCN + JK LSTM
0.8108
This paper
155
GCN + JK concat
0.8103
This paper
156
LDP extended
0.8102
This paper
158
GCN + JK max
0.8096
This paper
160
GraphSAGE + JK concat
0.8093
This paper
161
Deep Multisets + atom encoding
0.8093
This paper
164
LDP with additional features and feature selection
0.8082
This paper
169
LDP
actual #169 vs paper #173 4 places worse
0.8050 ±0.0030
best reported
actual #169 vs paper #173 4 places worse
paper 0.8016

Difference within reported uncertainty; not counted as underperformance.

paper 0.8016 Δ today 0.0034
171
GraphSAGE + JK LSTM
0.8039
This paper
179
GraphSAGE + JK max
0.7985
This paper
193
MFPs
0.7930
This paper
207
MFPs + atom encoding
0.7878
This paper
241
GAT + JK LSTM
0.7609
This paper
245
GAT + JK max
0.7573
This paper
267
Deep Multisets
0.7181
This paper
279
GIN + JK LSTM
0.7034
This paper
289
GAT + JK concat
0.6776
This paper
293
GIN + JK concat
0.6561
This paper
294
GIN + JK max
0.6515
This paper
298
SGC
0.6324
This paper
#ModelROC-AUC ↑Paper
1
MAGPrompt+
0.9432 ±0.0238
2
CamS-LLaMA
0.9420
3
SE(3)-I MPPH
0.9400 ±0.0210
34
GIN
actual #34 vs paper #293 259 places worse
0.8875 ±0.0049
best reported
actual #34 vs paper #293 259 places worse
paper 0.6258
38
GCN
actual #38 vs paper #188 150 places worse
0.8810 ±0.0360
best reported
actual #38 vs paper #188 150 places worse
paper 0.6861
57
GraphSAGE
actual #57 vs paper #290 233 places worse
0.8304 ±0.0038
best reported
actual #57 vs paper #290 233 places worse
paper 0.6309
128
MACCS
0.7054
This paper
133
Fingerprints concatenation
0.7047
This paper
176
RDKit
0.6910
This paper
179
GCN + JK concat
0.6901
This paper
192
GAT + JK max
0.6849
This paper
197
MFPs + atom encoding
0.6819
This paper
203
GIN + JK concat
0.6789
This paper
210
GAT
0.6776
This paper
228
Morgan
0.6716
This paper
231
LDP with additional features
0.6711
This paper
235
GraphSAGE + JK max
0.6698
This paper
236
LDP with additional features and feature selection
0.6694
This paper
241
GAT + JK concat
0.6667
This paper
247
GIN + JK max
0.6638
This paper
250
SGC
0.6621
This paper
251
MFPs
0.6617
This paper
253
GAT + JK LSTM
0.6608
This paper
256
GCN + JK LSTM
0.6599
This paper
267
Deep Multisets + atom encoding
0.6546
This paper
269
LDP extended
0.6506
This paper
277
GraphSAGE + JK LSTM
0.6453
This paper
276
GCN + JK max
0.6453
This paper
288
GraphSAGE + JK concat
0.6357
This paper
289
LDP
actual #289 vs paper #292 3 places worse
0.6330 ±0.0040
best reported
actual #289 vs paper #292 3 places worse
paper 0.6277

Difference within reported uncertainty; not counted as underperformance.

paper 0.6277 Δ today 0.0053
292
Deep Multisets
0.6286
This paper
303
GIN + JK LSTM
0.6016
This paper
#ModelRMSE ↓Paper
1
MemGNN
0.5560 ±0.0230
2
ProtoS-L2
0.5800 ±0.0160
3
ProtoW-L2
0.6040 ±0.0140
7
GCN
actual #7 vs paper #48 41 places worse
0.6550 ±0.0360
best reported
actual #7 vs paper #48 41 places worse
paper 0.7740
best at pub: Memory-Based Graph Networks (ICLR 2020)
paper 0.7740 Δ at pub 0.1190
9
GIN
actual #9 vs paper #53 44 places worse
0.6580 ±0.0190
best reported
actual #9 vs paper #53 44 places worse
paper 0.7790
paper 0.7790 Δ at pub 0.1210
25
GCN + JK concat
0.7150
This paper
29
GCN + JK LSTM
0.7210
This paper
30
GraphSAGE + JK LSTM
0.7260
This paper
33
GraphSAGE + JK concat
0.7320
This paper
35
GAT + JK max
0.7340
This paper
36
GIN + JK LSTM
0.7350
This paper
42
GIN + JK max
0.7470
This paper
43
GAT
actual #43 vs paper #43
0.7530
actual #43 vs paper #43
paper 0.7530

Improvement within reported uncertainty; not highlighted.

paper 0.7530 Δ at pub 0.0550
46
GCN + JK max
0.7630
This paper
50
GraphSAGE + JK max
0.7760
This paper
51
GraphSAGE
0.7770
This paper
52
GIN + JK concat
0.7770
This paper
55
GAT + JK LSTM
0.7910
This paper
58
GAT + JK concat
0.8080
This paper
61
Fingerprints concatenation
0.8200
This paper
66
Deep Multisets + atom encoding
0.8370
This paper
68
Deep Multisets
0.8480
This paper
71
Morgan
0.8570
This paper
77
MACCS
0.8830
This paper
78
SGC
0.8840
This paper
83
RDKit
0.9350
This paper
84
LDP with additional features
0.9650
This paper
85
LDP extended
0.9660
This paper
86
LDP with additional features and feature selection
0.9660
This paper
89
MFPs + atom encoding
0.9920
This paper
90
LDP
1.0060
This paper
91
MFPs
1.0220
This paper
SOTA performance (not significantly different from the best) Proposed method (this paper) Baseline / comparison (this paper) Global context (other papers) Best at pub — reported result worse than the best known when published (best reported in green) Best at pub — reported value improved over the best known when published Best today — OK at publication; today's leaderboard best is elsewhere