Application of Graph Neural Networks and graph descriptors for graph classification
Abstract
Graph classification is an important area in both modern research and industry. Multiple applications, especially in chemistry and novel drug discovery, encourage rapid development of machine learning models in this area. To keep up with the pace of new research, proper experimental design, fair evaluation, and independent benchmarks are essential. Design of strong baselines is an indispensable element of such works. In this thesis, we explore multiple approaches to graph classification. We focus on Graph Neural Networks (GNNs), which emerged as a de facto standard deep learning technique for graph representation learning. Classical approaches, such as graph descriptors and molecular fingerprints, are also addressed. We design fair evaluation experimental protocol and choose proper datasets collection. This allows us to perform numerous experiments and rigorously analyze modern approaches. We arrive to many conclusions, which shed new light on performance and quality of novel algorithms. We investigate application of Jumping Knowledge GNN architecture to graph classification, which proves to be an efficient tool for improving base graph neural network architectures. Multiple improvements to baseline models are also proposed and experimentally verified, which constitutes an important contribution to the field of fair model comparison.
Benchmark Results
Tested on 3 datasets SOTA at publication on 1 7 baseline values reported too low
Tables show presented results in global context (best 3 papers + GCN + MLP).
MoleculeNet
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | ProtoMol | 0.9140
±0.0030 | |
| 2 | MMSG | 0.9080
±0.0050 | |
| 3 | MemGNN | 0.9070 | |
| 43 | GIN | 0.8646
±0.0081 best reported | best at pub: Optimal Transport Graph Neural Networks (2020) |
| 47 | Morgan | 0.8626 | This paper |
| 50 | Fingerprints concatenation | 0.8617 | This paper |
| 52 | GAT | 0.8600
±0.0110 best reported | best at pub: Optimal Transport Graph Neural Networks (2020) |
| 64 | GCN | 0.8540
±0.0110 best reported | best at pub: Motif-based Graph Representation Learning with Application to Chemical Molecules (2022) best today: Atom-Motif Contrastive Transformer for Molecular Property Prediction (2023) |
| 82 | GraphSAGE | 0.8451 | prev best: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 92 | RDKit | 0.8386 | This paper |
| 113 | MACCS | 0.8278 | This paper |
| 141 | LDP with additional features | 0.8143 | This paper |
| 154 | GCN + JK LSTM | 0.8108 | This paper |
| 155 | GCN + JK concat | 0.8103 | This paper |
| 156 | LDP extended | 0.8102 | This paper |
| 158 | GCN + JK max | 0.8096 | This paper |
| 160 | GraphSAGE + JK concat | 0.8093 | This paper |
| 161 | Deep Multisets + atom encoding | 0.8093 | This paper |
| 164 | LDP with additional features and feature selection | 0.8082 | This paper |
| 169 | LDP | 0.8050
±0.0030 best reported | best today: Molecular Topological Profile (MOLTOP) - Simple and Strong Baseline for Molecular Graph Classification (2024) Difference within reported uncertainty; not counted as underperformance. |
| 171 | GraphSAGE + JK LSTM | 0.8039 | This paper |
| 179 | GraphSAGE + JK max | 0.7985 | This paper |
| 193 | MFPs | 0.7930 | This paper |
| 207 | MFPs + atom encoding | 0.7878 | This paper |
| 241 | GAT + JK LSTM | 0.7609 | This paper |
| 245 | GAT + JK max | 0.7573 | This paper |
| 267 | Deep Multisets | 0.7181 | This paper |
| 279 | GIN + JK LSTM | 0.7034 | This paper |
| 289 | GAT + JK concat | 0.6776 | This paper |
| 293 | GIN + JK concat | 0.6561 | This paper |
| 294 | GIN + JK max | 0.6515 | This paper |
| 298 | SGC | 0.6324 | This paper |
| # | Model | ROC-AUC ↑ | Paper |
|---|---|---|---|
| 1 | MAGPrompt+ | 0.9432
±0.0238 | |
| 2 | CamS-LLaMA | 0.9420 | |
| 3 | SE(3)-I MPPH | 0.9400
±0.0210 | |
| 34 | GIN | 0.8875
±0.0049 best reported | |
| 38 | GCN | 0.8810
±0.0360 best reported | best at pub: Motif-based Graph Representation Learning with Application to Chemical Molecules (2022) |
| 57 | GraphSAGE | 0.8304
±0.0038 best reported | best at pub: Improving Molecular Pretraining with Complementary Featurizations (2022) |
| 128 | MACCS | 0.7054 | This paper |
| 133 | Fingerprints concatenation | 0.7047 | This paper |
| 176 | RDKit | 0.6910 | This paper |
| 179 | GCN + JK concat | 0.6901 | This paper |
| 192 | GAT + JK max | 0.6849 | This paper |
| 197 | MFPs + atom encoding | 0.6819 | This paper |
| 203 | GIN + JK concat | 0.6789 | This paper |
| 210 | GAT | 0.6776 | This paper |
| 228 | Morgan | 0.6716 | This paper |
| 231 | LDP with additional features | 0.6711 | This paper |
| 235 | GraphSAGE + JK max | 0.6698 | This paper |
| 236 | LDP with additional features and feature selection | 0.6694 | This paper |
| 241 | GAT + JK concat | 0.6667 | This paper |
| 247 | GIN + JK max | 0.6638 | This paper |
| 250 | SGC | 0.6621 | This paper |
| 251 | MFPs | 0.6617 | This paper |
| 253 | GAT + JK LSTM | 0.6608 | This paper |
| 256 | GCN + JK LSTM | 0.6599 | This paper |
| 267 | Deep Multisets + atom encoding | 0.6546 | This paper |
| 269 | LDP extended | 0.6506 | This paper |
| 277 | GraphSAGE + JK LSTM | 0.6453 | This paper |
| 276 | GCN + JK max | 0.6453 | This paper |
| 288 | GraphSAGE + JK concat | 0.6357 | This paper |
| 289 | LDP | 0.6330
±0.0040 best reported | best today: Molecular Topological Profile (MOLTOP) - Simple and Strong Baseline for Molecular Graph Classification (2024) Difference within reported uncertainty; not counted as underperformance. |
| 292 | Deep Multisets | 0.6286 | This paper |
| 303 | GIN + JK LSTM | 0.6016 | This paper |
| # | Model | RMSE ↓ | Paper |
|---|---|---|---|
| 1 | MemGNN | 0.5560
±0.0230 | |
| 2 | ProtoS-L2 | 0.5800
±0.0160 | |
| 3 | ProtoW-L2 | 0.6040
±0.0140 | |
| 7 | GCN | 0.6550
±0.0360 best reported | best at pub: Memory-Based Graph Networks (ICLR 2020) |
| 9 | GIN | 0.6580
±0.0190 best reported | best at pub: Optimal Transport Graph Neural Networks (2020) |
| 25 | GCN + JK concat | 0.7150 | This paper |
| 29 | GCN + JK LSTM | 0.7210 | This paper |
| 30 | GraphSAGE + JK LSTM | 0.7260 | This paper |
| 33 | GraphSAGE + JK concat | 0.7320 | This paper |
| 35 | GAT + JK max | 0.7340 | This paper |
| 36 | GIN + JK LSTM | 0.7350 | This paper |
| 42 | GIN + JK max | 0.7470 | This paper |
| 43 | GAT | 0.7530 | prev best: Optimal Transport Graph Neural Networks (2020) Improvement within reported uncertainty; not highlighted. |
| 46 | GCN + JK max | 0.7630 | This paper |
| 50 | GraphSAGE + JK max | 0.7760 | This paper |
| 51 | GraphSAGE | 0.7770 | This paper |
| 52 | GIN + JK concat | 0.7770 | This paper |
| 55 | GAT + JK LSTM | 0.7910 | This paper |
| 58 | GAT + JK concat | 0.8080 | This paper |
| 61 | Fingerprints concatenation | 0.8200 | This paper |
| 66 | Deep Multisets + atom encoding | 0.8370 | This paper |
| 68 | Deep Multisets | 0.8480 | This paper |
| 71 | Morgan | 0.8570 | This paper |
| 77 | MACCS | 0.8830 | This paper |
| 78 | SGC | 0.8840 | This paper |
| 83 | RDKit | 0.9350 | This paper |
| 84 | LDP with additional features | 0.9650 | This paper |
| 85 | LDP extended | 0.9660 | This paper |
| 86 | LDP with additional features and feature selection | 0.9660 | This paper |
| 89 | MFPs + atom encoding | 0.9920 | This paper |
| 90 | LDP | 1.0060 | This paper |
| 91 | MFPs | 1.0220 | This paper |
